Ubuntu 20.04 loses network in AWS after updates 28 october

28 october I run a series of normal updates on one of my Amazon-EC2 instances. For some reason it locked up and I had to reboot it. It got up but it never got any network again. So after a lot of troubleshooting I restored it from an old snapshot. After updating it (that went smoothly) I rebooted as expected and it got up, but without network connectivity. This is quite annoying.

Is this a known problem? When I look in dpkg.log in the instance that crashed first, the following packages were updated (filtered dump from dpkg.log). I see quite a lot of network-related stuff there.

Edit 2: I found the offending package by interval halving the updates from the old image. If I allow this update:

 0.100-0ubuntu4~20.04.2 amd64 [upgradable from: 0.99-0ubuntu3~20.04.2]

the machine will not get up on the network again after a reboot. I should of course have suspected netplan all along but whatever. I get one error message in /var/log/dmsg that have note investigated enough but seems network-related:

[ 3.396831] kernel: ena 0000:00:05.0: LLQ is not supported Fallback to host mode policy.

Edit: I have now looked at the first instance and it seems as if it does not help if I downgrade the packages (I do that on another instance using chroot). So I suspect (but very uncertain) that it is just a reboot issue although seemingly very difficult to solve.

upgrade ca-certificates:all 20190110ubuntu1.1 20201027ubuntu0.20.04.1
upgrade distro-info-data:all 0.43ubuntu1.1 0.43ubuntu1.2
upgrade accountsservice:amd64 0.6.55-0ubuntu12~20.04.1 0.6.55-0ubuntu12~20.04.2
upgrade libaccountsservice0:amd64 0.6.55-0ubuntu12~20.04.1 0.6.55-0ubuntu12~20.04.2
upgrade bind9-dnsutils:amd64 1:9.16.1-0ubuntu2.3 1:9.16.1-0ubuntu2.4
upgrade bind9-libs:amd64 1:9.16.1-0ubuntu2.3 1:9.16.1-0ubuntu2.4
upgrade bind9-host:amd64 1:9.16.1-0ubuntu2.3 1:9.16.1-0ubuntu2.4
upgrade libplymouth5:amd64 0.9.4git20200323-0ubuntu6 0.9.4git20200323-0ubuntu6.1
upgrade plymouth-theme-ubuntu-text:amd64 0.9.4git20200323-0ubuntu6 0.9.4git20200323-0ubuntu6.1
upgrade plymouth:amd64 0.9.4git20200323-0ubuntu6 0.9.4git20200323-0ubuntu6.1

My (admittedly bad) forensics says that something goes wrong in cloud-init. That init is necessary as an instance gets a new IP every time it boots (it has a static IP but that is routed to the dynamic one).

I see two ways forward: Either I restore from my dump again and try to hold the packages above somehow or I find a way to fix the resulting network issue without having access to the live machine (as it has no network), I do have access to the file system when the machine is taken down though.

Any ideas or help appreciated! Thank you!

1 Answer

I don't know if this will help in your specific case, but I know it's helped a lot of other EC2 users with SSH problems.

Note: Change the MAC addresses in the example for your information...

# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network: version: 2 renderer: networkd ethernets: eth0: match: macaddress: 06:bf:0c:4a:a5:d6 set-name: eth0 bridges: br0: dhcp4: true macaddress: 06:bf:0c:4a:a5:d6 interfaces: - eth0

Note: You may need to create /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

network: {config: disabled}

sudo netplan --debug generate

sudo netplan apply

reboot # mandatory

See