4.7 KiB
| title | date | draft |
|---|---|---|
| Custom MTU on Amazon Linux 2023 | 2024-04-03T17:07:26.0684656+08:00 | false |
A quick search for "How to set MTU for Amazon Linux" would lead you straight here and you might think to yourself "This is easy! No problemo!" but upon following the instructions, you'll be shocked to learn your MTU is still 9001 and asking yourself why do machines not listen to the instructions we give them?
It turns out despite the best efforts of the machines, a brain worm everyone loves immensely called 'systemd' has made this an exciting learning experience.
For this guide, our target MTU is 1440 bytes and the NIC is ens5.
What Amazon tells you to do
The short summary is:
- Edit
/etc/sysconfig/network-scripts/ifcfg-eth0and appendMTU=1440 - Add
request subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-search, domain-name-servers, host-name, nis-domain, nis-servers, ntp-servers;to/etc/dhcp/dhclient.conf - Reboot and bask in your success
Wait a minute!
~ ip link
2: ens5 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 blahblah
How did this happen?
Inspect the logs
Thankfully there are logs and I hope you're a savvy journalctl user as you'll
need it for troubleshooting issues like this. Start by querying the logs for
systemd-networkd (excellent name, Poettering)
journalctl -u systemd-networkd
EC2 instances use DHCP for grabbing their IP configuration, so the log will be spammed with countless DHCP events, but the line which caught my eye was this:
systemd-networkd[pid]: ens5: Configured with /usr/lib/systemd/network/80-ec2.network
My love of systemd grows stronger with every passing day.
What is this file?
The ArchWiki team has an excellent article on systemd-networkd that explains
how these configuration files work, what order they're loaded in and where they
should be located. View it here
Open this file up in nano (accept no substitutes) and you'll see
[Link]
MTUBytes=9001
Unbelievable! My first instinct here was to avoid editing the system file as
the ArchWiki explains that files in /etc/systemd/network/ take precedence.
So therefore we should drop our modified .network files in /etc/ and be on our
our way, right?
Overriding systemd-networkd
Well this is what I tried and after a reboot I saw this:
systemd-networkd[pid]: ens5: Configured with /etc/systemd/network/80-ec2.network
Awesome!
~ ip link
2: ens5 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 blahblah
Noo! Back to the logs
systemd-networkd[pid]: ens5: Reconfiguring with /run/systemd/network/70-ens5.network
When I dump out the volatile config, I see it's configured for jumbo frames
[Link]
MTUBytes=9001
Back to the drawing board.
The desperate search begins
Now we've established there's a ghost in the machine, let's hunt for it.
journalctl | grep ens5
Something caught my eye
ec2net[pid]: Starting configuration for ens5
What the heck is ec2net? A quick search of the filesystem revealed it's a
helpful little script located at /usr/share/amazon-ec2-net-utils/lib.sh and
searching for that revealed it's open source. GitHub
The information on GitHub is enlightening:
The version 1.x branch of the amazon-ec2-net-utils package was used in Amazon Linux 2 and earlier releases. It has a long history and is tightly coupled to ISC dhclient and initscripts network configuration. Both of these components are deprecated and will not make up the primary network configuration framework in future releases of Amazon Linux or other distributions. The 2.x branch (released from the main branch in git) represents a complete rewrite targeting a more modern network management framework. The rest of this document describes the 2.x branch.
So the reason we're in this situation is the documentation is relevant to just Amazon Linux 2 and 2023 does things differently.
This script is the bridge between IMDS and systemd-networkd. It creates the
volatile configuration we saw above and sources its defaults from
/usr/lib/systemd/network/80-ec2.network.
In conclusion
Edit /usr/lib/systemd/network/80-ec2.network, change the MTU value from 9001
to your desired value and make sure to document your changes as it'll probably
come to haunt you next time amazon-ec2-net-utils updates.
You may undo any edits you made to /etc/dhcp/dhclient.conf and
/etc/sysconfig/network-scripts/ifcfg-eth0 as these have no effect.
p.s. This article was updated to correct references to AL2 as I didn't realise that AL2023 is an entirely different version from AL2.