hull-au/content/blog/amazon-linux-2-custom-mtu.md

---
title: "Custom MTU on Amazon Linux 2023"
date: 2024-04-03T17:07:26.0684656+08:00
draft: false
---

A quick search for "How to set MTU for Amazon Linux" would lead you straight 
[here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#set_mtu)
and you might think to yourself "This is easy! No problemo!" but upon following
the instructions, you'll be shocked to learn your MTU is still 9001 and asking
yourself why do machines not listen to the instructions we give them?

It turns out despite the best efforts of the machines, a brain worm everyone
loves immensely called 'systemd' has made this an exciting learning experience.

For this guide, our target MTU is 1440 bytes and the NIC is ens5.

# What Amazon tells you to do

The short summary is:
1. Edit `/etc/sysconfig/network-scripts/ifcfg-eth0` and append `MTU=1440`
2. Add `request subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-search, domain-name-servers, host-name, nis-domain, nis-servers, ntp-servers;` to `/etc/dhcp/dhclient.conf`
3. Reboot and bask in your success

Wait a minute!
```
~ ip link
2: ens5 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 blahblah
```
How did this happen?

# Inspect the logs

Thankfully there are logs and I hope you're a savvy `journalctl` user as you'll
need it for troubleshooting issues like this. Start by querying the logs for
`systemd-networkd` (excellent name, Poettering)

`journalctl -u systemd-networkd`

EC2 instances use DHCP for grabbing their IP configuration, so the log will be
spammed with countless DHCP events, but the line which caught my eye was this:

```
systemd-networkd[pid]: ens5: Configured with /usr/lib/systemd/network/80-ec2.network
```

My love of systemd grows stronger with every passing day.

# What is this file?

The ArchWiki team has an excellent article on `systemd-networkd` that explains
how these configuration files work, what order they're loaded in and where they
should be located. [View it here](https://wiki.archlinux.org/title/Systemd-networkd#Configuration_files)

Open this file up in `nano` (accept no substitutes) and you'll see

```
[Link]
MTUBytes=9001
```

Unbelievable! My first instinct here was to avoid editing the system file as
the ArchWiki explains that files in `/etc/systemd/network/` take precedence.
So therefore we should drop our modified .network files in /etc/ and be on our
our way, right?

# Overriding systemd-networkd

Well this is what I tried and after a reboot I saw this:
```
systemd-networkd[pid]: ens5: Configured with /etc/systemd/network/80-ec2.network
```

Awesome!

```
~ ip link
2: ens5 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 blahblah
```

Noo! Back to the logs

```
systemd-networkd[pid]: ens5: Reconfiguring with /run/systemd/network/70-ens5.network
```

When I dump out the volatile config, I see it's configured for jumbo frames

```
[Link]
MTUBytes=9001
```

Back to the drawing board.

# The desperate search begins

Now we've established there's a ghost in the machine, let's hunt for it.

`journalctl | grep ens5`

Something caught my eye

```ec2net[pid]: Starting configuration for ens5```

What the heck is `ec2net`? A quick search of the filesystem revealed it's a
helpful little script located at `/usr/share/amazon-ec2-net-utils/lib.sh` and
searching for that revealed it's open source. [GitHub](https://github.com/amazonlinux/amazon-ec2-net-utils)

The information on GitHub is enlightening:
```
The version 1.x branch of the amazon-ec2-net-utils package was used in Amazon Linux 2 and earlier releases. It has a long history and is tightly coupled to ISC dhclient and initscripts network configuration. Both of these components are deprecated and will not make up the primary network configuration framework in future releases of Amazon Linux or other distributions. The 2.x branch (released from the main branch in git) represents a complete rewrite targeting a more modern network management framework. The rest of this document describes the 2.x branch.
```

So the reason we're in this situation is the documentation is relevant to just
Amazon Linux 2 and 2023 does things differently.

This script is the bridge between IMDS and `systemd-networkd`. It creates the
volatile configuration we saw above and sources its defaults from 
`/usr/lib/systemd/network/80-ec2.network`.

# In conclusion

Edit `/usr/lib/systemd/network/80-ec2.network`, change the MTU value from 9001
to your desired value and make sure to document your changes as it'll probably
come to haunt you next time `amazon-ec2-net-utils` updates.

You may undo any edits you made to `/etc/dhcp/dhclient.conf` and
`/etc/sysconfig/network-scripts/ifcfg-eth0` as these have no effect.

p.s. This article was updated to correct references to AL2 as I didn't realise
that AL2023 is an entirely different version from AL2.