What is MTU?

MTU (maximum transmission unit) is the maximum size of one packet of data that can be transferred in a network. The default MTU size for Ethernet devices is 1500 bytes. This packet size contains the actual payload data as well as network overhead information needed for communication within the network.

All AWS EC2 instances support the default MTU size. But many current instance sizes support 9001 MTU, also referred to as jumbo frames. Enabling jumbo frames for supported EC2 instances can be beneficial because it improves network efficiency by allowing your instance to send fewer packets with bigger payloads while maintaining the same network overhead. This is especially helpful for instances inside placement groups wherein you want as much network performance as you can get.

When to enable Jumbo Frames?

Although jumbo frames bring the benefit of faster network throughput, it is not recommended to use this on Internet-bound traffic or traffic leaving your VPC because other network systems that do not support this traffic will fragment the packets, causing a slowdown.

To prevent possible slowdown of traffic, you can attach an additional elastic network interface (ENI) to your EC2 instances and configure it to route packets with a higher MTU, leaving the primary interface with the default MTU for communication outside the VPC. You can use jumbo frames for communication between your EC2 instances in your VPC, VPC peering network, or over the Direct Connect network.

In this post, we’ll show you how to enable Jumbo Frames to improve the network performance between your Linux EC2 instances.

Checking the Path MTU

Before changing the MTU of your EC2 instances, you need to determine the Path MTU, which is the maximum packet size supported by the network between the hosts. Path MTU Discovery is used to determine the path MTU. This is important because if the receiving hosts receive a larger MTU than it supports (or larger than the path supports), it will return an error with this ICMP message: “Destination Unreachable: Fragmentation Needed and Don’t Fragment was Set (Type 3, Code 4)”.

To check the path MTU, you can use the Linux command tracepath. If the package is not installed, you can install it first;

For Ubuntu based instances:

$ sudo apt install iputils-tracepath

For Amazon based instances:

$ sudo yum install iputils

From your source host, run the tracepath command to the destination host by using the IP address or the DNS name. Here, the source has IP 10.230.101.28, and the destination has IP 10.230.101.79.

[ec2-user@ip-10-230-101-28 ~]$ tracepath 10.230.101.79
1?: [LOCALHOST]                                         pmtu 1500
1:  ip-10-230-101-79.us-east-2.compute.internal         0.223ms reached
1:  ip-10-230-101-79.us-east-2.compute.internal         0.151ms reached
Resume: pmtu 1500 hops 1 back 1

You can see from the above output that path MTU is 1500. You may want to enable jumbo frames on this interface since the communication is within VPC. In this example, we are using an m5.large EC2 instance that supports MTU. Hence, we can enable jumbo frames for this interface.

Setting MTU on Your Linux Instance

1. In most EC2 instances, the default interface is already set to use jumbo frames. But you can verify this by using the ip command. You can check all instances on your VPC and ensure that jumbo frames are enabled.

$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group 
default qlen 1000
link/ether 06:88:e9:c0:e3:ae brd ff:ff:ff:ff:ff:ff

You can see from this output that the MTU is at 1500.

2. You can set the MTU value to 9001 by also using the ip command:

[ec2-user@ip-10-230-101-28 ~]$ sudo ip link set dev eth0 mtu 9001

3. You can verify the MTU again using the ip command and tracepath command:

[ec2-user@ip-10-230-101-28 ~]$ ip link show eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group 
default qlen 1000
link/ether 06:88:e9:c0:e3:ae brd ff:ff:ff:ff:ff:ff
[ec2-user@ip-10-230-101-28 ~]$ tracepath 10.230.101.79
1?: [LOCALHOST]                                         pmtu 9001
1:  ip-10-230-101-79.us-east-2.compute.internal         0.219ms reached
1:  ip-10-230-101-79.us-east-2.compute.internal         0.137ms reached
Resume: pmtu 9001 hops 1 back 1

4. This change is not permanent, once your instance is rebooted, the MTU will be set back to the default 1500. To make the change persist between reboots, you can update the following network-related files, based on operating system type.

For Amazon Linux 2:

$ echo "MTU=9001" | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-eth0
$ echo "request subnet-mask, broadcast-address, time-offset, routers, domain-name, 
domain-search, domain-name-servers, host-name, nis-domain, nis-servers, ntp-servers;" 
| sudo tee -a /etc/dhcp/dhclient.conf

For Amazon Linux:

$ echo 'interface "eth0" {
supersede interface-mtu 9001;
}' | sudo tee -a /etc/dhcp/dhclient-eth0.conf

5. You can reboot your EC2 instance and verify that MTU is set to 9001

Note: If you are studying for the AWS Certified Advanced Networking Specialty exam, we highly recommend that you take our AWS Certified Advanced Networking – Specialty Practice Exams and read our Advanced Networking Specialty exam study guide.

Sources:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html
https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html#path_mtu_discovery
https://docs.aws.amazon.com/directconnect/latest/UserGuide/set-jumbo-frames-vif.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html

This article is authored by: Kenneth Samonte, our resident AWS whiz/contributor.