Elastic Fabric Adapter (EFA) Cheat Sheet

• The Elastic Fabric Adapter (EFA) is a network interface for Amazon EC2 instances that enables you to run applications that require high levels of inter-node communication at scale on AWS.
• It is designed to accelerate High Performance Computing (HPC) and machine learning applications by providing lower and more consistent latency and higher throughput than traditional TCP channels.
• How It Works:

• EFA integrates with 
  • Libfabric (1.7.0+) and supports Open MPI (4.x+), Intel MPI, and
  • Nvidia Collective Communications Library (NCCL) for machine learning applications.

• With an EFA, HPC applications use Intel Message Passing Interface (MPI) or Nvidia Collective Communications Library (NCCL) to interface with the Libfabric API. The Libfabric API bypasses the operating system kernel and communicates directly with the EFA device to place packets on the network.
• Supported operating systems
  • Amazon Linux 2 and Amazon Linux 2023
  • RHEL 8 and RHEL 9
  • Debian 11 and 12
  • Rocky Linux 8 and 9
  • OpenSUSE Leap 15.5 and later
  • SUSE Linux Enterprise 15 SP2 and later
  • Ubuntu 22.04 and 24.04

• Examples of HPC Applications
  • computational fluid dynamics (CFD)
  • crash simulations
  • weather simulations

Features

• • EFA uses a custom-built operating system (OS) bypass hardware interface. This allows HPC and ML applications to communicate directly with the EFA device, bypassing the OS kernel to reduce latency and CPU overhead.
  • EFA relies on the SRD protocol, which is designed for consistent low latency. Unlike TCP, SRD provides reliable delivery but does not guarantee packet ordering (handled by the application layer) and uses multi-path routing to avoid network congestion.
  • Integration: Works seamlessly with AWS ParallelCluster and AWS Batch to orchestrate HPC clusters.
  • Cross-Subnet Support: EFA now supports communication across different subnets (and Availability Zones, though with higher latency implications), overcoming previous limitations.

Elastic Fabric Adapter Limitations

• • EFA traffic cannot cross Availability Zones or VPCs, and it is not routable. 
  • RDMA write is not supported on all instance types.
  • EFA traffic between P4d/P4de/DL1 instances and other instance types is currently not supported.
  • Instances that support multiple network cards can be configured with one EFA per network card. All other supported instance types support only one EFA per instance.
  • c7g.16xlarge, m7g.16xlarge, and r7g.16xlarge Dedicated Instances and Dedicated Hosts are not supported when an EFA is attached.
  • EFA is not supported on AWS Outposts.
  • The EFA device is supported on Windows instances only for AWS Cloud Digital Interface (CDI) SDK-based applications. For non-CDI applications on Windows, it functions as a standard ENA interface without EFA capabilities.

Elastic Fabric Adapter (EFA) Pricing

EFA is available as an optional EC2 networking feature that you can enable on any supported EC2 instance at no additional cost.

Elastic Fabric Adapter Cheat Sheet References:

https://aws.amazon.com/hpc/efa/
https://aws.amazon.com/hpc/faqs/
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html