EC2 Effective Throughput. The Full Stack of Constraints

The headline bandwidth figure on an instance spec sheet is the ceiling under ideal conditions — before you subtract the effects of flow-level limits, conntrack pressure, SG rule processing, PPS throttling, and traffic destination. Real effective throughput is the minimum across all of these simultaneously.

What is ENA?

The Elastic Network Adapter (ENA) is the virtual NIC used by all Nitro-based EC2 instances. It’s not just a paravirtual driver — it offloads the network data plane entirely to a dedicated Nitro Card (custom AWS silicon), bypassing the hypervisor for I/O. This gives EC2 instances near-bare-metal network performance, but it also means that all the enforcement constraints described below — bandwidth caps, PPS limits, conntrack allowances — are implemented in hardware on the Nitro Card, not in kernel software you can tune away.

ENA is the driver and hardware interface — the technology that defines how the instance talks to the network. An Elastic Network Interface (ENI), by contrast, is a logical network attachment — a virtual NIC with its own MAC address, private IP(s), security groups, and source/dest check flag, which you attach to an instance. You can have multiple ENIs on a single instance, each appearing as a separate network interface to the OS (eth0, eth1, etc.). The important thing to understand is that ENA is the mechanism underneath all of them: every ENI on a Nitro instance is backed by the same ENA hardware and subject to the same instance-level enforcement. Adding ENIs does not add bandwidth — it adds logical network attachment points, each independently addressable and policy-able.

ENA supports network speeds of up to 100 Gbps for supported instance types, and all Nitro-based instances use ENA for enhanced networking. The ENA Linux driver (available on GitHub, updated regularly) exposes per-instance performance metrics via ethtool -S eth0 — this is your primary observability surface for diagnosing which constraint is actually binding.

1. Instance Bandwidth: Baseline vs. Burst vs. Sustained

The burst trap: Instances with 16 vCPUs or fewer (≤4xlarge sizing) are documented as “up to X Gbps” — meaning they have a baseline and use a network I/O credit mechanism to burst. Burst duration is typically 5–60 minutes depending on instance size. When credits exhaust, the instance falls back to baseline. Burst is best-effort even with credits, as it’s a shared resource.

Larger instances (≥8xlarge for most families) get a hard, sustained bandwidth allocation tied to vCPU count. An m5.8xlarge (32 vCPUs) gets 10 Gbps sustained; an m5.16xlarge (64 vCPUs) gets 20 Gbps sustained.

The n-suffix family (c6in, m6in, r6in, c7gn, etc.) is the tier where network is the primary design axis — c6in.32xlarge hits 200 Gbps aggregate, though you must attach at least 2 ENIs to separate network cards to achieve that — each card caps at 170 Gbps.

Bandwidth weighting is a newer lever on 8th-gen instances (M8i, C8i, R8i, M8g, C8g, R8g, M9g, etc.): you can configure vpc-1 (more VPC bandwidth) or ebs-1 (more EBS bandwidth), which adjusts how the shared Nitro I/O pipeline is allocated between network and storage traffic. This is a live, no-reboot change via modify-instance-network-performance-options.

Cross-region/DX bandwidth — historically capped at 50% of aggregate for large instances or 5 Gbps for small ones. Since March 2025, EC2 now supports up to the full instance baseline bandwidth for inter-region VPC peering and Direct Connect traffic. The limit is now the full baseline or 5 Gbps, whichever is greater. Jumbo frames up to 8500B are also now supported for cross-region VPC peering.

2. Single-Flow (5-Tuple) Bandwidth Limits

This is one of the most operationally significant constraints and is frequently misunderstood.

Default single-flow limit: 5 Gbps — regardless of aggregate instance bandwidth. This applies to any unique 5-tuple TCP/UDP flow between instances not in a cluster placement group.

Exceptions and overrides:

ENA Express is powered by AWS Scalable Reliable Datagram (SRD), a proprietary transport that uses dynamic routing to increase throughput and minimize tail latency. It raises the single-flow ceiling from 5 Gbps to 25 Gbps within the same Region, up to the aggregate instance limit.

Flow definition for non-TCP/UDP: For protocols like GRE or IPsec that follow the IP header, the flow is defined by the 3-tuple of source IP, destination IP, and next protocol — not the 5-tuple. This directly impacts NVA/firewall designs using GWLB with GENEVE encapsulation: each VNI-keyed flow is a 3-tuple from the hypervisor’s perspective, which limits fan-out.

MPTCP workaround: Multipath TCP can be configured to establish multiple paths between two endpoints to achieve higher aggregate bandwidth beyond the single-flow cap.

As of June 2025, ENA Express now supports 120 EC2 instance types, including network-optimized, storage, high-memory, and accelerated computing instances.

3. Packets Per Second (PPS) — The Hidden Bottleneck

PPS is a separate enforcement dimension from bandwidth. You can be well under your bandwidth ceiling and still get throttled if your mean packet size is small (IMIX, lots of short flows, DPI inspection, etc.).

Each EC2 instance has a maximum PPS performance based on instance type and size. When traffic exceeds a maximum, AWS shapes traffic by queueing and then dropping packets.

Key points:

• PPS limits are not published — AWS does not document per-instance PPS values in the spec sheets. You find the limit empirically via ethtool -S eth0 | grep pps_allowance_exceeded.
• PPS allowance is separate from overall bandwidth allowance. Even if an instance is under overall bandwidth allowance, it might exceed the PPS allowance if the mean packet size is too small.
• Link-local PPS cap: Each EC2 instance can send 1,024 PPS per network interface to Route 53 Resolver (169.254.169.253 / .2 addresses). This quota cannot be increased. This bites DNS-heavy workloads (service meshes, container environments) hard.
• The Nitro card accelerates subsequent packets in an established flow by caching per-flow evaluation results (SG rules, ACLs, route table entries). The first packet of a new flow requires full evaluation — connections per second (CPS) is therefore a distinct performance axis from PPS.
• Asymmetric routing degrades PPS: Asymmetric routing (traffic in on one ENI, out on another) forces the Nitro system to do more detailed per-packet evaluation, which reduces overall packet rate before hitting the processing limit. This surfaces as pps_allowance_exceeded on smaller instances or tx_queue_stops on larger ones. NVA designs with separate ingress/egress ENIs need to account for this.

4. Connection Tracking (Conntrack) — The Stateful Tax

Security Groups are stateful — Nitro performs conntrack for all tracked flows. This consumes both memory and compute on the Nitro card, and the allowance is per-instance, not per-ENI.

Allowance visibility: The conntrack_allowance_available metric (via ethtool -S eth0 | grep conntrack) reports how many new tracked connections the instance can still establish. On a c6i.2xlarge, this example shows ~272,500 available entries.

What eats your allowance:

• Each connection to a DynamoDB gateway endpoint consumes 2 conntrack entries.
• Load balancers (NLB, GWLB, NAT Gateway) automatically create tracked connections, each with their own timeout values distinct from ENI defaults.
• Stale idle connections — if tcp_established_idle_timeout is high, orphaned connections hold entries indefinitely.

Nitro V6 timeout change (June 2025): With Nitro V6 instances (launched June 2025), the default TCP established idle timeout dropped from 432,000 seconds (5 days) to 350 seconds. Applications holding idle connections — database pools, IoT telemetry, persistent microservice connections — may see unexpected drops after migrating to these instances.

Configurable timeouts (available on all Nitro instances since Nov 2023):

Asymmetric routing + conntrack: Asymmetric routing is a special challenge for conntrack. When both directions of a flow don’t traverse the same network path, the Nitro system must do more detailed per-packet evaluation, reducing the overall packet rate achievable before hitting processing limits. This is the core design constraint for centralized GWLB-based firewall architectures.

Untracked connections (don’t consume allowance): Traffic where the SG allows all traffic in and out for the protocol (i.e., no statefulness needed) can be classified as untracked. Explicitly allowing 0.0.0.0/0 both inbound and outbound for a given port will cause those flows to be untracked. Worth knowing for high-CPS scrubbing/filtering workloads.

5. ENI Limits and Topology

Bandwidth is at the instance level, not the ENI level. Multiple ENIs don’t increase total bandwidth capacity — the limit is enforced at the EC2 instance level, not at the ENI.

ENI count by instance size (representative examples):

ENI counts matter primarily for: pod-per-ENI models (EKS VPC CNI), multi-homed NVA topologies, and traffic separation between management/data planes.

ECS awsvpcTrunking: With awsvpc trunking enabled, a c5.large increases from 3 ENIs to 12, via a managed trunk ENI. This allows running 10 tasks with dedicated ENIs instead of 2.

6. Security Group Rule Processing

Default limits: 60 inbound + 60 outbound rules per SG (separate counts for IPv4 and IPv6, so effectively 240 rules per SG). 5 SGs per ENI by default, raisable to 16 via support. The product of rules-per-SG × SGs-per-ENI cannot exceed 1,000.

Performance impact of SG rules: Each additional SG rule increases the first-packet evaluation overhead on the Nitro card. For new flows, the Nitro card evaluates the full flow including SG rules, ACLs, and route table entries. Subsequent packets in the same flow use cached evaluation results, so CPS (new flows/second) is more sensitive to rule count than steady-state PPS.

Prefix lists are the right tool at scale — they count as a single rule entry regardless of how many CIDRs they contain (up to 1,000 per customer-managed prefix list), and they’re evaluated efficiently on the Nitro card.

7. The ENA Metrics You Should Always Be Watching

Via ethtool -S eth0 or CloudWatch Agent:

AWS doesn’t provide out-of-the-box CloudWatch metrics for network bandwidth bursting. You need the CloudWatch Agent configured to export ENA driver metrics for visibility into which constraint is actually binding.

8. Effective Throughput Decision Tree (Architecture Heuristics)

NVA / firewall on EC2:

• Conntrack exhaustion is the dominant failure mode at scale. Size for conntrack_allowance_available headroom, not just bandwidth.
• Use n-suffix instances (c6in, m6in) for firewall/inspection appliances — they have disproportionately higher conntrack allowances relative to their CPU count.
• Reduce TCP established timeout aggressively (350s–3600s range) to drain stale entries.
• Watch for asymmetric routing killing your PPS headroom — GWLB GENEVE + ECMP can cause this if flow symmetry isn’t guaranteed.

High-CPS microservices:

• Each new TCP connection is expensive (full Nitro card evaluation). Connection pooling and keep-alive are your friends.
• DNS PPS cap (1,024/ENI) will bite service-mesh-heavy environments. Use local DNS caching (systemd-resolved, dnsmasq) and Route 53 Resolver Endpoints rather than hitting the .2 address directly from every pod.

High-bandwidth bulk transfer (single stream):

• 5 Gbps single-flow ceiling without ENA Express. Enable ENA Express on both endpoints for 25 Gbps.
• For inter-region or DX: full baseline bandwidth now available (post-March 2025 change).
• Use cluster placement groups + ENA Express for HPC/AI training fan-out.

Bandwidth weighting (8th-gen+):

• If your instance does heavy EBS I/O alongside network work, the default weighting may be starving one of them. Use vpc-1 for network-dominant workloads, ebs-1 for storage-dominant ones. No restart required.

Written by Paul Carvill

Enterprise → cloud → AI networking. I write the breakdowns I wish I’d had. New field notes roughly twice a month.