NICs, next-hop routing, and secondary IPs. where AWS, Azure, and GCP actually diverge

Every cloud lets you attach more than one NIC to a VM, route traffic to a NIC instead of a gateway, and stack extra IPs on top of all that. The consoles make all three look like checkboxes. They aren’t the same checkbox — and two of the three differences will break a design that was copied from the wrong cloud.

This is the version concentrated on topology first: how many NICs you can have and where, then how routing actually reaches them, and only then the secondary-IP/alias-IP layer that sits on top of a single NIC.

1. Multiple NICs on one VM

All three clouds support attaching more than one NIC to a VM. The split that actually matters is whether those NICs can sit in the same network — and whether they can sit in different networks. The two questions have different answers per cloud, which is the part worth holding onto.

AWS — fully supported in both directions. An EC2 instance can have multiple ENIs in the same VPC, even the same subnet, with max ENIs instance-type dependent (e.g. t3.medium = 3, c5.xlarge = 4, larger instances scale to 15+). AWS also supports multi-VPC ENI attachments (GA since late 2023): you can attach a secondary ENI from a completely different VPC to an instance, as long as the ENI is in the same Availability Zone as the instance. The constraint is account-scoped — you can’t multi-home across VPCs that live in different AWS accounts. This is the documented pattern for centralized appliances and databases that need to straddle otherwise-segregated VPCs, and for working around CIDR overlaps between VPCs that can’t be peered. Each ENI carries its own private IPs, security groups, and source/dest-check setting independently.

Azure is the one that’s actually locked to a single network. A VM’s NICs must all live in the same VNet — different subnets within that VNet are fine (size-dependent, e.g. Standard_D2s_v3 = 2 NICs, larger sizes up to 8), but Microsoft’s own guidance is explicit: you cannot attach NICs from different VNets to the same VM. If you need a VM logically present in two VNets, the path is VNet peering plus routing — not a second NIC sitting in the other network.

GCP only does it the other way around. A VM can have multiple NICs — up to 8, machine-type dependent — but each NIC must be in a different VPC network, not just a different subnet, and you cannot put two NICs in the same VPC on a GCP VM. GCP’s multi-NIC feature is “multi-VPC homing” exclusively — the native use case is an NVA dual-homed across an “untrust” VPC and a “trust” VPC.

If a design calls for two NICs in the same VPC on a GCP VM for redundancy or segmentation, that’s not buildable as stated. The fix depends on the actual goal: alias IP ranges (same-VPC, more addresses) or a genuine multi-VPC topology (real isolation). Conversely, a design that wants one VM physically dual-homed across two VNets on Azure isn’t buildable either — that’s a peering-and-routing problem, not a NIC-count problem.

The shared gotcha on all three: attaching a second NIC doesn’t solve in-guest routing for you. The guest OS still has exactly one default route unless you configure policy-based routing — ip rule plus per-table ip route on Linux. Traffic from a secondary NIC’s subnet will try to egress via the primary’s gateway and get dropped or come back asymmetric. This is the single most common multi-NIC failure mode across all three clouds, AWS’s cross-VPC case included.

2. Next-hop routing to a NIC, same VPC

This is the standard NVA/firewall-insertion pattern: a route table entry points at a NIC instead of an IGW/NAT/TGW/peering target. The mechanics differ more than the console suggests.

AWS — a route table target can be an eni-xxxx. The instance owning it needs source/destination check disabled, or it silently drops anything not addressed to itself. There’s no native ECMP across multiple ENIs in one route entry. The implicit local VPC route always wins over a more general route to an appliance, so you need more-specific prefixes to actually pull traffic toward it. Production NVA insertion almost always means Gateway Load Balancer rather than a raw ENI next-hop, since GWLB gives you symmetric hashing and failover natively.

Azure — the equivalent is a UDR with next-hop type “Virtual appliance.” The NIC needs enableIPForwarding: true instead of a check to disable — conceptually the same toggle, different name. Azure’s GWLB-equivalent is Standard Load Balancer with HA ports, load-balancing all ports and protocols to a backend pool. Azure Route Server adds something AWS has no direct equivalent for: BGP-speaking NVAs that inject routes dynamically instead of static UDRs maintained by hand.

GCP — static routes use next-hop-instance or next-hop-IP, and the forwarding toggle is --can-ip-forward, set at instance creation and immutable afterward — easy to forget and annoying to fix later. The real differentiator is native ECMP: multiple static routes with the same prefix and priority, pointed at different next-hop instances, get genuine load distribution — without the single-target ceiling AWS imposes. GCP’s GWLB-equivalent is ILB as next hop, which natively supports NVA HA with health checks.

3. Next-hop routing across VPC/VNet boundaries

Two of the three clouds say no outright to a direct cross-network next-hop. One says yes, conditionally — and that asymmetry is the most useful single fact in this whole comparison.

AWS — no. A route table target must reference a resource in the same VPC; the API rejects a direct cross-VPC ENI reference. You always insert an intermediary:

• Transit Gateway — the standard hub-and-spoke construct for centralizing NVA/firewall insertion across VPCs.
• GWLB + GWLB endpoints — purpose-built for this: the NVA sits behind GWLB in an inspection VPC, spoke VPCs get a GWLB endpoint as their route target, and traffic tunnels over Geneve to the centralized appliance.
• VPC Peering doesn’t help — peering only exposes CIDR routes within the peered VPC, never appliance insertion across the connection.

Azure — yes, with conditions. A UDR’s next-hop-IP can reference an NVA’s private address sitting in a genuinely different, peered VNet — directly, given appropriate peering settings and no overlapping address space. Hub-spoke NVA insertion commonly works exactly this way: spokes peer to a hub, UDRs on each spoke point straight at the firewall’s IP in that hub. The catch: peering is non-transitive by default, so spoke-to-spoke traffic through the hub NVA still needs explicit UDRs per spoke, or Azure Route Server with branch-to-branch enabled to propagate routes dynamically instead.

GCP — no, by default. A static route’s next-hop must live in the same VPC network. VPC Network Peering exists, but custom static routes are not exported across it by default — only subnet routes are, unless you explicitly enable custom route export/import on the peering connection. For more than a couple of peered VPCs, Network Connectivity Center is the hub-spoke construct that avoids a full peering mesh.

The asymmetry that matters: AWS and GCP both force cross-network NVA insertion through a purpose-built transit construct (TGW+GWLB, or NCC). Azure is the only one of the three where a route table can point straight at a resource sitting in a peered network, rather than requiring a dedicated transit layer.

4. Secondary IPs on a single NIC

Once the NIC/routing topology is settled, the next layer is density: how many addresses can live on one NIC, and what shape do they take. This is where the three clouds use genuinely different primitives, not just different limits.

AWS — limit is driven by instance type, the same table that governs ENI count (e.g. t3.medium: 6 IPv4 per ENI, 1 primary + 5 secondary; large instances up to 50). One primary IP is fixed for the ENI’s lifetime; secondaries assign/unassign dynamically via AssignPrivateIpAddresses. Prefix delegation is the newer option for claiming a /28 IPv4 or /80 IPv6 block instead of consuming individual secondary-IP slots one at a time — this is literally how the VPC CNI scales pod density on EKS.

Azure — limit is closer to flat: currently 256 private IP configurations per NIC. Every IP, primary or secondary, is wrapped in an ipConfiguration resource with its own optional public IP and NSG inheritance from the NIC. One primary config is required; it’s what’s used for outbound SNAT by default unless a Standard LB or NAT Gateway is doing explicit outbound.

GCP — structurally different. Up to 150 alias IP ranges per NIC, each a CIDR block rather than an individual address, sourced from the subnet’s primary range or from an explicitly defined secondary range (the per-network-interface count is independent of the netmask size — a single /24 range and a single /23 range both count as one). This is the same mechanism behind VPC-native GKE clusters: pod IPs are allocated from a subnet secondary range via alias IPs, one alias range per node covering that node’s pod CIDR slice. If you’ve set a “pod CIDR range” at GKE cluster creation, that’s this mechanism directly — and it’s why GCP never needed a bolted-on prefix-delegation feature the way AWS did.

5. Why you’d actually need a secondary IP

Worth pressure-testing before designing around it — a lot of the instinct to reach for secondary IPs is really a load-balancer problem wearing a different hat.

Legitimate cases:

• Container/pod IP-per-workload density — Kubernetes wants one routable IP per pod, not per node. Alias IPs (GKE), prefix delegation (EKS VPC CNI), and Azure CNI’s pod-subnet feature all exist because a node’s single NIC IP isn’t enough addresses for N pods, and they let pod traffic be natively routable in the VPC fabric instead of riding an overlay encapsulation.
• Multi-tenancy on one host — multiple SSL-terminated services each needing a distinct IP. Less critical now that SNI is ubiquitous, but still shows up in legacy or compliance-constrained stacks.
• NVA dual-personality interfaces — an appliance’s outside interface terminating several customer VPN tunnels, each expecting a distinct peer IP, without provisioning a whole second NIC per tunnel.
• Floating IP / active-passive failover — moving a service IP between primary and standby on failover, the cloud-VM substitute for VRRP since there’s no L2 broadcast domain for the real thing. Increasingly superseded by load-balancer health checks, but still common in lift-and-shift appliance designs.
• Segmentation without a second NIC — management-plane and data-plane addresses on one NIC in separate IP space, avoiding the policy-routing overhead a second NIC would force.

Where it’s the wrong tool:

• “We need more IPs to scale” — usually a load-balancer problem. One IP behind an ALB/NLB/Standard LB scales horizontally; stacking secondary IPs on one instance just centralizes risk.
• “Each microservice needs its own IP” — usually a Kubernetes Services-plus-ingress problem, if you’re not already running K8s that’s often the actual gap.
• DNS round-robin across secondary IPs on one box — adds complexity without adding capacity or resilience.

The litmus test: does the IP need to move independently of the instance, or attach to a workload rather than a host? If yes, it’s a legitimate secondary/alias IP case. If the real need is capacity or availability, that’s a load balancer or autoscaling group problem.

6. How a workload actually binds to the address

The cloud API assigning a secondary IP is half the job. The guest OS has to know it exists, and the application has to bind to it — neither happens automatically, and this gap is where most “I assigned the IP but it doesn’t work” tickets come from.

The guest OS has to be told

A cloud API call assigning a secondary IP to a NIC is a control-plane-only operation. It does not touch the guest. How automatically the address gets configured in-guest depends entirely on the image:

# manual fallback, works on all three clouds
ip addr add 10.0.1.50/24 dev eth0

# GCP alias IP range — a route, not an address, since it's a CIDR block
ip route add 10.0.2.0/28 dev eth0

• AWS — Amazon Linux ships ec2-net-utils, hooking into dhclient to auto-configure secondary IPs. Ubuntu generally needs a manual netplan entry or a cloud-init network config block.
• Azure — WALinuxAgent or cloud-init reads NIC metadata and configures secondary IP configs automatically on supported images.
• GCP — google-guest-agent can auto-configure alias IP routes, but only if the agent is running and the alias was set via --aliases at the API level; otherwise it’s a manual route, as above.

The application has to bind to it — or not

This is where most confusion lives, and it splits into two behaviors:

• Wildcard bind (0.0.0.0 / INADDR_ANY) — the app listens on every IP on the host, secondary included, with zero app-level config. Most simple services fall into this bucket automatically.
• Explicit bind — the app must bind the socket to the specific address. nginx differentiates SSL certs per IP this way:

server {
    listen 10.0.1.50:443 ssl;
    server_name tenant-a.example.com;
}

server {
    listen 10.0.1.51:443 ssl;
    server_name tenant-b.example.com;
}

That’s the multi-tenant-on-one-host pattern from section 5, made concrete.

Inbound is the easy direction. Outbound is where it gets real: if a workload needs to originate connections from the secondary IP rather than just receive them, the kernel’s source-address selection doesn’t pick it automatically — it typically defaults to the primary IP unless something forces the choice. Some applications expose SO_BINDTODEVICE or a bind-before-connect option; most don’t. The general-purpose fix is the same policy-routing mechanism from the multi-NIC section:

# force traffic FROM the secondary IP to use its own route table
ip rule add from 10.0.1.50 table 100
ip route add default via 10.0.1.1 dev eth0 table 100

The mental model worth keeping: a cloud API call means “this IP is now permitted to exist on this interface.” It does nothing to the guest OS and nothing to the application. The OS seeing the IP, the app listening on it, and outbound traffic sourcing from it are three separate failure points stacked on top of the cloud-side assignment — and they fail independently.

The throughline

Topology, routing, and addressing all look like the same feature across AWS, Azure, and GCP from the console. They aren’t:

• NICs — GCP forces one NIC per VPC and only does cross-VPC; Azure is locked to one VNet per VM and only does same-network; AWS is the only one of the three that does both.
• Same-VPC next-hop routing — all three support it, but only GCP gives you native ECMP across multiple appliances.
• Cross-VPC next-hop routing — Azure alone allows a direct route to a peered network’s NIC; AWS and GCP both require a dedicated transit construct.
• Secondary IPs — AWS and Azure assign individual addresses (with AWS bolting on block allocation later via prefix delegation); GCP’s alias IP ranges are block-based from day one, and back GKE’s pod-networking model directly.
• Binding — identical everywhere, and skipped by most runbooks: the cloud never configures the guest OS or the application for you.

Written by Paul Carvill

Enterprise → cloud → AI networking. I write the breakdowns I wish I’d had. New field notes roughly twice a month.