Both carry RDMA semantics. The delta is everything else — transport guarantees, congestion model, fabric scope, and operational posture.
IB Transport → UDP/IP — GRH replaced by outer IP/UDP headerQP state machines — no native fabric-layer ACKGID (IPv6-mapped) — routable across L3 boundaries. Subnet-agnosticLID (local) + GID (global) — LIDs are subnet-scoped, 16-bit flat space, managed by SM| Attribute | RoCEv2 | InfiniBand |
|---|---|---|
| Losslessness | Achieved via PFC (802.1Qbb) + DCQCN. Requires careful DSCP → CoS mapping and priority group configuration. One misconfigured switch breaks it | Native — link-layer credit flow control guarantees zero drops. No configuration required beyond subnet bring-up |
| Congestion signalling | ECN (DSCP CE bit) → CNP at receiver → DCQCN rate limiter at sender. Reaction loop adds ~1–2 RTT lag before rate reduction | FECN in data packets → CNP from receiver → rate reduced at sender. Similar mechanism, but without PFC dependency |
| QP types supported | RC, UD, UC, XRC — same Verbs model as IB. SRQ supported |
RC, UD, UC, XRC, DC (Dynamic Connected — NVIDIA ext). SRQ, DCT |
| Multipath / load balancing | ECMP via 5-tuple hash (src IP:port, dst IP:port, proto). UDP src port entropy seeded by QP number — ECMP-friendly by design | LID-based routing — per-flow path assigned by Subnet Manager. MLAG-style Adaptive Routing (AR) with NVIDIA SHArP and modern SMs |
| Fabric management | Standard SDN / network OS (Cumulus, SONiC, EOS). No dedicated fabric manager. OpenSM not required | Subnet Manager (SM) mandatory — typically UFM (NVIDIA) or OpenSM. SM owns LID assignment, routing tables, and fabric health |
| MTU | 4096 B IB payload + IP/UDP overhead. Jumbo frames (9000 B Ethernet) reduce per-packet overhead | 4096 B IB MTU per VL. No Ethernet overhead — wire efficiency ~3% higher than RoCEv2 per packet |
| Collective ops (MPI / NCCL) | NCCL over RoCEv2 — software collectives only (ring-allreduce). No in-network compute acceleration natively | NVIDIA SHArP (Scalable Hierarchical Aggregation and Reduction Protocol) — in-network allreduce on switch ASICs. Significant speedup for large-scale collective ops |
| Inter-fabric routing | Native IP routing — routers, BGP, any L3 device. Scales to arbitrary topology. Multi-site trivial | IB Router required at subnet boundaries. Inter-subnet latency penalty. GRH adds header overhead. Typically one SM per subnet (<48K end-nodes per LID space) |
| Tooling / observability | perftest, rdma-core, standard Ethernet counters (ethtool -S), DPDK telemetry, OpenTelemetry exporters |
perftest, ibstat, ibping, ibdiagnet, UFM telemetry, per-port credit counters, fabric-level heat maps |
| Hardware ecosystem | Broad — NVIDIA (CX-6/7), Intel E810, Marvell, Broadcom (co-packaged optics), any merchant silicon switching | NVIDIA near-monopoly — ConnectX HCAs, Quantum switches. Limited non-NVIDIA IB at HDR/NDR |
| Deployment complexity | High — PFC storm risk, DCQCN tuning (rp_dce_tcp_g, rp_min_rate), DSCP marking end-to-end, ECN thresholds per queue |
Medium — SM required, but fabric self-heals. No PFC. Operational complexity shifts to subnet management not per-switch QoS |
| Cost | Merchant silicon switches (Spectrum/Tomahawk) — lower capex. Ethernet NIC cost parity. Optics reuse | IB switches (Quantum-2) and HCAs at premium. NDR port cost 2–3× equivalent Ethernet. NVIDIA licensing for UFM/SHArP |