Overlays are sold as free abstraction. They aren’t. Every encapsulation adds a tax, and the bill arrives in three currencies: bytes, milliseconds, and engineer-hours.
The byte tax
Each header you add comes out of your payload. Wrap a frame in VXLAN and you’ve spent 50-plus bytes before any data moves:
- The packet gets bigger, so either you raise the underlay MTU or you fragment.
- Fragmentation on a fast path is a quiet performance killer.
- Path MTU discovery breaks in subtle ways the moment something drops ICMP.
If you take one thing from this: set the underlay MTU before you turn on the overlay, not after the tickets start.
The time tax and the human tax
Encapsulation and decapsulation cost cycles, and the indirection costs comprehension. The latency is usually small. The engineer-hours are not — every overlay adds a layer someone has to reason about at 2am.
That doesn’t mean don’t use overlays. It means budget for them honestly, the way you’d budget for any other piece of infrastructure that has to be operated, not just deployed.
Written by the 2× CCIE
Enterprise → cloud → AI networking. I write the breakdowns I wish I’d had. New field notes roughly twice a month.