VXLAN and EVPN explained: VNIs, VTEPs and BGP routes

A VLAN ID is 12 bits: about four thousand segments, in a layer 2 domain that has to be stretched across every switch, with spanning tree blocking links.

VXLAN keeps layer 2 for the servers, but builds it on top of a routed layer 3 underlay, where every link forwards and equal cost paths share the load. Its segment ID, the VNI, is 24 bits: about sixteen million segments.

A VTEP, the tunnel endpoint, wraps the original frame. First an 8 byte VXLAN header, with the I flag set and the 24 bit VNI. Then UDP, destination port 4789, with a source port hashed from the inner frame, so the underlay spreads flows across its paths.

Then an outer IP header from the local VTEP to the remote one, and an outer Ethernet header. RFC 7348 defines it, as an informational document, not a standard.

Those headers cost 50 bytes over IPv4: 14 of Ethernet, 20 of IP, 8 of UDP and 8 of VXLAN. Cisco says a 1500 byte overlay needs a transport of at least 1550. Juniper says 50 to 54 bytes.

The RFC forbids the VTEP from fragmenting, and Juniper and Aruba do not support path MTU discovery over VXLAN. So the underlay MTU is not a tuning option. It is a requirement, and Aruba calls 9000 the ideal.

The original VXLAN learned like a switch: flood unknown traffic to a multicast group per VNI, and learn which MAC sits behind which VTEP from the packets that come back. That needs multicast routing in the underlay, and it floods at scale.

BGP EVPN replaces it with a control plane. Each VTEP announces its own MAC and IP addresses in BGP, and every other VTEP learns them before the first packet.

EVPN has five route types you will meet. Type 2 carries a MAC address, and often its IP. Type 3 announces a VTEP for a VNI, so the others know where to send broadcast and unknown traffic.

Type 1 and Type 4 are for multihoming: Ethernet auto discovery, and Ethernet segment. Type 5 carries IP prefixes, whole subnets or external routes. With VXLAN, the VNI rides in the label field of these routes.

Broadcast, unknown unicast and multicast traffic still exist. With ingress replication, the sending VTEP copies the frame to every VTEP learned from Type 3 routes, and Cisco notes that this needs no multicast in the underlay. The alternative is a multicast group per VNI.

Juniper supports ingress replication, but not PIM bidir or PIM SSM with VXLAN. ARP suppression cuts the flooding further: the leaf answers ARP from what EVPN already told it.

Which VTEP imports what is decided by route targets. Cisco can derive them automatically from the AS number and the VNI, but RFC 8365 limits that to 2 byte AS numbers, and Cisco requires manual route targets for eBGP.

Get one wrong, and Huawei describes the result: one end accepts the route and builds a tunnel, the other does not, and the packets are dropped. A one way fabric.

Routing between subnets is called IRB. Every leaf is the gateway, with the same anycast address and MAC, so a server keeps its gateway wherever it moves.

In asymmetric IRB, the ingress leaf routes and then bridges, so every leaf needs every subnet. In symmetric IRB, both leaves route, through an L3 VNI per VRF, and each leaf keeps only its local subnets. Aruba uses symmetric IRB for distributed gateways. Juniper's default is asymmetric, with symmetric as an option.

A server can connect to two leaves with one link bundle. Both leaves share an Ethernet segment identifier, ten bytes long, and announce it with Type 4 and Type 1 routes.

In all active mode, both leaves forward. One of them is elected designated forwarder for broadcast traffic on each VLAN, and split horizon stops a frame from going back into the segment it came from.

On Cisco NX-OS, a VLAN maps to a vn-segment, and the nve interface carries the VNIs, with BGP for host reachability. On Huawei, a bridge domain gets a VNI, and the NVE interface replicates with a head end peer list.

On Aruba, the vxlan interface maps VNIs to VLANs and to a VRF. On Juniper, a MAC VRF instance with EVPN encapsulation VXLAN.

To check the tunnels: show nve peers on Cisco, display vxlan tunnel on Huawei, show interface vxlan vteps on Aruba, and show ethernet-switching vxlan-tunnel-end-point remote on Juniper.

Then look at the EVPN routes themselves. No Type 3 route from a VTEP means no tunnel to it. No Type 2 route for a MAC means it was never learned.

Three classic failures. A MAC that moves too often: RFC 7432 treats five moves in 180 seconds as a duplicate, and Cisco then holds it down, and after three times in a day, freezes it.

A route target mismatch: the one way fabric. And an underlay MTU that is too small: small pings succeed, while large packets vanish.

NacTrack reads the BGP EVPN sessions of every leaf and spine over SSH, whatever the vendor, and lists each one with its peer, remote AS, state, prefixes and uptime. A session that is down, or a leaf with far fewer prefixes than its neighbours, stands out.