Day 74 — VXLAN BGP EVPN: Underlay, VTEPs, VNIs and the Control Plane

1. Opening

VXLAN and EVPN solve different parts of the same fabric problem. VXLAN is an encapsulation/overlay data-plane framework carried over IP. EVPN is the BGP control plane that distributes reachability and membership information. A production fabric therefore has three independent health domains: the routed underlay, the EVPN overlay control plane, and the VXLAN/NVE forwarding plane.

2. Concept and standards behavior

RFC 7348 defines VXLAN and its VNI concept. RFC 8365 specifies how EVPN is used as a network-virtualization overlay control plane, including VXLAN encapsulation. RFC 7432 provides EVPN reachability semantics, and RFC 9572 updates parts of BUM procedures. Cisco Nexus 9000 NX-OS 10.5(x) uses the NVE interface as a VTEP function and MP-BGP L2VPN EVPN as the overlay control plane on supported hardware.

The troubleshooting hierarchy used throughout the batch is underlay IP reachability → VTEP/NVE reachability → BGP L2VPN EVPN family → expected EVPN route type → RT/VNI/VRF membership → MAC/IP/ARP/ND state → hardware forwarding → tenant service. This fixed order prevents a healthy upper-layer object from distracting the engineer from an earlier failed dependency.

3. Scenario

The common lab is a two-leaf, two-spine logical VXLAN EVPN fabric. Leaf-1 and Leaf-2 are VTEPs in AS65000 and peer for the L2VPN EVPN address family through a spine/route-reflector function. Host-A is attached to Leaf-1 and Host-B to Leaf-2. Documentation-safe values are used throughout: VTEP loopbacks 192.0.2.11 and 192.0.2.12, route reflector 192.0.2.254, tenant IPv4 203.0.113.0/24, tenant IPv6 2001:db8:100::/64, L2 VNI 10100 and L3 VNI 50100.

Success criteria

1. Underlay reaches every required VTEP endpoint.
2. BGP EVPN is Established and the expected family is active.
3. The correct EVPN route object appears with approved RT/VNI state.
4. NVE and tenant forwarding state match the control plane.
5. A positive service test passes.
6. A negative tenant-isolation or invalid-state test also passes.

4. Topology diagram

A matching editable SVG and PNG are stored under media/diagrams/.

5. Prerequisites

• Supported Nexus 9000 platform and NX-OS 10.5(x) maintenance release.
• Stable routed underlay among leaf/spine/VTEP loopbacks.
• BGP L2VPN EVPN overlay neighbors or RRs.
• VNI, VLAN, VRF, RD and RT allocation plan.
• NVE source-loopback plan.
• Hardware resource/TCAM prerequisites checked for the exact switch.
• Independent management access and time-synchronized logs.
• Feature support verified before enabling multihoming, multicast or mobility extensions.

6. Baseline configuration

Topic-specific NX-OS excerpt — not a complete deployable fabric configuration.

feature bgp
feature nv overlay
nv overlay evpn
!
vlan 100
  vn-segment 10100
!
interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10100
    ingress-replication protocol bgp

Exact syntax, hardware availability and caveats must be validated against the selected Nexus model and NX-OS maintenance release.

7. Verification before modification

show bgp l2vpn evpn summary
show bgp l2vpn evpn
show nve peers
show nve vni
show interface nve1
show mac address-table
show ip arp
show ip route 192.0.2.12
show nve peers
show nve vni
show bgp l2vpn evpn route-type 2
show bgp l2vpn evpn route-type 3
show interface nve1

The command list is evidence-oriented: EVPN CLI proves control-plane state; NVE CLI proves overlay endpoint/VNI state; MAC/ARP/VRF/FIB evidence proves service programming. None alone proves the whole fabric.

8. Controlled modification

Build the service in phases: prove VTEP loopback reachability, bring up EVPN BGP with no tenant VNI, add VNI 10100, verify Type-3 membership, attach Host-A, then verify Type-2 and VXLAN forwarding. Each phase should have its own acceptance evidence.

Predict the exact route type, VNI, NVE peer, gateway object or tenant RIB entry that should change before applying the modification.

9. Fault injection

Illustrative lab — not a real incident.

Remove the underlay route to Leaf-2 VTEP loopback while leaving the route reflector reachable. EVPN routes may remain visible through the RR while the NVE data path to Leaf-2 fails.

Capture the healthy state and change one variable only. Preserve logs and route evidence before correction.

10. Step-by-step troubleshooting

1. Verify the affected tenant/VNI/VRF and exact source/destination.
2. Verify VTEP underlay reachability.
3. Verify EVPN BGP state and capabilities.
4. Identify the EVPN route type expected for the service.
5. Inspect RD, RT, next hop and VNI/VRF mapping.
6. Verify NVE peer and VNI state.
7. Verify local/remote MAC and ARP/ND information.
8. Verify anycast gateway or IP-VRF state if routed traffic is involved.
9. Verify the hardware/FIB path where platform tooling allows it.
10. Test the intended positive flow.
11. Test an isolation/negative case.
12. Correct the smallest proven cause and repeat the same evidence set.

11. Root cause and correction

The first failed invariant is VTEP underlay reachability. Restore the route/ECMP path to the NVE source loopback and verify NVE peer state. Editing route targets or clearing EVPN BGP would not address the failing layer.

12. Post-fix verification

Verify underlay reachability, EVPN route type, RT/VNI/VRF membership, NVE state, MAC/IP/ARP/ND state, forwarding, tenant isolation and stability. If a feature is platform-specific, record the exact hardware/release in the execution record.

13. Rollback

Restore the last approved EVPN/VNI/VRF/NVE state. Never remove broad route-target or tenant-isolation controls just to force connectivity. Roll back immediately if a tenant imports another tenant's routes, an anycast gateway becomes inconsistent, a multihomed segment shows loop/duplicate symptoms, or unrelated VNIs are withdrawn.

14. Deep-dive engineering notes

Underlay

The underlay only needs to provide stable IP reachability between VTEP endpoints and overlay BGP peers according to the chosen architecture. It should converge independently of tenant state.

Overlay control plane

EVPN BGP distributes Type-2/3 and other routes. The route reflector is normally not in the VXLAN tenant data path. It can therefore be healthy while VTEP-to-VTEP forwarding is broken.

NVE/VXLAN data plane

NVE peer and VNI state prove the VTEP function has usable remote endpoints. A Type-2 route in BGP without a usable NVE peer is control-plane-only reachability.

VNI mapping

An L2VNI represents a tenant L2 segment. Wrong VLAN-to-VNI mapping can produce a leak or black hole. Treat the mapping as part of tenant security policy.

BUM versus known unicast

Test an unknown destination/ARP flow separately from an already learned MAC. This catches missing Type-3/ingress-replication state that a warmed-up ping may hide.

MTU

VXLAN adds outer headers. Validate underlay MTU using an encapsulation-aware test. A fabric can have perfect BGP and still drop larger tenant frames.

Route-reflector reachability trap

The RR may be reachable using loopback0 while NVE uses loopback1. Losing only loopback1 reachability creates a classic state where BGP EVPN is Established but tenant VXLAN traffic is dead.

15. Production workflow

1. Inventory all VTEP source loopbacks.
2. Verify bidirectional underlay routes and ECMP.
3. Validate underlay MTU.
4. Establish EVPN BGP before adding tenants.
5. Add one reference L2VNI.
6. Verify Type-3 membership.
7. Attach one host and verify Type 2.
8. Verify NVE peer and remote MAC programming.
9. Test BUM discovery and known unicast separately.
10. Fail one underlay path and observe independent convergence.
11. Store baseline outputs for comparison.

Operational exercise — isolate overlay from fabric transport

Create two monitoring views: one that knows nothing about tenants and checks only loopback reachability/ECMP/MTU, and a second that knows nothing about the IGP and checks EVPN/NVE/VNI state. During a planned underlay failure, the first view should show path reconvergence while the second should remain stable if VTEP reachability is preserved. During an EVPN route-policy fault, the underlay view should remain green while the overlay view changes. This separation prevents one giant “fabric health” dashboard from hiding the layer that actually failed.

Also verify the VTEP source address seen in VXLAN packets matches the NVE source-loopback design. A mis-sourced tunnel can work in a tiny lab because of incidental routes but fail when ECMP, ACLs, or routed boundaries are introduced. Record the VTEP source as part of the fabric inventory.

Finally, add one capacity check before onboarding a new VNI: compare current VNI, MAC, ARP/ND and EVPN route counts against the exact platform's verified scale guidance. A control plane can be architecturally correct yet fail operationally when hardware resources are exhausted.

16. Production lessons

• Underlay, EVPN BGP and VXLAN forwarding are separate failure domains.
• Do not use EVPN neighbor state as proof of VTEP reachability.
• Validate VLAN/VNI mapping and MTU explicitly.
• Test BUM and known unicast separately.
• Keep the RR role conceptually separate from the VTEP data path.

17. Knowledge check

1. Why can EVPN BGP be Established while tenant forwarding is down?
2. What should be identified before changing route targets or VNIs?
3. Why is a negative tenant-isolation test required after restoring connectivity?

Answers

1. Underlay/VTEP, route-type import, NVE, gateway or forwarding state can fail independently of the BGP session.
2. The expected EVPN route type and the first failed layer in the service chain.
3. An overbroad RT/VNI fix can restore connectivity by leaking another tenant, which is a security failure.

18. Sources

• RFC 7348 — VXLAN
• RFC 8365 — EVPN as an NVO solution
• RFC 7432 — EVPN
• RFC 9572 — updated EVPN BUM procedures
• Cisco Nexus 9000 NX-OS VXLAN Configuration Guide, Release 10.5(x)