1. Opening
The capstone is organized around symptoms rather than features. Users do not report “Type-3 is missing”; they report that a rack cannot reach another rack, new VMs cannot discover peers, one tenant loses north-south traffic, or only a moved workload fails. The runbook maps each symptom to the first control/forwarding layer that should be tested and preserves evidence before changes.
2. Concept and standards behavior
The capstone combines RFC 7432 route semantics, RFC 8365 VXLAN NVO procedures, RFC 9135 IRB, RFC 9136 Type-5 routes and RFC 9572 BUM updates. Cisco NX-OS verification uses EVPN route-type output, NVE peer/VNI state, MAC/IP tables, VRF routing and platform resources. Capacity values are intentionally not hard-coded; they must come from the exact NX-OS 10.5(x) Verified Scalability Guide for the Nexus model and maintenance release under test.
The troubleshooting hierarchy used throughout the batch is underlay IP reachability → VTEP/NVE reachability → BGP L2VPN EVPN family → expected EVPN route type → RT/VNI/VRF membership → MAC/IP/ARP/ND state → hardware forwarding → tenant service. This fixed order prevents a healthy upper-layer object from distracting the engineer from an earlier failed dependency.
3. Scenario
The common lab is a two-leaf, two-spine logical VXLAN EVPN fabric. Leaf-1 and Leaf-2 are VTEPs in AS65000 and peer for the L2VPN EVPN address family through a spine/route-reflector function. Host-A is attached to Leaf-1 and Host-B to Leaf-2. Documentation-safe values are used throughout: VTEP loopbacks 192.0.2.11 and 192.0.2.12, route reflector 192.0.2.254, tenant IPv4 203.0.113.0/24, tenant IPv6 2001:db8:100::/64, L2 VNI 10100 and L3 VNI 50100.
Success criteria
- Underlay reaches every required VTEP endpoint.
- BGP EVPN is Established and the expected family is active.
- The correct EVPN route object appears with approved RT/VNI state.
- NVE and tenant forwarding state match the control plane.
- A positive service test passes.
- A negative tenant-isolation or invalid-state test also passes.
4. Topology diagram
A matching editable SVG and PNG are stored under media/diagrams/.
5. Prerequisites
- Supported Nexus 9000 platform and NX-OS 10.5(x) maintenance release.
- Stable routed underlay among leaf/spine/VTEP loopbacks.
- BGP L2VPN EVPN overlay neighbors or RRs.
- VNI, VLAN, VRF, RD and RT allocation plan.
- NVE source-loopback plan.
- Hardware resource/TCAM prerequisites checked for the exact switch.
- Independent management access and time-synchronized logs.
- Feature support verified before enabling multihoming, multicast or mobility extensions.
6. Baseline configuration
Topic-specific NX-OS excerpt — not a complete deployable fabric configuration.
feature bgp
feature nv overlay
nv overlay evpn
!
! Healthy-state capstone objects:
! - VTEP loopback reachability
! - BGP L2VPN EVPN
! - VLAN100/VNI10100
! - VLAN200/VNI10200
! - VRF BLUE/L3VNI50100
! - NVE1 host-reachability protocol bgp
! - consistent anycast gateway
! Full image-validated configuration remains in the executable lab stage.
Exact syntax, hardware availability and caveats must be validated against the selected Nexus model and NX-OS maintenance release.
7. Verification before modification
show bgp l2vpn evpn summary
show bgp l2vpn evpn
show nve peers
show nve vni
show interface nve1
show mac address-table
show ip arp
show bgp l2vpn evpn summary
show bgp l2vpn evpn route-type 2
show bgp l2vpn evpn route-type 3
show bgp l2vpn evpn route-type 5
show nve peers
show nve vni
show ip route vrf BLUE
show mac address-table
show ip arp vrf BLUE
The command list is evidence-oriented: EVPN CLI proves control-plane state; NVE CLI proves overlay endpoint/VNI state; MAC/ARP/VRF/FIB evidence proves service programming. None alone proves the whole fabric.
8. Controlled modification
Capture a complete healthy evidence bundle first. Then inject six faults independently: VTEP underlay loss, missing BUM membership, wrong RT import, inconsistent anycast gateway, repeated host mobility, and missing routed Type-5 prefix. For each fault, identify the first failed layer and correct only that layer.
Predict the exact route type, VNI, NVE peer, gateway object or tenant RIB entry that should change before applying the modification.
9. Fault injection
Illustrative lab — not a real incident.
The capstone contains six separate fictional faults, never combined until the engineer has already proven each symptom independently. This preserves causality and creates a reusable symptom-to-layer matrix.
Capture the healthy state and change one variable only. Preserve logs and route evidence before correction.
10. Step-by-step troubleshooting
- Verify the affected tenant/VNI/VRF and exact source/destination.
- Verify VTEP underlay reachability.
- Verify EVPN BGP state and capabilities.
- Identify the EVPN route type expected for the service.
- Inspect RD, RT, next hop and VNI/VRF mapping.
- Verify NVE peer and VNI state.
- Verify local/remote MAC and ARP/ND information.
- Verify anycast gateway or IP-VRF state if routed traffic is involved.
- Verify the hardware/FIB path where platform tooling allows it.
- Test the intended positive flow.
- Test an isolation/negative case.
- Correct the smallest proven cause and repeat the same evidence set.
11. Root cause and correction
There is no single capstone root cause. The discipline is to locate the earliest failed invariant—underlay, EVPN route type, tenant membership, gateway/IRB, host location or routed-prefix advertisement—and repair that object without changing healthy layers.
12. Post-fix verification
Verify underlay reachability, EVPN route type, RT/VNI/VRF membership, NVE state, MAC/IP/ARP/ND state, forwarding, tenant isolation and stability. If a feature is platform-specific, record the exact hardware/release in the execution record.
13. Rollback
Restore the last approved EVPN/VNI/VRF/NVE state. Never remove broad route-target or tenant-isolation controls just to force connectivity. Roll back immediately if a tenant imports another tenant's routes, an anycast gateway becomes inconsistent, a multihomed segment shows loop/duplicate symptoms, or unrelated VNIs are withdrawn.
14. Deep-dive engineering notes
Incident A — EVPN up, rack unreachable
First test VTEP underlay reachability. The RR may still advertise remote EVPN routes while VXLAN cannot reach the destination VTEP. Wrong first action: clear BGP.
Incident B — existing unicast works, new-host discovery fails
Inspect Type-3/IMET, ingress replication and ARP/ND suppression. Wrong first action: change Type-2 policy.
Incident C — EVPN route visible, tenant missing it
Inspect route target, VNI and VRF import. The opposite error—an overbroad import—can leak tenants. Wrong first action: create another VNI at random.
Incident D — same-subnet works, inter-subnet fails
Inspect anycast gateway, L3VNI, IP-VRF and IRB state. Wrong first action: rebuild the access VLAN.
Incident E — only a moved host fails
Inspect Type-2 location/mobility sequence, duplicate detection, MAC table and ARP/ND. Wrong first action: clear the entire EVPN table.
Incident F — internal works, external prefix fails
Inspect Type-5/external route advertisement, tenant IP-VRF import and next-hop resolution. Wrong first action: change host MAC policy.
Evidence bundle
Collect timestamped EVPN summary, relevant route types, RDs/RTs, NVE peers/VNIs, underlay route to each VTEP, VRF route table, MAC and ARP/ND entries, configuration diff, mobility logs and platform resources. This gives multiple teams the same incident snapshot.
Capacity as a root cause
Resource exhaustion can mimic protocol faults. Compare MAC, ARP/ND, BGP path, VNI, NVE peer and hardware resource counts against the exact verified scalability guide for the model/release. Do not borrow limits from another Nexus SKU.
Escalation
Escalate when control-plane state is correct but hardware forwarding disagrees, when resources exhaust below documented limits, or when a reproducible DF/mobility state conflicts with documented behavior. Preserve a minimal reproducible topology and packet evidence where authorized.
15. Production workflow
- Capture underlay routes to every VTEP.
- Capture EVPN neighbor state.
- Export Type-2/3/5 tables.
- Export NVE peers/VNIs.
- Export tenant VRF route table.
- Export MAC and ARP/ND.
- Record anycast gateway state.
- Test same-subnet forwarding.
- Test inter-subnet forwarding.
- Test external-prefix reachability.
- Run tenant-isolation negative test.
- Inject one fault.
- Identify the first failed layer.
- Correct only that layer.
- Re-run the full acceptance suite.
- Store evidence as the production baseline.
Operational exercise — make the capstone repeatable
Turn the six incident scenarios into an automated evidence checklist rather than a one-time lab. Each run should timestamp the underlay route to every VTEP, EVPN route counts by type, NVE peer count, VNI state, tenant route count, duplicate-host alarms and selected hardware-resource counters. Store the results as structured text or JSON alongside the human-readable troubleshooting notes.
When a future software upgrade or fabric expansion occurs, re-run the capstone and compare the new evidence bundle with the previous baseline. This catches behavioral changes that a simple “all pings pass” test may miss—for example different BUM membership, unexpected Type-5 growth, reduced ECMP, or a new duplicate-host alarm.
The final sign-off should include an owner for every failed negative test. A tenant leak, excessive route count, or inconsistent gateway that does not break the primary ping is still a release blocker. This keeps the capstone aligned with production safety rather than demo success.
16. Production lessons
- Fixed layer order reduces EVPN mean time to isolate.
- Route type is the quickest control-plane discriminator.
- EVPN BGP health does not prove NVE/underlay health.
- RT/VNI errors can cause leaks as well as outages.
- Mobility failures require host-location evidence.
- Capacity claims must use current hardware-specific guides.
- Passing connectivity without tenant isolation is a failed capstone.
17. Knowledge check
- Why can EVPN BGP be Established while tenant forwarding is down?
- What should be identified before changing route targets or VNIs?
- Why is a negative tenant-isolation test required after restoring connectivity?
Answers
- Underlay/VTEP, route-type import, NVE, gateway or forwarding state can fail independently of the BGP session.
- The expected EVPN route type and the first failed layer in the service chain.
- An overbroad RT/VNI fix can restore connectivity by leaking another tenant, which is a security failure.
18. Sources
- RFC 7432 — EVPN route model
- RFC 8365 — EVPN/VXLAN NVO
- RFC 9135 — EVPN IRB
- RFC 9136 — EVPN Type-5
- RFC 9572 — updated BUM procedures
- Cisco Nexus 9000 NX-OS VXLAN Configuration Guide, Release 10.5(x)
- Cisco Nexus 9000 NX-OS Verified Scalability Guide, Release 10.5(x)
No comments:
Post a Comment