1. Opening
EVPN mobility is not ordinary BGP best-path tuning. When a workload moves, the fabric needs to replace the old host location with a newer one quickly enough that remote VTEPs stop encapsulating toward the previous leaf. Repeated or conflicting moves, however, can indicate a loop, duplicate host, hypervisor problem or broken orchestration. Mobility engineering therefore combines convergence with instability detection.
2. Concept and standards behavior
RFC 7432 defines MAC mobility procedures using the MAC Mobility extended community and sequence information so a newer location can supersede an older advertisement. RFC 9135 extends the context into EVPN IRB. RFC 9721, published in 2025, extends mobility procedures for EVPN-IRB scenarios in which IP-to-MAC bindings change across moves. Cisco NX-OS provides duplicate MAC/IP detection mechanisms, but exact thresholds and recovery behavior are release-specific and must be verified rather than copied from old documentation.
The troubleshooting hierarchy used throughout the batch is underlay IP reachability → VTEP/NVE reachability → BGP L2VPN EVPN family → expected EVPN route type → RT/VNI/VRF membership → MAC/IP/ARP/ND state → hardware forwarding → tenant service. This fixed order prevents a healthy upper-layer object from distracting the engineer from an earlier failed dependency.
3. Scenario
The common lab is a two-leaf, two-spine logical VXLAN EVPN fabric. Leaf-1 and Leaf-2 are VTEPs in AS65000 and peer for the L2VPN EVPN address family through a spine/route-reflector function. Host-A is attached to Leaf-1 and Host-B to Leaf-2. Documentation-safe values are used throughout: VTEP loopbacks 192.0.2.11 and 192.0.2.12, route reflector 192.0.2.254, tenant IPv4 203.0.113.0/24, tenant IPv6 2001:db8:100::/64, L2 VNI 10100 and L3 VNI 50100.
Success criteria
- Underlay reaches every required VTEP endpoint.
- BGP EVPN is Established and the expected family is active.
- The correct EVPN route object appears with approved RT/VNI state.
- NVE and tenant forwarding state match the control plane.
- A positive service test passes.
- A negative tenant-isolation or invalid-state test also passes.
4. Topology diagram
A matching editable SVG and PNG are stored under media/diagrams/.
5. Prerequisites
- Supported Nexus 9000 platform and NX-OS 10.5(x) maintenance release.
- Stable routed underlay among leaf/spine/VTEP loopbacks.
- BGP L2VPN EVPN overlay neighbors or RRs.
- VNI, VLAN, VRF, RD and RT allocation plan.
- NVE source-loopback plan.
- Hardware resource/TCAM prerequisites checked for the exact switch.
- Independent management access and time-synchronized logs.
- Feature support verified before enabling multihoming, multicast or mobility extensions.
6. Baseline configuration
Topic-specific NX-OS excerpt — not a complete deployable fabric configuration.
feature bgp
feature nv overlay
nv overlay evpn
!
! Dynamic host learning is used for the normal mobility case.
! Duplicate-host detection/recovery syntax is release/platform specific;
! verify current NX-OS 10.5(x) support before changing thresholds.
Exact syntax, hardware availability and caveats must be validated against the selected Nexus model and NX-OS maintenance release.
7. Verification before modification
show bgp l2vpn evpn summary
show bgp l2vpn evpn
show nve peers
show nve vni
show interface nve1
show mac address-table
show ip arp
show bgp l2vpn evpn route-type 2
show bgp l2vpn evpn | include 00aa.bbcc.0001
show mac address-table address 00aa.bbcc.0001
show ip arp vrf BLUE 203.0.113.10
show logging
The command list is evidence-oriented: EVPN CLI proves control-plane state; NVE CLI proves overlay endpoint/VNI state; MAC/ARP/VRF/FIB evidence proves service programming. None alone proves the whole fabric.
8. Controlled modification
Move Host-A once from Leaf-1 to Leaf-2 without changing its MAC/IP. Capture old and new Type-2 next hop plus mobility information, remote selected location, MAC table and ARP/ND state. Verify traffic follows the new VTEP.
Predict the exact route type, VNI, NVE peer, gateway object or tenant RIB entry that should change before applying the modification.
9. Fault injection
Illustrative lab — not a real incident.
Repeatedly move or duplicate the host between leaves in the isolated lab. Observe mobility/duplicate detection behavior and any suppression/freeze state. Do not increase thresholds as the first response.
Capture the healthy state and change one variable only. Preserve logs and route evidence before correction.
10. Step-by-step troubleshooting
- Verify the affected tenant/VNI/VRF and exact source/destination.
- Verify VTEP underlay reachability.
- Verify EVPN BGP state and capabilities.
- Identify the EVPN route type expected for the service.
- Inspect RD, RT, next hop and VNI/VRF mapping.
- Verify NVE peer and VNI state.
- Verify local/remote MAC and ARP/ND information.
- Verify anycast gateway or IP-VRF state if routed traffic is involved.
- Verify the hardware/FIB path where platform tooling allows it.
- Test the intended positive flow.
- Test an isolation/negative case.
- Correct the smallest proven cause and repeat the same evidence set.
11. Root cause and correction
The control plane is reporting excessive/conflicting host locations. Fix the underlying dual-active host, loop or orchestration issue first; then use supported recovery procedures and prove the Type-2 location remains stable.
12. Post-fix verification
Verify underlay reachability, EVPN route type, RT/VNI/VRF membership, NVE state, MAC/IP/ARP/ND state, forwarding, tenant isolation and stability. If a feature is platform-specific, record the exact hardware/release in the execution record.
13. Rollback
Restore the last approved EVPN/VNI/VRF/NVE state. Never remove broad route-target or tenant-isolation controls just to force connectivity. Roll back immediately if a tenant imports another tenant's routes, an anycast gateway becomes inconsistent, a multihomed segment shows loop/duplicate symptoms, or unrelated VNIs are withdrawn.
14. Deep-dive engineering notes
Mobility sequence
A newer MAC location can be advertised with higher mobility sequence information so remote PEs prefer it over stale state. This is EVPN host-location logic, not LOCAL_PREF or AS_PATH engineering.
Timeline method
At T0 capture stable Leaf-1 location. At T1 disconnect and move. At T2 capture Leaf-2 advertisement. At T3 verify stale Leaf-1 MAC/ARP state is removed. At T4 hold a stability window and confirm the sequence stops changing.
Duplicate versus legitimate movement
A planned VM move is not the same as two simultaneously active hosts using the same IP/MAC. Duplicate-IP behavior can involve different MAC bindings and is one reason RFC 9721 extends the IRB mobility model.
Cross-system correlation
Compare EVPN move timestamps with virtualization live-migration logs, NIC bonding, orchestration jobs and server events. The network may detect the symptom before the compute platform reveals the cause.
Threshold engineering
VDI/container environments may legitimately move more often than bare-metal servers. Set duplicate/move thresholds from workload behavior and platform limits. Too strict can suppress legitimate mobility; too loose can let a loop churn the fabric.
Recovery acceptance
After clearing duplicate state, require one active attachment, stable Type-2 next hop, stable sequence, consistent MAC/ARP tables, successful bidirectional traffic and no further alarms during the observation window.
15. Production workflow
- Capture the host's Type-2/MAC/ARP state on all leaves.
- Record current VTEP next hop and mobility sequence.
- Perform one controlled move.
- Verify remote control plane selects the new location.
- Verify stale local state disappears.
- Verify forwarding follows the new VTEP.
- Trigger repeated moves only in lab.
- Observe duplicate detection.
- Fix the attachment/orchestration cause.
- Recover duplicate state.
- Hold a stability interval before closure.
Operational exercise — build a mobility SLO
Define what “successful mobility” means before measuring it. Record the time from new attachment to updated Type-2 advertisement, the time until remote VTEPs select the new location, and the time until bidirectional traffic is restored. Do not publish a convergence number as a platform guarantee unless it was actually measured on the target hardware and topology.
Run a single move first, then a burst of legitimate moves, then an intentional duplicate-host condition. The expected control-plane response should differ across those cases. A protection threshold that treats a normal maintenance migration the same as a Layer-2 loop is poorly tuned.
After a duplicate event, preserve the last several mobility updates and system logs. They can reveal whether the host oscillated between exactly two leaves, whether the IP moved between different MAC addresses, or whether several VTEPs learned conflicting state. That pattern often points directly to a bond/LAG, hypervisor, or duplicate-address root cause.
16. Production lessons
- EVPN mobility is host-location state, not normal BGP policy.
- Correlate Type-2, MAC table and ARP/ND.
- Duplicate-host alarms are evidence, not obstacles to clear.
- RFC 9721 is the current EVPN-IRB mobility extension baseline.
- Thresholds are platform/workload specific.
- Require a post-recovery stability window.
17. Knowledge check
- Why can EVPN BGP be Established while tenant forwarding is down?
- What should be identified before changing route targets or VNIs?
- Why is a negative tenant-isolation test required after restoring connectivity?
Answers
- Underlay/VTEP, route-type import, NVE, gateway or forwarding state can fail independently of the BGP session.
- The expected EVPN route type and the first failed layer in the service chain.
- An overbroad RT/VNI fix can restore connectivity by leaking another tenant, which is a security failure.
18. Sources
- RFC 7432 — MAC Mobility procedures
- RFC 9135 — EVPN IRB
- RFC 9721 — Extended Mobility Procedures for EVPN-IRB
- Cisco Nexus 9000 NX-OS duplicate IP/MAC detection guidance
- Cisco Nexus 9000 NX-OS VXLAN Configuration Guide, Release 10.5(x)
No comments:
Post a Comment