Skip to content
Reticux Engineering by Reticux Engineering

Day 76 — Distributed Anycast Gateway and EVPN Integrated Routing and Bridging

Reticux Engineering —

1. Opening

Distributed anycast gateway moves the tenant first-hop router to every participating VTEP. Hosts use the same gateway IP and MAC after workload movement, while EVPN IRB carries enough host and prefix information to bridge within subnets and route between them. The price of that convenience is an additional routed overlay layer: MAC-VRF/L2VNI state must interact correctly with IP-VRF/L3VNI state.

2. Concept and standards behavior

RFC 9135 defines Integrated Routing and Bridging in EVPN and explains symmetric/asymmetric forwarding models. RFC 9136 defines the EVPN IP Prefix route. RFC 7432/8365 provide the EVPN/VXLAN foundation. Cisco NX-OS VXLAN EVPN on supported Nexus platforms uses distributed anycast gateway for extended VXLAN VLAN SVIs, with a common gateway MAC/IP identity across participating VTEPs.

The troubleshooting hierarchy used throughout the batch is underlay IP reachability → VTEP/NVE reachability → BGP L2VPN EVPN family → expected EVPN route type → RT/VNI/VRF membership → MAC/IP/ARP/ND state → hardware forwarding → tenant service. This fixed order prevents a healthy upper-layer object from distracting the engineer from an earlier failed dependency.

3. Scenario

The common lab is a two-leaf, two-spine logical VXLAN EVPN fabric. Leaf-1 and Leaf-2 are VTEPs in AS65000 and peer for the L2VPN EVPN address family through a spine/route-reflector function. Host-A is attached to Leaf-1 and Host-B to Leaf-2. Documentation-safe values are used throughout: VTEP loopbacks 192.0.2.11 and 192.0.2.12, route reflector 192.0.2.254, tenant IPv4 203.0.113.0/24, tenant IPv6 2001:db8:100::/64, L2 VNI 10100 and L3 VNI 50100.

Success criteria

  1. Underlay reaches every required VTEP endpoint.
  2. BGP EVPN is Established and the expected family is active.
  3. The correct EVPN route object appears with approved RT/VNI state.
  4. NVE and tenant forwarding state match the control plane.
  5. A positive service test passes.
  6. A negative tenant-isolation or invalid-state test also passes.

4. Topology diagram

A matching editable SVG and PNG are stored under media/diagrams/.

5. Prerequisites

  • Supported Nexus 9000 platform and NX-OS 10.5(x) maintenance release.
  • Stable routed underlay among leaf/spine/VTEP loopbacks.
  • BGP L2VPN EVPN overlay neighbors or RRs.
  • VNI, VLAN, VRF, RD and RT allocation plan.
  • NVE source-loopback plan.
  • Hardware resource/TCAM prerequisites checked for the exact switch.
  • Independent management access and time-synchronized logs.
  • Feature support verified before enabling multihoming, multicast or mobility extensions.

6. Baseline configuration

Topic-specific NX-OS excerpt — not a complete deployable fabric configuration.

feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
nv overlay evpn
!
fabric forwarding anycast-gateway-mac 0001.0001.0001
!
vrf context BLUE
  vni 50100
!
interface vlan100
  no shutdown
  vrf member BLUE
  ip address 203.0.113.1/24
  fabric forwarding mode anycast-gateway
!
interface nve1
  member vni 50100 associate-vrf

Exact syntax, hardware availability and caveats must be validated against the selected Nexus model and NX-OS maintenance release.

7. Verification before modification

show bgp l2vpn evpn summary
show bgp l2vpn evpn
show nve peers
show nve vni
show interface nve1
show mac address-table
show ip arp
show fabric forwarding anycast-gateway-mac
show bgp l2vpn evpn route-type 2
show bgp l2vpn evpn route-type 5
show ip route vrf BLUE
show nve vni
show ip arp vrf BLUE

The command list is evidence-oriented: EVPN CLI proves control-plane state; NVE CLI proves overlay endpoint/VNI state; MAC/ARP/VRF/FIB evidence proves service programming. None alone proves the whole fabric.

8. Controlled modification

Add a second tenant subnet/VNI and keep the anycast gateway identity identical on Leaf-1/Leaf-2. Verify same-subnet bridging first, then inter-subnet routing through VRF BLUE/L3VNI. Capture both host-oriented and prefix-oriented EVPN routes used by the design.

Predict the exact route type, VNI, NVE peer, gateway object or tenant RIB entry that should change before applying the modification.

9. Fault injection

Illustrative lab — not a real incident.

Configure a different anycast gateway MAC on Leaf-2 for the same tenant subnet while leaving EVPN BGP healthy. The failure becomes location-dependent and appears after hosts attach or move to the inconsistent leaf.

Capture the healthy state and change one variable only. Preserve logs and route evidence before correction.

10. Step-by-step troubleshooting

  1. Verify the affected tenant/VNI/VRF and exact source/destination.
  2. Verify VTEP underlay reachability.
  3. Verify EVPN BGP state and capabilities.
  4. Identify the EVPN route type expected for the service.
  5. Inspect RD, RT, next hop and VNI/VRF mapping.
  6. Verify NVE peer and VNI state.
  7. Verify local/remote MAC and ARP/ND information.
  8. Verify anycast gateway or IP-VRF state if routed traffic is involved.
  9. Verify the hardware/FIB path where platform tooling allows it.
  10. Test the intended positive flow.
  11. Test an isolation/negative case.
  12. Correct the smallest proven cause and repeat the same evidence set.

11. Root cause and correction

The distributed first-hop gateway invariant is broken. Restore the common gateway identity and verify SVI/VRF/L3VNI, ARP/ND and routed tenant forwarding before changing route-reflector policy.

12. Post-fix verification

Verify underlay reachability, EVPN route type, RT/VNI/VRF membership, NVE state, MAC/IP/ARP/ND state, forwarding, tenant isolation and stability. If a feature is platform-specific, record the exact hardware/release in the execution record.

13. Rollback

Restore the last approved EVPN/VNI/VRF/NVE state. Never remove broad route-target or tenant-isolation controls just to force connectivity. Roll back immediately if a tenant imports another tenant's routes, an anycast gateway becomes inconsistent, a multihomed segment shows loop/duplicate symptoms, or unrelated VNIs are withdrawn.

14. Deep-dive engineering notes

MAC-VRF and IP-VRF

The MAC-VRF represents bridging context; the IP-VRF represents routed tenant context. Inter-subnet traffic must cross from the ingress bridge/subnet into the IP-VRF and then to the egress segment.

Symmetric IRB packet walk

Host-A sends to the anycast gateway on Leaf-1. Leaf-1 routes in VRF BLUE, carries the packet across the overlay in the tenant L3VNI/IP-VRF context, and Leaf-2 routes/bridges into the destination subnet. This keeps routing symmetric at both VTEPs and can reduce the need to instantiate every remote L2 segment everywhere.

Asymmetric IRB

In asymmetric IRB, ingress routing may send directly into the remote destination L2 segment, which can require broader L2 state. Troubleshooting commands copied from a symmetric design may therefore be misleading.

Type-2 and Type-5

Host MAC/IP information commonly uses Type 2. Prefix-oriented reachability can use Type 5. Know which object the tenant design expects before declaring a route missing.

ARP/ND suppression

Distributed host bindings can reduce flooding. A stale control-plane binding can therefore cause a failure even when traditional broadcast behavior would have relearned the host.

Anycast gateway audit

Continuously compare gateway MAC, subnet IP, VRF membership, VLAN/VNI, L3VNI and RTs across leaves. A single inconsistent leaf produces workload-location-dependent outages.

Tenant-isolation proof

Build a second VRF with an overlapping prefix. Confirm BLUE routes between its own subnets but cannot import RED's reachability. Positive connectivity without the negative isolation test is incomplete acceptance.

15. Production workflow

  1. Define tenant VRF and L3VNI.
  2. Define L2VNIs/subnets.
  3. Configure the identical anycast gateway identity on every participating VTEP.
  4. Verify same-subnet Type-2/bridging.
  5. Verify IP-VRF/L3VNI.
  6. Test inter-subnet routing.
  7. Verify Type-5 where used.
  8. Inspect ARP/ND suppression state.
  9. Move a host between leaves.
  10. Prove the default gateway does not change.
  11. Run negative cross-VRF isolation tests.

Operational exercise — routed-overlay invariants

Build an automated comparison that exports the anycast gateway MAC, every tenant SVI gateway IP, VRF membership, L2VNI, L3VNI and route targets from all VTEPs. Flag differences before a change is committed. Distributed routing removes a centralized gateway bottleneck but makes consistency a distributed configuration requirement.

Test the routed service from four perspectives: Host-A to its gateway, Host-A to Host-B in another subnet, Host-A to an external Type-5 prefix, and Host-A to an overlapping address in another tenant that must fail. These tests prove local gateway behavior, IRB, external route integration and isolation separately.

If a route exists in the IP-VRF but forwarding fails, inspect recursive VTEP resolution and hardware programming before changing route targets. If the route is absent from the IP-VRF but visible as EVPN NLRI, inspect import policy. This simple distinction prevents many unnecessary overlay resets.

16. Production lessons

  • Anycast gateway identity is a fabric-wide invariant.
  • Separate MAC-VRF/L2VNI from IP-VRF/L3VNI.
  • Know whether the design uses symmetric or asymmetric IRB.
  • Type-2 and Type-5 answer different routing questions.
  • ARP/ND state is part of the EVPN routed service.
  • Tenant isolation is mandatory acceptance evidence.

17. Knowledge check

  1. Why can EVPN BGP be Established while tenant forwarding is down?
  2. What should be identified before changing route targets or VNIs?
  3. Why is a negative tenant-isolation test required after restoring connectivity?

Answers

  1. Underlay/VTEP, route-type import, NVE, gateway or forwarding state can fail independently of the BGP session.
  2. The expected EVPN route type and the first failed layer in the service chain.
  3. An overbroad RT/VNI fix can restore connectivity by leaking another tenant, which is a security failure.

18. Sources

  • RFC 9135 — Integrated Routing and Bridging in EVPN
  • RFC 9136 — IP Prefix Advertisement in EVPN
  • RFC 7432 — EVPN
  • RFC 8365 — EVPN/VXLAN overlay
  • Cisco Nexus 9000 NX-OS VXLAN Configuration Guide, Release 10.5(x)
Written by Reticux Engineering

Practical, lab-tested network engineering: Cisco, Juniper and Fortinet configurations, troubleshooting steps and exam-focused practice.

No comments:

Learn BGP, Cisco routing and network security with daily lab-driven lessons, from CCNA basics to CCNP level policy design.