Skip to content
Reticux Engineering by Reticux Engineering

Day 81 — EVPN Multicast and BUM Evolution: IMET, SMET and Tenant Routed Multicast

Reticux Engineering —

1. Opening

BUM delivery is necessary for Ethernet semantics, but replicating every multicast flow to every VTEP wastes bandwidth and receiver resources. EVPN multicast extensions add receiver-aware signaling so the network can become more selective while preserving the control-plane model.

The recurring theme in advanced EVPN design is scope. A route can be valid yet scoped to the wrong VRF. A VNI can be healthy inside one site yet fail across DCI. A multicast tree can work but replicate too broadly. A border gateway can restore connectivity while accidentally dissolving tenant isolation. Advanced design therefore requires the engineer to define exactly where reachability, replication, policy and failure domains begin and end.

2. Concept and standards behavior

RFC 7432 uses IMET routes for inclusive multicast/BUM membership. RFC 9251 adds optimized ingress replication procedures and the Selective Multicast Ethernet Tag (SMET) route so downstream interest in multicast groups can be signaled more selectively. RFC 9572 updates BUM procedures, while RFC 9856 addresses redundant multicast-source scenarios in EVPN. These are different layers of multicast optimization, not one universal feature toggle.

The safest EVPN operating model keeps five questions separate:

  1. What EVPN route or membership object represents the information?
  2. Which route target, VNI, site or VRF is allowed to consume it?
  3. Which device is the policy boundary?
  4. What forwarding object is programmed from that control-plane information?
  5. What failure should remain contained if that boundary breaks?

A BGP session being Established answers none of those questions by itself.

3. Scenario

Tenant BLUE has one multicast source behind Leaf-1 and receivers behind Leaf-2 but not Leaf-3. First observe inclusive replication behavior; then introduce selective receiver signaling in a standards-capable lab so only interested downstream PEs receive the flow.

Common advanced-fabric topology

       Site-1 Fabric                         Site-2 Fabric
 Host-A -- Leafs -- BGW-1 ===== DCI ===== BGW-2 -- Leafs -- Host-B
                 \                               /
                  \---- Shared/External Edge ----/

Documentation-safe values:

  • Fabric AS: AS65000
  • Site IDs: 1 and 2
  • Tenant BLUE L3VNI: 50100
  • Tenant RED L3VNI: 50200
  • Shared-services L3VNI: 50900
  • BLUE prefix: 203.0.113.0/24
  • RED prefix: 198.51.100.0/24
  • Shared-services prefix: 192.0.2.128/25
  • IPv6 documentation prefixes from 2001:db8::/32

Success criteria

  1. Only approved tenant prefixes cross each policy boundary.
  2. Site-local failures do not create a fabric-wide route leak.
  3. DCI loss preserves local-site operation.
  4. Shared services are reachable only through explicit policy.
  5. Scale/telemetry thresholds have measurable baselines.
  6. Negative tenant-isolation tests pass after every routing change.

4. Topology diagram

A matching editable SVG and PNG are stored under media/diagrams/.

5. Prerequisites

  • Validated Nexus 9000 platform and NX-OS 10.5(x) maintenance release.
  • Healthy routed underlay and EVPN overlay inside each site.
  • Route-target/VNI inventory under change control.
  • Border-gateway roles documented.
  • Independent management and out-of-band rollback access.
  • Exact platform scale limits taken from the matching Verified Scalability Guide.
  • Telemetry/logging configured before scale or DCI experiments.

6. Baseline configuration

Topic-specific NX-OS excerpt — not a complete deployable configuration.

! Baseline VXLAN EVPN BUM example
interface nve1
  member vni 10100
    ingress-replication protocol bgp
!
! Tenant Routed Multicast / SMET support is platform/release specific.
! Verify NX-OS 10.5(x) TRM and multicast feature support before configuration.

Do not treat conceptual snippets as universal syntax across every Nexus 9000 SKU. Validate feature support, licensing and hardware behavior on the target release.

7. Verification before modification

show bgp l2vpn evpn summary
show bgp l2vpn evpn
show bgp l2vpn evpn route-type 5
show nve peers
show nve vni
show ip route vrf BLUE
show ip route vrf RED
show logging
show bgp l2vpn evpn route-type 3
show nve vni
show ip mroute vrf BLUE
show ip igmp groups vrf BLUE

The purpose of verification is to identify the first policy/forwarding boundary at which healthy state diverges from the design.

8. Controlled modification

Create one receiver group behind Leaf-2 and compare inclusive VNI membership with receiver-specific multicast state. Verify Leaf-3 is not unnecessarily treated as a receiver for the selective flow where the platform supports it.

Before committing, predict the specific EVPN route set, route target, border-gateway advertisement, multicast membership or telemetry metric expected to change.

9. Fault injection

Illustrative lab — not a real incident.

Remove receiver membership but leave stale multicast control state in the lab scenario. Observe unnecessary replication or stale state until control-plane cleanup occurs.

Change one boundary at a time. A multi-site lab is especially vulnerable to ambiguous testing because one local change can alter route reorigination, VNI membership and remote best-path state simultaneously.

10. Step-by-step troubleshooting

  1. Identify source tenant, destination tenant/site and service type.
  2. Verify the source site's local service first.
  3. Verify the expected EVPN route type.
  4. Inspect RD, RT, VNI and next hop.
  5. Determine whether the route should remain site-local or cross DCI.
  6. Check border-gateway import/export/reorigination policy.
  7. Verify DCI/ISN reachability separately from site underlay.
  8. Check remote-site route import.
  9. Verify NVE/VNI forwarding.
  10. For multicast, verify receiver/membership state and replication scope.
  11. For external/shared services, verify leak/default-route direction.
  12. Check platform resources and telemetry for scale symptoms.
  13. Run positive forwarding tests.
  14. Run negative tenant/site-isolation tests.
  15. Correct the smallest proven cause.
  16. Re-run the entire evidence chain.

11. Root cause and correction

The fault is multicast membership/state synchronization, not unicast Type-2 reachability. Restore/refresh the legitimate receiver state and verify replication scope.

The correction must restore the intended scope. A reachability fix that broadens route-target import or distributes a default route to the wrong VRF is a new defect.

12. Post-fix verification

Verify local and remote-site tenant routing, route-target scope, DCI state, NVE peers/VNIs, shared/external reachability, multicast behavior where relevant, platform-resource health and negative isolation.

13. Rollback

Restore the last approved border, RT/VNI, multicast or telemetry policy. Do not roll back by applying broad route-target import or permit-all route policies. Preserve the route and resource evidence that triggered the rollback.

Rollback immediately if:

  • a route crosses into an unapproved VRF;
  • a default route is reflected/leaked in the wrong direction;
  • DCI failure affects unrelated local-site tenants;
  • multicast traffic replicates to nonmember sites/PEs unexpectedly;
  • hardware resource consumption approaches a validated hard limit faster than modeled;
  • the capstone loses tenant isolation.

14. Deep-dive engineering notes

IMET versus SMET

IMET is inclusive: it represents participation used for BUM delivery in the broadcast domain. SMET is selective: it can represent receiver interest for particular multicast traffic, allowing more targeted replication.

TRM

Tenant Routed Multicast extends multicast routing across tenant VRFs/subnets in a VXLAN EVPN fabric. It combines tenant multicast routing with EVPN/VXLAN distribution and requires careful PIM/IGMP/underlay design on Cisco platforms.

Source redundancy

RFC 9856 addresses duplicate multicast traffic from redundant sources in EVPN scenarios. Redundancy can improve availability but creates the possibility of duplicated traffic if source selection/forwarding is not coordinated.

BUM does not equal IP multicast

Ethernet BUM includes broadcast and unknown unicast in addition to multicast. Optimizing IP multicast does not eliminate the need to understand ordinary broadcast/ARP/ND behavior.

Underlay transport choices

Ingress replication duplicates at the ingress VTEP; multicast underlay can replicate in the network. The optimal choice depends on receiver scale, fabric capabilities and operational complexity.

Measuring efficiency

Compare packets/bytes replicated per source, number of downstream VTEPs, group membership and link utilization. “Multicast feature enabled” is not a scale result.

15. Production workflow

  1. Baseline IMET/Type-3 membership.
  2. Record BUM replication peers.
  3. Create one multicast source.
  4. Join one receiver on Leaf-2.
  5. Verify receiver-specific state.
  6. Confirm Leaf-3 has no receiver.
  7. Measure replication on uplinks.
  8. Remove the receiver.
  9. Verify state/traffic withdrawal.
  10. Introduce redundant-source scenario only after basic behavior is stable.
  11. Record the platform-specific TRM/SMET capability matrix.

Multicast acceptance thresholds

Before production rollout, define measurable multicast acceptance criteria: maximum receiver-join convergence time, expected number of downstream VTEPs per group, acceptable replication factor on DCI/uplinks, and maximum stale membership duration after a leave. These values should be derived from the application and platform, not copied from a generic design. Monitoring should alert when replication fan-out grows unexpectedly, because that can indicate stale receiver state or a policy change that converted selective delivery back into broad replication.

Failure-domain test

Fail the source-facing leaf while a receiver remains active. Observe whether the redundant source or alternate path produces one clean stream rather than duplicates. Then fail a receiver-side leaf and verify multicast state is withdrawn without affecting unrelated groups. These tests separate source redundancy, receiver membership and fabric transport into distinct failure domains.

16. Production lessons

  • IMET and SMET have different scopes.
  • Receiver-aware multicast should be proven by replication reduction.
  • TRM is a routed multicast service, not generic BUM handling.
  • Redundant multicast sources need duplicate-traffic controls.
  • Measure multicast scale and traffic, not only BGP route presence.
  • Keep platform feature support explicit.

17. Knowledge check

  1. Why is successful end-to-end reachability insufficient evidence that an advanced EVPN change is correct?
  2. Which boundary should be inspected before broadening route-target import?
  3. What makes a scale threshold operationally defensible?

Answers

  1. Connectivity can be restored by an unintended route leak or over-broad replication; isolation and scope must also pass.
  2. The first point where the expected route or membership state diverges from the design—often the border/leak/import policy.
  3. It is tied to measured baseline, growth rate, exact platform validated capacity, failure headroom and an alert/rollback action.

18. Sources

  • RFC 7432 — EVPN IMET base procedures
  • RFC 9251 — EVPN Optimized Ingress Replication / Selective Multicast Ethernet Tag
  • RFC 9572 — Updates to EVPN BUM procedures
  • RFC 9856 — Multicast Source Redundancy in EVPNs
  • Cisco Nexus 9000 NX-OS VXLAN/TRM documentation 10.5(x)
Written by Reticux Engineering

Practical, lab-tested network engineering: Cisco, Juniper and Fortinet configurations, troubleshooting steps and exam-focused practice.

No comments:

Learn BGP, Cisco routing and network security with daily lab-driven lessons, from CCNA basics to CCNP level policy design.