Day 41 — BGP Confederations: Sub-AS Design, External View and Migration

at

1. Opening

Confederations are another way to scale BGP inside a large administrative domain. They divide the domain into member autonomous systems while presenting a single confederation identifier to external peers. They are powerful, but their operational model is more complex than simply 'using private ASNs inside.'

The engineering goal is not to memorize another BGP command. It is to understand what information each speaker is allowed to propagate, what path information can be hidden, and what failure domain is created by the chosen control-plane architecture.

2. Concept and standards behavior

RFC 5065 defines AS_CONFED_SEQUENCE and AS_CONFED_SET and how member-AS relationships are represented. Confederation external sessions have eBGP-like properties inside the confederation, while the confederation is presented externally as one AS. Modern guidance must also account for the fact that RFC 9774 prohibits new origination of AS_SET/AS_CONFED_SET in ordinary aggregation contexts; that does not erase RFC 5065's confederation architecture, but it reinforces the need to distinguish attributes and mechanisms carefully.

A recurring rule in this module is that configuration scale and routing-information scale are different problems. A design can be easy to configure but still create poor path visibility, slow convergence, or an oversized blast radius. Conversely, a topology can carry complete information but be operationally expensive to maintain.

3. Scenario

Create confederation identifier 65000 with member AS65010 and AS65020. Establish a session between members and an external peer to AS65100. Verify the path representation internally and the single confederation identity seen externally.

Documentation-safe addressing is used. The common lab AS is 65010. Internal loopbacks use 192.0.2.0/24 host routes; external test prefixes use 203.0.113.0/24 and 198.51.100.0/24 where needed. All fictional failures are lab-only.

Success criteria

  1. Every speaker learns the routes it is supposed to learn.
  2. The propagation rule can be explained before commands are applied.
  3. Loop-prevention attributes are visible where the feature uses them.
  4. A negative test proves that an invalid or unintended propagation does not occur.
  5. Rollback returns the topology to a known-good control-plane state.

4. Topology diagram

A matching SVG/PNG diagram is stored in media/diagrams/.

5. Prerequisites

  • IOS XE 17.18.x documentation baseline.
  • Stable IGP reachability among BGP loopbacks.
  • Explicit update-source and appropriate neighbor reachability for loopback-based sessions.
  • Authentication only if the lab image and design have been validated for it.
  • Independent management access before changing route-reflection/confederation policy.
  • Pre-change capture of BGP summary, selected paths and relevant neighbor state.

6. Baseline configuration

Topic-specific configuration excerpt — not a complete device configuration.

router bgp 65010
 bgp confederation identifier 65000
 bgp confederation peers 65020
 neighbor 192.0.2.20 remote-as 65020
!
! Member AS65020 uses the same confederation identifier
! and declares AS65010 as a confederation peer.

Do not silently transfer this syntax to IOS XR or NX-OS. Those platforms must use their own policy/configuration model.

7. Verification before modification

Use a combination of topology, session and route evidence:

show bgp ipv4 unicast summary
show bgp ipv4 unicast
show bgp ipv4 unicast 203.0.113.0/24
show bgp ipv4 unicast neighbors
show ip route

For route-reflection topics also inspect the path detail for ORIGINATOR_ID and CLUSTER_LIST where present. The absence or presence of a route must be explained by the propagation rule, not guessed from neighbor state alone.

8. Controlled modification

Add a second member AS and trace a route across two member-AS boundaries to the external peer. Verify that external policy still sees the intended confederation AS.

Predict the exact peers whose Adj-RIB-In/Loc-RIB should change. Then apply one control-plane change, re-check path detail, and verify that no unrelated peer loses reachability.

9. Fault injection

Illustrative lab — not a real incident.

Misconfigure a member relationship as ordinary eBGP by omitting the confederation peer declaration. Observe changes in path handling and session semantics.

Capture the state before and after the fault. A useful fault must change one causal variable only.

10. Step-by-step troubleshooting

  1. Confirm IGP reachability among BGP endpoints.
  2. Confirm TCP/BGP sessions are Established.
  3. Identify where the route originated.
  4. Trace the route one BGP hop at a time.
  5. Determine whether the receiving neighbor is eBGP, ordinary iBGP, RR client, RR non-client, or confederation peer.
  6. Inspect best-path state before assuming propagation is broken.
  7. Inspect ORIGINATOR_ID/CLUSTER_LIST for reflected routes.
  8. Inspect policy and next-hop reachability.
  9. Verify whether an alternative path was never learned, learned but not selected, or selected but not advertised.
  10. Apply the smallest proven correction.
  11. Re-run the same positive and negative tests.
  12. Observe stability before closing the change.

11. Root cause and correction

The cause is inconsistent confederation membership configuration. Correct the member-AS and confederation-peer declarations on both sides, then re-check path attributes and external view.

The correction must address the architectural cause rather than adding random neighbor statements until the route appears.

12. Post-fix verification

Verify:

  • all intended sessions remain Established;
  • the expected prefix is learned by the intended speakers;
  • reflected attributes are correct where applicable;
  • next hop remains reachable;
  • no routing loop is created;
  • the negative propagation test still passes.

13. Rollback

Restore the previous neighbor relationship or policy, not an ad-hoc alternative. If the change affects route reflection, preserve enough connectivity that clients do not become isolated during rollback. Roll back for unexpected loss of route visibility, oscillation, unintended path change, or evidence that the design created a larger failure domain than approved.

14. Production lessons

Confederations are a control-plane architecture, not merely an ASN trick. Prefer the simplest design that meets scale, policy and failure-domain requirements; route reflection is more common and often easier to operate.

Deep-dive engineering notes

Confederation identity has two scopes

A confederation divides a large routing domain into member ASes for internal scaling and policy, while external peers see the confederation identifier rather than the internal member structure. This dual view is the core architectural idea. Engineers must know whether a policy is matching the member-AS representation used inside the confederation or the public/external AS representation seen outside.

Member-AS boundaries are neither ordinary iBGP nor ordinary Internet eBGP

Sessions between confederation members borrow eBGP-like behavior for scaling while preserving the confederation's single external identity. Treating them exactly like normal eBGP can lead to incorrect assumptions about AS-path representation and policy. Treating them like ordinary iBGP loses the reason confederations exist.

Route reflection versus confederations

Both reduce the universal iBGP full-mesh requirement, but their operational models differ. Route reflection changes propagation relationships inside one AS. Confederations introduce member-AS structure and explicit boundaries. A network with strong administrative regions and highly structured internal policy may benefit from confederation semantics, while many deployments favor route reflection because it is simpler and more common operationally.

Migration risk

A confederation migration changes session roles and path representation. Build coexistence carefully, define which routers move first, and test route policy that matches AS paths. A regex written for the pre-migration AS_PATH may behave differently when member-AS segments appear internally.

Oscillation awareness

RFC 7964 documents that route reflection or confederations can interact with certain topologies and policies to create persistent route oscillation. That is not a reason to avoid both mechanisms; it is a reason to validate policy ordering, MED behavior, topology and path visibility rather than assuming the scaling architecture is neutral.

15. Knowledge check

  1. Which problem in this post is a control-plane topology problem rather than a command-syntax problem?
  2. What evidence distinguishes “route was never learned” from “route was learned but not selected”?
  3. What negative test proves the scaling feature has not introduced unintended propagation?

Answers

  1. The relationship among BGP speakers and the rules governing propagation.
  2. Per-prefix BGP path detail and neighbor route evidence.
  3. Verify a route that should remain hidden/unadvertised is absent from the relevant neighbor's learned/advertised state.

16. Sources

  • RFC 5065 — Autonomous System Confederations for BGP
  • RFC 4271 — BGP-4
  • RFC 7964 — persistent route oscillation considerations
Labels:

Featured Post