Day 47 — BGP Session Authentication: TCP-MD5, TCP-AO and Key-Rollover Boundaries

1. Opening

BGP sessions are long-lived TCP connections and attractive disruption targets. Session authentication protects the TCP exchange from unauthenticated injection, but standards evolution and real platform support must be kept separate.

A session-engineering problem should be solved from the bottom up. Before modifying route policy, prove endpoint reachability, source address, TCP behavior, TTL/security controls, authentication and negotiated BGP parameters. An Established session is the result of all of those layers agreeing.

2. Concept and standards behavior

RFC 2385 defined the TCP MD5 Signature Option and was widely deployed with BGP. RFC 5925 later obsoleted it with TCP-AO, which supports stronger MACs, replay protection for long-lived connections and improved key handling. RFC 7454 notes the standards advantage of TCP-AO while acknowledging continuing MD5 deployment. Cisco availability and syntax must be verified on the exact product and release.

The engineering boundary matters: RFC behavior defines interoperable protocol rules, while dynamic-neighbor syntax, password configuration, peer templates and some security commands are implementation-specific. Exact syntax and feature support must therefore be verified on the actual IOS XE platform and release used in the lab or production network.

3. Scenario

Two eBGP peers use a lab-only placeholder key. First validate an MD5-protected session where supported. Separately document TCP-AO as the preferred standards mechanism without claiming it is identically available on every IOS XE 17.18.x platform.

Success criteria

1. The intended TCP endpoints and source addresses are explicit.
2. BGP reaches Established without weakening unrelated security controls.
3. Negotiated capabilities match the required address families/features.
4. A negative test proves an unauthorized or incorrectly formed session fails.
5. Rollback returns the neighbor relationship to the captured baseline.

4. Topology diagram

A matching editable SVG and PNG are stored under media/diagrams/.

5. Prerequisites

• Cisco IOS XE 17.18.x documentation baseline.
• Documentation-safe IPv4 addressing only.
• Stable underlay reachability before BGP troubleshooting.
• Independent management access.
• Time synchronization and logging for authentication/failure analysis.
• Pre-change capture of neighbor state and relevant TCP/BGP diagnostics.

6. Baseline configuration

Topic-specific configuration excerpt — not a complete deployable configuration.

router bgp 65010
 neighbor 192.0.2.2 remote-as 65020
 neighbor 192.0.2.2 password LAB-ONLY-PLACEHOLDER
 !
 address-family ipv4
  neighbor 192.0.2.2 activate
 exit-address-family
! Never reuse this placeholder in production.
! Verify TCP-AO support and exact key-chain syntax on the target platform.

Any authentication secret shown in a lab must be a non-production placeholder. Never copy a tutorial key into production.

7. Verification before modification

show bgp ipv4 unicast summary
show bgp ipv4 unicast neighbors
show tcp brief all
show ip route
show logging

When the session does not reach Established, identify the last successful layer: IP reachability, TCP establishment, BGP OPEN exchange, capability negotiation, or post-OPEN KEEPALIVE.

8. Controlled modification

Change the lab key using a synchronized maintenance procedure. Record expected session impact and compare with the platform supported rollover capabilities.

Capture the expected state transition before changing configuration. If the modification is expected to reset the TCP session, state that explicitly and preserve logs.

9. Fault injection

Illustrative lab — not a real incident.

Configure mismatched lab keys on the two peers. IP reachability remains valid, but the authenticated TCP session must not establish successfully.

Inject only one fault at a time. A failed session is useful evidence only if the experiment can identify which layer rejected it.

10. Step-by-step troubleshooting

1. Confirm the configured neighbor address or dynamic range.
2. Confirm local and remote source addresses.
3. Verify a valid route exists to the remote endpoint.
4. Verify return-path reachability.
5. Inspect TCP state and port 179 behavior.
6. Validate TTL, multihop and GTSM assumptions.
7. Validate authentication configuration and key-selection state where used.
8. Inspect BGP FSM state.
9. Inspect OPEN message parameters and negotiated capabilities.
10. Read the NOTIFICATION/error evidence rather than repeatedly clearing the session.
11. Correct the smallest proven cause.
12. Re-run both positive and negative tests.

11. Root cause and correction

The peer keys do not match. Restore the synchronized approved key. Do not fix the outage by removing authentication unless the change plan explicitly authorizes a temporary security downgrade.

12. Post-fix verification

Confirm TCP stability, BGP Established state, expected AFI/SAFI activation, route exchange, absence of recurring NOTIFICATION messages, and intended rejection of the negative test.

13. Rollback

Restore the exact prior neighbor/session configuration. Authentication or TTL changes need a synchronized rollback plan on both peers. Roll back if the session cannot be restored within the approved change window, if another peer is affected, or if the security posture is weakened beyond the approved design.

14. Deep-dive engineering notes

MD5 is obsolete in standards, not absent from networks

Obsolete means RFC 5925 replaced RFC 2385 as the standards recommendation. It does not mean MD5 disappeared from deployed routers or inter-provider agreements.

Key rotation is an operational problem

RFC 4808 exists because coordinated MD5 key changes can be difficult. Plan ownership, timing, rollback and dual-key or rollover capability according to what both implementations support.

TCP-AO support boundary

A standards document can define TCP-AO while a specific hardware/software train may have partial, different or absent support for BGP. Never convert Cisco TCP-AO documentation into a claim that every IOS XE image supports an identical BGP command.

Secret handling

Production keys should live in approved secret-management and change processes. Blog content must never contain real shared secrets, derived keys or customer key-chain names.

15. Production lessons

• Treat transport, security and BGP negotiation as separate diagnostic layers.
• Never weaken authentication or TTL controls merely to force Established state.
• Feature support is platform/release specific; standards support does not guarantee identical CLI support.
• Capture negative tests as part of session acceptance.
• Document which side initiates, what source address is expected, and what capabilities are required.

16. Knowledge check

1. Why can successful IP reachability coexist with a failed BGP session?
2. Which evidence distinguishes TCP failure from an OPEN/capability failure?
3. Why is temporarily removing authentication a poor troubleshooting default?

Answers

1. Reachability proves only IP forwarding; TCP, TTL controls, authentication, FSM parameters and capabilities can still fail.
2. TCP state plus BGP neighbor/error/NOTIFICATION evidence.
3. It changes the security boundary and can hide the actual mismatch instead of proving it.

17. Sources

• RFC 2385 — TCP MD5 Signature Option (obsolete)
• RFC 5925 — TCP Authentication Option
• RFC 4808 — TCP-MD5 key-change strategies
• RFC 7454 — BGP Operations and Security
• Cisco IOS XE 17.x — BGP TCP authentication guidance