Day 52 — BGP Graceful Restart: Capability, Stale Routes and End-of-RIB

1. Opening

Graceful Restart asks a neighboring router to keep forwarding on routes learned from a restarting BGP speaker while that speaker rebuilds its control-plane state. The mechanism can reduce churn, but it rests on a critical assumption: forwarding can safely continue while the control plane is recovering.

2. Concept and standards behavior

RFC 4724 defines the Graceful Restart capability, restart time, forwarding-state indications and End-of-RIB marker. A helper can retain routes from a restarting peer as stale instead of immediately withdrawing them. RFC 8538 later extends graceful-restart handling to certain NOTIFICATION scenarios. RFC 9494 defines Long-Lived Graceful Restart, which can retain stale routes beyond ordinary GR and marks them with LLGR_STALE. Longer retention increases the need to reason carefully about forwarding validity.

A recurring discipline in convergence engineering is to separate failure detection, control-plane reaction, RIB decision, and FIB repair. Different features accelerate different stages. A design that reports a fast protocol event but still forwards into a dead path has not achieved useful convergence.

3. Scenario success criteria

1. Define the failure signal before tuning convergence.
2. Capture control-plane and forwarding state before modification.
3. Change only one detection or HA mechanism at a time.
4. Use a positive traffic test and a negative/containment test.
5. Preserve logs and timing evidence through the fault.
6. Roll back if instability increases or forwarding safety becomes less certain.

4. Topology

                 +--------------------+
                 |   CORE / UNDERLAY   |
                 +----------+---------+
                            |
        +-------------------+-------------------+
        |                                       |
   EDGE-A / AS65010                        EDGE-B / AS65020
   Lo0 192.0.2.10                          Lo0 192.0.2.20
        |                                       |
        +----------- service / test ------------+
                     203.0.113.0/24

The exact node roles change by day; the matching SVG/PNG diagram is authoritative for the lab arrangement.

5. Prerequisites

• Cisco IOS XE 17.18.x documentation baseline unless the post explicitly calls out a broader 17.x feature document.
• Documentation-safe addressing and lab-only credentials.
• Stable IGP/underlay before enabling fast failure mechanisms.
• Time synchronization for event sequencing.
• Independent management access.
• Baseline captures of BGP summary, route detail, RIB/FIB state and traffic.

7. Verification before modification

show bgp ipv4 unicast summary
show bgp ipv4 unicast neighbors
show bgp ipv4 unicast 203.0.113.0/24
show ip route
show ip cef 203.0.113.1 detail
show logging

Where relevant, add BFD, redundancy, graceful-restart or platform-specific verification commands documented for the actual image.

10. Step-by-step troubleshooting

1. Establish the exact failure time from logs/telemetry.
2. Determine whether endpoint reachability changed.
3. Determine whether the BGP TCP session changed state.
4. Determine whether BFD/IGP/next-hop tracking changed first.
5. Inspect the selected BGP path and recursive NEXT_HOP.
6. Inspect RIB installation.
7. Inspect CEF/FIB programming.
8. Check stale/GR/NSR state if HA behavior is involved.
9. Compare the observed event order with the expected convergence design.
10. Correct the smallest proven cause.
11. Repeat the identical fault and measurements.
12. Observe stability after recovery.

12. Post-fix verification

Verify BGP state, selected path, route validity, RIB/FIB installation, end-to-end traffic, negative test, logs/counters, and a stability interval appropriate to the lab. A protocol-state recovery is not accepted as success if forwarding remains broken.

13. Rollback

Revert only the new timer, BFD, tracking, GR/NSR or PIC control introduced by the day. Restore the captured baseline and repeat the original verification set. Roll back immediately for repeated oscillation, unexpected multi-peer resets, persistent stale blackholing, loss of forwarding redundancy, or a platform behavior inconsistent with the documented prerequisite.

6. Baseline configuration

Topic-specific configuration excerpt — not a complete deployable configuration.

router bgp 65010
 bgp graceful-restart
 ! Exact neighbor/address-family behavior and timers must be verified on the target platform.

show bgp ipv4 unicast neighbors 192.0.2.2

Exact commands, defaults and support must be verified on the actual IOS XE hardware/software image before execution.

8. Controlled modification

Enable GR in a controlled lab, reset/restart the BGP control process in a way appropriate to the platform, and record whether the helper retains stale routes and when End-of-RIB clears the stale state.

State the expected detection source, expected BGP event, expected RIB change and expected FIB behavior before applying the change.

9. Fault injection

Illustrative lab — not a real incident.

Simulate a control-plane restart while separately breaking the data-plane path. This demonstrates the dangerous case where a helper might preserve stale routing information even though forwarding is no longer valid.

11. Root cause and correction

The failure is an invalid graceful-restart assumption: forwarding did not survive the control-plane event. Correct the HA design or disable/restrict GR for that failure mode instead of extending stale timers.

14. Deep-dive engineering notes

GR is negotiated capability

The neighbor must know the restarting speaker's capability and address-family forwarding state. Enabling a command after a session is already established may require session renegotiation on some Cisco workflows before the capability is exchanged.

Stale is not healthy

A retained route is marked stale because its control-plane owner is temporarily unavailable. Forwarding it is a calculated risk, not proof of reachability.

End-of-RIB matters

After re-advertising routes for an address family, the restarting speaker indicates completion with End-of-RIB so the helper can remove routes that were retained stale but not refreshed.

LLGR deserves separate caution

RFC 9494 allows longer stale retention than ordinary GR. The operational benefit is continuity through longer control-plane events; the corresponding risk is longer blackholing if forwarding is not actually preserved. A future dedicated day can explore LLGR policy and communities in more depth.

15. Production lessons

• Faster failure detection is useful only when the signal is trustworthy.
• Do not combine multiple fast-convergence mechanisms until each works independently.
• Preserve the distinction between control-plane recovery and forwarding-plane continuity.
• High-availability features often have platform, topology and address-family restrictions.
• Measure actual traffic loss and event ordering instead of advertising a theoretical convergence number.

16. Knowledge check

1. Which stage does today's feature accelerate: detection, BGP decision, RIB update, FIB repair, or more than one?
2. What failure could make the feature produce a false sense of availability?
3. What evidence would prove forwarding recovered before the BGP control plane fully reconverged?

Answers

1. See the day's mechanism: the answer must name the specific accelerated stage rather than saying simply 'BGP convergence.'
2. A false/unstable detection signal, unsupported HA state, invalid stale-route assumption, or absent precomputed forwarding alternate depending on the topic.
3. Timestamped traffic/FIB evidence showing forwarding switched while BGP path processing was still completing.

17. Sources

• RFC 4724: Graceful Restart Mechanism for BGP — https://www.rfc-editor.org/rfc/rfc4724.html
• RFC 8538: Notification Message Support for BGP Graceful Restart — https://www.rfc-editor.org/rfc/rfc8538.html
• RFC 9494: Long-Lived Graceful Restart for BGP — https://www.rfc-editor.org/rfc/rfc9494.html
• Cisco IOS XE 17.x: BGP NSF awareness / graceful restart — https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/ip-routing/b-ip-routing/m_irg-nsf-awareness.html