Introduction Link: #introduction
History and Background Link: #history
Recommendations Link: #recommendations
References and Further Reading Link: #references
This document discusses Route-flap Damping and recommends acceptable practices for ISPs who are considering deploying Route-flap Damping.
1.0 Introduction Link: #intro
1.1 Background Link: #1_1
1.2 Coordination of Flap Damping Parameters Link: #1_2
2.0 Current Status of Route-flap Damping Link: #2
2.1 Impeded Convergence Link: #2_1
2.2 Updates transiting the network Link: #2_2
3.0 Solutions Link: #sol
4.0 Recommendation Link: #recommendation
5.0 Conclusion Link: #conclusion
6.0 Acknowledgements Link: #acknowledgements
7.0 References Link: #aw
A small number of prefixes on the Internet continue to flap rapidly and cause a disproportionate number of updates to BGP and load on BGP speaking routers. This document uses experimental data gathered from an operational environment to suggest changes to the RFD parameters to suppress the prefixes that flap the most, while minimising the suppression of other prefixes.
This document suggests parameters which would make RFD usable and is based around the work of Cristel Pelsser, Olaf Maennel, Pradosh Mohapatra, Randy Bush, and Keyur Patel presented at PAM2011[3 Link: #ref3 ].
History andIn the early 1990s the accelerating growth in the number of prefixes being announced to the Internet (often due to inadequate prefix aggregation), prefix-aggregation), the denser meshing through multiple inter-provider paths, and increased instabilities started to cause significant impact on the performance and efficiency of some the Internet backbone routers. Every time a routing prefix altered state became unreachable because of a single line-flap, the withdrawal was advertised to the whole BGP-Speaking Zone (BSZ) core Internet and handled by every single router that carried the full Internet routing table.
The load this processing placed on the control planes of routers caused further instability as the routers were not able to process other BGP updates or they dropped traffic transiting the device. This could produce cyclic crashing behaviour.
To overcome this situation RFD was developed in 1993 and has since been integrated into most router BGP software implementations. RFD is described in detail in RFC 2439[1] Link: #ref1 . 2439. RFD is now used in many service provider networks in the Internet.
When RFD was first implemented in commercial routers, vendor implementations had different default values and different characteristics. As this inconsistency would result in different rates of flap damping, and therefore therefor introduce inconsistent path selection and thus behavior that was very hard to diagnose, the operator ISP community introduced a consistent set of recommendations for flap damping parameters, so that ISPs deploying RFD would treat flapping prefixes in the same way.
This call for consistency resulted in the RIPE Routing Working Group producing first ripe-178, then ripe-210, and finally the ripe-229 documents [2a Link: #ref2a ]. [2], following consensus of the Routing Working Group. The parameters documented in ripe-229 were considered, at time of publication in 2001, the best current practice.
In 2006, this was reviewed again and resulted in ripe-378 [2 Link: #ref2 ] which recommended to disable RFD because it created more harm than good.
Research in the years following the introduction of RFD into BGP implementations, and the publication of the RIPE Routing Working Group recommendations, has demonstrated that there are real and signficant problems with RFD as deployed on the Internet today.
Perhaps the best known work highlighting major problems with RFD is that by Zhuoquing Mao and colleagues, presented at Sigcomm in 2002. Following presentations by Randy Bush and colleagues explain the research work more accessibly.
The major issue is that if one path is withdrawn, all BGP speakers will use best path selection to pick the next best path, and advertise this best path to all their neighbours. These neighbours will see a change in path; a change in path is a change in attribute, so the prefix as seen on a neighbouring router will attract a flap penalty - even though that path is perfectly valid and there has been no disappearance of the prefix from the routing table [5].
And this path "hunting" goes on throughout the Internet - a simple prefix withdrawal can result in the appearance of a major flap event a few AS hops away in the Internet, with the result that vendor default and even the RIPE-229 recommended flap damping parameters will mark the prefix to be suppressed. While the operator can see this is an error, the routers are simply reacting to the circumstances presented to them.
Problems are not just caused by path "hunting". Each implementation of BGP either has differing values of the Minimum Route Advertisement Interval (MRAI) Timer (the amount of time a router waits before passing on a route update) or does not implement MRAI at all in favour of the vendor's own throttling algorithm.
Some implementations pass on the update without waiting at all, others wait for 30 seconds, etc. These differences mean that update messages transiting different ASNs using different vendor equipment will arrive at the target router at different times. This router will see these different messages, and will consider each one for best path options. This will more than likely result in a different best path offered to its neighbours for each message update arriving.
The result of this is that a simple update message from one ASN would be seen as a multiple route flap event a few ASN hops away - when in fact there was no instability whatsoever. There have been actual measurements where this resulted in a single prefix withdrawal producing 41 BGP events a few hops away!
Not only is the MRAI timer a potential source of problems, but also differences in CPU loadings and CPU speed will result in different update times for prefixes announcements passing from router to router. These differences will also contribute to the effects described above.
The default values used in current implementations of RFD apply a penalty of 1000 each time a route flaps, and suppresses the prefix when the penalty exceeds a figure in the region of 2000 (Cisco IOS) or 3000 (Juniper JunOS).
The table shows the percentage of prefixes above the suppress threshold and the percentage reduction in BGP churn for various values of suppress threshold. The current default suppress value of 2000 reduces BGP churn by 47%, but it suppressed 14% of the prefixes at some point over the lifetime of the experiment. Significantly larger values of suppress threshold such as 12000, 15000 or 18000 still reduced BGP churn, but suppressed far fewer prefixes which it is believed reduces the risk of penalising otherwise well-behaved prefixes.
Suppress Threshold | % prefixes suppressed | % reduction in BGP churn compared with no damping |
2000 4000 6000 12000 15000 18000 | 14 4.2 2.1 0.63 0.44 0.32 | 47 26 19 11.26 9.51 8.12 |
In order to punish the biggest offenders - those prefixes that flap the most – yet without punishing others, the RIPE Routing-WG recommends vendors raise the maximum suppress threshold in router implementations to 50,000 and operators configure a suppress threshold value of at least 6,000. The vendors might also change the default suppress threshold to 6,000. But this might surprise operators who use the default.
This has a number of advantages:
Changing the default suppress threshold could result in an increase in forwarding table size or announcement rate for operators who use RFD with the default settings. This warrants further discussion.
However, despite publication in 2002, there has since then been no desire expressed from the ISP industry for these modifications to be made to the BGP implementations. Nor has there been any activity by the BGP implementors to enhance their flap damping implementations to follow those recommendations.
As the power of routers has increased, the original needs for BGP Flap Damping is no longer a major concern for operators or router equipment vendors as it was in the mid-1990s when route flapping consumed a signficant percentage of the CPU of early routers. In fact, the negative effects of RFD, as described above, have become the major concern, the cure has become worse than the disease!
This Routing Working Group document proposes that with the current implementations of BGP flap damping, the application of flap damping in ISP networks is NOT recommended. The recommendations given in ripe-229 and previous documents [2] are considered obsolete henceforth.
If flap damping is implemented, the ISP operating that network will cause side-effects to their customers and the Internet users of their customers' content and services as described in the previous sections. These side-effects would quite likely be worse than the impact caused by simply not running flap damping at all.
With current vendor implementations, BGP flap damping is harmful to the reachability of prefixes across the Internet. We would like to encourage more work to correct some of the issues highlighted by the work of Mao et al [3], to allow the viewing of prefix flap statistics without applying flap damping, and permit more flexible per eBGP neighbour damping configuration features for network operators.
We would like to acknowledge valuable contributions and feedback from Randy Bush.
[1] Curtis Villamizar, Ravi Chandra, Ramesh Govindan
RFC2439: BGP Route-flap Damping (Proposed Standard)
ftp://ftp.ietf.org/rfc/rfc2439.txt
Most recent RIPE Document
ftp://ftp.ripe.net/ripe/docs/ripe-378.txt Link: ftp://ftp.ripe.net/ripe/docs/ripe-378.txt
[2a] Older RIPE Documentsftp://ftp.ripe.net/ripe/docs/ripe-178.txt
ftp://ftp.ripe.net/ripe/docs/ripe-210.txt Link: ftp://ftp.ripe.net/ripe/docs/ripe-210.txt
ftp://ftp.ripe.net/ripe/docs/ripe-229.txt
Cristel Pelsser, Olaf Maennel, Pradosh Mohapatra, Randy Bush and Keyur Patel. "Route Flap Damping Made Usable". PAM 2011, March 2011.
[4] Zhouqing Mao, Ramesh Govindan, George Varghese, Randy KatzRoute-flap Damping Exacerbates Internet Routing Congerence SIGCOMM Sigcomm 2002
http://www.eecs.umich.edu/~zmao/Papers/sig02.pdf
[5]
NANOG 26
http://www.nanog.org/mtg-0210/ppt/flap.pdf
[6]
Delayed Internet Routing Convergence SIGCOMM 2000