You are here: Home > Publications > RIPE Document Store > RIPE Routing Working Group Recommendations on Route Flap Damping

Changes to RIPE Routing Working Group Recommendations on Route Flap Damping

Legend (+) Added (-) Deleted
Changed Tag Added Tag Deleted
ripe-580: RIPE Routing Working Group Recommendations on Route Flap ripe-378: This document discusses Route-flap Damping and recommends acceptable practices for ISPs who are considering deploying Route-flap Damping.
delete: <p> delete: <a class="anchor-link" href="#introduction"> insert: <h2>

Abstract insert: </h2>

insert: <p>

This document discusses Route-flap Damping and recommends acceptable practices for ISPs who are considering deploying Route-flap Damping. insert: </p>

insert: <hr />
insert: <p>

insert: </p>

insert: <p>

insert: <a href="#intro"> 1.0 Introduction delete: </p> delete: <p> delete: <a class="anchor-link" href="#history"> History and insert: <br />
insert: <a href="#1_1"> 1.1 Background delete: </p> delete: <p> delete: <a class="anchor-link" href="#analysis"> Analysis delete: </a> delete: </p> delete: <p> delete: <a class="anchor-link" href="#recommendations"> Recommendations delete: </a> delete: </p> delete: <p> delete: <a class="anchor-link" href="#references"> References and Further Reading delete: </a> delete: </p> delete: <h2> delete: </h2> delete: <h3> delete: <a name="introduction"> delete: </a> Introduction delete: </h3> delete: <p> Route insert: <br />
insert: <a href="#1_2"> 1.2 Coordination of Flap Damping Parameters insert: </a> insert: <br />
insert: <a href="#2"> 2.0 Current Status of Route-flap Damping insert: </a> insert: <br />
insert: <a href="#2_1"> 2.1 Impeded Convergence insert: </a> insert: <br />
insert: <a href="#2_2"> 2.2 Updates transiting the network insert: </a> insert: <br />
insert: <a href="#sol"> 3.0 Solutions insert: </a> insert: <br />
insert: <a href="#recommendation"> 4.0 Recommendation insert: </a> insert: <br />
insert: <a href="#conclusion"> 5.0 Conclusion insert: </a> insert: <br />
insert: <a href="#acknowledgements"> 6.0 Acknowledgements insert: </a> insert: <br />
insert: <a href="#aw"> 7.0 References insert: </a> insert: </p>

insert: <p>

insert: </p>

insert: <hr />
insert: <h2>

insert: <a name="intro"> insert: </a> 1.0 Introduction insert: </h2>

insert: <p>

Route-flap Damping (RFD) [ delete: <a class="anchor-link" href="#ref1"> 1 delete: </a> ] [1] is a mechanism for BGP speaking routers that penalises prefixes that exhibit a large number of updates (‘flapping'), and suppresses a route when the accumulated penalty exceeds a given threshold. The penalty decays over time until it reaches a lower threshold at which point the route is unsuppressed. RFD is intended to improve the overall stability of the Internet routing table and reduce the load on BGP speaking the CPUs of the core routers. In ripe-378 [ delete: <a class="anchor-link" href="#ref2"> 2 delete: </a> ] it was stated that Unfortunately, due to the dynamics of BGP, especially a phenomenon called ‘path hunting,' the default the protocol, common simple configurations of flap damping can do more harm than good as it may suppress a prefix after it has only flapped a few times. Consequently RFD was deprecated due to the problem of over damping (see [ delete: <a class="anchor-link" href="#ref2"> 2 delete: </a> ] for more details). delete: </p> delete: <p> A small number of prefixes on the Internet continue to flap rapidly and cause a disproportionate number of updates to BGP and load on BGP speaking routers. This document uses experimental data gathered from an operational environment to suggest changes to the RFD parameters to suppress the prefixes that flap the most, while minimising the suppression of other prefixes. delete: </p> delete: <p> This document suggests parameters which would make RFD usable and is based around the work of Cristel Pelsser, Olaf Maennel, Pradosh Mohapatra, Randy Bush, and Keyur Patel presented at PAM2011[ delete: <a class="anchor-link" href="#ref3"> 3 delete: </a> ]. good, see [3,4].

delete: <a name="history"> delete: </a> History and insert: <a name="1_1"> insert: </a> 1.1 Background

In the early 1990s the accelerating growth in the number of prefixes being announced to the Internet (often due to inadequate prefix aggregation), prefix-aggregation), the denser meshing through multiple inter-provider paths, and increased instabilities started to cause significant impact on the performance and efficiency of some the Internet backbone routers. Every time a routing prefix altered state became unreachable because of a single line-flap, the withdrawal was advertised to the whole BGP-Speaking Zone (BSZ) core Internet and handled by every single router that carried the full Internet routing table. delete: </p> delete: <p> The insert: <br />
insert: <br />
It was soon realized that the increasing routing churn created significant processing load this processing placed on the control planes of routers caused further instability as the routers were not able to process other BGP updates or they dropped traffic transiting the device. This could produce cyclic crashing behaviour. delete: </p> delete: <p> routing engines, sometimes sufficiently high load to cause router crashes. insert: <br />
insert: <br />
To overcome this situation RFD was developed in 1993 and has since been integrated into most router BGP software implementations. RFD is described in detail in RFC 2439[ delete: <a class="anchor-link" href="#ref1"> 1] delete: </a> . delete: </p> 2439. RFD is now used in many service provider networks in the Internet. insert: </p>

insert: <h3>

insert: <a id="1_2" name="1_2"> insert: </a> 1.2 Coordination of flap damping parameters insert: </h3>

When RFD was first implemented in commercial routers, vendor implementations had different default values and different characteristics. As this inconsistency would result in different rates of flap damping, and therefore therefor introduce inconsistent path selection and thus behavior that was very hard to diagnose, the operator ISP community introduced a consistent set of recommendations for flap damping parameters, so that ISPs deploying RFD would treat flapping prefixes in the same way.

This call for consistency resulted in the RIPE Routing Working Group producing first ripe-178, then ripe-210, and finally the ripe-229 documents [ delete: <a class="anchor-link" href="#ref2a"> 2a delete: </a> ]. [2], following consensus of the Routing Working Group. The parameters documented in ripe-229 were considered, at time of publication in 2001, the best current practice. In 2006, insert: </p>

insert: <hr />
insert: <h2>

insert: <a name="2"> insert: </a> 2.0 Current Status of Route-flap Damping insert: </h2>

insert: <p>

Research in the years following the introduction of RFD into BGP implementations, and the publication of the RIPE Routing Working Group recommendations, has demonstrated that there are real and signficant problems with RFD as deployed on the Internet today. insert: </p>

insert: <h3>

insert: <a id="2_1" name="2_1"> insert: </a> 2.1 Impeded Convergence insert: </h3>

insert: <p>

Perhaps the best known work highlighting major problems with RFD is that by Zhuoquing Mao and colleagues, presented at Sigcomm in 2002. Following presentations by Randy Bush and colleagues explain the research work more accessibly. insert: </p>

insert: <p>

The major issue is that if one path is withdrawn, all BGP speakers will use best path selection to pick the next best path, and advertise this best path to all their neighbours. These neighbours will see a change in path; a change in path is a change in attribute, so the prefix as seen on a neighbouring router will attract a flap penalty - even though that path is perfectly valid and there has been no disappearance of the prefix from the routing table [5]. insert: </p>

insert: <p>

And this path "hunting" goes on throughout the Internet - a simple prefix withdrawal can result in the appearance of a major flap event a few AS hops away in the Internet, with the result that vendor default and even the RIPE-229 recommended flap damping parameters will mark the prefix to be suppressed. While the operator can see this is an error, the routers are simply reacting to the circumstances presented to them. insert: </p>

insert: <h3>

insert: <a id="2_2" name="2_2"> insert: </a> 2.2 Updates transiting the network insert: </h3>

insert: <p>

Problems are not just caused by path "hunting". Each implementation of BGP either has differing values of the Minimum Route Advertisement Interval (MRAI) Timer (the amount of time a router waits before passing on a route update) or does not implement MRAI at all in favour of the vendor's own throttling algorithm. insert: </p>

insert: <p>

Some implementations pass on the update without waiting at all, others wait for 30 seconds, etc. These differences mean that update messages transiting different ASNs using different vendor equipment will arrive at the target router at different times. This router will see these different messages, and will consider each one for best path options. This will more than likely result in a different best path offered to its neighbours for each message update arriving. insert: </p>

insert: <p>

The result of this is that a simple update message from one ASN would be seen as a multiple route flap event a few ASN hops away - when in fact there was reviewed again and no instability whatsoever. There have been actual measurements where this resulted in ripe-378 [ delete: <a class="anchor-link" href="#ref2"> 2 delete: </a> ] which recommended to disable RFD because it created more harm than good. delete: </p> delete: <h3> delete: <a name="analysis"> delete: </a> Analysis delete: </h3> delete: <p> In a single prefix withdrawal producing 41 BGP events a few hops away! insert: </p>

insert: <p>

Not only is the MRAI timer a potential source of problems, but also differences in CPU loadings and CPU speed will result in different update times for prefixes announcements passing from router to router. These differences will also contribute to the effects described above. insert: </p>

insert: <hr />
insert: <h2>

insert: <a id="sol" name="sol"> insert: </a> 3.0 Solutions insert: </h2>

insert: <p>

Possible solutions to the problems summarised above have been proposed and analysed in the work by Pelsser Zhouqing Mao and colleagues. insert: </p>

insert: <p>

However, despite publication in 2002, there has since then been no desire expressed from the ISP industry for these modifications to be made to the BGP implementations. Nor has there been any activity by the BGP implementors to enhance their flap damping implementations to follow those recommendations. insert: </p>

insert: <p>

As the power of routers has increased, the original needs for BGP Flap Damping is no longer a major concern for operators or router equipment vendors as it was in the mid-1990s when route flapping consumed a signficant percentage of the CPU of early routers. In fact, the negative effects of RFD, as described above, have become the major concern, the cure has become worse than the disease! insert: </p>

insert: <hr />
insert: <h2>

insert: <a name="recommendation"> insert: </a> 4.0 Recommendation insert: </h2>

insert: <p>

This Routing Working Group document proposes that with the current implementations of BGP flap damping, the application of flap damping in ISP networks is NOT recommended. The recommendations given in ripe-229 and previous documents [2] are considered obsolete henceforth. insert: <br />
insert: <br />
If flap damping is implemented, the ISP operating that network will cause side-effects to their customers and the Internet users of their customers' content and services as described in the previous sections. These side-effects would quite likely be worse than the impact caused by simply not running flap damping at all. insert: </p>

insert: <hr />
insert: <h2>

insert: <a id="conclusion" name="conclusion"> insert: </a> 5.0 Conclusion insert: </h2>

insert: <p>

With current vendor implementations, BGP flap damping is harmful to the reachability of prefixes across the Internet. We would like to encourage more work to correct some of the issues highlighted by the work of Mao et al [ delete: <a class="anchor-link" href="#ref3"> 3 delete: </a> ], it is shown that 3% of all prefixes cause 36% of BGP updates, and just 0.01% of the prefixes cause 10% of the BGP updates. The aim is to only penalise those prefixes with excessive numbers of updates. delete: </p> delete: <p> The default values used in current implementations of RFD apply a penalty of 1000 each time a route flaps, and suppresses the [3], to allow the viewing of prefix when the penalty exceeds a figure in the region of 2000 (Cisco IOS) or 3000 (Juniper JunOS). delete: </p> delete: <p> The table shows the percentage of prefixes above the suppress threshold and the percentage reduction in BGP churn flap statistics without applying flap damping, and permit more flexible per eBGP neighbour damping configuration features for various values of suppress threshold. The current default suppress value of 2000 reduces BGP churn by 47%, but it suppressed 14% of the prefixes at some point over the lifetime of the experiment. Significantly larger values of suppress threshold such as 12000, 15000 or 18000 still reduced BGP churn, but suppressed far fewer prefixes which it is believed reduces the risk of penalising otherwise well-behaved prefixes. delete: </p> delete: <p> delete: </p> delete: <table class="listing"> delete: <tbody> delete: <tr> delete: <td> delete: <p> delete: <b> Suppress delete: </b> delete: </p> delete: <p> delete: <b> Threshold delete: </b> delete: </p> delete: </td> delete: <td> delete: <p> delete: <b> % prefixes delete: </b> delete: </p> delete: <p> delete: <b> suppressed delete: </b> delete: </p> delete: </td> delete: <td> delete: <p> delete: <b> % reduction in BGP churn delete: </b> delete: </p> delete: <p> delete: <b> compared with no damping delete: </b> delete: </p> delete: </td> delete: </tr> delete: <tr> delete: <td> delete: <p> 2000 delete: </p> delete: <p> 4000 delete: </p> delete: <p> 6000 delete: </p> delete: <p> 12000 delete: </p> delete: <p> 15000 delete: </p> delete: <p> 18000 delete: </p> delete: </td> delete: <td> delete: <p> 14 delete: </p> delete: <p> 4.2 delete: </p> delete: <p> 2.1 delete: </p> delete: <p> 0.63 delete: </p> delete: <p> 0.44 delete: </p> delete: <p> 0.32 delete: </p> delete: </td> delete: <td> delete: <p> 47 delete: </p> delete: <p> 26 delete: </p> delete: <p> 19 delete: </p> delete: <p> 11.26 delete: </p> delete: <p> 9.51 delete: </p> delete: <p> 8.12 delete: </p> delete: </td> delete: </tr> delete: </tbody> delete: </table> delete: <h3> delete: <a name="recommendations"> delete: </a> Recommendations delete: </h3> delete: <p> In order to punish the biggest offenders - those prefixes that flap the most – yet without punishing others, the RIPE Routing-WG recommends vendors raise the maximum suppress threshold in router implementations to 50,000 and operators configure a suppress threshold value of at least 6,000. The vendors might also change the default suppress threshold to 6,000. But this might surprise operators who use the default. delete: </p> delete: <p> This has a number of advantages: delete: </p> delete: <ul> delete: <li> it is easy to implement delete: </li> delete: <li> it will reduce the churn compared to the situation we havenow where no RFD is applied delete: </li> delete: <li> it spares the smaller offenders. delete: </li> delete: </ul> delete: <p> Changing the default suppress threshold could result in an increase in forwarding table size or announcement rate for operators who use RFD with the default settings. This warrants further discussion. delete: </p> delete: <h3> delete: <a name="references"> delete: </a> network operators. insert: </p>

insert: <hr />
insert: <h2>

insert: <a id="acknowledgements" name="acknowledgements"> insert: </a> 6.0 Acknowledgements insert: </h2>

insert: <p>

We would like to acknowledge valuable contributions and feedback from Randy Bush. insert: </p>

insert: <hr />
insert: <h2>

insert: <a name="aw"> insert: </a> 7.0 References and Further Reading delete: </h3> delete: <p> delete: <a name="ref1"> delete: </a> insert: </h2>

insert: <p>

[1] Curtis Villamizar, Ravi Chandra, Ramesh Govindan delete: </p> delete: <p> insert: <br />
RFC2439: BGP Route-flap Damping (Proposed Standard) delete: </p> delete: <p> delete: <a href="ftp://ftp.ietf.org/rfc/rfc2439.txt"> insert: <br />
insert: <a href="ftp://ftp.ietf.org/rfc/rfc2439.txt" target="_blank"> ftp://ftp.ietf.org/rfc/rfc2439.txt

delete: </p> delete: <p> delete: <a name="ref2"> delete: </a> [2] Most recent RIPE Document delete: </p> delete: <p> delete: <a href="ftp://ftp.ripe.net/ripe/docs/ripe-378.txt"> ftp://ftp.ripe.net/ripe/docs/ripe-378.txt delete: </a> delete: </p> delete: <p> delete: </p> delete: <p> delete: <a name="ref2a"> delete: </a> [2a] Older RIPE Documents

delete: <a href="ftp://ftp.ripe.net/ripe/docs/ripe-178.txt"> insert: <a href="ftp://ftp.ripe.net/ripe/docs/ripe-178.txt" target="_blank"> ftp://ftp.ripe.net/ripe/docs/ripe-178.txt delete: </p> delete: <p> delete: <a href="ftp://ftp.ripe.net/ripe/docs/ripe-210.txt"> ftp://ftp.ripe.net/ripe/docs/ripe-210.txt delete: </a> delete: </p> delete: <p> delete: <a href="ftp://ftp.ripe.net/ripe/docs/ripe-229.txt"> insert: <br />
insert: <a href="ftp://ftp.ripe.net/ripe/docs/ripe-210.txt" target="_blank"> ftp://ftp.ripe.net/ripe/docs/ripe-210.tx insert: </a> t insert: <br />
insert: <a href="ftp://ftp.ripe.net/ripe/docs/ripe-229.txt" target="_blank"> ftp://ftp.ripe.net/ripe/docs/ripe-229.txt

delete: </p> delete: <p> delete: <a name="ref3"> delete: </a> [3] Cristel Pelsser, Olaf Maennel, Pradosh Mohapatra, Randy Bush and Keyur Patel. "Route Flap Damping Made Usable". PAM 2011, March 2011. delete: </p> delete: <p> delete: <a href="http://www.iij-ii.co.jp/en/lab/researchers/cristel/publications/Pelsser-RFD-PAM2011.pdf"> http://www.iij-ii.co.jp/en/lab/researchers/cristel/publications/Pelsser-RFD-PAM2011.pdf delete: </a> delete: </p> delete: <p> delete: </p> delete: <p> [4] Zhouqing Mao, Ramesh Govindan, George Varghese, Randy Katz delete: </p> delete: <p> insert: <br />
Route-flap Damping Exacerbates Internet Routing Congerence SIGCOMM Sigcomm 2002 delete: </p> delete: <p> delete: <a href="http://www.eecs.umich.edu/~zmao/Papers/sig02.pdf"> insert: <br />
insert: <a href="http://www.eecs.umich.edu/~zmao/Papers/sig02.pdf" target="_blank"> http://www.eecs.umich.edu/~zmao/Papers/sig02.pdf

delete: </p> delete: <p> [5] [4] Randy Bush, Tim Griffin, Zhouqing Mao insert: <br />
Route-flap Damping: Harmful? delete: </p> delete: <p> insert: <br />
NANOG 26 delete: </p> delete: <p> delete: <a href="http://www.nanog.org/mtg-0210/ppt/flap.pdf"> insert: <br />
insert: <a href="http://www.nanog.org/mtg-0210/ppt/flap.pdf" target="_blank"> http://www.nanog.org/mtg-0210/ppt/flap.pdf

delete: </p> delete: <p> [6] [5] Craig Labovitz, Abha Ahuja, Abhijit Bose, Farnam Jihanian delete: </p> delete: <p> insert: <br />
Delayed Internet Routing Convergence SIGCOMM insert: <br />
Sigcomm 2000 delete: </p> delete: <p> delete: <a href="http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigcomm2000-5-2.pdf"> http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigcomm2000-5-2.pdf delete: </a> delete: </p> insert: <br />
insert: <a href="http://www.acm.org/sigs/sigcomm/sigcomm2000/conf/paper/sigcomm2000-5-2.pdf"> http://www.acm.org/sigs/sigcomm/sigcomm2000/conf/paper/sigcomm2000-5-2.pdf insert: </a> insert: </p>

insert: <hr />
RIPE Routing Working Group Recommendations on Route Flap On Route-flap Damping
RIPE Documents Search