You are here: Home > Participate > Join a Discussion > RIPE Forum > Routing Working Group > [routing-wg] RPKI Route Origin Validation - Africa
RIPE Forum v1.4.1

Routing Working Group

Threaded
Collapse

[routing-wg] RPKI Route Origin Validation - Africa

Ben Maddison

2019-04-09 13:51:49 CET

Hello all.
In November 2018 during the ZAPF (South African Peering Forum) meeting in Cape Town, 3 major African ISP's announced that they would enable RPKI-based ROV (Route Origin Validation), including dropping Invalid routes as part of efforts to improve Internet routing security, on the 1st April, 2019.
On the 1st of April, Workonline Communications (AS37271) enabled ROV and began dropping Invalid routes. This applies to all eBGP sessions, both IPv4 and IPv6.
On the 5th of April, SEACOM (AS37100) enabled ROV and began dropping Invalid routes. This applies to eBGP sessions with public peers, private peers and transit providers, both for IPv4 and IPv6. eBGP sessions toward downstream customers will follow in 3 months time.
Implementation at the third ISP has yet to be completed. We are sure they will communicate with the community when they are ready to do so.
Please note that for the legal reasons previously discussed in various fora, neither Workonline nor SEACOM are utilising the ARIN TAL. As a result, any routes covered only by a ROA issued under the ARIN TAL will fall back to a status of Not Found. Unfortunately, this means that ARIN members will not see any improved routing security for their prefixes on our networks until this is resolved.
We will each re-evaluate this decision if and when ARIN's policy changes. We are hopeful that this will happen sooner rather than later.
If you interconnect with either of us and believe that you are experiencing any routing issues potentially related to this new policy, please feel free to reach out to either:
    - noc@workonline.africa
    - peering _at_ seacom _dot_ mu
Workonline Communications and SEACOM hope that this move encourages the rest of the ISP community around the world to ramp up their deployment of RPKI ROV and begin dropping Invalid routes. We appreciate the work that AT&T and others have carried out in the same vein.
In the mean time, we are happy to answer any questions you may have about our deployments.
Thanks,
Mark Tinka (SEACOM) & Ben Maddison (Workonline Communications).

Job Snijders

2019-05-15 10:28:58 CET

Dear Ben and Mark,

You are now almost 5 weeks into the deployment - can you share any
insights on issues (or lack of issues) you've faced in the last 5 weeks?

Did you have to create exceptions for "important destinations covered by
misconfigured ROAs"? Are you aware of incidents that were prevented
because of the actions you took? Any feedback from customers or
partners?

Kind regards,

Job

On Tue, Apr 09, 2019 at 11:51:49AM +0000, Ben Maddison via routing-wg wrote:
> Hello all.
> In November 2018 during the ZAPF (South African Peering Forum) meeting in Cape Town, 3 major African ISP's announced that they would enable RPKI-based ROV (Route Origin Validation), including dropping Invalid routes as part of efforts to improve Internet routing security, on the 1st April, 2019.
> On the 1st of April, Workonline Communications (AS37271) enabled ROV and began dropping Invalid routes. This applies to all eBGP sessions, both IPv4 and IPv6.
> On the 5th of April, SEACOM (AS37100) enabled ROV and began dropping Invalid routes. This applies to eBGP sessions with public peers, private peers and transit providers, both for IPv4 and IPv6. eBGP sessions toward downstream customers will follow in 3 months time.
> Implementation at the third ISP has yet to be completed. We are sure they will communicate with the community when they are ready to do so.
> Please note that for the legal reasons previously discussed in various fora, neither Workonline nor SEACOM are utilising the ARIN TAL. As a result, any routes covered only by a ROA issued under the ARIN TAL will fall back to a status of Not Found. Unfortunately, this means that ARIN members will not see any improved routing security for their prefixes on our networks until this is resolved.
> We will each re-evaluate this decision if and when ARIN's policy changes. We are hopeful that this will happen sooner rather than later.
> If you interconnect with either of us and believe that you are experiencing any routing issues potentially related to this new policy, please feel free to reach out to either:
>     - noc@workonline.africa
>     - peering _at_ seacom _dot_ mu
> Workonline Communications and SEACOM hope that this move encourages the rest of the ISP community around the world to ramp up their deployment of RPKI ROV and begin dropping Invalid routes. We appreciate the work that AT&T and others have carried out in the same vein.
> In the mean time, we are happy to answer any questions you may have about our deployments.
> Thanks,
> Mark Tinka (SEACOM) & Ben Maddison (Workonline Communications).

Ben Maddison

2019-05-21 16:14:17 CET

Hi Job,


On 2019-05-15 10:29:04+02:00 Job Snijders wrote:

Dear Ben and Mark,

You are now almost 5 weeks into the deployment - can you share any
insights on issues (or lack of issues) you've faced in the last 5 weeks?

Did you have to create exceptions for "important destinations covered by
misconfigured ROAs"? Are you aware of incidents that were prevented
because of the actions you took? Any feedback from customers or
partners?


For us it's remained mostly smooth sailing throughout. We're still seeing close to 0 support traffic as a result of the change, and we haven't had to create any local exceptions.

That's not to say that we're not seeing things that need fixing. These are mostly the result of vendor implementation issues. As a recent example, we've seen ios-xe occasionally accepting invalid routes on certain ebgp sessions contrary to configured policy, and despite the device correctly displaying the status==invalid at the cli. This has been an irritation rather than actually causing customer issues, but is another thing that occupies valuable engineering time.

Our observation has mostly been that features which get implemented and sit un-deployed for a considerable time are a recipe for hard-to-resolve trouble operator issues down the line.

The other time-sink that we need to address is our use of the ripe validator. It's been clear that this was going to be something that we would need to leave behind even before we began dropping invalids last month. The code requires an utterly un-reasonable amount of babysitting - we're finding that either the daemon or the OS needs some intervention (mostly restarts) on an almost daily basis. Our way forward almost certainly involves routinator and one other (tbd).

I would suggest that this community gives some serious thought to whether RIR resources should continue to be invested into a code base that appears to have outlived it's operational usefulness?

I'm around the meeting this week if anyone would like to ask questions or swap stories!

Cheers,

Ben



Kind regards,

Job

On Tue, Apr 09, 2019 at 11:51:49AM +0000, Ben Maddison via routing-wg wrote:
> Hello all.
> In November 2018 during the ZAPF (South African Peering Forum) meeting in Cape Town, 3 major African ISP's announced that they would enable RPKI-based ROV (Route Origin Validation), including dropping Invalid routes as part of efforts to improve Internet routing security, on the 1st April, 2019.
> On the 1st of April, Workonline Communications (AS37271) enabled ROV and began dropping Invalid routes. This applies to all eBGP sessions, both IPv4 and IPv6.
> On the 5th of April, SEACOM (AS37100) enabled ROV and began dropping Invalid routes. This applies to eBGP sessions with public peers, private peers and transit providers, both for IPv4 and IPv6. eBGP sessions toward downstream customers will follow in 3 months time.
> Implementation at the third ISP has yet to be completed. We are sure they will communicate with the community when they are ready to do so.
> Please note that for the legal reasons previously discussed in various fora, neither Workonline nor SEACOM are utilising the ARIN TAL. As a result, any routes covered only by a ROA issued under the ARIN TAL will fall back to a status of Not Found. Unfortunately, this means that ARIN members will not see any improved routing security for their prefixes on our networks until this is resolved.
> We will each re-evaluate this decision if and when ARIN's policy changes. We are hopeful that this will happen sooner rather than later.
> If you interconnect with either of us and believe that you are experiencing any routing issues potentially related to this new policy, please feel free to reach out to either:
>     - noc@workonline.africa
>     - peering _at_ seacom _dot_ mu
> Workonline Communications and SEACOM hope that this move encourages the rest of the ISP community around the world to ramp up their deployment of RPKI ROV and begin dropping Invalid routes. We appreciate the work that AT&T and others have carried out in the same vein.
> In the mean time, we are happy to answer any questions you may have about our deployments.
> Thanks,
> Mark Tinka (SEACOM) & Ben Maddison (Workonline Communications).

Mark Tinka

2019-05-21 16:58:06 CET

Thanks, Job.
So we hit the below issue in the first week:
https://lists.uknof.org.uk/cgi-bin/mailman/private/uknof/2019-April/006470.html
Also in the first week, 2 customers reported an issue accessing some IP
addresses in Europe and the U.S. These turned out to be broken ROA's
which took a while to fix. Finding the right person in the remote
networks (one being a cloud provider) was a bit of a mission. But
eventually, it got fixed. Recently, AMS-IX sent out 2 notifications for
Invalid ROA's of some of our customers. It was also an incorrect ROA
setup, which was fixed in 2 days, each. We are now rolling out ROV to
our downstream BGP customers. I don't expect any issues as: - Most of
our customers will have a NotFound state. - We will manually check each
customer's setup before we turn on ROV for them. We should be done
within 3 months or so. Mark.
> Dear Ben and Mark, > > You are now almost 5 weeks into the deployment
- can you share any > insights on issues (or lack of issues) you've
faced in the last 5 weeks? > > Did you have to create exceptions for
"important destinations covered by > misconfigured ROAs"? Are you aware
of incidents that were prevented > because of the actions you took? Any
feedback from customers or > partners? > > Kind regards, > > Job > On
Tue, Apr 09, 2019 at 11:51:49AM +0000, Ben Maddison via routing-wg
wrote: >/Hello all. />/In November 2018 during the ZAPF (South African
Peering Forum) meeting in Cape Town, 3 major African ISP's announced
that they would enable RPKI-based ROV (Route Origin Validation),
including dropping Invalid routes as part of efforts to improve Internet
routing security, on the 1st April, 2019. />/On the 1st of April,
Workonline Communications (AS37271) enabled ROV and began dropping
Invalid routes. This applies to all eBGP sessions, both IPv4 and IPv6.
/>/On the 5th of April, SEACOM (AS37100) enabled ROV and began dropping
Invalid routes. This applies to eBGP sessions with public peers, private
peers and transit providers, both for IPv4 and IPv6. eBGP sessions
toward downstream customers will follow in 3 months time.
/>/Implementation at the third ISP has yet to be completed. We are sure
they will communicate with the community when they are ready to do so.
/>/Please note that for the legal reasons previously discussed in
various fora, neither Workonline nor SEACOM are utilising the ARIN TAL.
As a result, any routes covered only by a ROA issued under the ARIN TAL
will fall back to a status of Not Found. Unfortunately, this means that
ARIN members will not see any improved routing security for their
prefixes on our networks until this is resolved. />/We will each
re-evaluate this decision if and when ARIN's policy changes. We are
hopeful that this will happen sooner rather than later. />/If you
interconnect with either of us and believe that you are experiencing any
routing issues potentially related to this new policy, please feel free
to reach out to either: />/- noc at workonline.africa
 />/- peering at
seacom.mu 
/>/Workonline Communications and SEACOM hope that this move encourages
the rest of the ISP community around the world to ramp up their
deployment of RPKI ROV and begin dropping Invalid routes. We appreciate
the work that AT&T and others have carried out in the same vein. />/In
the mean time, we are happy to answer any questions you may have about
our deployments. />/Thanks, />/Mark Tinka (SEACOM) & Ben Maddison
(Workonline Communications)./