[db-wg] NRTM replication inefficiencies
- Previous message (by thread): [db-wg] NRTM replication inefficiencies
- Next message (by thread): [db-wg] NRTM replication inefficiencies
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Job Snijders
job at instituut.net
Thu Nov 9 17:57:21 CET 2017
On Thu, Nov 09, 2017 at 04:28:03PM +0100, Tim Bruijnzeels wrote: > > On 7 Nov 2017, at 23:11, Job Snijders via db-wg <db-wg at ripe.net> wrote: > > > > I would also welcome an investigation into alternative approaches, (some > > not-via-WHOIS replication mechanisms), perhaps something over HTTPS can > > be done? Either way, something more robust would be useful. > > We recently developed and implement a standard for something similar for RPKI: > https://datatracker.ietf.org/doc/rfc8182/ > > I believe this approach can be useful here as well. Without going into all the RPKI specifics, it works a little something like this: > > Starting points: > = The state of the rpki repository (or whois) at a given point in time can represented by a ‘snapshot’ > - This snapshot is “immutable” - therefore they may be cached indefinitely and we can give it a unique URL and deliver it through a distributed CDN > = The delta between two consecutive snapshots is also “immutable” data - so again we can cache it and give it a unique URL and distribute > = We can publish a notification file (which should NOT be cached) that points to: > - the CURRENT snapshot > - a list of deltas (each for 1 increment) - total size of deltas MUST not exceed size of snapshot > > Clients can then just poll the notification file and work out for > themselves whether a list of deltas is available to them, or that they > need to get the latest snapshot instead. > > Yes, we use a session_id and hashes of referenced files for additional > checks (details in the RFC). > > The idea behind this design was that we wanted to minimise the impact > on the server. In a chatty protocol (like rsync which is still used in > RPKI) the server and client need to work out their differences > together to determine what needs to be transferred. This is fine in > one on one relations, but when a server needs to serve a multitude of > clients this doesn’t scale. We want to be able parallelise as much as > we can (Amdahl’s law), so we push the computational burden to the > clients. The server just needs a one-off investment to create the > snapshot and delta and latest notification which it can then offload. > Using HTTPS allows us to leverage one of the many, many CDNs out > there. This problem has been solved in the industry. So we do not need > to invent our own infrastructure for this. > > Note that in the case of RPKI the protocol is XML based. This made > sense because it leveraged existing definitions in the RPKI space that > were also XML based. For whois it may make more sense to look at JSON > and/or RDAP. > > Please let me know if you see merit in this kind of ‘delta’ protocol > in the whois space. yes, I think there may be merit to replacing NRTM, and DELTA would certainly be a good source of inspiration. Would it be fair to ask for a two-pronged approach? DELTA-WHOIS + WHOIS-END-OF-BLURP markings? How much work (or complexity?) is involved for RIPE NCC to develop a marking that is send to the client at the end of a '-g' query that also had '-k' enabled? Kind regards, Job
- Previous message (by thread): [db-wg] NRTM replication inefficiencies
- Next message (by thread): [db-wg] NRTM replication inefficiencies
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
[ db-wg Archives ]