[mat-wg] Probe and measurement archives vs. the APIs

Dear All,

We've observed recently that some users are using scripts to "crawl" through
all the metadata provided by the probe and measurement APIs. This means that
some people out there download basically the entire probe and/or measurement
meta data set the system provides. As you can imagine, this is highly
inefficient and keeps our servers busy. We believe that there could be
better solutions.

We have been providing the downloadable archive for probe metadata for some
time now, and recently we've added the measurement metadata too. You can now
reach these at http://ftp.ripe.net/ripe/atlas/

This way one can even get complete sets of metadata as far back as January
2015 for measurement specifications and March 2014 for probe data.

Given the existence of these archives, we will apply a sane limit on how
much metadata can be reached via the APIs. In practice this means that the
API will not allow fetching results beyond, say, 10.000 (i.e. 500 pages of
20) entries. This is after the application of filters, so for example with
no filters at all you could still access metadata of the 10k newest
measurements, while if you filter for DNS measurements only, you could get
the 10k newest of those, and so on.

We believe this keeps the metadata API useful for the majority of the users,
while the existence of the archive serves the remaining cases.

Our plan is to make this change around the end of July.

Regards,
Robert Kisteleki