[mat-wg] RIPE NCC measurement data retention

I'd like to go out on the record and say that RIPE RIS is invaluable
to the internet research and routing analytics world, and a large part
of the invaluableness comes from it's longevity in its data archives.

That being said, it's understandable that the storage costs to store
bgp ribs and update messages would soon become untenable. However I
would like to point out (Obviously excluding redundancy) that the
currently quoted "raw" data volume for RIS is not that much.

(From the blog post linked in the OP)

> This dataset currently weighs in at roughly 50 TB of compressed dump and update files, with 80% accounting for the data collected in the last five years

50TB is 12x 4TB drives, 12 3.5" drive slots is a standard size for a 2
Rack Unit sized server.

It sounds like the larger concern (at least for the next 5 years) is
that the RIPEStat use case is so large (800 TB according to the post).

Could you provide more information on what is potentially causing such
a large amplified storage usage?

Is there a better argument to potentially deprecate some ripe stat
tools if they are costing a huge amount of backend storage to provide?
Rather than risk degrading RIS's historical archive storage.

As a side point I would propose to remove the existence of private
right atlas measurements, As almost all things on the internet ( and
thus being measured by RIPE Atlas ) are public, the measurements
towards such inherently public things probably shouldn't need to be
private.


On Mon, Dec 11, 2023 at 4:45 AM Stephen Strowes via mat-wg
<mat-wg at ripe.net> wrote:
>
> Hi folks,
>
> Bumping this thread in case you missed it:
>
> The NCC is keen to hear the community's thoughts on the principles they should follow for measurement data retention. We had a good discussion on this topic at the WG session during RIPE87, and it'd be good to see it continue here, or on the forum, or privately back to the NCC if you prefer.
>
> The relevant materials are here:
>
> - Recording: https://ripe87.ripe.net/archives/video/1176/
> - Slides: https://ripe87.ripe.net/presentations/13-kisteleki-MATWG-RIPE87.pdf, slides 16 & 17
> - Labs post: https://labs.ripe.net/author/kistel/ripe-ncc-measurement-data-retention-principles/
>
> Specifically, the principles in question are the bullets on slide 17, or the bullets at the end of the Labs article.
>
>
> Cheers,
>
> S.
>
>
>
> On Wed, Nov 22, 2023 at 11:43 AM Robert Kisteleki <robert at ripe.net> wrote:
>>
>> Dear all,
>>
>> We've just published a proposal about establishing principles around how
>> the RIPE NCC retains and publishes Internet measurement data,
>> specifically in RIS and RIPE Atlas:
>> https://labs.ripe.net/author/kistel/ripe-ncc-measurement-data-retention-principles/
>>
>> We would be very happy to see discussions about this here on the mailing
>> list, on the RIPE NCC Forum, or live at RIPE87.
>>
>> Regards,
>> Robert Kisteleki
>> RIPE NCC
>>
>> --
>>
>> To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/mat-wg
>
> --
>
> To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/mat-wg