[mat-wg] Changes to the rpki archives on ftp.ripe.net

Dear colleagues,

A while ago we announced changes to the RPKI archives on our FTP server [0].

Based on your feedback, we have compressed all historic roa.csv files (to
roa.csv.xz) and repo.tar.gz files (to repo.tar.xz). This results in a uniform
set of files that is easier to process than a mix of compressed and uncompressed
files.

As announced earlier, we will stop creating the roa.csv and repo.tar.gz files.
We will do so after the 12th of February. Afterwards, we will clean up the old,
now recompressed files (roas.csv, repo.tar.gz).

Kind regards,

Ties de Kock
RIPE NCC

[0]: https://ftp.ripe.net/rpki/


> On 11 Oct 2023, at 08:59, Ties de Kock <tdekock at ripe.net> wrote:
> 
> Dear colleagues,
> 
> As you may know, the RIPE NCC publishes daily RPKI dumps for all trust anchors
> on ftp.ripe.net/rpki. Today, we have made some changes to this longitudinal
> dataset.
> 
> We have added:
>  * The compressed JSON output of a validator (output.json.xz). We added this
>    because it contains validated ASPA and BGPsec payloads.
>  * A compressed version of the CSV file of ROAs (roas.csv.xz)
>  * A xz compressed archive of the validated and unvalidated objects (repo.tar.xz)
> 
> We made this change because it significantly reduces the size of the files. At
> this point, we do not plan to re-compress historic files. However, to realise
> the savings, we must stop producing the larger version at some point and clean
> up the "duplicate" files from the transition period.
> 
> For now, we will create the old (roas.csv, repo.tar.gz) and new (roas.csv.xz,
> repo.tar.xz) side by side during a transition period. We plan to stop creating
> the old format after a transition period of at least a month, taking into
> account the feedback we receive. We will inform you when this happens.
> 
> Ties de Kock
> Specialist Software Engineer
> RIPE NCC
>