[atlas] Friday's events on RIPE Atlas
- Previous message (by thread): [atlas] Friday's events on RIPE Atlas
- Next message (by thread): [atlas] atlas probe usb power usage question
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Robert Kisteleki
robert at ripe.net
Wed Sep 23 12:37:56 CEST 2015
On 2015-09-23 12:29, Colin Johnston wrote: > maybe compress the data info on the probes before it is sent and save to backend storage with the data already compress’d ? > > just a thought > > Colin It may be interesting to know, although not at all surprising: compressing individual results achieves almost nothing (and has the drawback of having to unpack/repack while the result moves in the pipeline). Compressing small chunks, say dozens of related results (e.g. from the same measurement) achieves 40-50% saving (a lot depends on what kind of result it is), while compressing large blocks (multi-megabyte chunks) can achieve 90-95% in most cases. It makes sense to compress data once it's at rest, especially if it's not often accessed. Therefore our processing pipeline is constructed such that it applies compression once a large enough batch has been consumed. As Romeo explained, this part was stretched a lot on Friday. Cheers, Robert
- Previous message (by thread): [atlas] Friday's events on RIPE Atlas
- Next message (by thread): [atlas] atlas probe usb power usage question
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]