ZSTD Compression

[VincentJ]

Is there any way we could drop in zstd compression into fog and see if we get faster imaging/better compression ratios?

http://facebook.github.io/zstd/

[Junkhacker]

from what i’m reading looks like this is optimized for small files, fog images are huge files…
have you tried decompressing a fog image and compressing it with this to see how well it compresses/decompresses with this and how quickly?

[VincentJ]

Disk image of a windows 10 system with some applications - 7,512,046KB (PIGZ 9)
Unpacked - 16,390,624KB (I can’t unpack further with the tools on the system)
zstd lvl5 - 7,286,951KB
zstd lvl11 - 6,967,155KB
zstd lvl17 - 6,781,375KB
zstd lvl22 - 6,214,702KB

I tested in a VM so compression/decompression times/speeds are not useful measurements in my case.

[george1421]

@VincentJ We have to remember here that the target client does all of the heavy lifting during image capture/deploy. (I’m only speaking in general terms here) You can have the greatest compress / decompression ratio, but if the penalty is time then what you are saving in over the wire time is lost because the decompression at the target is too slow because of the target hardware.

If the OP is interested in testing this its possible to unpack the inits and swap out gzip for zstd to see if imaging rates are better or worse.

This post should help you narrow in on the spot where fog uses gzip in the imaging process: https://forums.fogproject.org/topic/9525/dedupe-storage-how-to-best-bypass-pigz-packaging/4

[VincentJ]

https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/

They did a comparison between the gzip cli and the zstd cli and compression with zstd was around 5 times faster and decompression was around 3.5 times faster…

zstd is tunable, which is why i included four levels, which all beat gzip9 for compression ratio. If using zstd, we can deliver the image faster (because it’s smaller) and decompress the data faster once it arrives at the client… where is the loss?

I don’t have a testing infrastructure i can use at the moment that would yield verifiable speed results. My storage array is not built for IOPS and there would be live VMs also consuming bandwidth on the array and the network which would affect the result. I would also be reading an image off a drive while trying to write the same image to the same drive within a VM.

[Junkhacker]

@VincentJ if you’re comparing for performance, you need to compare zstd with pigz. pigz is what we use, and it is multithreaded. it is significantly faster than regular gzip while maintaining compatibility with it’s compression algorithm.

[x23piracy]

interesting: https://news.ycombinator.com/item?id=12399804

[VincentJ]

I am trying to setup something to test speed with local disks and a ramdisk…

I have a copy of pigz for windows and the zstd version of 7zip…

I’m setting up a quad core VM on one of my hosts with over 32GB RAM. CPU in the host is dual E5-2603V3 and storage is HDD Raid 1.

Which compression levels in pigz do you want me to try and test?

I am not entirely sure this gets round all of the problems speed wise which i could have that would make results not ideal… but it’s the best i can do.

[Junkhacker]

@VincentJ default compression level is 6. testing found that it had the best compression/performance ratio for us. i don’t think many people change it.

[Jaymes Driver]

@Junkhacker I have never had to adjust the compression. 3-5 minutes to image a client machine is well within my acceptance.

[george1421]

@Jaymes-Driver Same here. 3-5 minutes is more than sufficient considering that MDT and WDS take much longer to produce a finished product. No need to change it from the defaults unless you are dealing with under powered target systems.

[Junkhacker]

for what it’s worth, i’m testing a few things with this new compression method. i’ll share my results.

[VincentJ]

I have more results in progress on the RAMdisk. Hopefully i’ll get them done tonight.

[Quazz]

@VincentJ From what I’ve read this compression algorithm is targetted specifically at modern CPUs, leveragering their instructions sets, which older CPUs will lack.

Meaning that tests will be needed on older hardware, as there will be people using it for quite some time. If there’s a huge time penalty there, it will still be a no go for a lot of people.

[Junkhacker]

ok, here are my pseudo scientific results:

pigz vs pzstd (parallel implementation of zstandard, experimental)
tests were performed using a windows 7 image
(larger of 2 partitions only)
uncompressed image file size:34650439624

(de)compression tests were performed as closely as i could to emulate fog’s operation methods (without doing too much work :P). files were cat-ed from a mounted nfs share and piped into the programs, with results saved to an SSD. test machine was running Lubuntu instead of a custom FOS init, because i’m lazy.

pigz -6 compression
duration: 6:06
file size: 17548659028

pigz -6 decompression
duration: 6:00

pzstd default (3?) compression
duration: 5:16
file size: 16967988207

pzstd decompression default compression file
duration: 3:17

pzstd -6 compression
duration: 6:11
file size: 16247155611

pzstd decompression -6 compression file
duration: 3:16

pzstd -9 compression
duration: 10:00
file size: 16084180231

pzstd decompression -9 compression file
duration: 3:21

Edited to add zst compression level 6

[Quazz]

@Junkhacker Interesting, what kind of CPU did the test device have?

[Junkhacker]

@Quazz it’s an Optiplex 3020. i5

it may be worth pointing out that the pigz performance from my tests is not really representative of what i typically see when actually fog imaging this machine. testing was faster at compression by a significant amount, and a bit slower at decompression, than my experience. not sure what that says about the usefulness of these tests.

[VincentJ]

SO…

16,390,624KB file removed from the compressed windows 10 image.
on a 34GB RAMdisk.

Copying the file on the RAMdisk is 740MB/s so that is well above what we need for imaging for most people.

Lets try some things to get some numbers.

Compression - Compressed size - Compression time - Decompression time
zstd lvl1 - 7,940,779KB - 50 seconds - 38 seconds
zstd lvl3 - 7,420,268KB - 75 seconds - 40 seconds
zstd lvl5 - 7,286,951KB - 128 seconds - 40 seconds
zstd lvl8 - 7,070,670KB - 261 seconds - 41 seconds
zstd lvl11 - 6,967,155KB - 425 seconds - 41 seconds
zstd lvl14 - 6,942,360KB - 674 seconds - 42 seconds
zstd lvl17 - 6,781,375KB - 1,618 seconds - 42 seconds
zstd lvl20 - 6,471,945KB - 2,416 seconds - 43 seconds
zstd lvl22 - 6,214,702KB - 3,970 seconds - 45 seconds

pigz.exe --keep -0 a:\d1p2 - 16,393,125KB - 72 seconds - 80 seconds
pigz.exe --keep -3 a:\d1p2 - 7,783,303KB - 292 seconds - 158 seconds (157 seconds)
pigz.exe --keep -6 a:\d1p2 - 7,535,149KB - 518 seconds - 149 seconds
pigz.exe --keep -9 a:\d1p2 - 7,512,046KB - 1,370 seconds - 149 seconds

Windows 10 Pro, 4 vCPU 42GB RAM with 34GB RAM Disk.
Host XenServer 7.0, Dual E5-2603 v3, 64GB RAM, HDD Raid 1.
Other VMs moved to the other hosts in the pool.

Decompression seems to not use all CPU with PIGZ… around 50%…
Compression does use all 100% CPU
Decompression with zstd does use all CPU - but most were around 400MB/s so possibly I’m hitting some other limit.

[VincentJ]

So, both of us have results that show zstd decompressing quicker and having better ratios.

I’m going to reconfigure my VM to only have 1 vCPU at 1.6GHz to see if i can get more useful decompression results.

I redid the pigz -3 decompression test twice to confirm it was slower than the others… Not what i was expecting but that is what happened.

In my tests for compression the standard -6 on PIGZ is beaten by zstd for ratio by zstd lvl3 and completes 443 seconds faster… We could up to zsd lvl11 and have it 93 seconds quicker and save around 550MB.

[Junkhacker]

i’ve been talking with @Tom-Elliott about this, and we don’t think it would be worth the effort it would take to implement zstandard. the thing is, faster decompression is kind of irrelevant for FOG at the moment. what slows down deployments at the moment is transfer speed. the only way fog would get faster is if the file size was very significantly decreased. while the compression ratio is a better with zstandard, the difference isn’t very significant until you get to the higher compression levels, where processing time becomes a big issue.

there are other issues that deter us from adoption, but that’s the most significant reason. in fact, the single greatest reason TO adopt it would be because i think it’s really cool, lol.