ZSTD Compression

[Tom Elliott]

imgFormat is how we determined Partclone VS. Partimage
from upgrading 0.32 -> current, image definitions are automatically associated with imgFormat of 1 (which equals partimage)
All I’ve done is extend the capability of this field
so the way it works now:
0 = partclone gzipped
1 = partimage (gzipped of course)
2 = partclone gzipped split 200MiB
3 = partclone uncompressed
4 = partclone uncompressed split 200mib
5 = zstd compressed
6 = zstd compressed split 200mib

[Wayne Workman]

@Tom-Elliott said in ZSTD Compression:

I mean, To be usable to multiple systems, a compression bearing of at least half the size would make it a suitable alternative. 200 MB not worth it.

It probably doesn’t matter so much to us, most people don’t have over 1TB of images.

But that little difference to a company like Facebook or Google or youtube - well they might be bucking against the chains to get it implimented because it may save them 100PB of space (Taking into account very redundant storage plus backups plus mirror sites).

[Tom Elliott]

So I realized that while I had built and the zstd compression binary is working, the pzstd library was more of use as it allowed realtime and multiple core compression.

I am having a problem building pzstd for 32 bit systems though :(.

[A Former User]

@Wayne-Workman We actually have about 1.5 TB worth of FOG images.

We have a “Standard Lab” image, but we have numerous labs with custom software or just custom configurations. Then, we also have a “Standard Employee” image, too.

But honestly, space still isn’t the biggest consideration for us; storage is cheap, but time is not. It sounds like the time savings may not be worth it, or there may actually be a loss of time?

[Tom Elliott]

@Wayne-Workman When I was referring to ZSTD Compression and not being worth it, I’m not thinking other “uses” of zstd. I’m aware the larger the disk space and usage of it could be greatly enhanced. I’m not thinking in terms of “space” itself though. I’m thinking in whole.

For example, if a 2.0 gb image causes issues distributing to multiple items at the same time, a 1.0 gb image would make the same number of multiple item much faster as the data across the network would be greatly minimized. Even with smaller sizes this should old true though as you’re saving that amount of data. 200MB to one host may not be the best, but 200MB to 10 hosts would save nearly 2 gb of data being passed across the network. However, the time it takes to deploy to each of those hosts has many considerations to be thought of (but that would remain the case no matter the size of the file).

[Quazz]

@Tom-Elliott said in ZSTD Compression:

Alright, so I got bored.

I added all the “capabilities” as requested.

What I have found, so far, is the zstd doesn’t appear any better in compression even on -19 but with a SIGNIFICANT amount of overhead to wait. My image with pigz -6 was 2.8GB with 10 minute capture time. Same image deployed, and captured under zstd -19 was 2.6GB with 35 minute capture time. This was all done in NATIVE capture/deploy to ensure the inits were capable.

Mind you my test system was 1 CPU so multiple CPU’s may have helped in the capture time, but was it worth it? I mean, To be usable to multiple systems, a compression bearing of at least half the size would make it a suitable alternative. 200 MB not worth it.

This was kind of my fear with ZSTD, it seems very slow on older CPUs (and even more so on single core ones)

[compman]

@Tom-Elliott this is really unexpected. zstd tends to save a ton of time compared to gzip, on top of a better compression ratio, especially on large files (and anything > 100 MB certainly qualifies).

Except for cases where compression time doesn’t matter, zstd good use range seems to be levels 1 - 8. Default is 3, though I tend to prefer level 5 for my own use. It always compresses better than default gzip 6.

Your test image may contains large uncompressible sections (such as already-compressed files in the image, that no algorithm can compress further). This is typical of a freshly installed Windows OS. But is it representative of fog images ?

[Tom Elliott]

Remember, the speed was due to finding out after that I was using a non-multi-threading version of zstd. The speeds are much faster when paired with multiple CPU’s. The size, however, isn’t much different with or without multithreading.

[Wayne Workman]

@loosus456 said in ZSTD Compression:

We actually have about 1.5 TB worth of FOG images.

Then you are not most people.

[Wayne Workman]

@Tom-Elliott said in ZSTD Compression:

the pzstd library was more of use as it allowed realtime and multiple core compression.
I am having a problem building pzstd for 32 bit systems though :(.

32 bit systems can use the zstd one. Can’t wait to see the compression/decompression speeds when you get pzstd working.

I can setup some tests for multiple unicast tasks simultaneously and I’m sure I’ll be able to note greater performance with the better compression.

[Junkhacker]

@Wayne-Workman we have 3.7TB

[Wayne Workman]

@Junkhacker You’re for-sure not most people.

[Tom Elliott]

I want to say for people, using the zstdmt (vs pzstd as I can’t get it to build on 32 bit) the imaging is MUCH faster even at -19.

It’s compressing at about the same rate, more or less, but it’s much faster than the 35 minutes I originally had tested against. Using zstdmt on the same image with all cores in use took the 35 minute time and got it all done in 2 minutes 58 seconds.

[Junkhacker]

@Tom-Elliott i was just performing some testing at zstandard -11 to compare to pigz -6
image size went from 16.4GB gz to 14.6GB zstdmt
capture time 14 minutes 15 seconds pigz to 11 minutes 4 seconds zstdmt
deploy time 3 minutes 33 seconds pigz to 2 minutes 38 seconds zstdmt

[Wayne Workman]

geeze that is flippin fast… ok I’m sold.

[Junkhacker]

i have not been able to get “ultra” levels (>=20) tasks to work without crashing. anyone else seeing the same?

[A Former User]

Anyone tried like a 40 GB image? That’s in the range where I’d like to see a comparison in deployment time.

[george1421]

@loosus456 I would say that a 40GB disk image is uncommon. If you have a source, then I might suggest that you update to the latest RC release and test the results.

[A Former User]

@george1421 40 GB is uncommon? Uh what? Are you guys deploying XP or something?

[Tom Elliott]

@Junkhacker I’m seeing exactly the same problem, and It feels like it’s writing the data and filling the RAM space too quickly. It’s doing so far faster than it can compress it, and overruns the memory space that partclone is using. (Just my theory).