ZSTD Compression

[Tom Elliott]

@Junkhacker I’m seeing exactly the same problem, and It feels like it’s writing the data and filling the RAM space too quickly. It’s doing so far faster than it can compress it, and overruns the memory space that partclone is using. (Just my theory).

[Tom Elliott]

@loosus456 My base images with all software had a disk usage of 48gb at it’s largest, but that was EVERYTHING I could put on. My “average” was about 25gb.

[compman]

@Junkhacker “ultra” compression ratio (>=20) use a lot of memory.
So, either stick to “normal” compression levels (<= 19), or limit the number of threads, to ensure it doesn’t consume too much memory.

[A Former User]

@Tom-Elliott We have images with Autodesk and Solidworks near 100 GB.

[Tom Elliott]

@loosus456 I’m not disagreeing, I’m just saying what I imagine “most” people are working with (on average).

[A Former User]

@Tom-Elliott I dunno. Most educational professionals I work with have had similar sizes.

[Wayne Workman]

@loosus456 said in ZSTD Compression:

40 GB is uncommon?

My Win10 images were around 20GB. Linux images are even smaller, 3 or 4 GB - and that’s all I deploy anymore, just all Linux.

Understand when I build an image - it’s lean. No flab. I literally throw out the manufacturer’s bloated crappy flabby unoptimized image and build my own from scratch. Not just for size & performance reasons but for security reasons too. Computer manufacturers don’t vet the bloatware they sign contracts for - to install into their images - many times these crappy bloated pieces of software are found to have vulnerabilities. But I make images lean. In Windows that means a vanilla installation & using device manager to install ONLY the drivers and no extra flab/bloatware crap. When I put MS Office on the image, I only installed the pieces that people used, not everything. The fat 10GB recovery partition gets thrown out, don’t need that. I also rebuilt all my images every summer - because if all you do is take an old image and patch it up, it gets bloated & slow.

[falko]

Tom-Elliott said in ZSTD Compression:

@loosus456 My base images with all software had a disk usage of 48gb at it’s largest, but that was EVERYTHING I could put on. My “average” was about 25gb.

I use a single win7 image at my place (nearly all software etc.), it’s just under 30gb. Will make some time to do some tests

[sudburr]

@Junkhacker The majority of my images are ~55 GiB compressed, 85-115 GiB deployed. And yah, Autodesk and Adobe products suck it up real good.

[Junkhacker]

@loosus456 the image i was testing with here was 33.87 GB “on client”

[VincentJ]

So… Deploy with old fog, capture with new fog on a different image set to zstd 11

I noticed on the upload, the screen still says:
Starting to clone device (/dev/sda2) to image (/tmp/pigz1)

The image is Multiple Partition Single Disk (non re-sizeable)

On my old image it has two files… but the new one has three…

I am pulling the image off the NAS now to unpack it and test it, but thought i would pitch in on what i’ve seen so far.

svn revision 6066

[Junkhacker]

@VincentJ “/tmp/pigz1” is just the name of the fifo that the data is being piped into to be sent to compression. maybe we should rename it for purely aesthetic reasons, but that’s working as it should. was the previous image also Multiple Partition Single Disk (non re-sizeable)?

[VincentJ]

@Junkhacker Yes it was. All of my images are.

[JJ Fullmer]

So I just updated to RC 15 and made a new image at zstd level 9. I was using gzip level 5 before. The image is still currently downloading, estimating the full time to be about 12 minutes. With gzip the download was taking around 3 minutes. This is a 6th gen i5 client and the fogserver is on a vm with 8 vcpus (intel xeon something something) and 16 GB ram. There is also a storage node on a older server with a little less power, but still imaged around 2-3 minutes.
The size of the image uncompressed is ~15 GB and it is 7.5 GB compressed with zstd. My old image was ~18 GB and with gzip level 5 it compressed to 6.7 GB.

So what did I do wrong? Is the difference between 9 and 11 really that substantial? Should I re-upload that image? Also split vs non split? I saw the post on the topic of it being better for multiple images, but are there any other considerations in regard to speed?

[Tom Elliott]

@JJ-Fullmer ZSTD “mid” range is 11, compared to PIGZ “mid” range being 5-6.

I set my images to 19 now and it’s pretty awesome.

[JJ Fullmer]

@Tom-Elliott But at 19 are you sacrificing speed substantially?

[Tom Elliott]

@JJ-Fullmer Not at all.

[JJ Fullmer]

@Tom-Elliott I will give it a try then

[JJ Fullmer]

@Tom-Elliott What about split vs not so split?

[Tom Elliott]

@JJ-Fullmer Either or, shouldn’t matter in terms of performance.