PartImage faster than PartClone?

scgsg

@Tom-Elliott Don’t suppose you have any suggestions? be it server config or switch config or something? I am somewhat at a loss as to why higher compression equals faster deployment (or so it seems) but the size on server remains pretty much the same so the increased speed is not due to smaller file transfer. Not entirely sure how this impacts on deployment to multiple computers.

Junkhacker

@scgsg if you want to optimize for speed, i suggest switching to zstd compression. it works better with modern multi-core multi-thread processors than the previously used types. as for whether partimage or partclone is faster by themselves, i consider it a moot point since partimage has not been under active development in 7 years. partclone might be slower due to it’s built in checksums.

scgsg

@Junkhacker I did try zstd in one of my earlier tests and it clocked in at 15min 33secs so it is quicker. I’m not 100% sure but i think i left compression at default 6. I didnt test this any further because it took 1 hour 6 mins to capture the image. Testing this one for optimal compression is going to take a while.

Fair enough, if thats expected behavior between partimage and partclone, it is what it is and as you say a moot point. Furthermore its not like i can capture with partimage anyway. That being said, i have a real head scratcher with my latest test. By chance I captured a new image with image manager set to partimage and it captured using partclone so no surprises there. However heres where it gets odd, I deployed this image and its clocking in at just over 15 mins. I’ve done this 3 times now, twice to 2 computers at the same time and once to just 1 computer. Deployment times were between 15-16 mins. In between these tests i did retest deploying an image that had been captured with image manager set to partclone, and the result was 17min 35sec (similar times as before). This is confusing, why would setting image manager to partimage create an image that is faster to deploy? this is even more confusing as I checked the file size on the server for the captured images (one set to Partclone and the other set to Partimage for capture) and they are exactly identical. I really dont understand how I can get image deployment 2 mins faster just because I set it to partimage for the capture, given that it doesnt capture using partimage and uses partclone anyway. Any idea what the heck is going on there?

Junkhacker

@scgsg since we no longer support capturing with partimage, a capture image set to partimage is captured using partclone with literally the exact same code as if it had been properly set to partclone to begin with. I suspect external factors are at play causing any difference in speed you are seeing.

Wayne Workman

Using FOG 1.4 and ZSTD with partclone, my entire deploy time is 3 minutes. That’s from the moment the power button is pressed until the moment the task completes in FOG Task Manger. Using a Win10 image about 11GB in size, server disk is mechanical, host disk is SSD.

x23piracy

i have similar conditions but my image is a bit bigger around 15-16gb and i deploy in 2-3 minutes.

Bob Henderson

@x23piracy
I can also confirm, with multiple pieces of hardware (In this case, Dell Optiplex 790, Dell E7450, Lenovo Thinkpad T520, and Thinkpad X140e), that zstd and 1.4 will give me sub 5 min deploys of a fully configured Win10 image, including Office, Adobe Suite, and all Win. Updates.

I routinely deploy to 100+ devices a week, in both uni and multi cast. Fog server is an old P4 with 4 gigs of ram and 4 nics in a LACP bridge.

scgsg

Thank you for all the responses, i’ve been away for a couple of day and didnt get a chance to look at this. ZSTD is certainly quicker on the 1 test I did but i havent tested any further as it took over an hour to capture. Still it was only as quick as my most recent test with partclone gzip.

@Junkhacker yes I am aware captures are no longer done with partimage and that is my point, if i configure it as partimage, the capture is done in partclone (and it will change the setting to partclone after capture) but deployment is quicker than an image captured with image manager set to partclone. I dont know why that is, I’ve done a number of tests now and each time the deployment is 2 mins quicker. I cant fathom why 1 setting, which ultimately make no difference in the capture of the image, will deploy quicker.

Tom Elliott

@scgsg There are may things that impact the speed, regardless of the manager in use.

Network, the most obvious one here, would play a big factor.
System, as decompression is handled on the client.
Disk read/write speeds, this one is probably the largest factor on most modern systems.
Compression type, ZSTD compresses and decompresses much faster than gzip when in multithreading.
Compression Ratio, 19 is the maximum I’d recommend for any zstd as the memory required for any higher makes it nearly impossible to perform on most systems. 0 is the minimum, it still compresses but is nearly nothing. This impacts speeds because of the data that would need to transfer over the network. 1GB plain on a 1gbps will take anywhere between 5-10 seconds on a “perfect” network. 1GB compressed to 50% would take half the time to transfer because the data size would be only about 500MB. The speed it writes to disk would primarily be slowed down by the system decompressing the system. You must think of this when capturing too because while it will ultimately mean faster deploy’s, this setting can slow the capture process quite a lot.
Hops over network. The most direct route to a system will give you the fastest speed. If you have to jump through 10 switches to get to the host, your delays are somewhat related to the network as the data has to traverse the different points to reach the target.

These, obviously, aren’t everything that might impact speeds, but a basic list of what I can think of for right now.

As you’ve seen with the many other posts however, your case appears to be the exception. I understand there may be others with similar results to your own, but please understand we aren’t making these changes to make your day go longer. If anything we’ve made these changes so the vast majority of people will have a faster deploy/capture time. It also helps that partclone is in active development where partimage seems to have lost their develoment.

Wayne Workman

@scgsg said in PartImage faster than PartClone?:

if i configure it as partimage, the capture is done in partclone (and it will change the setting to partclone after capture) but deployment is quicker than an image captured with image manager set to partclone.

Most people on the internet are aware of the scientific method. How were you testing? How many times? What were the constants? The variable here is what the image manager is set to - all other things should be constant, and all other things would include every bullet that @Tom-Elliott posted below. And this would be repeatable. I would be able to conduct your test at home here and find the same results.

Junkhacker

@Wayne-Workman let me save you the trouble of testing:

    case $imgFormat in
        6)
            # ZSTD Split files compressed.
            zstdmt --ultra $PIGZ_COMP < $fifo | split -a 3 -d -b 200m - ${file}. &
            ;;
        5)
            # ZSTD compressed.
            zstdmt --ultra $PIGZ_COMP < $fifo > ${file}.000 &
            ;;
        4)
            # Split files uncompressed.
            cat $fifo | split -a 3 -d -b 200m - ${file}. &
            ;;
        3)
            # Uncompressed.
            cat $fifo > ${file}.000 &
            ;;
        2)
            # GZip/piGZ Split file compressed.
            pigz $PIGZ_COMP < $fifo | split -a 3 -d -b 200m - ${file}. &
            ;;
        *)
            # GZip/piGZ Compressed.
            pigz $PIGZ_COMP < $fifo > ${file}.000 &
        ;;
esac

this is the code that the image format setting gets used for on uploads. the default partclone image format is “1” partimage is “0”
as you can see, literally the same thing is done on an upload with either of those two settings set with regard to how the image is captured.

scgsg

@Tom-Elliott Wanted to say that I hope haven’t in anyway (either by implication or statement) come across as negative or complaining. If I have, that’s my bad as I think this project is great and I am very thankful for all your hard work. My intention was actually to see if it was expected behavior and/or maybe find the best way to get the most efficient deployment strategy.

Thank you for pointing out the main things that can/will have an impact on speed.

The gig switch only has the 2 test pc, vsphere host (that the fog server sits on and there are no other VMs on this host) and a connection to the rest of the network. The gig switch has default config on it, is there any particular config i need to pay attention to? given that I havent used multicast deployment in any of my tests.
I will keep that in mind particularly with our older machines but at the moment all tests are done on the same pc
Will keep that in mind too.
I think i intend on using ZSTD ultimately but havent for my tests at the moment as capture speeds are very slow i.e. with compression set at 6, it was over an hour to capture.
Interesting, can I clarify your statement ref max compression, is it not recommended for any zstd or recommended for zstd. I assume its not recommended due to the reasoning given, and what would you say is the optimal compression level on average (I realise this is dependent individual cases and environment and you probably cant give an answer).
Will keep this in mind too but plan is to have client on the same switch where possible.

@Wayne-Workman Well initially i was testing difference between old partimage deployment vs a new partclone deployment so initial partclone captured deployments times were used as a comparison against the later partimage setting captured deployments (yes i know its actually a partclone capture so there should be no difference). Everything is set the same and the only thing different is the image manager setting. Currently this would be 4 deployments where partclone was set and 4 deployments where partimage was set. I wouldnt say the sample is big enough to be conclusive but is what i have at the moment. At the moment I am in the middle of retesting all of it by creating another 2 images, 1 image is captured with partimage set and the other is with partclone set. Everything else remains the same i.e. same client, same image settings (except image manager setting), same switch with the same config, same process of capture (as described by Tom earlier, post 8 ) etc.

Junkhacker

@scgsg i would like to point out that, unless the client you’re using for testing is similar to the clients you’re deploying to, your benchmarks aren’t going to be very useful for you. your test client is using a processor that was a economy model when it was released almost 7 years ago. zstd and pigz are optimized for modern efficient multi-threading systems, and i suspect your Pentium isn’t taking advantage of them very well.

personally i use zstd compression level 11, as i find it has nearly the same upload speed as gzip at compression level 6 while making the images 26% smaller and deploy 36% faster. again, that is on more modern hardware than you’re using, your results will vary.

zstd compression level 19 is the highest normal compression level. above 19 are “ultra” compression levels that require massive amounts of ram.

scgsg

@Junkhacker Yes fair point and the client I’m using is one from the network with the majority clients being the same or equivalent.

Interesting, that certainly gives me an idea as how to approach zstd images when i start looking at this.

Ah ok, thank you for clarifying.

scgsg

@Wayne-Workman Ok this really is odd, as stated I’ve created 2 images with everything the same except the Image Manager setting before capture and there is a difference in deployment times. Here is what I did:

Capture Images:

Deployed the old image (created originally in partimage from fog 0.32) and set to shutdown after deployment
Create a new Image in Fog with the following setting to match the old image:
a. For the purpose of testing lets call it Image1
b. Default Storage group
c. Windows 7
d. Multiple Partition Image - Single Disk (Not Resizable)
e. Partition = Everything
f. Level 3 compression
g. Image Manager = PartImage
Capture new image and set to shutdown after deployment
Deployed old image again and shutdown after deployment
Create another image in Fog with the same settings as above but call this Image2 and Image Manager set to PartClone
Capture this image and set to shutdown after deployment

NOTE: Captured images are exactly the same size on the server (35061064).

Test Deployments:
Each deployment is set to shutdown after deployment.

Deploy Image1
Deploy Image1
Deploy Image2
Deploy Image2
Deploy Image1
Deploy Image2

Here are the results:
Image1 Image2
16min 29sec 18min 30sec
16min 27sec 18min 30sec
16min 29sec 18min 27sec

Again still not really a large enough sample to be definitive but does seem to imply an odd pattern.

Wayne Workman

@scgsg said in PartImage faster than PartClone?:

16min 29sec 18min 30sec
16min 27sec 18min 30sec
16min 29sec 18min 27sec

Which settings did these use?

scgsg

@Wayne-Workman Not sure what you mean, I’ve given details for the settings for Image1 and Image2 so what settings are we talking about?

Wayne Workman

@scgsg said in PartImage faster than PartClone?:

mage1 and Image2 so what settings are we talking about

What times are image one, what times are image two?

Tom Elliott

@scgsg I’m confused what this is showing. From what you’re saying, the deploy times are the “Updated” images? What version of FOG are you running? Level 3 compression is “small” comparatively. We have “benchmarks” for a reason. Your mileage may vary. But it’s up to you to find the “goldilocks” configuration.

You haven’t shown us any useful benchmarks.

If you’re trying to show us variance, we need to SEE the variance.

I’m assuming by your stuff below, the image deployment times for all three are the “averages” but what is this compared with/to? What other variables are present?

is the network working optimally?
Is there a single hop to the client’s receiving the images or multiple?
Are all machines identical in specs (Maybe use a single machine to perform the tests?)
Is your FOG 0.32 server configured and identically spec’d to your newer FOG Server?

Like I said earlier, there are MANY reasons for speeds to vary between one another. Once you rule out ALL of those variances you can get a solid benchmark.

If your FOG 0.32 server has 8 CPU’s and 16GB of ram, then your “newer” system should be identical in EVERY way possible. (If a VM, use one with 0.32 grab snapshot. Change out with 1.x.x and grab snapshot. Switch between the two snapshots)
Your network should be identical during your benchmarks.
Your machines should be identical during your benchmarks.

When presenting, you should show the same testing from the different servers to present the information accordingly.

For example:
Test set 1 FOG 0.32: (I know you can’t do partclone testing here)
Image: Configuration: Time Capture: Time Deploy: Number of times tested.

Test set 2 FOG 1.x.x (Import your 0.32 image and also present information less the capture.
Image: Configuration: Time Capture: Time Deploy: Number of times tested.

Deploy Image 1 (What is this, one partimage, one partclone?)
Deploy Image 1 (What is this, one partimage, one partclone?)

Quazz

From my understanding of it:

He captured the exact same image with the same image definition, except for the image manager (partimage vs partclone) and then noticed different deploy times.

PartImage faster than PartClone?

157

11.7k

17.2k

154.7k