Very slow unicast deploy when imaging 2 or more machines

[Tom Elliott]

@Haz I’m assuming you’re using “Partimage” for the images then?

I would almost think this being a part of the problem. Partimage does native decompression of the gzipped data, where Partclone we’re passing through a highly improved item.

Maybe try a re-uploading the image after it is on one system and give it a go? I’m just grasping at thoughts and while it shouldn’t seem to impact things, we’re essentially (as I’m hearing it) doing two times the work on the partimage files.

[george1421]

@Haz said in Very slow unicast deploy when imaging 2 or more machines:

Well that is a pretty comprehensive test. I’m still thinking it could be that single sata disk that is causing a problem.

One question comes to mind, when you do a 1 minute stagger start, after unit 1 finishes does unit 2 increase from 500MB/s up to 6GB/m or does it stay at 500MB for the duration of the imaging.

<edit> OK second question can you get the details (model number) of that hard drive. If you are using a dell then you can get this info from the bios screens.

[Haz]

Thank you again for the replies.

@Tom-Elliott said in Very slow unicast deploy when imaging 2 or more machines:

I’m assuming you’re using “Partimage” for the images then?

As far as I can remember, it is Partimage. I am out of the office now and will not be back until tomorrow (UK time) so will confirm it then. It is just the default imaging software that came with my installation (I installed FOG 1.2 from scratch and then used git pull to update to the latest version). If it does turn out to be this, would somebody be able to point me towards some instructions on how to install it to work with FOG?

@george1421 said in Very slow unicast deploy when imaging 2 or more machines:

I’m still thinking it could be that single sata disk that is causing a problem.

I am also starting to think the same thing. I’ll get the model number for you tomorrow but I can tell you it’s a 160GB Western Digital. Sometimes the performance on these drive can be pretty horrible, and simply running the WD Drive Utilities on the HDD in question can work wonders for drive performance - even when the drive is completely unreadable.

Yes, unit 2 does slowly increase in speed after unit one has finished imaging. Very slowly.

Just to add some confusion to this matter, I am actually going to be building a brand new FOG server for when we move premises in a weeks time. The current one was just a test run really, and wanted to try and iron out any potential problems before we get into our new office. I’ll be using a Precision T3500 with at least 1x Xeon processor in and I’m pretty sure the board is SATA3 too. I’ll be very mindful of the HDD I choose this time around…

I’ll obviously update everyone on progress of the current server and will let you know how the Precision compares once it is built.

[george1421]

@Haz In reference to Tom’s post. Was the image recorded with FOG 1.2.0 or newer or is this an old 0.3x image? The newer version of FOG and new captured image with 1.2.0+ will net better results than and older captured image with newer FOG.

If you have the option (if you only have a single disk) use an SSD instead of a HDD. The cost is trivial and the speed is great. Your specs on the 380 is sufficient, just the HDD is in question (in my mind only). FWIW I was able to deploy ~4.5GB/m for a dual unicast deployment using a Dual Core celeron in an intel nuc running off an SSD. So I know its possible to get pretty good speeds out of a low end system. The fog server itself is not really taxed much at all during deployment. The only thing the fog server does during a deployment is move data from disk to the network. The target computer does all of the heavy lifting during imaging.

[Wayne Workman]

@george1421 said in Very slow unicast deploy when imaging 2 or more machines:

I’m personally a bit suspicious of the 9GB/min simply because the theoretical maximum for a single GbE link is 7.5GB/min.

That speed is not transfer rate, it’s write rate, the speed at which partclone is writing to disk. If you want to see actual network speeds, install iftop on the FOG server, and then run iftop -n during imaging on the server.

[Wayne Workman]

@Haz said in Very slow unicast deploy when imaging 2 or more machines:

I repeated the process above with all 3 machines, however I “staggered” the deploy task to give around 1 minute between each one before the deploy starts.
This is where the speed issues began.
The first started normally around 6GB/m, but when the second machine started roughly 1 minute later, it was getting no more than 500MB/m for the first 15 minutes of deploying. The third machine started a minute after again, with no more than 350MB/m for the WHOLE deploy.

That is what I’ve found too honestly. This I believe is normal. A while back, I debated for changing the default Max Clients setting to be 2 or 3 instead of 10. Why? Because this prevents one computer from taking 2 hours to image, by limiting how many are going at once to just 2. I’d recommend you limit your max clients to 2 or 3 and you will most likely see the same overall duration decreases for imaging a lab of 30 computers as I did. Computers that try to image when 3 are already going will simply wait in line for their turn via FOG’s queuing system (you don’t have to do anything for this)- and that’s perfectly fine.

[Haz]

@george1421 said in Very slow unicast deploy when imaging 2 or more machines:

Was the image recorded with FOG 1.2.0 or newer or is this an old 0.3x image?

The image I have been using for these tests was captured using the new 1.3.0. I was thinking of transferring our old images to the new server, however I didn’t have much faith in it actually working or at best there would be issues somewhere down the line so I just captured from fresh.
Using an SSD was discussed, and I noticed an old thread here where somebody was thinking about using one, but I don’t think the results were actually shared. Now that you have clarified this for me I believe I will be going down the SSD route. Thanks.

@Wayne-Workman said in Very slow unicast deploy when imaging 2 or more machines:

That speed is not transfer rate, it’s write rate, the speed at which partclone is writing to disk.

Thanks for confirming this. I was getting rather confused after the server was continuing to display this speed after you stated it was not possible!

@Wayne-Workman said in Very slow unicast deploy when imaging 2 or more machines:

I’d recommend you limit your max clients to 2 or 3

That is a good idea. It will be a much better option for hosts to simply wait in line so that they can image at full speed rather than pull the rest of the hosts down. Considering the vast speed increase of 1.3.0 on just 1 or 2 hosts, this will be just as fast, if not faster than our old method on 0.32 anyway.

Many thanks to all of you for your input and suggestions, great help as usual.
Keep up the good work!

Harrison

[Quazz]

This thread is strange to me, because we can unicast like 7 hosts with barely any speed lost (7-8GB/min). Our Fog server has a refurbished HDD, so I don’t know if putting an SSD in your FOG server would help much.

Sounds more like your clients are being very slow for some reason (or network derping)

[george1421]

@Quazz said in Very slow unicast deploy when imaging 2 or more machines:

This thread is strange to me, because we can unicast like 7 hosts with barely any speed lost (7-8GB/min)

Please describe your fog server then (hardware, and number of nic connected to your lan from fog server)

[Quazz]

@george1421 It’s an HP Elite 6200 I think? Core i5 3470 or something, 6GB RAM, refurbished 5200RPM HDD, one 1gbps NIC, nothing fancy.

[george1421]

@Quazz Hmm, its less than I expected. But impressive speeds none the less.

[Quazz]

@george1421 I should note, that we don’t use FOG client and storage nodes or anything, so it literally only does network boot menu and imaging, which might explain it a bit.

[Wayne Workman]

@Quazz I’ve not had the same luck you’ve had. Our bottleneck was always the network connection to the FOG server, I could clearly see the NIC being maxed out by watching iftop -n

[george1421]

@Wayne-Workman Create a LAG trunk of 2 or more links. Its not fool proof but will help if you are using a static lag or lacp (802.3ad).

Depending on the algorithm used it will select a lag channel based on host target IP or MAC hash. This will spread the traffic out. I have to say it carefully because LAGs don’t multiplex the traffic over all links only allocate the traffic between two devices to different links.

[Wayne Workman]

@Haz The thread has gone off topic and has been forked to here:
https://forums.fogproject.org/topic/9085/host-already-registered-and-memtest-issue