Imaging computers at 2.6Gbps

  • Yes, FOG can do it. I just built a new server with 1TB SSD, 32GB memory and 10Gbps fiber network adapter for well under $2,000. It’s a 1U Supermicro barebones system with an i3-based processor running Ubuntu Server 16.10. All my computers are SSD-based finally, and following are the results of my first test download of an image (157GB image across two separate switches with 1GbE port at the client).

    Entire image completed in under 15 minutes, never dropped below 9.5GB/min. Top speed was about 10.9GB/min, or 1.5Gbps [edit: up to 19.75GB/min or 2.6Gbps thanks to zstd, see more recent post below). This was only possible over a 1Gbps link due to the compression of the image. I ran this test in the middle of the day while about 400 people were on the network. I didn’t look at the network port utilization but it had to be close to 100%. Client computer is an i5-3570K (3.4Ghz) with 256GB MSATA SSD and 8GB memory. The server’s 10Gbps link isn’t all that necessary for single deployments like this but for deploying to multiple classrooms at once it should allow me to image dozens of computers at once in unicast mode (most of our edge ports are still 100Mbps).

    No RAID on the server, I figure if it croaks I can rebuild. One backup of the server and database and backups of my images are enough for me to sleep soundly at night, which I will be able to do more now that I can image this quickly! Anyone else seeing these kind of speeds? I continue to be impressed with FOG.


  • @Joe-Schmitt Just dropping from NFS over to HTTP/SPDY/etc is huge.

  • Senior Developer

    @Bob-Henderson there’s really 2 factors going into it (keep in mind this is a very simplified / high level overview of what we’re doing). First is the removal of NFS (it’s still an image storage option if you want, but not how FOS reads the images now). The server is now the one responsible for streaming an image to the client, this means we can use protocols such as http/2 streams, SPDY, on-the-fly packet compression, so on.

    Secondly is making the server and client more intelligent (giving it a basic ai of sorts). FOS will basically be running a custom build of the linux FOG client to give us access to our existing framework, and real-time server communication. This means the server / client can automatically take care of throttling on congested networks and on-the-fly transferring which server (assuming you have a multi server setup) FOS is receiving the image from incase of high workload, or if a more optimal server frees up.

    TL;DR By making FOS smarter, and upgrading our transport protocols.

  • @Joe-Schmitt Any documentation or such available for this? I’m curious about how you’re doing it from a technical perspective.

  • Senior Developer

    @Obi-Jon @Junkhacker it would be interesting to see the speed difference using our new fog 2.0 transfer protocols / techniques when it’s more ready for testing. Generally speaking our new approach we’re working on is far more stable in transit (less packet loss), and capable of a much higher throughput / network efficiency.

  • @Obi-Jon

    Ah, yeah, I remember that game. We ended up doing a disconnect/reconnect system myself, to alleviate the issues.

    good thing there are more colors available, huh?

  • @Bob-Henderson My color scheme for network cables has been based on length (5’ = white, 7’ blue, 10’ pink, 14’ yellow, etc). Maintenance disconnects everything every summer, waxes the floors or shampoos the carpets, then hooks everything up again, so the different colors on each row of computers is useful for figuring out which cables go where. Not sure I can do away with that for network cables, but I’ll definitely incorporate your idea for everything else.

    Good idea to post laminated signs too.

  • @Quazz Yes, the target device does appear to be the deciding factor for deployment speed when imaging 1 device. That said, my old FOG box was running 0.32 (didn’t upgrade it due to some customization I had made to it) and was WAY slower than this, even with the same clients. The server was no slouch, even for 5 year old hardware, but the newer versions of FOG (partclone, etc) are making a big different too.

    From now on I think the bottleneck (if you can call it that) will be endpoint network bandwidth. We’re pretty much saturating 1Gbps links as this test shows, so going concurrent with 10Gbps link at the server is the next logical step. For me, 10Gbps is overkill since I have mostly 100Mbps endpoints with 1Gbps uplinks, but as I upgrade endpoint switches client speeds will improve a lot. Heck, with 100Mbps endpoints and SSD on the server I am thinking I can saturate 50-100 clients simultaneously at 100Mbps each. Can’t wait to try it.

  • Moderator

    From my experience:

    Your FOG machine doesn’t need to be that fancy to accomplish this. It will be useful for concurrency and what not, of course, but I’ve had similar results with a 2nd generation i5, 6GB RAM, 500GB HDD (!!!) and 1gbps link

    The key to fast imaging is primarily down to the target device (assuming your imaging 1 device of course). Modern multi-core CPUs tend and faster storage solutions tend to get deployed to quite a bit faster.

  • @Obi-Jon Thats how it started with us too. Labels didn’t work, cuz who can asked to read something written in big letters on each end…

    So we stock 10 colors now, and they all have a reason for them. We have a sign made up for the colors, and posted it in every room laminated.

  • @Bob-Henderson That’s a really good idea. Gives me ideas for audio/video stuff that gets plugged in wrong every time they wax the floors…

  • @Obi-Jon eww ya, no, don’t do that.

    Color code the jacks and cables. blue jack, blue cable, orange jack, orange cable. It’s what we do for our users. Another school here, but we’re a 1:1 shop so all my stuff are laptops. Snapins are my saving grace, tied with PDQ Deploy

  • @Bob-Henderson Good point on the Intel nics, that’s what I’m using as well. My new clients are based on H110T chipset, which has both Intel and Realtek nics. I made sure to only enable PXE on the Intel nic and disabled the Realtek nic entirely. Now to keep my users from plugging into the wrong jack and generating more help desk tickets. I’m actually contemplating gluing a punched down RJ45 plug into the Realtek ports, lol.

  • @Obi-Jon That’s great to see. We see very similar speeds with ours, though it’s just a VM on ZFS based storage with 4g of ram and 2 cores. I think the key is making sure you’ve got solid clients, and ssds on the client side. And Intel nics.

    Some of our broadcom equipped thinkpads max out at around 7, the intel Dells hit the 10,11 mark.

    This is all on gzip. Haven’t made a new image since the new compression.

  • @Tom-Elliott Lol, my sentiments as well, been pumped all day. Can’t wait to try this over multiple unicasts simultaneously.

  • @Obi-Jon I feel I must add, once again.

    “Holy Shit”

  • @Tom-Elliott Ah, I borked the log then when I uploaded the image and redownloaded after the error to compare image file sizes. No errors the second time since the images matched. If I see that error again I’ll revisit this. Thanks!

  • @Obi-Jon Partclone.log is related to the client.

    Anything presented on the client in regards to FOS (Inits) will be in respect to the client.

    You could add isdebug=yes to that host’s Kernel Arguments field. This will drop the host into a terminal prompt where we can step through and obtain more information.

  • @Junkhacker “locate partclone.log” does not find a log file. Thinking a log was not generated, or am I looking for the wrong log file? I tried “locate *.log” but didn’t see anything promising:


    0_1491518979404_Speedtest error.JPG

  • Developer

    @Obi-Jon the error is most likely completely unrelated to the compression level. if you can capture more information, we’ll see if we can find the cause.