FOG mobile deployment server for $200 USD (finished)

[george1421]

This is going to be a bit blogish tutorial since right now this is more of a concept than a working model. Buy the last post here I should know if this is a reality or a fools errand.

I have two use cases in mind for this project.

1. A friend of mine has a good size dentist office with around 40 computers, no computer room, the dental software is hosted by their software vendor so there are no real servers on site, and he has a desire to harmonize the software installed on all computers. Luckily there are only 3 different models of computers in his office and they are all made by Dell. Oh, and he doesn’t want to spend any money to do this. While I could cheap out on this and just use an old desktop, I want to find a solution that is cheap to install, operate and is under warranty.

2. I talked to my brother who has a mobile deployment situation that he asked if we can work on a solution while he is here over the holidays. Just for a little background on this, he works for a company that has sales offices around the world. They will have a global sales meeting in March where all of the sales people [~30] will be meeting at a hotel somewhere. Historically, for convenience most of these users have OEM copies of Win 7, 8.0 and 8.1. He has been instructed that all of these computers must be wiped and reloaded with the company standard image and applications. He has 2 days to re-image these systems before the sales people fly back to their homes.

For both of these projects I’m looking at using an Intel NUC Dual DN2820FYKH Dual core Celeron. We currently use this model for our on site digital signage system. I have loaded win7 on these boxes. It does run Win7 and is “usable” for common office applications and web browsing. The use user would do much more I might move them to the i3 or i5 based system. But for this project one of my constraints are “cost no money” (fyi my dentist friend is just going to have to deal with the $200 cost). This system should perform very nicely for the FOG server since there will be no X server shell to power with the CPU. Just for giggles I ran the windows 7 performance index on this system and it came in with a 4.4 with the limiting factor being the GPU.

[sidebar] I had briefly considered to use the Raspberry Pi 2 for this. It should work without issue, but I have concerns with how fast I can move the image from the built in micro ssd to the target computer. I know from past experiences that the quality of the micro flash drive has a direct impact on the performance of raspbian on the PI. I might try that project later, just to see if I could make it work. But for now I’ll stick with a known working x86 based platform [/sidebar]

The DN2820FYKH is what I would call a kit computer. The DN2820FYKH comes with the enclosure, mobo, wireless/bluetooth module, CPU and international power supply. You will need to add a RAM stick and a hard drive for storage. (I can’t remember if this NUC has the built in 4GB flash hard drive or not, but in this case I’m not going to use it). For this project I’m going to use my trusty sandisk 128GB SSD drive.

Shopping around a bit I found this kit on Amazon. http://www.amazon.com/Intel-DN2820FYKH-Celeron-N2820-support/dp/B00HVKLSVC If you look at the add on section it actually lists all of the parts I’ll need. These are exactly the same bits as we purchase for our signage project. In case the URL fails in the future I’ll itemize them below.

(1) Intel NUC DN2820FYKH ($125 USD)
(1) Crucial 8GB Single DDR3 CT102464BF160B ($35 USD)
(1) SanDisk SanDisk 128GB SATA 6.0GB/s SDSSDP-128G-G25 ($55 USD)*

• The SSD drive listed in the amazon link is for Kingston. For me its worth the extra $20 to go with the Sandisk. I can go with a little better SSD drive and still come in under my $200 constraint.

Items I’ll need for this project

1. The NUC kit from above fully assembled
2. Centos 7 ISO image
3. 8GB USB flash drive (to install centos 7)
4. FOG install tar ball. (I’ll probably end up with a current trunk build since FOG 1.2.0 is pretty old in computer terms).
5. Win32DiskImager http://sourceforge.net/projects/win32diskimager/ (to “burn” the Centos ISO image to the USB flash drive)
6. ??

[george1421]

(default placeholder)

[george1421]

Post date 12/27/15

Just for grins and giggles, I replace the seagate rotating hard drive in the e6400 with a Crucial MX100 256GB SSD I had laying around. I again redeployed the same image as in the previous tests, this time the transfer rate was 7.8GB/m (130MB/s) according to partclone. As compared to 5.1GB/m with a rotating disk in the target. I booted the e6400 back into debug mode and ran hdparm -Tt /dev/sda hdparm reported 242MB/s for buffered disk reads as compared to 80MB/s with rotating media.

In the end I’m marking this project a success and case closed.

For reference here is the NUC DN2820FYKH attached to the back of a 23" Asus wide screen monitor as we’ll setup in the dentist’s office part of the project.

[george1421]

Post date 12/27/15

I booted the e6400 into debug mode and ran hdparm -Tt /dev/sda to get the hard drive performance just on the e6400. The results were 80MB/s for buffered disk reads (as compared to 252MB/s for the NUC with an SSD drive). So that tells me the hard drive on the target computer is the limiting factor in this test. Which is good to know since the NUC as it sits should drive multiple unicast deployments at the same time. The limiting factor will be saturating that single GbE network interface on the NUC.

Just for grins I grabbed another computer to see if my bandwidth speculation from yesterday would hold true. So in this setup I use 1 e6400 and 1 e6410 and scheduled two unicast deployments at the same relative time.

The e6400 still continued to pull 5.1GB/m (85MB/s) transfer rate according to partclone. With the e6410 pulling at 6.1GB/m (101MB/s) during a simultaneous transfer. I’m surprised and a bit confused how I can transfer (consume) more bandwidth than I have available on a single GbE link. But either way the NUC continued to serve both target computers at a normal rate.

So for this I think this Dual Core Celeron NUC is a suitable for a mobile deployment server and for a small office deployment server.

Target computer stats
e6400 CPU is core 2 duo P8800 @ 2.66Ghz 4GB RAM 80GB rotating hard drive
e6410 CPU is i7 M620 @ 2.67Ghz with 2GB RAM 160GB rotating hard drive

[george1421]

Post date 12/26/15

To keep this thread from being miles long, I broke out some of the setup to external technical posts. The OS was installed on the NUC according to this document: https://forums.fogproject.org/topic/6384/install-fog-centos-7-on-intel-nuc-dn2820fykh

Since my home router doesn’t have the ability to set/change dhcp options 66 and 67 I had to install dnsmasq on my FOG server using what I outlined in this post https://forums.fogproject.org/topic/6376/install-dnsmasq-on-centos-7

The test FOG server was installed using the latest FOG trunk at the time of this writing (Cloud 5798).

For the target computer I used an old Dell e6400 that my brother discarded a few years ago. I use the image that was on that system as the reference image. The captured image is about 29GB in size. I captured the image to the NUC FOG server and then deployed that same image back to the target.

I would like to say that my initial upload and download test were really unimpressive. I had transfer rates about 10MB/s. I was about to scrap this whole idea because that is an unrealistic transfer rate and knowing my brother project would not be fast enough to deploy to the number of systems he had in the amount of time he had to complete. As I was thinking about it, 10MB/s is about the max transfer rate for 100Mb/s network.

I checked the NUC and confirmed that it’s link state was 100Mb/s. Then I checked the switch the NUC was connected to. That is when I realized the switch was 100Mb switch. While this was disappointing it was not the fault of the NUC or the e6410.

My brother had a spare 8 port GbE switch at his house so it brought it with him.

Test this time with the GbE switch gave us a better idea of the capability of the NUC deploying to the e6410. This time we had deploy rates of 5GB/m or 83MB/s according to partclone. This is more of an acceptable transfer rate since it is closer to the max theoretical transfer rate for GbE at 125MB/s.

I tried to tweak/squeak a bit more throughput by modifying several kernal and nfs parameters. None had any significant improvement over the 83MB/s transfer rate.

For grins I ran hdparm on the NUC. Here are the stats of hdparm

# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   3072 MB in  2.00 seconds = 1536.39 MB/sec
 Timing buffered disk reads: 758 MB in  3.01 seconds = 252.05 MB/sec

As you can see I have a transfer rate of 252MB/s per second between the NUC and the 128GB SSD drive. So the disk channel on the NUC has sufficient capacity and isn’t the bottleneck.

I’m speculating that the bottleneck is actually on the target computer, either with decompressing the image or the disk subsystem on the target computer. I’m actually leaning towards the target computer’s hard drive since it is a spinning disk and not a SSD so a 83MB/s data path to a spinning disk is reasonable.

At this point I think I’m happy with the NUC and its single unicast deployment to the e6400. I would like to see if I can get an answer to the target computer hard drive stats and to see what happens when I try to deploy two systems at the same time. I suspect that is where the ssd in the NUC will have an advantage. CPU usage during the single unicast deployment was about 12% utilization with the highest process NFS.

I’m totally guessing at this point.
If one computer can pull 83MB/s then if I add a second one I would expect both to saturate the GbE network interface. This should give me a about 60MB/s per system.

83 - (((83*2)-120)/2) = 60MB/s
83 = MB/s for a single computer
120 = GbE max bandwidth minus a little overhead.

[george1421]

Post date 12/22/15
Before I continue, I do have to say this is NOT the way I would build a FOG setup for a SMB or enterprise deployment. In the two cases above the number of devices is < 50 and this is more like a load and go setup than using FOG to its fullest potential. A proper FOG server should always be setup as an enterprise class deployment either in a virtual environment or in the physical realm use a properly backed up server running on a raid array to protect the deployment system.

1. Download Cento 7 from your favorite mirror
2. Create a bootable Centos 7 image on the USB flash drive. If you are windows centric user, follow the reference link below. Since I’m using a zorin based laptop I’ll just use the dd comand.
   ref: http://www.sysadminguide.net/how-to-create-bootable-usb-key-for-centos-7-installation/
3. Insert the flash drive into your linux computer.
   To determine where the flash drive is mounted issue the following command: lsblk. Your output should look similar to this:

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 298.1G  0 disk 
├─sda1   8:1    0 294.3G  0 part /
├─sda2   8:2    0     1K  0 part 
└─sda5   8:5    0   3.8G  0 part [SWAP]
sdb      8:16   1    15G  0 disk 
└─sdb1   8:17   1    15G  0 part /media/johndoe/BEA0-A4EA

4. As you can see from the printout my 16GB flash drive is device sdb and its mounted on /media/johndoe/BEA0-A4EA. Armed with this information we’ll need to unmount (not remove from the computer) the flash drive before we use the dd command. To unmount the flash drive issue the following umount /media/johndoe/BEA0-A4EA
5. Once the drive has been unmounted I use the dd command to write the iso image to the flash drive. The syntax for dd is dd bs=4M if=CentOS-7-x86_64-DVD-1511.iso of=/dev/sdX where X is the device number of the usb flash drive. Get this right or you could overwrite something you might need later. Using the information from the lsblk command I know my flash drive is /dev/sdb so the proper dd command to write the iso image to my flash drive is: sudo dd bs=4M if=CentOS-7-x86_64-DVD-1511.iso of=/dev/sdb Depending on how fast the flash drive is it may take several minutes to write the iso image to the flash drive. On my laptop this was the output from dd.

1032+1 records in
1032+1 records out
4329570304 bytes (4.3 GB) copied, 521.836 s, 8.3 MB/s

6. Remove and reinsert the flash drive into your computer. Your computer should properly mount this flash drive. Viewing the contents of the flash drive should be successful.

7. Before we boot the DN2820FYKH for the first time lets go and pick up the latest bios, because we will want to update the bios early in the process. The latest bios version for the 2820 (at the time of this writing) is from 10/2015. You can get the bios directly from this link https://downloadcenter.intel.com/product/78953/Intel-NUC-Kit-DN2820FYKH. In my case I’m going to download the FY0052.BIO file. You don’t need any of the downloads with the updaters. We are going to do this from within the bios. So all you need is the .bio file.

8. Download and move the FY0052.BIO file to the root of the flash drive we just created.

9. Remove the flash drive from your computer and insert it into the NUC

10. Waiting on my amazon order before continuing.

[george1421]

As I anxiously wait for the FedEx delivery tomorrow of the NUC and other bits I don’t really think I covered why I choose the NUC over the other possibilities available. There are quite a few manufacturer that make the tiny/micro platform computers from Dell, HP, Lenovo, and ASUS. I could have went with any of those manufactures because their offerings are roughly equivalent.

My justification may get a bit mushed together because I have to potential use cases for the same product. (Actually I have a third use case in that my company is setting up a remote sales office along the gulf coast and this nuc may be a good fit or a remote storage node. I surely has the right price point where I can have a warm spare sitting on the shelf at the remote office [consider using wol to wake up the warm spare once a month (or when major image updates are released) running the sync and then powering off the device once the update is done].

1. Go with what you know.
   I have personally used these exact devices before. I know their capability and how to manage them. They have a solid track record and a 3 year warranty.
2. The Intel NUCs are pretty small.
   The DN2820FYKH dual core Celeron is roughly 4.5" square and 2" (115 mm square and 51.5 mm) tall. These will easily fit into a carry on or pack into a suit case for my brothers travels (i can see the TSA having an issue here but we have time to work that out). For the doctor’s office they don’t have a computer room to speak of. These NUCs come with a VESA mounting bracket so they will mount onto the back of most desktop monitors and tuck out of sight.
3. Hardware available internationally
   While it isn’t important for the scope of my project, my company does have specific requirements for hardware availability so that all divisions can share in the technology developed in their tech division.
4. Covered by manufacturers warranty.
   These NUCs come with a 3 year manufacturers warranty. I can’t say good this warranty is since I haven’t needed to use it at all. But when it comes to warranties, (this is my personal belief) the manufacturers build in this time-period for their benefit not the customers. Based on their testing they feel this NUC should run at least 3 years (on average) without a warranty call. I have more faith in a system that has a 3 year warranty than one that has a 30 or 90 day warranty. I have to think when I see a manufacturer offer a 90 day warranty, do they really believe in their product?
5. Hit a specific price point.
   For this project this is a primary constraint. Can I built up a solid system that will function at an acceptable level for about $200 USD. Based on what I know of this NUC I think I have a good chance to satisfy this requirement. I could have just picked up a 2 year old laptop and been assured that it would work. But then I would have not had a 3 year warranty, hit my price point, or the size constrains.

[george1421]

6. Power consumption.
   This Intel NUC consumes about 36 watts of power (about the same as a netbook). The 24x7x365 (yearly) expected cost of running this NUC is less than $32 USD in energy costs. A 52 watt laptop will cost about $45 USD per year. And finally an old desktop computer running for year continuously will cost abut $131 USD. While there isn’t a huge saving difference in consumed electricity, there is a difference. In the scope of the my brothers project it isn’t quite as important as the dentist’s office.
7. Contains a standard x86 processor
   While is isn’t specifically a requirement it makes it easier to use off the shelf pre bundled packages instead of having to cross compile packages for the target hardware (thinking Raspberry PI here). FOG should run on a PI because you can run linux on a PI without issue.
8. Uses commodity hardware
   The NUC uses off the shelf common bits for add on and expansion devices. There are no special device specific memory or storage modules that only work in device x or product y. Actually I for my business project I used an 8GB memory stick from a Dell e7440 and it worked without issue. NOTE: In regards to memory you MUST purchase DDR3L (low power 1.5v) memory. I made the mistake of ordering DDR3 (5.0v) memory once and it didn’t work.
9. Kit of bits and bytes
   The NUCs are classed as a kit computer. Intel provides the foundation hardware so you can purchase the amount of memory and disk storage you need for your project. If your project needs 2GB of ram and a 2TB hard drive you can purchase off the shelf components to meet these system requirements. Another thing to note is that the NUCs do not come with any OS license. Since we will be running FOG under linux, having a windows OEM license baked into the hardware costs of the NUCs is just a waste of money. The other hardware platforms I looked into all come with windows OEM licenses so it pushed the unit costs over the self imposed $200 limit. But conversly, if this project required a Windows OS license then having to purchase a reail license for the NUC would put the total unit cost in the same ballpark as the other hardware manufacturers offerings.

I’m not saying this is the only hardware that meets the requirements. But from a size and costing standpoint it is in line with the expectations.

I do have concerns about the NUCs. Can I deliver the image at a high enough bandwidth to the target computer to make this a viable option. My brother has to image 30 computers in 2 days and have enough time with the computers post deployment to ensure that they work as intended before the sales guys fly back to where ever they came. This is a big risk, but technically FOG should be able to lay the image onto the target computer in 6 to 8 minutes (I can do it in about 4 minutes with my production servers). Once the image is on the laptop FOG is out of the picture (in this case we will not be loading the FOG client so no interaction is needed by FOG once the image is on the computer). For the dentist office, speed is not important at all.