Hardware upgrades for server
-
Truthfully what you have is more than what is needed for FOG. The fog server really doesn’t do a lot of the work in imaging. It manages the process and transfers the image from disk to the network adapter. Its been a while since I measured it, but I was getting about those transfer rates with my FOG-Pi server (Raspberry Pi 3). Yes your transfer rate is a bit slow, I would expect numbers in the 4.5 to 6.1 Gb/min range assuming you are on a solid GbE network using enterprise switches. As I said the heavy load in transfer is not on the FOG server but on the target computer itself. The point being if you are transferring to a dual core processor with an ssd or nvme drive you will get a faster deployment than transferring to a quad core processor with a sata low power drive.
I do have some additional questions about your setup. Since you have a 2950, you are probably running either SAS or SATA disks, now many disks do you have in that server? Are they backed by a raid controller like a Perc4 or Perc5?
If I gave you a few commands to run could you execute them on your FOG server?
-
@george1421 good thing I was about to spend some money on more ram and better cpus.
Yes I’m running 2 SAS Drives. 15K rpm they are in raid 0 so speeds on the drive themselves should be really good. We just upgraded our core switches but our computer lab switches are 3560 so bit old. I thinking that’s the bottle neck.
I’m not at the office right now, but you can give me the commands and i will try them when I’m back.
Fog 1.4.4 running on CentOS 7 -
@tesparza OK give me a second. What you need to do is benchmark what you have. In your case you have 3 metrics on the fog server that will impact performance.
-
Disk subsystem
-
Network stack
-
NFS stack
-
(I know I said 3) the number of clients you have checking into the fog server and the check in cycle time.
-
-
@tesparza Lets start with these commands:
We’ll use the
ddlinux command to create a sequential 1GB file on disk and then to read it back. This process is designed to simulate the single unicast workload. The command used to write the 1GB file is this:
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=directThe command to read it back is:
echo 3 | tee /proc/sys/vm/drop_caches && time dd if=/tmp/test1.img of=/dev/null bs=8k
The echo command is intended to disable the read cache so we get a true read back value.For a Raid 0 dual sas 15K drives I might expect to see 280-350 MB/s disk speeds. I might expect read and write speeds to be near consistent (within a few percentages).
-
@george1421
did the test and these are the results
[root@localhost tesparza]# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 4.99397 s, 215 MB/s
[root@localhost tesparza]# echo 3 | tee /proc/sys/vm/drop_caches && time dd if=/tmp/test1.img of=/dev/null bs=8k
3
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB) copied, 4.27885 s, 251 MB/sreal 0m4.366s
user 0m0.042s
sys 0m1.058s
[root@localhost tesparza]# -
@tesparza Those numbers are not bad. They are a little lower than expected, but more than adequate. With those numbers that disk subsystem can produce (theoretical max) of about 13GB/min. So your bottle neck is not with your disk subsystem.
The next step is to check the network stack. Ideally we’d like to get a computer connected to the same physical network switch as your FOG server.
-
@george1421 Okay Im gonna try to capture again, this time im directly connected to the same switch as the fog server. 1Gbps link
-
@tesparza Understand that capture rates will be different then deployment rates. On capture you take the penalty for faster deployments.
I haven’t had a chance to get back to this thread, but the next steps are to test the network stack. Here is the concepts: We will do that using a target computer connected to the same switch as your fog server. We will register the target computer then schedule a debug capture or deploy it doesn’t matter. Once we are at the command prompt we will manually mount the /images/dev share on the fog server and then use the iperf3 utility to measure bandwidth between the target computer and FOG. That will give us an idea of bandwidth availability. Then you will repeat the same process from the far end of your network to see if there is a difference.
The last bit is testing nfs performance. But lets see what the network tests tell us. I need to create clear instructions for the above to ensure we get the numbers we might expect.
-
@george1421
My core switches are 1gbps. And my classroom switch are 100mbps. That’s the bottle neck. Can’t do nothing about it until I get my upgrades later this year. -
@tesparza OK knowing that I might expect to see a ~700MB/min transfer rates using FOG.
So what can you do?
- Upgrade your network <smirk>
- Change your image compression from standard gzip to zstd and set the compression level to 11-15. zstd is a newer, tigher and faster decompression tool than gzip. The tighter you can pack the image the easier it will be on bandwidth. But its also a sliding scale, the tigher you pack the data the more CPU usage you will have on the client during image decompression.
We don’t have any base line numbers using 100Mb/s networking so you will have to find the right fit for your setup. In your case a faster server won’t help. If you have multiple computers to image at the same time, I would try to multicast the image to several at a time. Because you don’t have speed, you have to manage the quantity.
-
@george1421 Yeah i just tried a deployment on the switch where the fog server is connected and im getting 8GB/min
I just gonna have to wait for the upgrade that we are getting, for now I’m gonna just register the host and use the snapins. Multicast and cloning will have to wait until later this year. Thank you so much guys -
@tesparza On the plus side you know that your server is fine and up to the job. No expense there needed.
-
I use a VM as my central fog server.
The storage ‘nodes’ are NAS’ (Usually FreeNAS) so if you have Synology, FreeNAS or Qnap you could just set those up as where the data actually moves from.
My FOG Server is over the other side of an IPSEC VPN so I cannot pull images directly from it.
-
@george1421 zstd decompression speed is supposed to stay more or less the same whatever the compression level. Only compression becomes slower.
-
@compman while that’s true, in practice zstd at a higher compression level will return a faster overall process when the bottleneck is the network.
