Uploading an image from Hyper-V is extremely slow

[A Former User]

We purchased a small Xeon-based server with 16 GB of RAM for the purpose of making images in Hyper-V.

It’s worked out well so far, but when we went to upload a 40-gigabyte image, it took about 100 minutes. Uploading from the real hardware takes about 4 minutes.

I dedicated 7 processors to the virtual machine, along with most of the RAM.

Is there any way to fix this?

Oh, and we are using a trunk that is about 1 month old.

[george1421]

While this will not help your performance issue, please update to the latest trunk build. There has been major improvements in the platform in the last month.

With that said, if your current setup was running on vmware, adding extra vCPUs would actually kill performance on a busy vm host than help. It has to do with have time slices available where all of the 7 cpus are in a ready state. More complicated to explain than I’m going to post here.

I would say for your setup the vm client should only need 2 vCPUs and about 2GB of RAM memory for the FOS (Fog Operating System) engine (the software that uploads and downloads disk images). What your actual virtual machine needs when you create the client install is up to you.

Now for performance issues. I can imagine a few areas where you could have performance issues.

1. You could have a slow vm host disk subsystem.
2. You could have a network bottleneck between the vm host server and the FOG server
3. You could have a slow fog server disk or network link.

First of all fog server performance. The performance of the FOG server is really minimal in the typical image capture and deployment process. In this setup the FOG server only really moves data from the network interface to the FOG disk and from the FOG disk back to the target computer.
The real work is being done by the target computer for both capture and deploying images. In one installation I have the FOG server running on a dual core celeron Intel NUC. For small deployments it runs great, but thats another story.

To help debug your issue we may need to do a few tests.

Configure your vm target computer with 2 vCPU and 4GB of ram. This is only to help build a consistent backup environment.

1. If you feel up to it recapture your reference image using the above settings to establish a baseline. What I’m interested in is the GB per minute number. Let the capture run for 5 minutes to make sure you have a solid transfer rate. Note the transfer rate. You don’t need to let it run the full 100 minutes unless you note that the transfer gets slower and slower the longer it runs after the 5 minute mark.

2. For your image you want to capture, drop the compression on the image to 0. Then recapture the image from the target computer. Note the speed difference.

3. Now lets try to capture a physical computer (preferred core 2 duo 4GB of ram) at the initial compression ratio. Again we want to note the transfer rate after ~5 minutes of running.

4. Capture that physical computer again at 0 compression ratio. Again we want to note the transfer rate after ~5 minutes of running.

Report back the numbers and maybe we can focus on a specific area.

[Sebastian Roth]

@loosus456 Which version of FOG (see in the blue cloud on the web interface) and kernel version (FOG configuration -> Kernel update) do you use? Maybe related to this: https://forums.fogproject.org/topic/6695/performance-decrease-using-hyper-v-win10-clients

[A Former User]

I’ll check all this when I get back to work tomorrow.

I will say just a few things now:

1. The performance of the VM is gorgeous until I use FOG. I usually get really good performance in the VM when it’s loaded with Windows 7 or Windows 10.

2. One weird thing that I noticed is that the reason the upload seems slow is that it sits on a group of blocks for long periods of time. It’ll upload a bunch of blocks, and then it’ll just sit for like 5 seconds and upload absolutely nothing. It does this throughout the upload. So it seems to have bursts of activity and then nothing, over and over.

3. From what I can remember the upload speed was about 600 MB/min most of the time. The compression on the image was set to 6.

4. If I use the exact same FOG settings on the exact same same hardware (but on the physical hardware instead of a VM in Hyper-V), I can usually get about 8 GB/min upload speed.

I’ll try to get more specifics tomorrow if I get a chance.

[ch3i]

@loosus456 Hi, the 600MB/min looks like a FastEthernet speed, not a Giga.

[george1421]

@ch3i said:

@loosus456 Hi, the 600MB/min looks like a FastEthernet speed, not a Giga.

I agree with ch3i’s conclusion. 600MB/min translates to 10MB/s which is just below 100Mhz (fast) ethernet’s theoretical max of 12.5MB/s.

If this was a GbE network, I might expect slightly less than 125MB/s or (7GB/min) transfer rates. As you noted with physical hardware I would expect between 4-6GB/m download rate with less depending on the hardware and compression ratio on the upload side.

I would be sure to update FOG first then check your network path for a 100Mhz links.

[Andy Abplanalp]

@loosus456 does this server have Broadcom NICs in it by chance? If so, you can look into disabling Virtual Machine Queues.

[A Former User]

Haven’t had a chance to look at everything just yet, but the version of FOG is 6321.

In regards to it being FastEthernet: it sometimes bursts as high as gig does, which is what makes me believe there is an issue.

[Sebastian Roth]

@loosus456 Any news on this? Did you have time to look into all the suggestions and try things out?

[rdw]

@loosus456 Any resolution? Exact same symptoms here. Thanks.

[Wayne Workman]

@rdw I remember that there is a specific linux kernel argument for Hyper-V. I would guess that this would really help out - definitely wouldn’t hurt anything. I’ll see if I can find it…

[Wayne Workman]

https://technet.microsoft.com/en-us/library/dn720239.aspx

Since FOS (fog operating system) is linux, all that stuff could apply potentiall.

There is a kernel argument though. I have to get on a actual computer to find it…

[sudburr]

If it is Hyper-V 2012 R2 and you are running a Generation 1 VM which uses the Legacy Network Adapter for PXE booting then 100 Mb is all you will get as the Legacy Network Adapter is only a 100 Mb NIC.

Having said that, some kernels in the past have on occasion switched to the 1Gb NIC for uploading … including right now. I just rolled back to 4.1.4 from 4.6.2 and I’m getting Gb.

[Wayne Workman]

I know it’s been a while, but I just came across this: http://www.funtoo.org/HyperV_Kernel_Configuration

These are the Hyper-V kernel arguments listed on that page, just so they aren’t lost in the depths of the internet:

HyperV Kernel Configuration

For a Hyper-V generation 2 system, you’ll need certain options enabled in the Kernel in this order because latter options aren’t available until earlier options are enabled:

    CONFIG_HYPERVISOR_GUEST: Processor type and featueres > Linux Guest Support
    CONFIG_PARAVIRT: Processor type and features > Linux Guest Support > Enable paravirtualization code
    CONFIG_PARAVIRT_SPINLOCKS: Processor type and features > Linux Guest Support > Paravirtualization layer for spinlocks
    CONFIG_HYPERV: Device Drivers > Microsoft Hyper-V guest support > Microsoft Hyper-V client drivers
    CONFIG_HYPERV_UTILS: Device Drivers > Microsoft Hyper-V guest support > Microsoft Hyper-V Utilities driver
    CONFIG_HYPERV_BALLOON: Device Drivers > Microsoft Hyper-V guest support > Microsoft Hyper-V Balloon driver
    CONFIG_HYPERV_NET: Device Drivers > Network device support > Microsoft Hyper-V virtual network driver
    CONFIG_HYPERV_STORAGE: Device Drivers > SCSI device support > SCSI low-level drivers > Microsoft Hyper-V virtual storage driver
    CONFIG_HYPERV_KEYBOARD: Device Drivers > Input device support > Hardware I/O ports > Microsoft Synthetic Keyboard driver
    CONFIG_FB_HYPERV: Device Drivers > Graphics support > Frame buffer Devices > Microsoft Hyper-V Synthetic Video support
    CONFIG_HID_HYPERV_MOUSE: Device Drivers > HID support > Special HID drivers > Microsoft Hyper-V mouse driver

[Sebastian Roth]

@Wayne-Workman This is what we have in the current config:

grep -e HYPERV -e PARAVIRT TomElliott.config.asc 
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_DEBUG is not set
# CONFIG_PARAVIRT_SPINLOCKS is not set
# CONFIG_PARAVIRT_TIME_ACCOUNTING is not set
# CONFIG_PCI_HYPERV is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_HYPERV_STORAGE=y
CONFIG_HYPERV_NET=y
CONFIG_HYPERV_KEYBOARD=y
CONFIG_FB_HYPERV=y
# CONFIG_HID_HYPERV_MOUSE is not set
CONFIG_HYPERV=y
CONFIG_HYPERV_BALLOON=y

Are you keen to compare those and let us know if we should any option.

[Scott Adams]

Any update on this? I have just started universal imaging using Hyper-V, which is working great. However, the upload speed from VM to FOG-server is slow. Currently uploading now, and am getting >300MB/min