Performance decrease using Hyper-V Win10 clients
-
@Tom-Elliott Got them. Give me a few minutes to fight another fire, then I’ll test and report.
-
@Tom-Elliott
I didn’t run the full deploy (probably not needed at this point), but the “Formatting initialized partition” step during deploy takes 3:34, which is on par with 4.4.1. So, the new build doesn’t appear to solve this issue.Wonder if something changed in the device block size or cache? Or maybe it does full zero out of the partition now, when it used to do a quick format?
-
@jkozee that would mean it was a binary of the ntfs-progs of which changing the kernel out would not have any impact.
-
@Tom-Elliott Ah, yes for ntfs. So, perhaps the block driver.
-
@Tom-Elliott I kicked off a script to build the kernels. Assuming they build and boot, I’ll report my findings.
-
@Tom-Elliott Hmm. 3.3.2 built but wouldn’t boot. I got a kernel panic, not sycning VFS. I used the config from https://svn.code.sf.net/p/freeghost/code/trunk/kernel/TomElliott.config.64. Is there another one I should use for 3.3.2?
-
@jkozee 3.3 is very old. I thought 4.3 worked and 4.4 doesn’t so I would suspect somewhere between those would be enough to start to figure out.
-
@Tom-Elliott Um, yeah. That’s what I get for trying to multitask and trying to script the builds. Let me see what I’m actually doing. Sorry for wasting space here…
-
@Tom-Elliott Ok, I think I scripted builds 4.3.2 to 4.3.5 and 4.4.1, but I’ll start over just to be sure. I see the config for 4.3 on the repo at r4316. Let me start there and see what I get…
-
@jkozee look on wiki for build tomelliott kernel
Follow instructions and please test with the additional patches. Speed up build time by adding -j $(nproc) to the make commands
-
@jkozee said:
I see the config for 4.3 on the repo at r4316. Let me start there and see what I get…
Don’t bother too much about getting the exact config Tom used for a particular version. I’d suggest using the newest config for all the builds. As far as I know - hope this is correct -
make oldconfigwill ask you on the console if there are settings missing. Older ones will just be tossed.As well, using the same config as a base is wise to properly compare the different kernels versions. Otherwise you end up wondering if a change in config made the difference!
-
@Sebastian-Roth
Looks like my script wasn’t copying the .config file, so I was building with the defaults.I updated it and it built 4.3.2 and it boots fine now. I used the latest config and my script does “yes ‘’ | make oldconfig”. I’ll let it build the ones I mentioned earlier and test them. I’m about out of time for now, so I’ll post the results later.
@Tom-Elliott
I did not have time to write a sed script to include the additional patches from the wiki, but I can do that later or apply them by hand, once I have a chance to test the scripted builds.Sound reasonable?
-
Build script finished quicker than I expected. Looks like it was introduced between 4.3.5 and 4.4.1. I’ll look at git bisect when I can make the time.
-
@jkozee Great work! I am sure you will see what’s exactly causing it and when it was introduced! bisect is your friend.
-
@Sebastian-Roth and @Tom-Elliott
The change to the kernel is actually in the scsi driver.
The commit that introduced the delay is 81988a0e6b031bc80da15257201810ddcf989e64, which applies changes to drivers/scsi/storvsc_drv.c.
I can confirm that reverting the diff on 4.4.2 brings the performance on the hyper-v client on par with 4.3.2. I can’t speak to the commit itself, as I just blindly reverted it and didn’t spend any time on digesting the patch itself.
My timings on the patched 4.4.2 was 2:14 for the deploy and 18:20 for the capture. That means the deploy is 50% faster and the capture is 27% slower than my tests for 4.3.2. @Tom-Elliott I did not include the additional patches you mentioned either, so I would need to retest both kernels under the same server conditions (and with the additional patches applied to 4.4.2) for more accurate results.
Hope this proves useful.
-
Still seems more like the issue should be with the block device, rather than the scsi driver. Seems like it would be related to caching or block size/block alignment of the ssd.
-
To me, these lines from the commit look most interesting:
/* Ensure there are no gaps in presented sgls */
blk_queue_virt_boundary(sdevice->request_queue, PAGE_SIZE - 1); -
So, adding that line to 4.3.2 results in performance degradation and removing it from 4.4.2 results in performance increase. Guess all that’s left is to figure out (understand) what it actually does…
-
Maybe this makes more sense: blk_queue_virt_boundary(sdevice->request_queue, sdevice->page_size - 1);
-
@jkozee You are great man! This is what I love about open source and the people knowing how to go with it…
I didn’t even know that there is a storage driver for Hyper-V (and VMware by the way) right in the linux kernel. And we have it enabled: CONFIG_HYPERV_STORAGE
Looking at the scsi_device (sdevice) struct I don’t see page_size. So I don’t think your change is gonna work. Probably wouldn’t compile at all. Looking through the driver code I see PAGE_SIZE used several times. So I feel like this seams ok - although I don’t know much about this particular driver!
You just might want to get in contact with the authors of the storvsc_drv driver (line 18ff) and as well the author of the patch! Tell them that you bisected a major slowdown issue to that particular patch. I guess that their input is a lot more helpful than trying to understand the whole driver by yourself! Please keep us posted.
Edit: Seams like the function blk_queue_virt_boundary was added only a few months back - intended to be used with NVMe devices from what it looks to me (I don’t think this has anything to do with you seeing slower speeds on the SSD backend though).
Edit2: Further interesting things to read on this are here and here - infiniband driver using this as well, interesting part is:
(Very) nice cleanup – so what’s the actual deal here? is that as long
as we plant a slave alloc callback into out scsi host template which
further invokes a
blk_queue_virt_boundary(sdev->request_queue, ~MASK_4K) call, we’re
100% safe/sure what all SGs we get meet the alignment criteria?Correct, the nvme driver has the same alignment constraints for its PRPs
and uses the queue virt_boundary to have the block layer enforce the
SG alignment.
