Performance decrease using Hyper-V Win10 clients
-
I pulled version 4.1.4 of bzImage and bzImage32 from another server.
Without replacing the init’s, “Resizing Filesystem” now completes in about a minute.
-
@jkozee said:
Something has changed in the kernel build in regards to a VM running on Hyper-V with SSD backed storage
From my point of view the kernel in the VM has absolutely no knowledge of the underlaying filesystem/disk outside the VM. I thought it could be a fragmentation problem on the backend storage device but then you wouldn’t see a difference in speed just by booting different kernel versions. From what you said I think your test setup is pretty good (just changing the kernel parameter in the host setting and leaving everything else untouched).
There is a great way to pin this kind of issue down to exactly one version/commit. It’s called git bisect. Please read through this article and see if you want to dive into this. I am more than happy to help you along the way! Have you ever compiled a (FOG) kernel? It’s actually not to complicated. Just give it a try following this article: https://wiki.fogproject.org/wiki/index.php/Build_TomElliott_Kernel (the second half is talking about current FOG version). Instead of
make menuconfig(after downloading Tom’s kernel config) you can just runmake oldconfiginstead where you don’t need to bother about the menu stuff.You can build the kernels on your FOG server if you like. Just needs some disk space for the kernel git repo and some tools. As I don’t know which linux you are on I will leave this open. Ask google which packages you need to install on CentOS/Debian/… to compile the kernel. There are lots of tutorials out there.
-
@Sebastian-Roth I’m pretty short on time right now (I’m sure everyone here can say the same thing), but compiling and testing kernels shouldn’t be a problem. I’ll try to make time this weekend, but it may have to wait untill next weekend. I’ll post here if/when I make any progress.
My FOG server is slow storage backed, so I’ll need to build a new VM to make kernel compiling tolerable. My plan would be to script building the incremented versions between 4.3.2 and 4.4.0, to narrow it down. Once we have that, we can bisect between them to find what changed.
@Tom-Elliott Are the .config’s available for download for the 4.3.2 and 4.4.0 builds that you released? Do you build with the defaults, or do you tweak the .config for FOG?
@sudburr Ouch, 45-60 minutes is way more painful. Looks like the FTP issue is now resolved. How does the performance compare with 4.1.4, 4.3.2, and 4.4.1 ?
-
@jkozee Configs can be downloaded as I improved/edit the kernel configs I update them on SVN/GIT, though I can’t possibly tell you which specific revisions these 4.3.0 to 4.4.x changes were made.
I’m about to try building the 4.4.2 kernel (didn’t know it released) and I will pull in a 4.3.0 kernel and rip the config out of it.
-
@Tom-Elliott Thanks Tom. I’ll look through the repo after I get a VM up to compile on. If you get 4.4.2 built and available, I’ll test it first, as there will be really no point in testing the other builds if it is fixed now…
-
@jkozee I am building the 4.4.2 using the 4.3.0 config (adding the nics that were not part, and patches to other areas as needed for other support thing. I’ll let you know when they’re complete. Rather than immediately publish them, I’d just like you to run a few tests to see if they are working better.
-
@Tom-Elliott NP, just let me know.
-
-
@Tom-Elliott Got them. Give me a few minutes to fight another fire, then I’ll test and report.
-
@Tom-Elliott
I didn’t run the full deploy (probably not needed at this point), but the “Formatting initialized partition” step during deploy takes 3:34, which is on par with 4.4.1. So, the new build doesn’t appear to solve this issue.Wonder if something changed in the device block size or cache? Or maybe it does full zero out of the partition now, when it used to do a quick format?
-
@jkozee that would mean it was a binary of the ntfs-progs of which changing the kernel out would not have any impact.
-
@Tom-Elliott Ah, yes for ntfs. So, perhaps the block driver.
-
@Tom-Elliott I kicked off a script to build the kernels. Assuming they build and boot, I’ll report my findings.
-
@Tom-Elliott Hmm. 3.3.2 built but wouldn’t boot. I got a kernel panic, not sycning VFS. I used the config from https://svn.code.sf.net/p/freeghost/code/trunk/kernel/TomElliott.config.64. Is there another one I should use for 3.3.2?
-
@jkozee 3.3 is very old. I thought 4.3 worked and 4.4 doesn’t so I would suspect somewhere between those would be enough to start to figure out.
-
@Tom-Elliott Um, yeah. That’s what I get for trying to multitask and trying to script the builds. Let me see what I’m actually doing. Sorry for wasting space here…
-
@Tom-Elliott Ok, I think I scripted builds 4.3.2 to 4.3.5 and 4.4.1, but I’ll start over just to be sure. I see the config for 4.3 on the repo at r4316. Let me start there and see what I get…
-
@jkozee look on wiki for build tomelliott kernel
Follow instructions and please test with the additional patches. Speed up build time by adding -j $(nproc) to the make commands
-
@jkozee said:
I see the config for 4.3 on the repo at r4316. Let me start there and see what I get…
Don’t bother too much about getting the exact config Tom used for a particular version. I’d suggest using the newest config for all the builds. As far as I know - hope this is correct -
make oldconfigwill ask you on the console if there are settings missing. Older ones will just be tossed.As well, using the same config as a base is wise to properly compare the different kernels versions. Otherwise you end up wondering if a change in config made the difference!
-
@Sebastian-Roth
Looks like my script wasn’t copying the .config file, so I was building with the defaults.I updated it and it built 4.3.2 and it boots fine now. I used the latest config and my script does “yes ‘’ | make oldconfig”. I’ll let it build the ones I mentioned earlier and test them. I’m about out of time for now, so I’ll post the results later.
@Tom-Elliott
I did not have time to write a sed script to include the additional patches from the wiki, but I can do that later or apply them by hand, once I have a chance to test the scripted builds.Sound reasonable?
