WOL Firmware Linux Kernel Breaks - Power pulled or boot to windows fixes
-
I have a problem that I do not explain.
I have Dell Optiplex 3060 computers.
I have a Windows 10 master. imageSince I updated FOG from 1.5.2 to 1.5.7, on each PC I remastered, Windows turn off the network card when it shut down, preventing further WOL.
This is Windows because if I remove the power and I put it back, the network card is turned on (so setting bios out of cause).
You will tell me that it is my image, what I thought as well except that here are the facts: (checked several times to be certain):I restored from the backup my FOG while it was still in 1.5.2
I captured back my master image updated last week.I therefore have:
FOG 1.5.2 with:
-the master image of January (currently deployed everywhere)
-the master image updated last weekFrom a FOG 1.5.7 with:
-the master image of January (currently deployed everywhere)
-the master image updated last weekIf I deploy any of my 2 masters images from the fog 1.5.2 the network card remains on after shutting down Windows.
If I deploy any of my 2 masters images from the fog 1.5.7 the network card turns off after shutting down WindowsThe problem does not come from the image, nor the bios or AD GPO.
I do not understand how the way used to deploy the same image can affect how Windows turns off the machine …If you have an idea?
Thank you
-
@Gael Hey one of the senior FOG developers just chatted with me on this issue. So we have a few more things to try here.
First of all on the 1.5.2 version of the fog server do the following.
cd /var/www/html/fog/service/ipxe` file bzImageThis will tell us the linux kernel version that was shipped with FOG 1.5.2
Now scp that file to the new fog server into the correct path. Power off (unplug) the target computer then pxe boot and image with the image you copied from the 1.5.2 version of the fog server.
See if the system responds differently?
The logic here is depending on the nic used, the linux kernel may load (patch) firmware onto the nic controller that would be persistent between system reboots, but would be erased after a power outage. With the different versions of the linux kernel its possible different firmware could be loaded causing the discrepancy.
Once all of the other things have been tested out (so we are not changing a bunch of things all at once), make sure the firmware (bios) is updated on the 3060 just in case that is causing the root of the issue.
-
@Gael Well I’m going to tell you its windows. What you are describing is exactly what windows does. When you shutdown windows, its not really a power off (the same as when you removed the power cord). Its an enhanced sleep mode which windows disables wol. This is not the case when you remove the power. You should be able to test this by removing the fast start option in windows then it should power off the hardware correctly so you can boot via WOL.
I understand what you are saying about the FOG version upgrade seemed to have triggered this, but once FOS Linux exits its no longer in memory and doesn’t have an impact on the hardware.
So to start debugging this disable fast start in windows and see if that changes the way wol works for you. A quick google–fu for “windows 10 wol not working” will show you “its a thing” with win10.
-
@george1421 Hi, thank for answer but fast startup is already off in all my master image.
Also understand this exemple:
-> I take one of my Dell Optiplex 3060 computer, he is running ok, (faststartup is disabled), I shutdown windows and i see network card is keep on.
-> I power on my Fog 1.5.2 server and ask it to deploy my master image (same image that is already running on the computer, also with faststartup disabled)
Wol work computer was reimaged and I shutdown windows and see network card is keep on.
-> Now I shutdown my Fog 1.5.2 server and I power on my Fog 1.5.7 server and ask it to deploy my master image (same image that is already running on the computer, also with faststartup disabled)
Wol don’t work, i power on computer myself, computer was reimaged BUT when I shutdown windows I see network card is shutdown…
->Now I shutdown my Fog 1.5.7 server and I power on my Fog 1.5.2 server and ask it to deploy my master image (same image that is already running on the computer, also with faststartup disabled)
Wol work computer was reimaged and I shutdown windows and see network card is keep on.Like you, i don’t understand…
-
@Gael said in Incomprehensible network adapter issue that goes out since the update to Fog 1.5.7:
(same image that is already running on the computer, also with faststartup disabled)
Can you please do me a favor?
- Start up your 1.5.2 server and save a copy of that image to an external drive
- Shut down the 1.5.2 server
- Startup the 1.5.7 server rename the image and move that from the external drive in place
- Deploy using that image
- Test WOL
-
@Sebastian-Roth I think I can say that I already did it because, at first, I thought that my updated master image was the problem. I recovered a copy of my Fog server before it was updated (and before my image was updated) from backup, then i deployed the image to a VM, then I captured it (without powering-on the image) with my Fog 1.5.7 server.
-
@Gael Is it possible to power on both fog servers at the same time? Maybe move the older fog server’s IP address? With both on at the same time, copy the image directly from the old fog server to the new fog server over nfs?
Like you I’m confused why the fog server would have any impact on windows unless the wol code on the new fog server is faulty for some reason. I still want to blame windows for some reason. When fog clones an image it doesn’t know anything (really) about the target OS. It only moves disk blocks from the fog server to the target computer. It doesn’t touch the target computer’s registry or drivers. After a power outage (really a reboot) there should be no trace that FOG was ever near the target computer. Also with both fog servers on you would have the chance to issue a wol from each server to compare the differences. -
@george1421 It’s not a WOL packet problem because the network card is shutdown (leds powered off)
Can the fog client have any impact? -
@Gael said in Incomprehensible network adapter issue that goes out since the update to Fog 1.5.7:
because the network card is shutdown
Two things come to mind.
- If the 3050 has vpro it should not power off the network card. vPro is for out of band management.
- Windows has the driver check box ticked to power off the network adapter to energy (or batter life, I can’t remember the exact term at the moment).
I still have to think there is a difference in the image with the one working vs the one not. Are you sure no windows updates have been applied on the one not working?
There is a (very) slight chance the fog client could have an impact if the version(s) are different. But I’m only saying its a very slight chance. I don’t think the fog client code has been updated since 1.5.2 but the difference is almost 1.5 years between releases.
-
@Gael said in Incomprehensible network adapter issue that goes out since the update to Fog 1.5.7:
then i deployed the image to a VM, then I captured it (without powering-on the image) with my Fog 1.5.7 server
Not exactly what I asked you to do!
-
I will scp image from old fog to new and report here the result.
-
This post is deleted! -
@Sebastian-Roth Done. Network card was shutdown too
-
@Gael This is very troubling since the issue (via process of elimination) appearing to be a FOG Server created problem. Which I’m still not convinced is possible. The only possibility I can see is if the updated FOS Linux kernel in 1.5.7 its leaving the hardware in a strange state, but a power removal should reset anything that FOS Linux would leave behind. I guess you could test by grabbing bzImage and init.xz from the 1.5.2 fog server and temporarily installing them on the 1.5.7 fog server. Those are the only thing from the fog site that actually touches the target hardware.
I have a 3050 around here somewhere, I can try deploying a copy of 1903 or 1809 to that system to see if we have similar results. But if vPro is on and fast windows startup is disabled it should leave the network adapter powered on. I’m pretty sure that the 3050s we have here do have vPro installed.
-
@george1421 Sorry typo, Optiplex 3060 not 3050
-
@Gael Hey one of the senior FOG developers just chatted with me on this issue. So we have a few more things to try here.
First of all on the 1.5.2 version of the fog server do the following.
cd /var/www/html/fog/service/ipxe` file bzImageThis will tell us the linux kernel version that was shipped with FOG 1.5.2
Now scp that file to the new fog server into the correct path. Power off (unplug) the target computer then pxe boot and image with the image you copied from the 1.5.2 version of the fog server.
See if the system responds differently?
The logic here is depending on the nic used, the linux kernel may load (patch) firmware onto the nic controller that would be persistent between system reboots, but would be erased after a power outage. With the different versions of the linux kernel its possible different firmware could be loaded causing the discrepancy.
Once all of the other things have been tested out (so we are not changing a bunch of things all at once), make sure the firmware (bios) is updated on the 3060 just in case that is causing the root of the issue.
-
Ok, i will do that.
But for information, another test i have juste done:
I only asked fog 1.5.7 to capture a working dell optiplex 3060 (without sysprep or anythink else), after capture was done i shutdown computer and: the network card was shutdown!
-
1.5.2 :
file bzImage
bzImage: Linux kernel x86 boot executable bzImage, version 4.19.1 (jenkins-agent@Tollana) #1 SMP Mon Nov 12 18:23:08 CST 2018, RO-rootFS, swap_dev 0x8, Normal VGA
file bzImage32
bzImage32: Linux kernel x86 boot executable bzImage, version 4.15.2 (builder@c38bc0acaeb4) #5 SMP Tue Feb 13 18:32:54 UTC 2018, RO-rootFS, swap_dev 0x7, Normal VGA1.5.7 :
file bzImage
bzImage: Linux kernel x86 boot executable bzImage, version 5.1.16 (sebastian@Tollana) #2 SMP Wed Aug 28 17:12:41 CDT 2019, RO-rootFS, swap_dev 0x8, Normal VGA
file bzImage32
bzImage32: Linux kernel x86 boot executable bzImage, version 4.19.64 (jenkins-agent@Tollana) #1 SMP Mon Aug 5 09:59:10 CDT 2019, RO-rootFS, swap_dev 0x7, Normal VGA -
@george1421 That’s work!
-
@Gael Ok, now we are sure it’s a FOS kernel issue causing this, probably some seldom issue with firmware on your NIC model. Can you tel us which NIC exactly you have (windows device manager hardware IDs)?
-
@Gael See we also learned something that we didn’t know before, not only did you install 1.5.7, but you updated the FOS Linux kernel to version 5.x.x, where 1.5.7 should be shipped with linux-4.19.64 or there about. The 5.x.x kernels really haven’t been tested well enough and imo should not be used until we’ve had more field experience with them.
So what happens if you roll the fos linux kernel back to 4.19.64? Does the behavior correct itself? Rolling back to the fos linux 4.19.1 seems to have done the trick.
From windows can you get us the vendor ID and hardware ID from the device manager for this NIC as Sebastian suggested?
