WOL working too good

[tmerrick]

I have FOG 1.4.4 on RedHat and most everything seems to be working okay. Except that I have a number of hosts that get turned back on by FOG about 10-20 min after I shut them down (either manually, remote desktop, or via FOG power management). I have not been able to find anything in FOG’s logs to indicate that this is happening. I assume that I have something in my database that is causing this since is only happening on 35 of 280 computers. Any ideas of where to look?

[Wayne Workman]

@tmerrick Look at the power management schedules, and scheduled tasks also. Also, if there is any task waiting for a host, fog will send WOLs to that host every minute until it starts the task.

[tmerrick]

There are only two scheduled tasks for deploying snapins to other computers. Power management only shows a scheduled 11pm shutdown, but then turn back on 15 min later.

[Wayne Workman]

@tmerrick Run these db maintenance commands and see if they help:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_MySQL#Database_Maintenance_Commands

[tmerrick]

Found about 140 rows, but they still turn back on. I even rebooted to make sure that nothing was in memory.

[Sebastian Roth]

@tmerrick Do you actually see Wake On LAN packets in your network?? Maybe it’s something completely different. What if you shut down the FOG server? PCs still turn back on?

[george1421]

I agree with Sebastian, shut the FOG server down and see if this behavior still exists. I know these win10 machines will turn them selves on at random times to do system “stuff” then power off. It is a bit unnerving when that happens.

Thinking about it a bit more you can probably capture if its the fog server doing this by installing tcpdump onto your fog server then launching it with the following command.
sudo tcpdump ether proto 0x0842 or udp port 9
You should see no output from the above command, unless there are wol packets floating about.

or to dump it into a pcap file for later review. You may have to run the command for 24 hrs to see when and what is generating wol commands.
sudo tcpdump -w wol.pcap ether proto 0x0842 or udp port 9

Since WOL is ethernet based and not IP based you should see the mac address of the device sending the magic packets.

[tmerrick]

I did not receive any packets from the server subnet using your tcpdump command. Unfortunately the computers in question are on a different subnet. I know that it is a WOL packet since the following shows up in the server messages log:

Dec 16 23:04:02 clstfogi in.tftpd[16975]: tftp: client does not accept options
Dec 16 23:04:02 clstfogi in.tftpd[16976]: Client 10.4.134.75 finished undionly.kpxe
Dec 16 23:04:07 clstfogi in.tftpd[17006]: tftp: client does not accept options
Dec 16 23:04:07 clstfogi in.tftpd[17007]: Client 10.4.134.78 finished undionly.kpxe
Dec 16 23:04:09 clstfogi in.tftpd[17053]: Client 10.4.134.75 finished default.ipxe
Dec 16 23:04:14 clstfogi in.tftpd[17111]: Client 10.4.134.78 finished default.ipxe

The computers all boot to the SSD, but have WOL+PXE boot enabled. So I should not see anything unless it is a WOL boot. So it does not appear to be a FOG issue, but rather a rogue server on that subnet.

[george1421]

@tmerrick Then take that same the same filter (without the -w wol.pcap flag) and use wireshark on a laptop. Connect it to that subnet. WOL packets are broadcast packets so if its being generated on that subnet you should trap it.

[Sebastian Roth]

@tmerrick The logs you posted don’t show anything WOL related. Any PC doing a PXE boot will request files via TFTP, so that’s not an argument for WOL causing the PCs to startup.

If it’d be the FOG server sending WOL packets you would see those in tcpdump on the FOG server!! Maybe connect a Windows PC in the client subnet and look at the packets using Wireshark from there. If the network switch forgot about the client MAC address it needs to send out the WOL as broadcast I think.

[Tom Elliott]

This really doesn’t sound right.

Can you post the FOGScheduler Logs? This is where “WOL” happens, but it seems suspicious to me, too say the least, that it turns on at “15 minute” increments. This would lead me to think the WOL is coming (if it is indeed WOL from FOG) from this particular service. Is it always 15 minutes after shutdown? As @Sebastian-Roth and @george1421 have asked, are you sure this is the FOG Server (disable the fogscheduler service systemctl stop FOGScheduler)?

[tmerrick]

The FOGScheduler logs just show my 7 scheduled cron wake up jobs. I have disabled all of the FOG services and they still started up. Thus it does not appear to be FOG. I am trying to get a hold of a network analyzer to trace who is really sending this signal. Thanks for your help in understanding how FOG works, it has helped me narrow down who is causing the problem.

I know it is a WOL being sent as my boot is SSD, then IP/PXE and it is PXE booting. Also, it is only happening in one of my three buildings. And only about 1/3 to 1/2 of the computers that I have in that building. So it looks like I have a rogue server somewhere in that building.