Daily wake-up routine not working after update
-
Hi Team,
I updated to the latest dev branch earlier this week and now our scheduled WoL isn’t working. I can still trigger it manually so I know the WoL packets are not the issue.
I tried deleting the task and creating a new one but no change.Just looked at the /var/log/fog/fogscheduler.log and the latest entry is from the date I updated the system:
[03-22-21 10:30:55 am] * Task run time: Tue, 23 Mar 2021 06:00:00 -0500
Looks like a service isn’t running, but checking the scheduler it appears fine:
[Cent-7:root@fog ~]# systemctl status FOGScheduler.service ● FOGScheduler.service - FOGScheduler Loaded: loaded (/usr/lib/systemd/system/FOGScheduler.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2021-03-19 14:31:56 CDT; 5 days ago Main PID: 2955 (php) Memory: 16.6M CGroup: /system.slice/FOGScheduler.service ├─2955 php /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler └─3124 php /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
Using my favorite command I restarted the FOG services:
for i in FOGSnapinHash.service FOGScheduler.service FOGImageSize.service FOGSnapinReplicator.service FOGMulticastManager.service FOGImageReplicator.service FOGPingHosts.service;do systemctl restart $i; done
Now I have this at the beginning of the log:
[03-25-21 9:06:22 am] * Starting TaskScheduler Service [03-25-21 9:06:22 am] * Checking for new items every 60 seconds [03-25-21 9:06:22 am] * Starting service loop [03-25-21 9:06:22 am] * 61 tasks found.
Do I need to worry about the 61 tasks found? Anything else I need to check to keep the service active?
Thank you!
-
@Tom-Elliott any other places to check? I don’t mind clearing out all scheduled tasks since I don’t have much in there. Just not sure how to do a safe reset.
-
MariaDB [fog]> UPDATE tasks SET taskStateID=5 WHERE taskStateID IN ('0','',NULL,'1','2','3') AND taskTypeID=14; Query OK, 0 rows affected, 1 warning (0.000 sec) Rows matched: 0 Changed: 0 Warnings: 1
And the logs after restarting the service:
/var/log/fog/fogscheduler.log [03-31-21 7:04:29 am] ================================== === ==== ===== ==== === ========= == === == === === ======== ==== == ==== === === ======== ==== == ========= === ==== ==== == ========= === ======== ==== == === === === ======== ==== == ==== === === ========= == === == === === ========== ===== ==== ================================== ===== Free Opensource Ghost ====== ================================== ============ Credits ============= = https://fogproject.org/Credits = ================================== == Released under GPL Version 3 == ================================== [03-31-21 7:04:29 am] Interface Ready with IP Address: [localIP] [03-31-21 7:04:29 am] Interface Ready with IP Address: 127.0.0.1 [03-31-21 7:04:29 am] Interface Ready with IP Address: 127.0.1.1 [03-31-21 7:04:29 am] Interface Ready with IP Address: fog.domain.com [03-31-21 7:04:29 am] * Starting TaskScheduler Service [03-31-21 7:04:29 am] * Checking for new items every 60 seconds [03-31-21 7:04:29 am] * Starting service loop [03-31-21 7:04:29 am] * 61 tasks found. [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 05:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500 [03-31-21 7:04:29 am] * Task run time: Thu, 01 Apr 2021 06:00:00 -0500
-
@altitudehack If you can try:
UPDATE tasks SET taskStateID=5 WHERE taskStateID IN ('0','',NULL,'1','2','3') AND taskTypeID=14;
Then restart FOGScheduler
If you can let us know what your log shows?
Thank you,
-
@altitudehack All of the tasks are purely WOL tasks, correct?
Just want clarification as I want to try to clear out the DB of erroneous entries that I suspect are playing a bit of a role in the issue.
-
@Tom-Elliott I rebooted the server this morning about six minutes prior to the scheduled wake-up time. Still nothing showing in the schedule log:
/var/log/fog/fogscheduler.log [03-31-21 5:54:00 am] ================================== === ==== ===== ==== === ========= == === == === === ======== ==== == ==== === === ======== ==== == ========= === ==== ==== == ========= === ======== ==== == === === === ======== ==== == ==== === === ========= == === == === === ========== ===== ==== ================================== ===== Free Opensource Ghost ====== ================================== ============ Credits ============= = https://fogproject.org/Credits = ================================== == Released under GPL Version 3 == ================================== [03-31-21 5:54:00 am] Interface Ready with IP Address: [privateIP] [03-31-21 5:54:00 am] Interface Ready with IP Address: 127.0.0.1 [03-31-21 5:54:00 am] Interface Ready with IP Address: 127.0.1.1 [03-31-21 5:54:00 am] Interface Ready with IP Address: fog.domain.com [03-31-21 5:54:00 am] * Starting TaskScheduler Service [03-31-21 5:54:00 am] * Checking for new items every 60 seconds [03-31-21 5:54:00 am] * Starting service loop [03-31-21 5:54:00 am] * 61 tasks found. [03-31-21 5:54:00 am] * Task run time: Thu, 01 Apr 2021 05:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500 [03-31-21 5:54:00 am] * Task run time: Wed, 31 Mar 2021 06:00:00 -0500
I still have the entries showing up in scheduled tasks:
-
# systemctl status -l FOGScheduler.service ● FOGScheduler.service - FOGScheduler Loaded: loaded (/usr/lib/systemd/system/FOGScheduler.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2021-03-30 09:24:36 CDT; 2h 20min ago Main PID: 14299 (php) Memory: 13.6M CGroup: /system.slice/FOGScheduler.service ├─14299 php /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler └─14300 php /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler Mar 30 09:24:36 fog.my.domain.com systemd[1]: Started FOGScheduler.
-
@altitudehack What about output of:
systemctl status -l FOGScheduler.service
? -
@tom-elliott said in Daily wake-up routine not working after update:
log/php-fpm/www-error.log
There was no output from this or any logfile in that directory during a restart of the service, which happened at 09:24:36:
# ls -lht /var/log/php-fpm/ total 276K -rwxr-xr-x. 1 root root 15K Mar 30 07:50 error.log -rwxr-xr-x. 1 root root 59K Mar 28 06:59 error.log-20210328 -rwxr-xr-x. 1 root root 58K Mar 21 03:27 error.log-20210321 -rwxr-xr-x. 1 root root 60K Mar 14 07:53 error.log-20210314 -rwxr-xr-x. 1 root root 60K Mar 7 06:20 error.log-20210307 -rwxr-xr-x. 1 apache apache 0 Mar 4 05:13 www-error.log -rwxr-xr-x. 1 apache apache 1.6K Mar 3 08:03 www-error.log-20210304 -rwxr-xr-x. 1 apache apache 5.6K Feb 19 11:25 www-error.log-20210221 -rwxr-xr-x. 1 apache apache 1.6K Feb 16 08:17 www-error.log-20210217 -rwxr-xr-x. 1 apache apache 1.6K Jan 31 17:07 www-error.log-20210201
-
@altitudehack I’m not quite sure whats going on but it feels like a php-fpm error.
Can you restart FOGScheduler (
systemctl restart FOGScheduler
) while tailing the output of:
/var/log/php-fpm/www-error.log
( at least that’s where it is on Centos, unsure for debian ) -
Nothing has processed since the service restart on the 25th. This is the entirety of the scheduler log as of March 30th:
/var/log/fog/fogscheduler.log [03-25-21 2:39:17 pm] ================================== === ==== ===== ==== === ========= == === == === === ======== ==== == ==== === === ======== ==== == ========= === ==== ==== == ========= === ======== ==== == === === === ======== ==== == ==== === === ========= == === == === === ========== ===== ==== ================================== ===== Free Opensource Ghost ====== ================================== ============ Credits ============= = https://fogproject.org/Credits = ================================== == Released under GPL Version 3 == ================================== [03-25-21 2:39:17 pm] Interface Ready with IP Address: [privateIP] [03-25-21 2:39:17 pm] Interface Ready with IP Address: 127.0.0.1 [03-25-21 2:39:17 pm] Interface Ready with IP Address: 127.0.1.1 [03-25-21 2:39:17 pm] Interface Ready with IP Address: [fqdn] [03-25-21 2:39:17 pm] * Starting TaskScheduler Service [03-25-21 2:39:17 pm] * Checking for new items every 60 seconds [03-25-21 2:39:17 pm] * Starting service loop [03-25-21 2:39:17 pm] * 61 tasks found. [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 05:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500 [03-25-21 2:39:17 pm] * Task run time: Fri, 26 Mar 2021 06:00:00 -0500
Is there anywhere else I can look?
Thanks!