503 Service unavailable after Debian upgrade

[EdCooper]

I haven’t quite been able to find the answer in previous posts. I just upgraded Debian 10. I’m running Fog 1.5.9. After upgrade, I couldn’t access the webpage. I just get a 503 error. The apache log is showing all the clients are unable to connect as well. Restarting apache, mysql, linux don’t help.
Apache log:

tech@eastfog:~$ sudo tail /var/log/apache2/error.log
[Wed Mar 17 12:23:57.245931 2021] [proxy:error] [pid 734] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Mar 17 12:23:57.245955 2021] [proxy_fcgi:error] [pid 734] [client 10.34.21.9:60737] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Mar 17 12:23:58.430682 2021] [proxy:error] [pid 733] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Mar 17 12:23:58.430719 2021] [proxy_fcgi:error] [pid 733] [client 10.38.27.77:54180] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Mar 17 12:23:58.599523 2021] [proxy:error] [pid 876] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Mar 17 12:23:58.599547 2021] [proxy_fcgi:error] [pid 876] [client 10.38.27.77:54181] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Mar 17 12:23:58.639461 2021] [proxy:error] [pid 1139] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Mar 17 12:23:58.639485 2021] [proxy_fcgi:error] [pid 1139] [client 10.38.27.77:54182] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Mar 17 12:23:59.513198 2021] [proxy:error] [pid 737] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Mar 17 12:23:59.513221 2021] [proxy_fcgi:error] [pid 737] [client 10.34.16.99:3898] AH01079: failed to make connection to backend: 127.0.0.1

Php-fpm log:

tech@eastfog:/var/log$ tail php7.3-fpm.log
[17-Mar-2021 11:32:43] NOTICE: Terminating ...
[17-Mar-2021 11:32:43] NOTICE: exiting, bye-bye!
[17-Mar-2021 11:33:01] NOTICE: fpm is running, pid 587
[17-Mar-2021 11:33:01] NOTICE: ready to handle connections
[17-Mar-2021 11:33:01] NOTICE: systemd monitor interval set to 10000ms
[17-Mar-2021 12:00:01] NOTICE: Terminating ...
[17-Mar-2021 12:00:01] NOTICE: exiting, bye-bye!
[17-Mar-2021 12:00:20] NOTICE: fpm is running, pid 568
[17-Mar-2021 12:00:20] NOTICE: ready to handle connections
[17-Mar-2021 12:00:20] NOTICE: systemd monitor interval set to 10000ms

Mysql error log:

root@eastfog:/var/log/mysql# tail error.log
2021-03-17 12:00:20 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-03-17 12:00:20 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-03-17 12:00:20 0 [Note] InnoDB: Buffer pool(s) load completed at 210317 12:00:20
2021-03-17 12:00:21 0 [Note] Server socket created on IP: '::'.
2021-03-17 12:00:21 0 [Note] Reading of all Master_info entries succeeded
2021-03-17 12:00:21 0 [Note] Added new Master_info '' to hash table
2021-03-17 12:00:21 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.3.27-MariaDB-0+deb10u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 10

Servicemaster log:

tech@eastfog:/var/log/fog$ tail servicemaster.log
[03-17-21 12:00:21 pm] FOGPingHosts forked child process (853).
[03-17-21 12:00:21 pm] FOGImageSize forked child process (850).
[03-17-21 12:00:21 pm] FOGSnapinReplicator forked child process (854).
[03-17-21 12:00:21 pm] FOGImageReplicator child process (852) is running.
[03-17-21 12:00:21 pm] FOGSnapinHash forked child process (851).
[03-17-21 12:00:21 pm] FOGTaskScheduler child process (849) is running.
[03-17-21 12:00:21 pm] FOGTaskScheduler forked child process (849).
[03-17-21 12:00:21 pm] FOGSnapinHash child process (851) is running.
[03-17-21 12:00:21 pm] FOGSnapinReplicator child process (854) is running.
[03-17-21 12:00:21 pm] FOGPingHosts child process (853) is running.

I’m not sure where to go from here. I have about 200 hosts running. It’s not really an option to rebuild. Of course there’s no backup because the boss said it backs up the images also, and there are several terabytes worth of images backed up.

Thanks for the help.

[Tom Elliott]

@edcooper I’m going to guess that somebody has setup php-fpm to run in socket mode, while Apache appears to be looking for it in localhost:port mode.

NOTE: You will likely need to sudo to root first as well

Please post the contents of:
grep -r 'listen = ' /etc

You should get a return similar to:

[root@fogserver ~]# grep -r 'listen = ' /etc/
/etc/php-fpm.d/www.conf:listen = 127.0.0.1:9000

[Tom Elliott]

@edcooper The other “best” option would be to reinstall FOG.

No, reinstalling fog does not mean “Delete all images, delete everything in the database, etc…” It just means: re-run the fog installer. This can fix many issues just on its own.

[EdCooper]

Thank you so much for your replies. I reinstalled and it fixed it. I didn’t realize a reinstall wouldn’t affect the existing install. It installed “php-gettext” and did all the checks. Database is perfect.

Thanks for the assist. I appreciate it.