503 Service Unavailable



  • Fog Version: 1.5.4.8
    OS: Debian 9

    I am having an issue where my Fog server will get 503 service unavailable. I first saw this issue when I was booting 64 computers to my Fog server (1.5.2 at the time) to multicast. TFTP worked but was timing out at HTTP.
    Updated to 1.5.4 but still had the issue so I switched to the working branch of 1.5.4.8 which resolved that issue. After the multicast was completed, the PCs were stuck at updating database. The web portal of Fog became unresponsive with 503 Service Unavailable. Eventually the PCs timed out and booted to the OS and the Fog web portal returned to normal.
    0_1534976174153_IMG_0781.jpg

    I have reviewed other posts and I have verified that my www.conf file had the recommend settings where users reported 504 gateway timeout issues when they had updating database failures.

    pm.max_children  = 35
    pm.max_requests = 2000
    php_admin_value[memory_limit] = 256
    

    I have not seen this before with older versions of Fog (1.5.0) which I was successfully able to multicast 130 computers at once without any issues (same hardware).

    Any suggestions would be awesome!

    Thanks!



  • @george1421 Alright. Ran the test again and got the logs.

    Apache Log (sample)

    [Thu Aug 23 14:52:54.299174 2018] [proxy_fcgi:error] [pid 12423] [client 172.25.49.13:37260] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:52:54.299174 2018] [proxy_fcgi:error] [pid 12367] [client 172.25.49.13:36928] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:52:54.299296 2018] [proxy_fcgi:error] [pid 12423] (104)Connection reset by peer: [client 172.25.49.13:37260] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:52:54.299333 2018] [proxy_fcgi:error] [pid 12367] (104)Connection reset by peer: [client 172.25.49.13:36928] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:52:56.343241 2018] [proxy_fcgi:error] [pid 12314] [client 172.25.49.13:37312] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:52:56.343242 2018] [proxy_fcgi:error] [pid 12324] [client 172.25.49.13:37154] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:52:56.343334 2018] [proxy_fcgi:error] [pid 12314] (104)Connection reset by peer: [client 172.25.49.13:37312] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:52:56.343372 2018] [proxy_fcgi:error] [pid 12324] (104)Connection reset by peer: [client 172.25.49.13:37154] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:52:58.391273 2018] [proxy_fcgi:error] [pid 13269] [client 172.25.49.13:37284] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:52:58.391305 2018] [proxy_fcgi:error] [pid 13263] [client 172.25.49.13:37244] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:52:58.391279 2018] [proxy_fcgi:error] [pid 13261] [client 172.25.49.13:37238] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:52:58.391286 2018] [proxy_fcgi:error] [pid 13266] [client 172.25.49.13:37274] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:52:58.391432 2018] [proxy_fcgi:error] [pid 13269] (104)Connection reset by peer: [client 172.25.49.13:37284] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:52:58.391447 2018] [proxy_fcgi:error] [pid 13263] (104)Connection reset by peer: [client 172.25.49.13:37244] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:52:58.391502 2018] [proxy_fcgi:error] [pid 13266] (104)Connection reset by peer: [client 172.25.49.13:37274] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:52:58.391486 2018] [proxy_fcgi:error] [pid 13261] (104)Connection reset by peer: [client 172.25.49.13:37238] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:02.487262 2018] [proxy_fcgi:error] [pid 12363] [client 172.25.49.116:40982] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:02.487262 2018] [proxy_fcgi:error] [pid 12364] [client 172.25.49.108:52168] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:02.487356 2018] [proxy_fcgi:error] [pid 12363] (104)Connection reset by peer: [client 172.25.49.116:40982] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:02.487391 2018] [proxy_fcgi:error] [pid 12364] (104)Connection reset by peer: [client 172.25.49.108:52168] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:04.535082 2018] [proxy_fcgi:error] [pid 13249] [client 172.25.49.13:37160] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:04.535082 2018] [proxy_fcgi:error] [pid 13250] [client 172.25.49.13:37162] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:04.535187 2018] [proxy_fcgi:error] [pid 13249] (104)Connection reset by peer: [client 172.25.49.13:37160] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:04.535208 2018] [proxy_fcgi:error] [pid 13250] (104)Connection reset by peer: [client 172.25.49.13:37162] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:06.583087 2018] [proxy_fcgi:error] [pid 12425] [client 172.25.49.13:37266] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:06.583115 2018] [proxy_fcgi:error] [pid 12430] [client 172.25.49.13:37302] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:06.583115 2018] [proxy_fcgi:error] [pid 12427] [client 172.25.49.13:37268] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:06.583111 2018] [proxy_fcgi:error] [pid 12429] [client 172.25.49.13:37304] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:06.583162 2018] [proxy_fcgi:error] [pid 12425] (104)Connection reset by peer: [client 172.25.49.13:37266] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:06.583202 2018] [proxy_fcgi:error] [pid 12427] (104)Connection reset by peer: [client 172.25.49.13:37268] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:06.583190 2018] [proxy_fcgi:error] [pid 12430] (104)Connection reset by peer: [client 172.25.49.13:37302] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:06.583230 2018] [proxy_fcgi:error] [pid 12429] (104)Connection reset by peer: [client 172.25.49.13:37304] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:10.679240 2018] [proxy:error] [pid 12279] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.679240 2018] [proxy:error] [pid 12301] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.679240 2018] [proxy:error] [pid 12365] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.679270 2018] [proxy:error] [pid 13228] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.679404 2018] [proxy_fcgi:error] [pid 12279] [client 172.25.49.13:37034] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.679423 2018] [proxy_fcgi:error] [pid 12301] [client 172.25.49.13:37050] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.679469 2018] [proxy_fcgi:error] [pid 12365] [client 172.25.49.155:54020] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.679490 2018] [proxy_fcgi:error] [pid 13228] [client 172.25.49.13:36996] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.679852 2018] [proxy:error] [pid 12277] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.679852 2018] [proxy:error] [pid 12272] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.679931 2018] [proxy_fcgi:error] [pid 12277] [client 172.25.49.13:37012] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.679912 2018] [proxy:error] [pid 12304] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.679912 2018] [proxy:error] [pid 12278] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.679948 2018] [proxy_fcgi:error] [pid 12272] [client 172.25.49.13:37014] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.680005 2018] [proxy_fcgi:error] [pid 12278] [client 172.25.49.13:37018] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.679992 2018] [proxy_fcgi:error] [pid 12304] [client 172.25.49.13:37056] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.680241 2018] [proxy:error] [pid 12271] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.680240 2018] [proxy:error] [pid 12303] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.680304 2018] [proxy_fcgi:error] [pid 12271] [client 172.25.49.13:37002] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.680316 2018] [proxy_fcgi:error] [pid 12303] [client 172.25.49.13:37054] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.680510 2018] [proxy:error] [pid 12302] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.680573 2018] [proxy_fcgi:error] [pid 12302] [client 172.25.49.13:37036] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.680804 2018] [proxy:error] [pid 12275] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.680881 2018] [proxy_fcgi:error] [pid 12275] [client 172.25.49.13:37004] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:10.681115 2018] [proxy:error] [pid 12273] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:10.681182 2018] [proxy_fcgi:error] [pid 12273] [client 172.25.49.13:36994] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.727045 2018] [proxy:error] [pid 12315] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.727052 2018] [proxy:error] [pid 12310] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.727052 2018] [proxy:error] [pid 12313] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.727168 2018] [proxy_fcgi:error] [pid 12315] [client 172.25.49.13:37110] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.727193 2018] [proxy_fcgi:error] [pid 12310] [client 172.25.49.13:37094] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.727202 2018] [proxy_fcgi:error] [pid 12313] [client 172.25.49.13:37106] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.727191 2018] [proxy:error] [pid 12323] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.727316 2018] [proxy_fcgi:error] [pid 12323] [client 172.25.49.13:37148] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.727431 2018] [proxy:error] [pid 12418] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.727475 2018] [proxy_fcgi:error] [pid 12418] [client 172.25.49.13:37214] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.727455 2018] [proxy:error] [pid 11819] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.727510 2018] [proxy_fcgi:error] [pid 11819] [client 172.25.49.13:37146] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.727726 2018] [proxy:error] [pid 12309] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.727785 2018] [proxy_fcgi:error] [pid 12309] [client 172.25.49.13:37098] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.728015 2018] [proxy:error] [pid 12311] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.728064 2018] [proxy_fcgi:error] [pid 12311] [client 172.25.49.13:37092] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.728060 2018] [proxy:error] [pid 12317] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.728108 2018] [proxy_fcgi:error] [pid 12317] [client 172.25.49.13:37120] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.728268 2018] [proxy:error] [pid 12307] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.728317 2018] [proxy_fcgi:error] [pid 12307] [client 172.25.49.13:37084] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.728317 2018] [proxy:error] [pid 12316] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.728369 2018] [proxy_fcgi:error] [pid 12316] [client 172.25.49.13:37112] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.728546 2018] [proxy:error] [pid 12305] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.728596 2018] [proxy_fcgi:error] [pid 12305] [client 172.25.49.13:37076] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.728584 2018] [proxy:error] [pid 12308] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.728634 2018] [proxy_fcgi:error] [pid 12308] [client 172.25.49.13:37078] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.728826 2018] [proxy:error] [pid 12412] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.728856 2018] [proxy:error] [pid 12306] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.728879 2018] [proxy_fcgi:error] [pid 12412] [client 172.25.49.13:37200] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.728927 2018] [proxy_fcgi:error] [pid 12306] [client 172.25.49.13:37068] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.729140 2018] [proxy:error] [pid 12410] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.729163 2018] [proxy:error] [pid 12417] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.729192 2018] [proxy_fcgi:error] [pid 12410] [client 172.25.49.13:37196] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.729214 2018] [proxy_fcgi:error] [pid 12417] [client 172.25.49.13:37212] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.729445 2018] [proxy:error] [pid 12413] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.729495 2018] [proxy_fcgi:error] [pid 12413] [client 172.25.49.13:37172] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.729717 2018] [proxy:error] [pid 12414] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.729768 2018] [proxy_fcgi:error] [pid 12414] [client 172.25.49.13:37170] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.729985 2018] [proxy:error] [pid 12408] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.730035 2018] [proxy_fcgi:error] [pid 12408] [client 172.25.49.13:37166] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:12.732232 2018] [proxy:error] [pid 12320] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:12.732257 2018] [proxy_fcgi:error] [pid 12320] [client 172.25.49.13:37142] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:14.775041 2018] [proxy:error] [pid 13267] (110)Connection timed out: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
    [Thu Aug 23 14:53:14.775162 2018] [proxy_fcgi:error] [pid 13267] [client 172.25.49.13:37278] AH01079: failed to make connection to backend: 127.0.0.1
    [Thu Aug 23 14:53:18.871129 2018] [proxy_fcgi:error] [pid 12359] [client 172.25.49.182:38394] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:18.871206 2018] [proxy_fcgi:error] [pid 12359] (104)Connection reset by peer: [client 172.25.49.182:38394] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:18.871368 2018] [proxy_fcgi:error] [pid 12362] [client 172.25.49.111:48116] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:18.871408 2018] [proxy_fcgi:error] [pid 12362] (104)Connection reset by peer: [client 172.25.49.111:48116] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:20.919068 2018] [proxy_fcgi:error] [pid 12419] [client 172.25.49.13:37216] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:20.919070 2018] [proxy_fcgi:error] [pid 12420] [client 172.25.49.13:37230] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:20.919067 2018] [proxy_fcgi:error] [pid 12415] [client 172.25.49.13:37210] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:20.919067 2018] [proxy_fcgi:error] [pid 12421] [client 172.25.49.13:37228] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:20.919151 2018] [proxy_fcgi:error] [pid 12419] (104)Connection reset by peer: [client 172.25.49.13:37216] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:20.919161 2018] [proxy_fcgi:error] [pid 12420] (104)Connection reset by peer: [client 172.25.49.13:37230] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:20.919193 2018] [proxy_fcgi:error] [pid 12415] (104)Connection reset by peer: [client 172.25.49.13:37210] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:20.919205 2018] [proxy_fcgi:error] [pid 12421] (104)Connection reset by peer: [client 172.25.49.13:37228] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:22.967076 2018] [proxy_fcgi:error] [pid 12434] [client 172.25.49.13:37316] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:22.967076 2018] [proxy_fcgi:error] [pid 12435] [client 172.25.49.13:37318] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:22.967146 2018] [proxy_fcgi:error] [pid 12434] (104)Connection reset by peer: [client 172.25.49.13:37316] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:22.967157 2018] [proxy_fcgi:error] [pid 12435] (104)Connection reset by peer: [client 172.25.49.13:37318] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:25.015166 2018] [proxy_fcgi:error] [pid 12436] [client 172.25.57.80:51375] AH01067: Failed to read FastCGI header, referer: http://172.25.49.13/fog/management/index.php?node=task&sub=active
    [Thu Aug 23 14:53:25.015312 2018] [proxy_fcgi:error] [pid 12436] (104)Connection reset by peer: [client 172.25.57.80:51375] AH01075: Error dispatching request to : , referer: http://172.25.49.13/fog/management/index.php?node=task&sub=active
    [Thu Aug 23 14:53:51.639042 2018] [proxy_fcgi:error] [pid 12276] [client 172.25.49.13:37010] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:51.639052 2018] [proxy_fcgi:error] [pid 12274] [client 172.25.49.13:37008] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:51.639117 2018] [proxy_fcgi:error] [pid 12276] (104)Connection reset by peer: [client 172.25.49.13:37010] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:51.639142 2018] [proxy_fcgi:error] [pid 12274] (104)Connection reset by peer: [client 172.25.49.13:37008] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:53.687052 2018] [proxy_fcgi:error] [pid 12411] [client 172.25.49.13:37206] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:53.687094 2018] [proxy_fcgi:error] [pid 12416] [client 172.25.49.13:37186] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:53.687149 2018] [proxy_fcgi:error] [pid 12411] (104)Connection reset by peer: [client 172.25.49.13:37206] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:53.687168 2018] [proxy_fcgi:error] [pid 12416] (104)Connection reset by peer: [client 172.25.49.13:37186] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:55.735175 2018] [proxy_fcgi:error] [pid 12431] [client 172.25.49.13:37310] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:55.735256 2018] [proxy_fcgi:error] [pid 12431] (104)Connection reset by peer: [client 172.25.49.13:37310] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:53:55.735258 2018] [proxy_fcgi:error] [pid 13265] [client 172.25.49.13:37272] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:53:55.735499 2018] [proxy_fcgi:error] [pid 13265] (104)Connection reset by peer: [client 172.25.49.13:37272] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:54:10.071304 2018] [proxy_fcgi:error] [pid 12438] [client 172.25.49.103:4802] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:54:10.071391 2018] [proxy_fcgi:error] [pid 12438] (104)Connection reset by peer: [client 172.25.49.103:4802] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:54:14.167115 2018] [proxy_fcgi:error] [pid 12439] [client 172.25.49.184:1025] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:54:14.167187 2018] [proxy_fcgi:error] [pid 12439] (104)Connection reset by peer: [client 172.25.49.184:1025] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:54:14.167431 2018] [proxy_fcgi:error] [pid 12437] [client 172.25.49.200:57962] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:54:14.167476 2018] [proxy_fcgi:error] [pid 12437] (104)Connection reset by peer: [client 172.25.49.200:57962] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:54:14.167472 2018] [proxy_fcgi:error] [pid 12328] [client 172.25.49.171:5606] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:54:14.167537 2018] [proxy_fcgi:error] [pid 12328] (104)Connection reset by peer: [client 172.25.49.171:5606] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:54:24.407081 2018] [proxy_fcgi:error] [pid 12329] [client 172.25.49.149:27292] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:54:24.407174 2018] [proxy_fcgi:error] [pid 12329] (104)Connection reset by peer: [client 172.25.49.149:27292] AH01075: Error dispatching request to : 
    [Thu Aug 23 14:54:34.647026 2018] [proxy_fcgi:error] [pid 12374] [client 172.25.57.80:51444] AH01067: Failed to read FastCGI header, referer: http://172.25.49.13/fog/management/index.php?node=task&sub=active
    [Thu Aug 23 14:54:34.647069 2018] [proxy_fcgi:error] [pid 12381] [client 172.25.49.168:12119] AH01067: Failed to read FastCGI header
    [Thu Aug 23 14:54:34.647107 2018] [proxy_fcgi:error] [pid 12374] (104)Connection reset by peer: [client 172.25.57.80:51444] AH01075: Error dispatching request to : , referer: http://172.25.49.13/fog/management/index.php?node=task&sub=active
    

    PHP-fpm Error Log (Sample)

    [23-Aug-2018 10:33:37] NOTICE: systemd monitor interval set to 10000ms
    [23-Aug-2018 12:23:53] NOTICE: [pool www] child 740 exited with code 0 after 6616.766926 seconds from start
    [23-Aug-2018 12:23:53] NOTICE: [pool www] child 16544 started
    [23-Aug-2018 12:23:54] NOTICE: [pool www] child 742 exited with code 0 after 6617.066685 seconds from start
    [23-Aug-2018 12:23:54] NOTICE: [pool www] child 16595 started
    [23-Aug-2018 12:24:26] NOTICE: [pool www] child 838 exited with code 0 after 6641.945870 seconds from start
    [23-Aug-2018 12:24:26] NOTICE: [pool www] child 16758 started
    [23-Aug-2018 12:25:56] NOTICE: [pool www] child 741 exited with code 0 after 6739.049484 seconds from start
    [23-Aug-2018 12:25:56] NOTICE: [pool www] child 16890 started
    [23-Aug-2018 12:26:56] NOTICE: [pool www] child 739 exited with code 0 after 6799.084111 seconds from start
    [23-Aug-2018 12:26:56] NOTICE: [pool www] child 16965 started
    [23-Aug-2018 12:27:06] NOTICE: [pool www] child 743 exited with code 0 after 6809.011899 seconds from start
    [23-Aug-2018 12:27:06] NOTICE: [pool www] child 16972 started
    [23-Aug-2018 12:30:26] NOTICE: [pool www] child 2261 exited with code 0 after 6995.000073 seconds from start
    [23-Aug-2018 12:30:26] NOTICE: [pool www] child 17204 started
    [23-Aug-2018 12:31:36] NOTICE: [pool www] child 2260 exited with code 0 after 7065.031347 seconds from start
    [23-Aug-2018 12:31:36] NOTICE: [pool www] child 17285 started
    [23-Aug-2018 12:32:16] NOTICE: [pool www] child 2088 exited with code 0 after 7105.962251 seconds from start
    [23-Aug-2018 12:32:16] NOTICE: [pool www] child 17333 started
    [23-Aug-2018 14:23:47] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 16 total children
    [23-Aug-2018 14:23:48] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 21 total children
    [23-Aug-2018 14:23:49] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 26 total children
    [23-Aug-2018 14:23:50] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 31 total children
    [23-Aug-2018 14:23:51] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 36 total children
    [23-Aug-2018 14:26:37] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 17 total children
    [23-Aug-2018 14:26:38] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 22 total children
    [23-Aug-2018 14:26:39] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 27 total children
    [23-Aug-2018 14:26:40] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 32 total children
    [23-Aug-2018 14:26:41] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 37 total children
    [23-Aug-2018 14:26:42] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 42 total children
    [23-Aug-2018 14:45:58] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 17 total children
    [23-Aug-2018 14:45:59] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 22 total children
    [23-Aug-2018 14:46:00] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 27 total children
    [23-Aug-2018 14:46:01] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 32 total children
    [23-Aug-2018 14:46:02] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 37 total children
    [23-Aug-2018 14:46:03] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 42 total children
    [23-Aug-2018 14:46:04] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 47 total children
    [23-Aug-2018 14:46:05] WARNING: [pool www] server reached pm.max_children setting (50), consider raising it```

  • Moderator

    @LJedi Well then I’m a bit confused why you have the 503 error then. I might suspect that when the 503 error is displayed, that php-fpm hasn’t returned what its working on back to apache.

    When that 503 error happens again, please check the php-fpm and apache error logs for clues to why the server is unavailable, and then time fixes it.



  • @george1421 Thanks for your feedback. I did verify that I did have 256M and not 256. I did run the top command and sorted by process and I do have php-fpm7.0 running in the top 4 services. I will also look to downgrading the linux kernels to resolve the 3-5 minute create disk structure issue.


  • Moderator

    Let me preface this with my standard response first. Hopefully in your post you just omitted the M in 256M for the memory limit, otherwise you allocated 256 bytes, which WILL cause an isssue.

    ////
    Lets assume is the issue we’ve found after FOG 1.5.4 has been released.

    1. Change to the /etc directory from the fog server linux command prompt.
    2. Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
      find /etc -name www.conf (hopefully you will only find one)
    3. Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except php_admin_value[memory_limit] = 256M you will need to add that entry.
    php_admin_value[memory_limit] = 256M
    pm.max_requests = 2000
    pm.max_children = 35
    pm.min_spare_servers = 5
    pm.start_servers = 5
    
    1. Save and exit your text editor.
    2. Reboot the fog server.
    3. See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.

    Also we found there is something strange going on in the linux kernels after 4.15.2, I’m going to recommend that you downgrade your FOG/FOS kernel to 4.15.2. The issue with later kernels is that its taking 3-5 minutes to create the disk structure under certain circumstances, where with 4.15.2 and older its only seconds to create the structure.

    Now the kernel will not impact your issue, but processing is incomplete might be related to the missing php-fpm configuration setting.
    ////

    In addition to the above, lets confirm that php-fpm is working correctly on debian. There was a bug in the installer, where it didn’t take account of debian different config file.

    From the linux console, run the top command. Sort by Process. Confirm that php-fpm is running in the top 4 services. If apache is there then php-fpm didn’t load correctly.


Log in to reply
 

348
Online

6.0k
Users

13.4k
Topics

126.2k
Posts