Intermittent error 504

[Kagashe]

hi Guys,

My server keeps going offline with the following error, what could be causing it?

[george1421]

What version of FOG are you running? I see you are running ubuntu from the error message, what version is ubuntu?

If its fog 1.5.3 or 1.5.4 there is a setting adjustment we need to make to php-fpm that should address your issue.

[Kagashe]

Hi George,

FOG is on 1.5.4 running on Ubuntu 16.04.
Where do i have to make the adjustment?

[george1421]

@Kagashe Let me start with the standard response.

///
Lets assume is the issue we’ve found after FOG 1.5.4 has been released.

1. Change to the /etc directory from the fog server linux command prompt.
2. Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
   find /etc -name www.conf (hopefully you will only find one)
3. Search for php_admin_value[memory_limit] = 32M. It should have a comment mark in front of the line. Remove the comment mark and replace 32MB with 256MB. Your entry should look like below.

php_admin_value[memory_limit] = 256M

4. Save and exit your text editor.
5. Reboot the fog server.
6. See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.
   ///

That should address the gateway timeout issue under load.

[Kagashe]

@george1421

Found it, let me give it a run and see how it goes, thanks for the help.

[Kagashe]

Hi George,

Made the changes and still seem to get the error but getting less, seems to happen most when I add another storage location, also having very long load times in the browser.

[Sebastian Roth]

@Kagashe How many clients with installed fog-client do you have?

As well please take a look at the apache error log and post here. See my signature on where to find the log.

[george1421]

@Kagashe What does top sorted by Processes say as your highest cpu user?

[Kagashe]

Hi Sebastian and George,

I currently have around 300 hosts and around 10 Mirrors with the locations add-on registered but aim to get to 600 odd hosts and 36 mirrors in the coming months.
Top 3 processes CPU usage is php-fpm: pool www with at most about 12% each.
Load on the system is showing as 0.34 0.33 0.29
System is an i3 Processor with 16GB of DDR4 Ram a 120GB solid state and a 1TB Mechanical drive mounted at /images.
Running Ubuntu Server 16.04.

Apche Log shows the following error every 2 odd seconds

[Fri Nov 02 14:13:10.001117 2018] [proxy_fcgi:error] [pid 3236] [client 10.10.3.7:41454] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n

php-fpm log loops the following statement

[25-Sep-2018 15:01:01] NOTICE: fpm is running, pid 29251
[25-Sep-2018 15:01:01] NOTICE: ready to handle connections
[25-Sep-2018 15:01:01] NOTICE: systemd monitor interval set to 10000ms
[25-Sep-2018 15:03:40] NOTICE: Terminating ...
[25-Sep-2018 15:03:40] NOTICE: exiting, bye-bye!

[george1421]

@Kagashe First on the errors you see about the bandwidth.php, while its a problem its just an annoyance message at the moment.

The fpm message is a bit troubling and I need to look into it.

Your configuration is a bit abnormal (based on what we typically see) with the number of mirrors. That is fine, it might mean that we need to tweak our configuration a bit.

There is a timeout setting I would like you to make/change to see if we can address this issue. Specifically to tell apache to wait a bit longer for php-fpm to respond than its default settings. I can see the potential if you have your mirrors behind a slow network connection that it may take a little longer to get a response.

    <Proxy "fcgi://127.0.0.1:9000">
        ProxySet timeout=300
    </Proxy>

https://forums.fogproject.org/topic/11713/503-service-unavailable-error/40
and
https://forums.fogproject.org/topic/12057/fog-unresponsive-under-heavy-load/17

[Kagashe]

@george1421 Sorry haven’t answered been a hectic few days, have updated my config with the script you posted, will test it and feedback. Thanks for all the help so far, it’s much appreciated.

[Kagashe]

@george1421 This seems so have solved the issue so far, I’ve added to more mirrors to the mix and still haven’t had the 504 error again.

[george1421]

@Kagashe so what solved it. The timeout value being added to the apache config file?

[Kagashe]

@george1421 Yes that was the last change I made to the config.

[Kagashe]

@george1421 Aaaaaaand it just happened again

[Sebastian Roth]

@Kagashe Trying to reach you on chat. See the speech bubble in the top right corner.

[Sebastian Roth]

Marking as unsolved for now as the issue was worked around by George’s good advice to adjust proxy timeout but I am working on fixing the code to not have those very long timeouts at all.

[Sebastian Roth]

@Kagashe Ok here we go. Can you please upgrade all your nodes to the latest dev-branch version and see if you still get timeouts? I’d even suggest reverting the proxy timeout change (if not reverted by upgrage anyway) to see if the code improvements really do make a difference.

[Kagashe]

@Sebastian-Roth Have upgraded all the sites to 1.5.5.1 except one where I’m getting the following error

Adding Needed Repository…Failed!

Apart from that, all seems to be going well.

[Sebastian Roth]

@Kagashe Please check the install logs which reside in the bin directory from which you run the installer.

Are all your servers Ubuntu 16.04?