503 Service Unavailable Error

Greg Plamondon

@tom-elliott said in 503 Service Unavailable Error:

@greg-plamondon Can you grab a video or pic of the error as it’s trying to get into a deploy task?
Sure thing:

george1421

@greg-plamondon I spent a few hours working on this issue last night. I’m proud to report I can’t duplicate this error. That doesn’t really mean a hill of beans here. That just means my environment is a bit different than yours. My dev box only has a handful of clients and is on a pretty fast server. I do have a few ideas that I would like you guys to test. We do need php-fpm to work well to help improve web ui performance. The UI Developers are working on an optimized version of the new ui for 1.6.0 release, but that is a few months out yet.

What I would like both of you to do is:

Rerun the fog installer to reset everything back to a known configuration (hopefully).
Once the installer has completed I want both of you to modify the following file: /etc/httpd/conf.d/fog.conf
Insert the following section:

    <Proxy "fcgi://127.0.0.1:9000">
        ProxySet timeout=300
    </Proxy>

To make the file look like this:

<VirtualHost *:80>
    <Proxy "fcgi://127.0.0.1:9000">
        ProxySet timeout=300
    </Proxy>
    <FilesMatch "\.php$">
        SetHandler "proxy:fcgi://127.0.0.1:9000/"
    </FilesMatch>
    KeepAlive Off
    ServerName 192.168.100.1
    DocumentRoot /var/www/html/
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]
    <Directory /var/www/html/fog/>
        DirectoryIndex index.php index.html index.htm
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-d
        RewriteRule ^/(.*)$ /fog/api/index.php [QSA,L]
    </Directory>
</VirtualHost>

Save and exit the editor
Issue the following command to restart apache:
systemctl restart httpd

This directive should tell apache to wait up to 5 minutes for php-fpm to complete its operations. (if it takes that long there is something else really wrong)

Now Greg, since you had a memory exhaustion issue I want you to do these additional setup.

Edit this file /etc/php-fpm.d/www.conf
Look for a line that reads:
;pm.max_requests = 500
Uncomment that line and change the parameter to 2000 to make it look like this:
pm.max_requests = 2000
Save and exit the editor.
Restart php-fpm and apache.
systemctl restart php-fpm
systemctl restart httpd

I think we actually have 2 issues here, and they are both relevant. Both are hitting the timeout issue, while Greg is hitting the next issue too.

Thank you guys for helping to work this out. I’m pretty sure we can get a resolution, if not we can always make 2 changes to apache and be back to the way 1.5.1 and earlier worked using the built in apache php engine.

Greg Plamondon

@george1421 said in 503 Service Unavailable Error:

/etc/php-fpm.d/www.conf

I dont have a /etc/php-fpm.d/www.conf
The only file in the php-fpm.d directory is fog.conf

george1421

@greg-plamondon Ok then you are using/still have my config in place. That is fine, change the line from 500 to 2000, save and restart both services. Hopefully the 500 value did address your memory exhaustion issue. This value is commented out in the FOG installed version.

Just make sure you don’t have two configuration files in the directory www.conf and fog.conf, you should be OK.

Greg Plamondon

@george1421 said in 503 Service Unavailable Error:

@greg-plamondon Ok then you are using/still have my config in place. That is fine, change the line from 500 to 2000, save and restart both services. Hopefully the 500 value did address your memory exhaustion issue. This value is commented out in the FOG installed version.

Just make sure you don’t have two configuration files in the directory www.conf and fog.conf, you should be OK.

That worked, I didn’t get the 503 error that time when imaging a PC!
Thanks! I will submit a new ticket for my other issues…

george1421

@greg-plamondon Thank you for the feedback, please keep (ab)using the FOG server. I’m interested in the durability of the configuration over time.

I am interested in what Troye’s experiences with just the timeout adjustment and to see if he does run into a memory exhaustion issue. I did see one other post from the github side that ran into the memory exhaustion issue and if I remember correctly he bumped his config to 3GB and still had the issue.

Greg Plamondon

@george1421 said in 503 Service Unavailable Error:

@greg-plamondon Thank you for the feedback, please keep (ab)using the FOG server. I’m interested in the durability of the configuration over time.

I am interested in what Troye’s experiences with just the timeout adjustment and to see if he does run into a memory exhaustion issue. I did see one other post from the github side that ran into the memory exhaustion issue and if I remember correctly he bumped his config to 3GB and still had the issue.

I am impressed with the speed of the GUI with these changes. Nice work guys keep up the good Work @george1421 @Tom-Elliott

Troye Johnson

hey, guys, none of this helped me so far. Still getting the 503 error. When I drop my database though everything works site moves faster and everything.

george1421

@troye-johnson So just to confirm you updated the apache fog.conf file with the proxy timeout setting. Then restarted apache and still have the 503 error?

What errors are you seeing in the tail of the /var/log/httpd/error.log file and in /var/log/php-fpm/www-error.log file? Are you seeing a memory depletion message too? Hang in there, we’ll get this worked out.

@greg-plamondon php-fpm is a really powerful php processor. With new fog 1.5.1or .2 gui and php-fpm the entire fog ui should be faster than the 1.4.1 web ui, with less load on the FOG CPU.

Troye Johnson

This is what I have seen in the error log.

[12-Apr-2018 08:56:18] NOTICE: [pool fog] child 16982 exited with code 0 after 10257.570038 seconds from start
[12-Apr-2018 08:56:18] NOTICE: [pool fog] child 23554 started
[12-Apr-2018 08:56:34] NOTICE: [pool fog] child 17085 exited with code 0 after 10251.368654 seconds from start
[12-Apr-2018 08:56:34] NOTICE: [pool fog] child 23636 started
[12-Apr-2018 08:56:41] NOTICE: [pool fog] child 17076 exited with code 0 after 10264.530989 seconds from start
[12-Apr-2018 08:56:41] NOTICE: [pool fog] child 23690 started
[12-Apr-2018 08:56:43] NOTICE: [pool fog] child 17179 exited with code 0 after 10242.630129 seconds from start
[12-Apr-2018 08:56:43] NOTICE: [pool fog] child 23697 started
[12-Apr-2018 08:56:47] NOTICE: [pool fog] child 17244 exited with code 0 after 10240.578956 seconds from start
[12-Apr-2018 08:56:47] NOTICE: [pool fog] child 23744 started
[12-Apr-2018 08:56:49] NOTICE: [pool fog] child 17188 exited with code 0 after 10247.308275 seconds from start
[12-Apr-2018 08:56:49] NOTICE: [pool fog] child 23757 started
[12-Apr-2018 08:57:43] NOTICE: [pool fog] child 17395 exited with code 0 after 10275.514540 seconds from start
[12-Apr-2018 08:57:43] NOTICE: [pool fog] child 24055 started
[12-Apr-2018 09:06:14] NOTICE: [pool fog] child 20983 exited with code 0 after 10261.635580 seconds from start
[12-Apr-2018 09:06:14] NOTICE: [pool fog] child 27599 started
[12-Apr-2018 09:38:52] NOTICE: [pool fog] child 2136 exited with code 0 after 10275.005038 seconds from start
[12-Apr-2018 09:38:52] NOTICE: [pool fog] child 9559 started
[12-Apr-2018 10:10:17] NOTICE: Terminating ...
[12-Apr-2018 10:10:17] NOTICE: exiting, bye-bye!
[12-Apr-2018 10:10:54] NOTICE: fpm is running, pid 23466
[12-Apr-2018 10:10:54] NOTICE: ready to handle connections
[12-Apr-2018 10:10:54] NOTICE: systemd monitor interval set to 10000ms
[12-Apr-2018 10:30:26] NOTICE: Terminating ...
[12-Apr-2018 10:30:26] NOTICE: exiting, bye-bye!
[12-Apr-2018 10:31:46] NOTICE: fpm is running, pid 1007
[12-Apr-2018 10:31:46] NOTICE: ready to handle connections
[12-Apr-2018 10:31:46] NOTICE: systemd monitor interval set to 10000ms
[12-Apr-2018 10:32:46] WARNING: [pool fog] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 3 idle, and 19 total children```

george1421

For others that find this thread and have the 503 error. Please post here. The developers need to understand how widespread this issue is. The default settings should be correct for all installs, but as we see here at least 2 installs has timeout issues.

Costas

OS: Centos 7 build 1708
Fog Version: 1.5 upgraded from 1.4.4

Hello.
Havent posted here before.
Got redirected from here

https://github.com/FOGProject/fogproject/issues/248

I get a “HTTP ERROR 500” error when attempting to do a user tracking report.
As Tom recommended on that page I did increase the memory limit in fog setting up to 4GB before I gave up as the error would still not go away.
This happens whether you search for a user or hostname.
I tried a few other reports such as snapin log report and history log report and they seem to be working fine.
Let me know if i need to provide anything else.

george1421

@costas Will you reset the memory limits back to FOG’s default value.

Then lets inspect /var/log/php-fpm/www-error.log for a memory exhaustion warning message like Greg’s below. If that is found then lets make the adjustment that appears to have fixed Greg’s issue.

Edit this file /etc/php-fpm.d/www.conf
Look for a line that reads:
;pm.max_requests = 500
Uncomment that line and change the parameter to 2000 to make it look like this:
pm.max_requests = 2000
Save and exit the editor.
Restart php-fpm and apache.
systemctl restart php-fpm
systemctl restart httpd

Costas

@george1421 Hey George.
My www-error.log is not very useful.
Only has one line in it.
“[15-Apr-2018 03:45:01] NOTICE: error log file re-opened”
just to be cleat the error I get when running a user tracking report in chrome is
“This page isn’t working
10.214.14.27 is currently unable to handle this request.
HTTP ERROR 500”
should i still go ahead and perform the changes you asked for?

Daniel Miller

Don’t know if this relevant to this issue (if not, let me know and I"ll open another thread), but I’m encountering http 500 responses on the user tracking report page with similar log entries on an ubuntu based system I just upgraded from 1.4.4 to 1.5.2 from a git pull off an existing checkout.

After changing the the pm.max_request setting in /etc/php/7.1/fpm/pool.d/www.conf and changing the memory limit to 2048 (the box only has 2GB), I see a warning about the server reaching the pm.max_children value in /var/log/php7.1-fpm.log followed by a proxy_fcgi:error in /var/log/apache2/error.log citing an out of memory condition in fogcontroller.class.php with referrer /fog/management/index.php?node=report&sub=file&f=dXNlciB0cmFja2luZw==. There is a reference to a line number for the oom error that seems to vary between lines 126 (in __construct()) and 260 (in set()), but I would expect that depends on when it actually ran out. I enabled error reporting in the php settings for apache, but it doesn’t seem to want to get past the 500.

Raising the max_children to 20 shows entries in php7.1-fpm.log about the pool seeming busy and it spawning additional children until the limit is reached then it reports the reaching of the max_children limit again. The proxy_fcgi:error in this case is a timeout dispatching a request to (polling) with the same referrer.

Scaling the memory limit back down to default results in a reference to pdodb.class.php line 602 in the memory exhaustion message.

I also attempted altering the apache configuration, but I only had to add the ProxySet timeout. No change in the error message.

Happy to poke at anything else you might have an idea about. Have a few days before I need to decide to backrev or not.

george1421

@daniel-miller How many fog clients do you have in your environment?

You shouldn’t need to change the memory limit in pgp-fpm the pm.max_requests at 2000 should address the issue.

Now your fog server having only 2GB of memory may be amplifying the issue. If you run the top command how much free memory do you have?

Tom Elliott

@daniel-miller et al. Usertacking is useful for knowing who is logging on and off systems. Beyond that there is really not much value in it. I don’t mean it is unuseful, but many times now I’ve heard of the memory exhaustion issues and it’s related to user tracking. This is because every login/logout is recorded for every user. Imagine the number of entries created in one year. Compound that with many people having fog servers spanning many versions ( I’ve been doing this for about 4.5 years now) and you can imagine the shear number of entries. Now pho is decent but it must load the database for every request. It typically only takes the tables being requested at the time so user edit page may work without a problem, but goto host edit and it fails

If recommend pruning the usertracking table just for sanity reasons. The simplest way I can suggest is create a backup of your db and just start fresh with the usertracking table.The reason for the backup is you can load it to another database if you need it later on. Once you have the backup truncate the table. The command in mysql for this is: truncate table fog.usertracking

Hopefully this helps.

flipwalker

So I’m also having this problem. I think. I thought it was working fine after the update, having done several unicast images in the process of updating the image I use. However today when I started doing a push to 20 computers I’ve had numerous issues. Currently trying a couple of the recommended fix’s here to see if I can get back up and running.

flipwalker

So after reading the whole thread, I’m not entirely sure my issue is the same. I can’t find the log files for fpm. I don’t know if that’s because I’m being stupid(Not a pro with Linux by any stretch) or if my issue is unrelated. Pretty frustrated at the moment.

Tom Elliott

@flipwalker what Linux is are you using? It matters. Typically fpm logs are found in /var/log/php-fpm/www-error.log I believe or very close to that path

503 Service Unavailable Error

203

11.6k

17.1k

154.6k