503 Service Unavailable Error
-
@costas Will you reset the memory limits back to FOG’s default value.
Then lets inspect
/var/log/php-fpm/www-error.logfor a memory exhaustion warning message like Greg’s below. If that is found then lets make the adjustment that appears to have fixed Greg’s issue.- Edit this file
/etc/php-fpm.d/www.conf - Look for a line that reads:
;pm.max_requests = 500 - Uncomment that line and change the parameter to 2000 to make it look like this:
pm.max_requests = 2000 - Save and exit the editor.
- Restart php-fpm and apache.
systemctl restart php-fpm
systemctl restart httpd
- Edit this file
-
@george1421 Hey George.
My www-error.log is not very useful.
Only has one line in it.
“[15-Apr-2018 03:45:01] NOTICE: error log file re-opened”
just to be cleat the error I get when running a user tracking report in chrome is
“This page isn’t working
10.214.14.27 is currently unable to handle this request.
HTTP ERROR 500”
should i still go ahead and perform the changes you asked for? -
Don’t know if this relevant to this issue (if not, let me know and I"ll open another thread), but I’m encountering http 500 responses on the user tracking report page with similar log entries on an ubuntu based system I just upgraded from 1.4.4 to 1.5.2 from a git pull off an existing checkout.
After changing the the pm.max_request setting in /etc/php/7.1/fpm/pool.d/www.conf and changing the memory limit to 2048 (the box only has 2GB), I see a warning about the server reaching the pm.max_children value in /var/log/php7.1-fpm.log followed by a proxy_fcgi:error in /var/log/apache2/error.log citing an out of memory condition in fogcontroller.class.php with referrer /fog/management/index.php?node=report&sub=file&f=dXNlciB0cmFja2luZw==. There is a reference to a line number for the oom error that seems to vary between lines 126 (in __construct()) and 260 (in set()), but I would expect that depends on when it actually ran out. I enabled error reporting in the php settings for apache, but it doesn’t seem to want to get past the 500.
Raising the max_children to 20 shows entries in php7.1-fpm.log about the pool seeming busy and it spawning additional children until the limit is reached then it reports the reaching of the max_children limit again. The proxy_fcgi:error in this case is a timeout dispatching a request to (polling) with the same referrer.
Scaling the memory limit back down to default results in a reference to pdodb.class.php line 602 in the memory exhaustion message.
I also attempted altering the apache configuration, but I only had to add the ProxySet timeout. No change in the error message.
Happy to poke at anything else you might have an idea about. Have a few days before I need to decide to backrev or not.
-
@daniel-miller How many fog clients do you have in your environment?
You shouldn’t need to change the memory limit in pgp-fpm the
pm.max_requestsat 2000 should address the issue.Now your fog server having only 2GB of memory may be amplifying the issue. If you run the
topcommand how much free memory do you have? -
@daniel-miller et al. Usertacking is useful for knowing who is logging on and off systems. Beyond that there is really not much value in it. I don’t mean it is unuseful, but many times now I’ve heard of the memory exhaustion issues and it’s related to user tracking. This is because every login/logout is recorded for every user. Imagine the number of entries created in one year. Compound that with many people having fog servers spanning many versions ( I’ve been doing this for about 4.5 years now) and you can imagine the shear number of entries. Now pho is decent but it must load the database for every request. It typically only takes the tables being requested at the time so user edit page may work without a problem, but goto host edit and it fails
If recommend pruning the usertracking table just for sanity reasons. The simplest way I can suggest is create a backup of your db and just start fresh with the usertracking table.The reason for the backup is you can load it to another database if you need it later on. Once you have the backup truncate the table. The command in mysql for this is:
truncate table fog.usertrackingHopefully this helps.
-
So I’m also having this problem. I think. I thought it was working fine after the update, having done several unicast images in the process of updating the image I use. However today when I started doing a push to 20 computers I’ve had numerous issues. Currently trying a couple of the recommended fix’s here to see if I can get back up and running.
-
So after reading the whole thread, I’m not entirely sure my issue is the same. I can’t find the log files for fpm. I don’t know if that’s because I’m being stupid(Not a pro with Linux by any stretch) or if my issue is unrelated. Pretty frustrated at the moment.
-
@flipwalker what Linux is are you using? It matters. Typically fpm logs are found in /var/log/php-fpm/www-error.log I believe or very close to that path
-
@tom-elliott Ubuntu 16.04. In var/log/ I have a file php7.1-fpm.log, but there’s no php-fpm directory, and no www-error.log that I can find.
-
@tom-elliott Alright. Some progress. I found the step I missed. I assumed that fpm was default, but I see now that @george1421 had people follow his steps to get it installed. I’ve got it installed now and when I restarted apache and php as part of that process magically everything is working again. For the moment. But now I sit and watch computers image for the next 5+ hours to see if they start erroring again.
-
@flipwalker Just for clarity, can you confirm the steps you took to resolve (hopefully the issue)? I provided several steps here trying this and that to find a common thread.
-
@george1421 I originally didn’t follow your guide to install and activate fpm, I had assume it was default. Once I followed the guide(not without cursing and much looking up of VIM guides), and got it installed and activated my web interface came back up and my clients waiting to image started communicating with FOG again.
I’m still not 100% that it’s fixed or if the last step(restarting PHP and Apache) simply reset the problem and it’s going to reappear in a few hours. I’ve been watching it run for 45 mins or so now and it at least appears to be stable. But we’ll see if it completes overnight or not.
Just wish I had remote capability so I could monitor from home instead of having to come to the office.
Philip
-
@george1421 This particular machine is only servicing about 30 units with 4 distinct named users and currently reporting about 375MB free. Looking through its history over the past 2 years (reported through snmp), it historically has had anywhere from 350MB to 500MB free, depending on the day of the week. Truncating the userTracking table does make it happy for now. If you are loading that table with PHP on every request, the behavior does make a lot of sense.
@Tom-Elliott I completely understand that, and for this particular environment it is not an important feature. We have some other larger environments (250-ish) where the user tracking feature does get used from time to time (“Hey, was the account of student X used to log in yesterday?”), so It is something I check in my canary system.
@flipwalker The box I have is also Ubuntu 16.04. Check to see if your config file is in /etc/php/7.1/fpm/pool.d/.
-
Not sure if this is related I have an older post where my web ui stopped working but my error logs are just
non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109 and 110. -
Hi, it sounds like the same issue I have had.
My solusion was to remove the www.conf in the /etc/php-fpm.d directory.
this directory must contane one file only (fog.conf) -
@pijltje said in 503 Service Unavailable Error:
Hi, it sounds like the same issue I have had.
My solusion was to remove the www.conf in the /etc/php-fpm.d directory.
this directory must contane one file only (fog.conf)FWIW, my instructions say to use fog.conf, the fog installer created the www.conf. So the fog installer owns www.conf and will be created each time the installer is run. You should remove the fog.conf file since that was hand created or you will have problems each time you upgrade fog.
-
@pijltje Hmm I don’t see that directory I am using Ubuntu (I know I know, first time fog when I have time I WILL switch been a pain honestly)
