503 Service Unavailable Error

[george1421]

@troye-johnson I’m not so concerned about the timezone messages, but the rest of them look pretty interesting. Your issue doesn’t seem to be a memory issue as with Greg’s. Your’s looks a bit more troubling.

[george1421]

@troye-johnson its in /etc/php.ini search for memory

The value looks like this:

memory_limit = 128M

Change it to 256M save it then reboot the fog server. I really don’t think your issue is memory since the log file doesn’t say anything about memory exhaustion.

[george1421]

@greg-plamondon Greg, if you still have memory exhaustion issues with my config I want you to change the memory allocation in the www.conf file. We need you guys to help us debug this issue.

I’m not getting a clear answer (my google-fu is weak today) on if php-fpm looks at the php.ini config file for its memory settings, or uses its own.

[Tom Elliott]

The fog settings overrides the php ini setting. So you can set whatever you want in the php.ini, but the fog settings overrides it. This is intentional. This way you can limit when memory usage should be expanded to only usage on fog.

[Troye Johnson]

changing the limit did not help still getting the 503 error.

[Tom Elliott]

https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast/3

Please try following this in regards to your 503 error.

The memory limit wouldn’t play an impact on the 503, it’d most likely do a 500 as it would crash.

[Greg Plamondon]

ok, I thought it was fixed for me but I still get the 503 when doploying an image.
If i refresh and look in tasks the task was created.

here is my httpd error.log

[Wed Apr 11 15:44:33.208616 2018] [proxy_fcgi:error] [pid 31003:tid 140654097147648] (70007)The timeout specified has expired: [client 192.168.10.170:50376] AH01075: Error dispatching request to :, referer: http://10fogserver.mtstrans.com/fog/management/index.php?node=host&sub=deploy&id=698&type=1

I dont want to Hijack @Troye-Johnson thread… Should a create a new one?

[george1421]

@tom-elliott said in 503 Service Unavailable Error:

The fog settings overrides the php ini setting.

This probably should be taken off line, but if I understand correctly php-fpm may not look or even care about the memory limts set on php. It may only pay attention to its own settings. I need do some more research tonight to see if I can get php-fpm to spit out its settings like we can do for …

[george1421]

@greg-plamondon what does /var/log/php-fpm/www-error.log say? I’m not seeing just yet what is causing the 503 error.

I don’t know if its apache not waiting for php-fpm to finish or php-fpm timing out before its able to deploy the image.

[Troye Johnson]

@tom-elliott

I am getting dependency errors when trying to do “yum install php-pecl-memcache”

[Greg Plamondon]

@george1421 said in 503 Service Unavailable Error:

/var/log/php-fpm/www-error.log

There are no new entries in /var/log/php-fpm/www-error.log

[Tom Elliott]

@greg-plamondon You said your 503 stuff went away after following the guide? Is this true, or the 503 returned?

[Greg Plamondon]

@tom-elliott said in 503 Service Unavailable Error:

@greg-plamondon You said your 503 stuff went away after following the guide? Is this true, or the 503 returned?

it went away for the most part, now I only get it when deploying an image.

[Tom Elliott]

@greg-plamondon Can you grab a video or pic of the error as it’s trying to get into a deploy task?

[Greg Plamondon]

@tom-elliott said in 503 Service Unavailable Error:

@greg-plamondon Can you grab a video or pic of the error as it’s trying to get into a deploy task?
Sure thing:

[george1421]

@greg-plamondon I spent a few hours working on this issue last night. I’m proud to report I can’t duplicate this error. That doesn’t really mean a hill of beans here. That just means my environment is a bit different than yours. My dev box only has a handful of clients and is on a pretty fast server. I do have a few ideas that I would like you guys to test. We do need php-fpm to work well to help improve web ui performance. The UI Developers are working on an optimized version of the new ui for 1.6.0 release, but that is a few months out yet.

What I would like both of you to do is:

1. Rerun the fog installer to reset everything back to a known configuration (hopefully).
2. Once the installer has completed I want both of you to modify the following file: /etc/httpd/conf.d/fog.conf
3. Insert the following section:

    <Proxy "fcgi://127.0.0.1:9000">
        ProxySet timeout=300
    </Proxy>

4. To make the file look like this:

<VirtualHost *:80>
    <Proxy "fcgi://127.0.0.1:9000">
        ProxySet timeout=300
    </Proxy>
    <FilesMatch "\.php$">
        SetHandler "proxy:fcgi://127.0.0.1:9000/"
    </FilesMatch>
    KeepAlive Off
    ServerName 192.168.100.1
    DocumentRoot /var/www/html/
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]
    <Directory /var/www/html/fog/>
        DirectoryIndex index.php index.html index.htm
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-d
        RewriteRule ^/(.*)$ /fog/api/index.php [QSA,L]
    </Directory>
</VirtualHost>

5. Save and exit the editor
6. Issue the following command to restart apache:
   systemctl restart httpd

This directive should tell apache to wait up to 5 minutes for php-fpm to complete its operations. (if it takes that long there is something else really wrong)

Now Greg, since you had a memory exhaustion issue I want you to do these additional setup.

1. Edit this file /etc/php-fpm.d/www.conf
2. Look for a line that reads:
   ;pm.max_requests = 500
3. Uncomment that line and change the parameter to 2000 to make it look like this:
   pm.max_requests = 2000
4. Save and exit the editor.
5. Restart php-fpm and apache.
   systemctl restart php-fpm
systemctl restart httpd

I think we actually have 2 issues here, and they are both relevant. Both are hitting the timeout issue, while Greg is hitting the next issue too.

Thank you guys for helping to work this out. I’m pretty sure we can get a resolution, if not we can always make 2 changes to apache and be back to the way 1.5.1 and earlier worked using the built in apache php engine.

[Greg Plamondon]

@george1421 said in 503 Service Unavailable Error:

/etc/php-fpm.d/www.conf

I dont have a /etc/php-fpm.d/www.conf
The only file in the php-fpm.d directory is fog.conf

[george1421]

@greg-plamondon Ok then you are using/still have my config in place. That is fine, change the line from 500 to 2000, save and restart both services. Hopefully the 500 value did address your memory exhaustion issue. This value is commented out in the FOG installed version.

Just make sure you don’t have two configuration files in the directory www.conf and fog.conf, you should be OK.

[Greg Plamondon]

@george1421 said in 503 Service Unavailable Error:

@greg-plamondon Ok then you are using/still have my config in place. That is fine, change the line from 500 to 2000, save and restart both services. Hopefully the 500 value did address your memory exhaustion issue. This value is commented out in the FOG installed version.

Just make sure you don’t have two configuration files in the directory www.conf and fog.conf, you should be OK.

That worked, I didn’t get the 503 error that time when imaging a PC!
Thanks! I will submit a new ticket for my other issues…

[george1421]

@greg-plamondon Thank you for the feedback, please keep (ab)using the FOG server. I’m interested in the durability of the configuration over time.

I am interested in what Troye’s experiences with just the timeout adjustment and to see if he does run into a memory exhaustion issue. I did see one other post from the github side that ran into the memory exhaustion issue and if I remember correctly he bumped his config to 3GB and still had the issue.