Solved Slow Fog 1.5.X WebUI due to Storage Node

[ElNatcho]

Hello,
I recently installed fog 1.5.1 on a Debian 9.4 system. Until then I used FOG 1.2.0.
Everything worked fine until I added a Windows Storage Node. Since then the
WebUI is incredible slow (Especially when I log in or when I open the Storage-Tab. Most of the time the login page will time out).

I found some forum entries and articles which suggest that something is wrong
with the php Code which accesses the storage node but I didn’t find a fix.
I also tried downgrading to 1.4.4. That solved the speed issue but brought overall more problems than solutions.

So can anyone imagine what the problem is and how I could fix it?

Any help would be greatly appreciated and thanks in advanced.

[Sebastian Roth]

@ElNatcho We are working on this issue. Just in case you are still around. I pushed some major changes to the dev-branch a few hours ago. Should fix the flow web UI issues altogether. Please give it a try if you can and let us know.

[ElNatcho]

@george1421
Fog failed to connect to the backend (FCGI: attempt to connect to 127.0.0.1:9000 (*) failed)

[george1421]

@elnatcho OK I think I remember what I did on the commute into the office.

First lets start by having you revert all of the changes you’ve made so far to the php7.0.conf files.

Now lets go back into the mods-enabled/php7.0.conf file and edit it.

Locate the line that has SetHandler application/x-httpd-php in it and place a hashtag in front to make it look like this:

#SetHandler application/x-httpd-php
SetHandler "proxy:fcgi://127.0.0.1:9000"

Now paste in the other handler for php-fpm just below it. To make it look like above.
Then restart apache2 with: sudo systemctl restart apache2

That should fix the issue.

[ElNatcho]

@george1421
I tried both (commenting only “SetHandler appplication/x-httpd-php” out and commenting both out)

[george1421]

@elnatcho said in Slow Fog 1.5.X WebUI due to Storage Node:

mods-enabled/php7.0.conf: SetHandler application/x-httpd-php

This is the only one I would have commented out. That is / was the enabled handler.

Please review the FOG Forum chat bubble for additional information.

[ElNatcho]

@george1421
The grep command gave the following output:

mods-available/php7.0.conf:    SetHandler application/x-httpd-php
mods-available/php7.0.conf:    SetHandler application/x-httpd-php-source
mods-enabled/php7.0.conf:    SetHandler application/x-httpd-php
mods-enabled/php7.0.conf:    SetHandler application/x-httpd-php-source

I’ve commented all of them out and now only the “php -q …” processes are on top of top (and logically the login page won’t display properly any more)

[george1421]

@elnatcho OK great, now we are down to the differences with debian 9.4. I found the solution once before but I can’t remember exactly what I found.

I think it was that Debian 9.4 had the handler for php listed in a different location than other distributions. Lets see if we can find it.

1. From the fog server linux conssole navigate to /etc/apache2
   cd /etc/apache2
2. Lets look for the php handler
   grep -R -e "SetHandler application/x-httpd-php"
3. If that returns a value then edit that file and place a comment marker (pound or hashtag) in front of that line to make it look like this
   #SetHandler application/x-httpd-php
4. Save and exit the editor
5. Restart apache
   systemctl restart apache

When you run top you should see php-fpm at the top 2 or 3 processes. If this is the case the system is configured correctly.

[ElNatcho]

@george1421
I doubled the “pm.” entrys by mistake and after deleting the surplus entries the service started without errors. But now even accessing the login page is very slow. My current settings are:

php_admin_value[memory_limit] = 256M
pm.max_requests = 0
pm.max_children = 5
pm.min_spare_servers = 1
pm.start_servers = 2

I also tried altering all setting to your values and then I could access the login page normally again. But the login still wont work

[george1421]

@elnatcho Make sure you don’t have 2 entries of the same (like you just copied and pastes my text in). The only new value should have been php_admin_value[memory_limit] = 256M The rest of the values should have been just check on the current setting.

[ElNatcho]

@george1421
I’ve added your lines to the php-fpm but after that the php7.0-fpm.service wont start at all (code=exited status=78).

[george1421]

@elnatcho Based on what you just posted, I’m still of a mind to say that you don’t have php-fpm configured correctly (not your fault, but the fog installer’s fault) on your fog server.

I did find a reference to this: https://forums.fogproject.org/topic/12057/fog-unresponsive-under-heavy-load/30 but it appears I did not provide a solution.

Once we find out why php-fpm isn’t being loaded correctly, we need to update a few settings that were realized after FOG 1.5.4 was released.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 35
pm.min_spare_servers = 5
pm.start_servers = 5

[ElNatcho]

@george1421
I looked up the CPU usage and it appears that apache is on top of the process list often but not always. php-fpm is shown but needs very little processing power.
I noticed that while logging in apache is on the top the most of the time. But I also see “/usr/bin/php -q /opt/fog/service/<fogstuff>” very often.

[george1421]

@elnatcho said in Slow Fog 1.5.X WebUI due to Storage Node:

I noticed the slow UI I upgraded it to 1.5.4.

This is typically an indication that php-fpm is not working correctly. php-fpm was added to fog to address the slow ui and to prepare the system for when FOG 1.6 is released in the future.

[ElNatcho]

Hey george1421,
sorry for my late reply but I wasn’t able to access the fog server in the last week and won’t be able to access it in the next two days. But as soon as I can I’ll test if the php-fpm is the alleged cause of my problem.
One thing which I forgot to mention is that after I noticed the slow UI I upgraded it to 1.5.4.

[george1421]

Why did you install fog 1.5.1 instead of 1.5.4?

I can say I think there is an issue with Debian 9.4 installer for FOG that does not create the config file properly for php-fpm. If this is the case you will have a very slow UI in 1.5.2 and later version of FOG.

There is a quick test to see if php-fpm is configured correctly. Open a linux console and run top then press P to sort by CPU usage. You should have php-fpm at the top of the processes. There should be 3 or 4 instances running. If apache is at the top then php-fpm is not configured correctly on your FOG server.

There are a few tweaks needed to FOG 1.5.4 that was discovered after 1.5.4 was released. Once we are sure that php-fpm is running then we can look at those settings.