FOG 1.5.4 Adding Snapin Not Working

pietroaretino

I just built a new FOG server from scratch running the latest version.

All is working fine except I am unable to add any snapins.

When I finish creating a new snapin and hit “add” nothing happens.

FOG sits idle, however it seems as though its attempting to run some processes when I check on the back-end.

I looked in the apache error logs and found the following:

[proxy_fcgi:error] [pid 595] (70008)Partial results are valid but processing is incomplete: [client 10.2.16.231:57427] AH01075: Error dispatching request to : (reading input brigade), referer: http://10.2.17.6/fog/management/index.php?node=snapin&sub=add

I’m not sure what this error means, however I do know its related to me attempting to add a snapin as you can see in the URL “node=snapin&sub=add”.

Tried simply rebooting the entire server, restarting apache and also trying different browsers, but the add doesn’t seem to work.

Could this be a permission issue?

george1421

Lets assume is the issue we’ve found after FOG 1.5.4 has been released.

Change to the /etc directory from the fog server linux command prompt.
Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
find /etc -name www.conf (hopefully you will only find one)
Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except php_admin_value[memory_limit] = 256M you will need to add that entry.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 35
pm.min_spare_servers = 5
pm.start_servers = 5

Save and exit your text editor.
Reboot the fog server.
See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.

Also we found there is something strange going on in the linux kernels after 4.15.2, I’m going to recommend that you downgrade your FOG/FOS kernel to 4.15.2. The issue with later kernels is that its taking 3-5 minutes to create the disk structure under certain circumstances, where with 4.15.2 and older its only seconds to create the structure.

Now the kernel will not impact your issue, but processing is incomplete might be related to the missing php-fpm configuration setting.

pietroaretino

@george1421

Hello George, thank you for the quick response.

I changed all the variables to the numbers you suggested, however it seems it is unable to handle it.

After rebooting when I attempt to go to my FOG server the web interface gives me this error:

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.4.25 (Debian) Server at 10.2.17.6 Port 80

Checked on the apache error log and had a list full of these errors, just the process id changed:

[Wed Jul 25 12:01:42.112292 2018] [proxy:error] [pid 1560] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Jul 25 12:01:42.112346 2018] [proxy_fcgi:error] [pid 1560] [client 10.2.17.6:47948] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Jul 25 12:01:42.120322 2018] [proxy:error] [pid 1556] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Jul 25 12:01:42.120368 2018] [proxy_fcgi:error] [pid 1556] [client 10.2.17.6:47952] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Jul 25 12:01:42.120874 2018] [proxy:error] [pid 1559] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Jul 25 12:01:42.120893 2018] [proxy_fcgi:error] [pid 1559] [client 10.2.17.6:47956] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Jul 25 12:01:52.029425 2018] [proxy:error] [pid 1558] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Jul 25 12:01:52.029486 2018] [proxy_fcgi:error] [pid 1558] [client 10.2.17.6:47962] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Jul 25 12:01:52.031208 2018] [proxy:error] [pid 1557] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Jul 25 12:01:52.031229 2018] [proxy_fcgi:error] [pid 1557] [client 10.2.17.6:47966] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Jul 25 12:01:52.031732 2018] [proxy:error] [pid 1560] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Jul 25 12:01:52.031751 2018] [proxy_fcgi:error] [pid 1560] [client 10.2.17.6:47970] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Jul 25 12:01:52.042110 2018] [proxy:error] [pid 1556] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Jul 25 12:01:52.042195 2018] [proxy_fcgi:error] [pid 1556] [client 10.2.17.6:47976] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Jul 25 12:01:52.045883 2018] [proxy:error] [pid 1559] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Jul 25 12:01:52.045908 2018] [proxy_fcgi:error] [pid 1559] [client 10.2.17.6:47980] AH01079: failed to make connection to backend: 127.0.0.1
[Wed Jul 25 12:01:52.046400 2018] [proxy:error] [pid 1558] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed
[Wed Jul 25 12:01:52.046417 2018] [proxy_fcgi:error] [pid 1558] [client 10.2.17.6:47984] AH01079: failed to make connection to backend: 127.0.0.1

I am running: Linux gafog 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3+deb9u1 (2017-12-23) x86_64 GNU/Linux

I will try downgrading the kernel.

george1421

@pietroaretino Double check the settings for php-fpm, it sounds like its not running. Don’t just blindly copy these setting in there. If there are duplicate any setting. If you look at /var/log there should be a php-fpm directory with an error log in it.

You can check to see if php-fpm is running by keying in ps aux|grep php-fpm or by running top and sort by 'P’rocess (CPU). You should see 3 or 4 top processes being php-fpm.

The kernel I’m referring to is the FOS image (bzImage) that gets downloaded to the target computer to capture and deploy images. This is not the host FOG server’s kernel.

Tom Elliott

@pietroaretino is selinux by chance running? sestatus should tell you permissive or disabled.

pietroaretino

@george1421

Thanks George,

I did not duplicate the variable entries, I simply changed their values to the ones you suggested in your previous post. It seems the values you suggested for the pm.* variables were too high. I reset them back to their previous values, rebooted and I am now able to get back into my fog server no problem. The php_admin_value, I was able to keep at 256M as you suggested, it initially was a 36M.

I utilized the browser-GUI to downgrade to the older kernel, however when I check I see this info:

bzImage Version: 4.15.2
bzImage32 Version: 4.16.6

Should the bzImage32 also be on 4.15.2?
Can that also be done through the GUI?

Regardless, still unable to add snapins.

Got this when grepping for php-fpm:

root       442  0.0  0.5 370876 36376 ?        Ss   12:02   0:00 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)
www-data  3324  1.7  0.2 371380 15948 ?        S    12:22   0:01 php-fpm: pool www
www-data  3414  7.0  0.3 371492 19784 ?        S    12:22   0:02 php-fpm: pool www
www-data  3480  6.9  0.3 371492 19776 ?        S    12:22   0:02 php-fpm: pool www
www-data  3667  3.1  0.3 371492 19608 ?        S    12:23   0:00 php-fpm: pool www

Here is my FPM log:

[25-Jul-2018 11:51:26] NOTICE: Terminating ...
[25-Jul-2018 11:51:26] NOTICE: exiting, bye-bye!
[25-Jul-2018 11:51:55] NOTICE: fpm is running, pid 444
[25-Jul-2018 11:51:55] NOTICE: ready to handle connections
[25-Jul-2018 11:51:55] NOTICE: systemd monitor interval set to 10000ms
[25-Jul-2018 11:52:16] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[25-Jul-2018 11:55:44] NOTICE: Terminating ...
[25-Jul-2018 11:55:44] NOTICE: exiting, bye-bye!
[25-Jul-2018 11:56:12] ALERT: [pool www] pm.max_spare_servers(3) must not be less than pm.min_spare_servers(5)
[25-Jul-2018 11:56:12] ERROR: failed to post process the configuration
[25-Jul-2018 11:56:12] ERROR: FPM initialization failed
[25-Jul-2018 12:02:25] NOTICE: fpm is running, pid 442
[25-Jul-2018 12:02:25] NOTICE: ready to handle connections
[25-Jul-2018 12:02:25] NOTICE: systemd monitor interval set to 10000ms
[25-Jul-2018 12:02:34] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[25-Jul-2018 12:02:46] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[25-Jul-2018 12:23:04] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

This is running as a VM on a Windows Hyper-V server. Don’t know if that would impact it, however I had the version 4.* of FOG running on it and it always worked well.

I suppose I should listen to the log and attempt raising the pm.max_children variable.

Also, don’t know if this has anything to do with the same error, but the “Search” field and page that should load on the Host Management, Image Management and Snapin Management menus do not load, I see them for a split second and then they disappear.
0_1532546873720_6735395c-374e-41f3-aedb-30620a83afba-image.png image url)

pietroaretino

@tom-elliott

Hello Tom,

No I am running Debian 9-Core it does not come pre-installed, however I ran the command just in case and it cannot find the command, so no selinux.

pietroaretino

Seems to be an issue with Snapin “packs”. When I tested a fake upload of a “Normal Snapin” it took the file and uploaded it and created it.

I’m just trying to upload a zip file of MS-Office 2016 install. I had done it on previous versions of FOG on version 1.4.X and it worked like a charm.

Fiddled with the various fpm-php entries and can’t seem to find any error log that is explaining what the hell it wants or why its upset.

Though I do know that when I try to add the snapin-pack, if I click “add” button twice, the load average on my VM goes up pretty high and theres lag even in typing through an SSH session.

The only error I consistently get when attempting a “snapin pack” upload is in the apache error log, as follows:

[proxy_fcgi:error] [pid 624] (70008)Partial results are valid but processing is incomplete: [client 10.2.16.231:62166] AH01075: Error dispatching request to : (reading input brigade), referer: http://10.2.17.6/fog/management/index.php?node=snapin&sub=add

Also, I completely rebuilt the server from scratch, again. Still running version 1.5.4, on a fresh clean install of Debian 9 core.

I have /images, /opt and / on separate partitions:

/ = 20GB partition (bootable)
/opt = 30GB partition
/images = 1TB partition
swap = 4GB
Total RAM = 8GB

Sebastian Roth

@pietroaretino Sorry for the late reply. We’ve lost track of this issue. Let’s see if we can work this out for you.

I’m just trying to upload a zip file of MS-Office 2016 install.

I am wondering if it’s just file size issue? I have just installed FOG 1.5.5.3 on Debian an created a couple of SnapinPacks. First smaller ones than bigger ones. All working great up to 250 MB. So I can’t replicate the issue as you describe it.

Would you mind updating to FOG 1.5.5 or even the dev-branch (currently 1.5.5.3) to see if you still see the issue? We’ve adjusted some of the php-fpm config values since then. So maybe this is fixed for you already.

FOG 1.5.4 Adding Snapin Not Working

314

11.8k

17.2k

154.9k