Fog HTTP completely unresponsive
-
I am no longer able to access the GUI or deploy images to machines.
I successfully captured a task this morning and have been using the GUI all day. I set up a group of machines, assigned them an image, and started a multicast. When they booted, they tftp booted okay, but http was not responding. When I tried to get to the GUI to stop the task and check things out, I could no longer reach the GUI. After a server reboot, it is still not working.
I’m unsure of what logs to look through to decipher what is happening.
-
@zpoling This is really strange. I would have expected just after this “Updating Database…” message to see a message about ftp login.
Lets assume is the issue we’ve found after FOG 1.5.4 has been released.
- Change to the /etc directory from the fog server linux command prompt.
- Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
find /etc -name www.conf(hopefully you will only find one) - Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except
php_admin_value[memory_limit] = 256Myou will need to add that entry.
php_admin_value[memory_limit] = 256M pm.max_requests = 2000 pm.max_children = 35 pm.min_spare_servers = 5 pm.start_servers = 5- Save and exit your text editor.
- Reboot the fog server.
- See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.
-
Just attempted to update Fog to 1.5.4 from 1.5.2. When instructed to install/update the database schema, I was able to get to the GUI. However, it simply took me to the login page rather than the database update page. After attempting to log in, it stopped loading and went back to the 504 error.
The install is stuck at “Ensuring node username and passwords match” which I assume has something to do with this https://wiki.fogproject.org/wiki/index.php/Troubleshoot_FTP#Credentials_.2F_Passwords. However, I have not changed any passwords and the server has been fine for months.
-
I left the server off during the weekend. I have seen issues before where tasks have been stuck and caused the server to do strange things. Last time all I did to fix it was unplug the damned ethernet cable and plugged it back in. This time I left the server off over the weekend. Came back in this morning, started the server up, and everything was working as expected. Very frustrating.
-
More oddities. I tried imaging one of the machines that I tried on Friday. Pxe boot pulled the correct IP, but then once Fog booted, it was trying to pull an IP from a different VLAN. I changed the switch port to the VLAN that it was trying to pull from, Pxe boot pulled the address that fog was attempting to pull before, and everything worked fine. Never had to have my machines on a specific VLAN before.
-
One machine is stuck after imaging at “Updating Database” and the original issue is occurring once again.
-
@zpoling I have no clue on the vlan bits. I know the FOS engine doesn’t natively support vlans. So this issue has to be related to your infrastructure for some reason. It does sound a bit strange at that.
One machine is stuck after imaging at “Updating Database” and the original issue is occurring once again
Is this only one machine out of X or the first machine you have imaged? Can you provide a picture of the error you see about updating the database?
What OS is the host OS for the fog server?
-
Quick thing about the VLANS. We added a voice VLAN recently, but the port was not set to that, nor was it trunked with it. I think it was just a goof because non of the other machines did it. Unless there’s some odd multicast issue due to adding a voice VLAN.
The server is Ubuntu Server 16.04 LTS
I originally said that only one machine was stuck here, but there were actually two of six stuck here. The others made it past this point then got stuck at PXE boot. The second image is what was happening back on Friday when I made this post.
After a few reboots of the server, I was able to login again. I never shut the machines off, so the machines that were stuck at “updating database” just retried over and over. Once the server came back up, they all were able to make it past this step and completed successfully.
-
@zpoling This is really strange. I would have expected just after this “Updating Database…” message to see a message about ftp login.
Lets assume is the issue we’ve found after FOG 1.5.4 has been released.
- Change to the /etc directory from the fog server linux command prompt.
- Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
find /etc -name www.conf(hopefully you will only find one) - Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except
php_admin_value[memory_limit] = 256Myou will need to add that entry.
php_admin_value[memory_limit] = 256M pm.max_requests = 2000 pm.max_children = 35 pm.min_spare_servers = 5 pm.start_servers = 5- Save and exit your text editor.
- Reboot the fog server.
- See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.
-
@george1421 Done. The memory_limit line already existed, but was set to 32M. The other lines existed as well, but were set to lower numbers. pm.max_requests was the only correct one.
However, this was happening before I updated to 1.5.4. Or did the issue exist before then as well?
9am-5pm Eastern | Monday-Friday
-
I now can’t reach http://fogserver/fog. I get a 503 error. I am able to reach http://fogserver and /html/
Should I change the settings back?
Edit: I changed pm.min_spare_servers to 5.
pm.max_spare_servers was set to 3. I changed it to 5 and it began working again.
-
@zpoling You only found one instance of the www.conf file?
Also the 32MB one might have existed, but it was commented out. Just be aware of that.
And the last tweak to fog 1.5.4 is that you need to downgrade the FOG kernel to 4.15.2 from 4.17.0.
The reason we are going through these steps is to ensure we have your FOG server configured to a know good state now that you are on fog 1.5.4. The error messages you have posted so far could be either php or mysql related.
-
@george1421 Yes, there was only one www.conf file.
I also have to apologize. I have no idea where to begin to downgrade the kernel. I’ve cringed every time I ever see kernels mentioned because I have no clue what it even is.
-
@george1421 Okay I think I found what you were requesting. I installed 4.15.2 x64 and x32, but I’m not sure what to do now. I assume I have to tell Fog to use one of those specifically rather than the most current?
-
@zpoling If you go into fog settings -> fog kernel locate the PC versions of the 4.15.2 kernel and select the download button. That should update (down grade) your versions of bzImage and bzImage32 to the older (yet working) release.
-
I has a similar issue when we had sent to about 30 machines. While it shouldn’t make a difference as we only let 10 image concurrently, it did. When I edited the php.ini file in /etc/php/7.1/apache2 and changed the memory_limit from 128M to 256M, it worked as soon as I rebooted.
I am on Ubuntu 16.04 LTS and FOG 1.5.4
-
@nt_tech said in Fog HTTP completely unresponsive:
When I edited the php.ini file in /etc/php/7.1/apache2
This is interesting. From what I found, that php-fpm was to ignore what was in php.ini and take its config file settings. But if this method works too, that is fine. It is a bit annoying that now each version of php the .ini file moves about. I can understand why they did it that way, its still annoying none the less.
-
@george1421 Fog has been working, but I haven’t had any machines to test a multicast on since we last spoke. I am going to attempt to image a classroom today and will report back.
