Web GUI error after clicking List Hosts

tesparza

I get this error after clicking List Hosts it hangs for a longtime then that error appears. We are nearing 2000+ hosts. Im guessing it a capacity issues. An ideas where to look?

george1421

@tesparza There is a short answer and a bit longer and messy one.

Lets first check the apache error logs in /var/logs/apache2 directory. I’m going to suspect you will see a php-fpm timeout.

Then look at the linux top utility and sort by Processor

If your top task is mysql at about 20% or more and you see many (many) php-fpm task below. Lets stop here and let me know.

tesparza

@george1421 mysql is running at 70.5 CPU Usage.

[09-Nov-2021 17:28:55] WARNING: [pool www] server reached pm.max_children setting (50), consider raising it

php-fpm error.log file

tesparza

@george1421
In /etc/php-fpm.d/www.conf:

pm.max_requests = 500

This worked.

george1421

@tesparza said in Web GUI error after clicking List Hosts:

pm.max_requests = 500

You can do this but it doesn’t fix the root of the problem. What we need to do is modify your database configuration. What’s happening is the database can’t keep up with the requests so the php-fpm requests are being queued up. The workers are being depleted and eventually apache will start timing out. Setting the max requests to 500 only forces the children workers to restart after 500 requests. Restarting the workers just reboots them and discards anything they were working on like request from your FOG clients.

tesparza

@george1421 Okay how can I fix my database issue?

george1421

@tesparza I really need to write up a step by step on addressing this. I was working (recently) with a forum member on doing this and he ghosted me so I stopped.

The first we need to do is adjust a few mysql/mariadb database parameters because a few of the FOG default values will cause the migration to fail if these new settings exist with the innodb engine.

So first log into the mysql client program as root. Then run this command and post the results. Please post the text not a picture of the results because we are going to remove a few parameters.
SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session;

The output should look like this:

mysql> SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session;
+-------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| global                                                                                                                                    | session                                                                                                                                   |
+-------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+-------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

Once we have that we’ll go into the next step.

tesparza

@george1421 The command is not returning anything. Its blank Global and Session boxes

tesparza

@george1421 I copy pasted the command but its not giving the results. Im running CentOS and MariaDB

george1421

@tesparza Ok then you have an older version of mariadb so we can just keep moving forward with the migration.

I’m referencing this post: https://forums.fogproject.org/post/133490

At this point we need to temporarily stop apache so we don’t have any transactions going on when you run the migration command. Do this with: systemctl stop httpd on centos.

The next steps are to run this command via the mysql cli program:

SELECT TABLE_NAME,ENGINE
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'fog' and ENGINE = 'MyISAM';

The output should show that all tables are in MyISAM format.

Now paste this section into the cli. You might want to do it a few lines at a time in case there is an error message.

USE FOG
ALTER TABLE LDAPServers ENGINE=InnoDB;
ALTER TABLE clientUpdates ENGINE=InnoDB;
ALTER TABLE dirCleaner ENGINE=InnoDB;
ALTER TABLE globalSettings ENGINE=InnoDB;
ALTER TABLE greenFog ENGINE=InnoDB;
ALTER TABLE groupMembers ENGINE=InnoDB;
ALTER TABLE groups ENGINE=InnoDB;
ALTER TABLE history ENGINE=InnoDB;
ALTER TABLE hookEvents ENGINE=InnoDB;
ALTER TABLE hostAutoLogOut ENGINE=InnoDB;
ALTER TABLE hostMAC ENGINE=InnoDB;
ALTER TABLE hostScreenSettings ENGINE=InnoDB;
ALTER TABLE hosts ENGINE=InnoDB;
ALTER TABLE imageGroupAssoc ENGINE=InnoDB;
ALTER TABLE imagePartitionTypes ENGINE=InnoDB;
ALTER TABLE imageTypes ENGINE=InnoDB;
ALTER TABLE images ENGINE=InnoDB;
ALTER TABLE imagingLog ENGINE=InnoDB;
ALTER TABLE inventory ENGINE=InnoDB;
ALTER TABLE ipxeTable ENGINE=InnoDB;
ALTER TABLE keySequence ENGINE=InnoDB;
ALTER TABLE location ENGINE=InnoDB;
ALTER TABLE locationAssoc ENGINE=InnoDB;
ALTER TABLE moduleStatusByHost ENGINE=InnoDB;
ALTER TABLE modules ENGINE=InnoDB;
ALTER TABLE multicastSessions ENGINE=InnoDB;
ALTER TABLE multicastSessionsAssoc ENGINE=InnoDB;
ALTER TABLE nfsFailures ENGINE=InnoDB;
ALTER TABLE nfsGroupMembers ENGINE=InnoDB;
ALTER TABLE nfsGroups ENGINE=InnoDB;
ALTER TABLE notifyEvents ENGINE=InnoDB;
ALTER TABLE os ENGINE=InnoDB;
ALTER TABLE oui ENGINE=InnoDB;
ALTER TABLE plugins ENGINE=InnoDB;
ALTER TABLE powerManagement ENGINE=InnoDB;
ALTER TABLE printerAssoc ENGINE=InnoDB;
ALTER TABLE printers ENGINE=InnoDB;
ALTER TABLE pxeMenu ENGINE=InnoDB;
ALTER TABLE scheduledTasks ENGINE=InnoDB;
ALTER TABLE schemaVersion ENGINE=InnoDB;
ALTER TABLE snapinAssoc ENGINE=InnoDB;
ALTER TABLE snapinGroupAssoc ENGINE=InnoDB;
ALTER TABLE snapinJobs ENGINE=InnoDB;
ALTER TABLE snapinTasks ENGINE=InnoDB;
ALTER TABLE snapins ENGINE=InnoDB;
ALTER TABLE supportedOS ENGINE=InnoDB;
ALTER TABLE taskLog ENGINE=InnoDB;
ALTER TABLE taskStates ENGINE=InnoDB;
ALTER TABLE taskTypes ENGINE=InnoDB;
ALTER TABLE tasks ENGINE=InnoDB;
ALTER TABLE userCleanup ENGINE=InnoDB;
ALTER TABLE userTracking ENGINE=InnoDB;
ALTER TABLE users ENGINE=InnoDB;
ALTER TABLE virus ENGINE=InnoDB;
ALTER TABLE wolbroadcast ENGINE=InnoDB;

And the end rerun this command

SELECT TABLE_NAME,ENGINE
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'fog';

You should now see all of the tables in innodb format. If you get an error message around an invalid date format then we will need to dig a bit deeper. I think on the debian based systems there was 3 possible tables. When I first did this on centos it worked correctly.

Now that all of the tables have been updated, exit the mysql cli and restart apache. systemctl start httpd

That should be it. Monitor top for about 10-20 minutes. You should not see 70% cpu on mysql again. Monitor for the next few hours and everything stay normal.

tesparza

@george1421 Okay I did the commands, some did error out saying they dont exist but this are the results of the final command. Website seems to be running smoother. Will continue to monitor.

+------------------------+--------+
| TABLE_NAME             | ENGINE |
+------------------------+--------+
| clientUpdates          | InnoDB |
| dirCleaner             | InnoDB |
| fileChecksums          | MyISAM |
| globalSettings         | InnoDB |
| greenFog               | InnoDB |
| groupMembers           | InnoDB |
| groups                 | InnoDB |
| history                | InnoDB |
| hookEvents             | InnoDB |
| hostAutoLogOut         | InnoDB |
| hostMAC                | InnoDB |
| hostScreenSettings     | InnoDB |
| hosts                  | InnoDB |
| hoststatus             | MyISAM |
| imageGroupAssoc        | InnoDB |
| imagePartitionTypes    | InnoDB |
| imageTypes             | InnoDB |
| images                 | InnoDB |
| imagingLog             | InnoDB |
| inventory              | InnoDB |
| ipxeTable              | InnoDB |
| keySequence            | InnoDB |
| location               | InnoDB |
| locationAssoc          | InnoDB |
| moduleStatusByHost     | InnoDB |
| modules                | InnoDB |
| multicastSessions      | InnoDB |
| multicastSessionsAssoc | InnoDB |
| nfsFailures            | InnoDB |
| nfsGroupMembers        | InnoDB |
| nfsGroups              | InnoDB |
| notifyEvents           | InnoDB |
| os                     | InnoDB |
| oui                    | InnoDB |
| plugins                | InnoDB |
| powerManagement        | InnoDB |
| printerAssoc           | InnoDB |
| printers               | InnoDB |
| pxeMenu                | InnoDB |
| scheduledTasks         | InnoDB |
| schemaVersion          | InnoDB |
| snapinAssoc            | InnoDB |
| snapinGroupAssoc       | InnoDB |
| snapinJobs             | InnoDB |
| snapinTasks            | InnoDB |
| snapins                | InnoDB |
| supportedOS            | InnoDB |
| taskLog                | InnoDB |
| taskStates             | InnoDB |
| taskTypes              | InnoDB |
| tasks                  | InnoDB |
| userCleanup            | InnoDB |
| userTracking           | InnoDB |
| users                  | InnoDB |
| virus                  | InnoDB |
+------------------------+--------+
55 rows in set (0.00 sec)

george1421

@tesparza OK I see two tables either I missed or there was an error converting.

Can you run these commands

use fog
ALTER TABLE fileChecksums ENGINE=InnoDB;
ALTER TABLE hoststatus ENGINE=InnoDB;

You can do this now or later. You got the big tables that cause resource contention already, these are just for completeness.

I’m glad the innodb switch has settled down your FOG server, lets see how it runs for the next day or so to see if we need to tune it a bit more.

tesparza

@george1421 Okay now everything is InnoDB, Server is running bad, web gui taking long time to load, can’t image. CPU usage is around 40%

FYI this is running of a Optiplex 7010 i5 quad core, 8gb ram

Do I maybe need to switch to a server?

george1421

@tesparza Ok the innodb engine is much more efficient than myisam, so the effort is not wasted. It will never be worse with innodb format.

So what is your top process now? Post a screen shot of top sorted by Process.

How many computers with FOG clients?
What is your client check in interval? Still the default of 5 seconds?

Don’t change anything lets see where we are at first.

tesparza

@george1421

2000 + clients

Interval checkin time it 60 seconds

george1421

@tesparza For testing lets change the check in interval to 300 seconds (5 minutes) and then lets wait 15 minutes to see if things settle down. This isn’t the answer at the moment, its just to see if having 2000 hosts hit (if we lineralize it, 34 per second) is a bit too much for your hardware.

The screen shot you shown is indicative of what a myisam engine configuration will give us. We know we already fixed that.

Looking at top, it also appears you only have 4GB of ram. Can you do a uname -a and post the results here?

tesparza

@george1421 Okay interval changed to 300

george1421

@tesparza well that’s a good thing you have a 64 bit OS. Since you are telling me 8GB is installed but top is reporting only 4GB then I was thinking a 32 bit OS.

If the 300 second check in doesn’t settle things down, then please confirm you have 8GB installed. Since is a 7010 you might be able to find some old memory to top it out at 16GB. I know I have at least 10 sticks of 4GB 7010 memory in the bottom of my junk bin. I’m hoping you have similar.

Again when things settle down a bit we’ll check also disk performance. Right now the sql sever is having a high cpu load. This could be either disk, or not enough free ram for caching to happen. We need to make small changes to see where things pick up the most.

tesparza

@george1421 Okay it seems to be running smoother, I will add more RAM, eventually I will transfer to a server.

george1421

@tesparza I made a post a few years ago to test different hardware and its impact on imaging speed. We may need to reference it as we work through this.

https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast

If your fog server has a sata spinning hard drive that may also be the culprit. But again testing will show us.

Web GUI error after clicking List Hosts

187

11.6k

17.1k

154.6k