Login takes 5minutes+

Greg Plamondon

@george1421 I changed the CPU cout from 8 to 4 like you suggested.
Initial load:
alt text

Then it calmed down after a bit:
alt text

The Centos CDROM is already disconnected:
alt text
I am going to change the check-in time to 300 as suggested.

I ran these commands against the DB

# No password:
mysql -D fog

#Password:
mysql --user=UsernameHere --password=PasswordHere -D fog


DELETE FROM `hosts` WHERE `hostID` = '0';
DELETE FROM `hostMAC` WHERE hmID = '0' OR `hmHostID` = '0';
DELETE FROM `groupMembers` WHERE `gmID` = '0' OR `gmHostID` = '0' OR `gmGroupID` = '0';
DELETE FROM `snapinGroupAssoc` WHERE `sgaID` = '0' OR `sgaSnapinID` = '0' OR `sgaStorageGroupID` = '0';
DELETE from `snapinAssoc` WHERE `saID` = '0' OR `saHostID` = '0' OR `saSnapinID` = '0';
DELETE FROM `hosts` WHERE `hostID` NOT IN (SELECT `hmHostID` FROM `hostMAC` WHERE `hmPrimary` = '1');
DELETE FROM `hosts` WHERE `hostID` NOT IN (SELECT `hmHostID` FROM `hostMAC`);
DELETE FROM `hostMAC` WHERE `hmhostID` NOT IN (SELECT `hostID` FROM `hosts`);
DELETE FROM `snapinAssoc` WHERE `saHostID` NOT IN (SELECT `hostID` FROM `hosts`);
DELETE FROM `groupMembers` WHERE `gmHostID` NOT IN (SELECT `hostID` FROM `hosts`);
DELETE FROM `tasks` WHERE `taskStateID` IN ("1","2","3");
DELETE FROM `snapinTasks` WHERE `stState` in ("1","2","3");


quit

I will report back with results later.
Thanks for your help.

Greg Plamondon

I forgot to mention that the Fogserver is also handleing dhcp for 4 subnets.

george1421

@greg-plamondon The data store file is what I should have said. The VM holds open a connection to that iso file even if its not mounted. You can see this by looking at the datastore you will see a bunch of .lck files associated with that iso.

In regards to mysql, we will probably need to rebuild and reorg the indexes and tables. Lets see how the other changes impacted the environment first.

Greg Plamondon

@george1421
Ok, I removed the CDROM device. I don’t know how long it takes for the changes to speed things up but I haven’t seen any noticeable differences yet.

george1421

@greg-plamondon Well for the clients to get the new checkin interval it should be 60 seconds to switch over to the 5 minute timing.

So is your sql process still at 21% with the matching http process?

If you don’t see any better performance understand that dropping your vCPUs didn’t hurt the process, even if it didn’t help it much. There is a documented penalty for over committing vCPUs to VMs either way.

Greg Plamondon

@george1421

Ok, I found the issue I was having with logging in. I had about 5 storage nodes enabled on the home screens graph, Once i disabled those I was able to login immediately. The submit a task still takes about 1 minutes or 2 but its allot better!

Thanks

george1421

@greg-plamondon OK it WAS the dashboard doing it to you.

I still think you can optimize your FOG install more. BUT if your issue is addressed then you can save that activity for another time. Switching over to php-fpm should lower the vCPU requirements too. BUT the biggest driver for performance was lowering the check in time to once every 5 minutes especially with 300+ nodes.

Greg Plamondon

@george1421
I would love to optimize our FOG install if you have the time to tell me what to do.

Thanks!

george1421

@greg-plamondon Here is a thread I was working on to see if moving to a dedicated php engine (php-fpm) over letting apache just do it, helped any. At the time I didn’t have the volume of check-ins needed to see if there was much of a change. I could tell from the responsivness of the web gui that php-fpm was working correctly. Also adding memcache would help with caching session data in memory over writting it to disk.

There is two parts here php-fpm and memcache.
https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast

Beyond that I feel that mysql database could also use some tuning to help with performance. But that is a personal feeling without a solid basis of evidence.

Wayne Workman

@greg-plamondon said in Login takes 5minutes+:

I forgot to mention that the Fogserver is also handleing dhcp for 4 subnets.

Ok, gotta say it… You should put MySQL onto a dedicated VM, and point the FOG Web Server to it instead of the local one. Separate out stuff like this to scale horizontally (more VMs) instead of vertically (bigger VMs).

Sebastian Roth

@george1421 Well done! Great you followed this so closely. Lot’s good advices…

Wayne Workman

@lpetelik said in Login takes 5minutes+:

following for more solutions.

I wanted to tell you that you can disable the graph for all of the storage nodes, and this really does make login pretty snappy, but of course with this you no longer have graphs for all the nodes on the dashboard. With graphs disabled, you can still see tasks in Task Management.

191

11.6k

17.1k

154.6k