High RAM, CPU and 'Too Many Processes' mainly when running reports

[nick_v]

Hi,

We have just upgraded our FOG Server from 1.4.4 to 1.5.5. When we were on 1.4.4 it was running fine. However since the upgrade we are having serious issues with high RAM usage when running reports. And every so often (every few days) FOG will stop responding due to ‘Too Many Processes’. RAM usually sits at 1.7GB, however if I try and run a user report to see where a user has logged in, RAM shoots up to 10GB+ and CPU sits at 100% on an apache thread. Once PHP runs out of RAM, then we get a Internal 500 error.

A bit of information about our setup.

FOG Servers: 1 (1.5.5 on CentOS 6.10 - 8 Virtual Cores, 16GB RAM, Virtual machine on SSDs)
Storage nodes: 2 (1.5.5 on CentOS 6.10)

Hosts: 1300
User Tracking: 200,000 records (limited to the last year)
Database size: 25mb

Any help/ideas would be greatly appreciated.

Cheers,
Nick

[george1421]

Are you really running Centos 6? If so what version of php are you using php -v from the fog server’s linux prompt? Both of these may contribute to the slowness using version 1.5.x of FOG. In 1.5.x FOG moved the usage of the php engine out of apache and into a dedicated php engine called php-fpm. This was for performance reasons.

Now lets talk about the VM and its VM Host. How many physical cores does the VM Host server have? I see you have 8 allocated to the FOG VM. IMO that maybe too many depending on what your VM Host looks like.

[Tom Elliott]

If ram usage is the suspected culprit, what if beyond the 128M that is used for fog by default, upping it to use 256M? This can be done in the GUI.

Of note I might suggest using the working-1.6 branch as we have added proper pagination that would certainly help you out in this situation. I know it’s still in kind of testing but most things should work without issue and I can work on improving and fixing things you may find.

[nick_v]

Hi,

Thanks both for the advice. I have upped the memory limit in FOG to 256mb to see if that helps.

Yes we are still unfortunately running Centos 6.10, PHP is still on 5.6 and yes it does need updating. Its just been working so didn’t want to mess until we really needed to.

The host has 28 cores and all VM’s on that host doesn’t use all the cores, so that shouldn’t be the issue.

When I get chance I will re-build the server on a Centos 7 box and migrate the database across to see if performance improves. If not, we will install the working branch.

Thanks both once again.

Nick

[george1421]

@nick_v said in High RAM, CPU and 'Too Many Processes' mainly when running reports:

I have upped the memory limit in FOG to 256mb

The memory I’m referring to is php-fpm allocated memory. But either way I would recommend spinning up a new FOG server for your main server, the storage nodes can (could) remain on centos 6.1 but that’s not a supported OS anymore. On your main fog server that will run centos 7.5 install php 7.2 before installing FOG. You are almost at a point where creating a dedicated mysql server may add value. But I would take it one step at a time. Get a new server spun up and migrate your database and images to the new FOG server. Then look into performance issues. There are a few others that have large client base installs that we can ping in case we need to.

[Fernando Gietz]

Some reports, like “imaging log”, don’t have pre filters and try to show all the entries in the database, normally you have a lot of entries in the database and the server doesn’t have memory to run it.

In the dev-branch, not stable branch, I made some changes in the reports to have some pre-filters, with these pre-filters you can limit more the search parameters and be more exhaustive and use less memory