Allowed memory size of 536870912 bytes exhausted

[AndrewG78]

FOG 1.5.4
Linux: CENTOS 7
PHP 7.2.8
FPM/FastCGI : Enabled
I was unable to work on 1.5.4 until I changed memory_limit from 128 to 512M in php.ini
I can work with 1.5.4 now, but some pages are loading extremely slow and I keep getting errors in php-fpm file
"
[10-Aug-2018 16:28:23 UTC] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 10489856 bytes) in /var/www/html/fog/lib/router/route.class.php on line 1059
"
Is there some known memory leakage bug or I missed something?

[Tom Elliott]

@andrewg78 depends on what you’re after.

If you don’t care about any history:

Truncate table snapinJobs
Truncate table imagingLogs
Truncate table tasks
Truncate table multicastSessions
Truncate table multicastSessionsAssoc

[george1421]

Lets assume is the issue we’ve found after FOG 1.5.4 has been released.

1. Change to the /etc directory from the fog server linux command prompt.
2. Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
   find /etc -name www.conf (hopefully you will only find one)
3. Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except php_admin_value[memory_limit] = 256M you will need to add that entry.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 35
pm.min_spare_servers = 5
pm.start_servers = 5

4. Save and exit your text editor.
5. Reboot the fog server.
6. See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.

Also we found there is something strange going on in the linux kernels after 4.15.2, I’m going to recommend that you downgrade your FOG/FOS kernel to 4.15.2. The issue with later kernels is that its taking 3-5 minutes to create the disk structure under certain circumstances, where with 4.15.2 and older its only seconds to create the structure.

Now the kernel will not impact your issue, but processing is incomplete might be related to the missing php-fpm configuration setting.

[AndrewG78]

Hi,
Thx for the suggestion. I applied it, but is is even worse now. Page stopped loading half the way.
I think I should mention I host my FOG server on Vmware

[george1421]

@andrewg78 It should NOT be worse with those settings.

When you look at top and sort by Processor what are the top 3 tasks? They should be php-fpm.

Just for clarity lets get a few background bits.

1. How many vCPU for the fog host server?
2. How many imaged computers have the fog client installed?
3. What is your client check in interval?
4. Did you only find one www.conf file?
5. You confirmed that this setting pm.max_requests = 2000 was in the www.conf file?

[AndrewG78]

@george1421
2x vCPU, 4GB RAM
20 computers
Interval check was not changed, so it’s default one
Only one www.conf
I set pm.max_requests = 2000
When I set memory_limit to 512M I can work, but some pages are very slow.
THis is quite strange.
When I click first 14 computer to edit, the HOST manahement Edit opens in 1s.
But last 4(of 20) computers responds very slowly. Page is loading 20s or even longer.
Same when I click on Tasks from main menu. Sometimes page is not loaded at all, but when I click again after 20 or more seconds it pops up.

When I set 256M I got this

[AndrewG78]

OK. I think I found the reason.
Last 4 machines have more then 1 year of imaging history.
There are a lot of records.
Is there any way to delete snapin and imaging history ?

[Tom Elliott]

@andrewg78 depends on what you’re after.

If you don’t care about any history:

Truncate table snapinJobs
Truncate table imagingLogs
Truncate table tasks
Truncate table multicastSessions
Truncate table multicastSessionsAssoc

[AndrewG78]

@tom-elliott big thanks!
It helped. Problem is resolved.
However, please do check why is the huge history making such a big memory leakage.
thx again

[Tom Elliott]

@andrewg78 it’s because of the huge history. There is no fixing that can cure it beyond cleaning it out. With 1.6, however, it will be somewhat alleviated to the use of proper sql pagination. This isn’t a leak persay, it just can’t store the amount of information in the memory space.