[FOG 1.6] “Attempting to check in… Failed”

jmeyer

@Sebastian-Roth
I run version 1.5.8.960

I have these 2 type of error in Apache log :

[Wed Feb 26 13:33:08.728059 2020] [proxy_fcgi:error] [pid 7204] [client 10.121.84.39:49338] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Exception: Error multiple hosts returned for list of mac addresses in /var/www/html/fog/lib/fog/hostmanager.class.php:299\nStack trace:\n#0 /var/www/html/fog/lib/fog/fogbase.class.php(592): HostManager->getHostByMacAddresses(Array)\n#1 /var/www/html/fog/lib/client/registerclient.class.php(46): FOGBase::getHostItem(true, false, false, true)\n#2 /var/www/html/fog/lib/fog/fogpage.class.php(2117): RegisterClient->json()\n#3 /var/www/html/fog/lib/fog/fogpage.class.php(274): FOGPage->requestClientInfo()\n#4 /var/www/html/fog/lib/fog/fogpagemanager.class.php(255): FOGPage->__construct()\n#5 /var/www/html/fog/lib/fog/fogpagemanager.class.php(73): FOGPageManager->loadPageClasses()\n#6 [internal function]: FOGPageManager->__construct()\n#7 /var/www/html/fog/lib/fog/fogbase.class.php(493): ReflectionClass->newInstanceArgs(Array)\n#8 /var/www/html/fog/management/index.php(23): FOGBase::getClass('FOGPageManager')\n#9 {main}\n  thrown in /var/www/html/fog/lib/fog/hostmanager.class.php on lin...\n'

[Wed Feb 26 13:33:36.859890 2020] [proxy_fcgi:error] [pid 8925] [client 10.121.84.13:64391] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 483\n'

jmeyer

I edited /etc/php/7.2/fpm/pool.d/www.conf to set php_admin_value[memory_limit] = 1024M and end with this :

[Wed Feb 26 15:51:37.870983 2020] [proxy_fcgi:error] [pid 873] [client 10.121.84.43:59200] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 483\n'

Sebastian Roth

@jmeyer said in [FOG 1.6] “Attempting to check in… Failed”:

... Error multiple hosts returned for list of mac addresses ...

While the links I post are about the fog-client or snapins you can still go through those to see how to find and remove the duplicated entry - all have the same thing causing this message:
https://forums.fogproject.org/topic/14081/fog-client-authentication-issue
https://forums.fogproject.org/topic/14019/snapin-queued-forever

jmeyer

@Sebastian-Roth I just see that my OS partition is full. I have fix it before.

Each time I have the check in error, i have the memory exhausted error.
Endless error… I have set -1 as limit and it’s same…

[Thu Feb 27 10:38:35.348972 2020] [proxy_fcgi:error] [pid 30368] [client 10.121.86.78:36270] AH01071: Got error 'PHP message: PHP Fatal error:  Out of memory (allocated 4118974464) (tried to allocate 20480 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 483\n'

Sebastian Roth

@jmeyer Are you on the very latest FOG working-1.6 version? Probably not as there is constantly being worked on and new things committed. Please update to the very latest and see if it’s fixed.

I have looked through the current code and it just doesn’t make sense to error out on line 483 in fogbase.class.php on the latest code in working-1.6. Update and try again. Either it’s gone or you’ll probably see a different line of code in the error message.

jmeyer

@Sebastian-Roth

Fresh install of version 1.5.8.964 on Debian 10 bring the same message.
10.121.84.110 is the host with “check in failed”.
Other one is a powered registered host with fog client.

[Thu Feb 27 16:44:51.107987 2020] [proxy_fcgi:error] [pid 4573] [client 10.121.84.13:55316] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/db/pdodb.class.php on line 617'
[Thu Feb 27 16:44:58.220066 2020] [proxy_fcgi:error] [pid 4296] [client 10.121.86.110:45384] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 483'
[Thu Feb 27 16:45:07.058804 2020] [proxy_fcgi:error] [pid 8835] [client 10.121.84.43:62335] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 962560 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'

Sebastian Roth

@jmeyer Fresh install using an empty DB or importing an existing DB?

@Tom-Elliott Probably something you need to look into.

jmeyer

@Sebastian-Roth

It’s an empty db.
Only hosts and groups have been imported by csv. I have manually added images.
I have deleted my testing host and add it again in gui (name and mac) to remove any imported data.
I have run hardware inventory without problem.

I also notice that php-fpm7.3 use 100% of cpu when FOG do task such tell computer to start from lan, load pxe and just before check in failed. Between these, CPU go back to less than 10% use.

Tom Elliott

As this is handled from the inits, there is no log available for you to “get”.

If you can run the task in Debug, when this errors out can you get us a photo of the exact error in context? (I know the check in’s happen every 10 seconds until it fully fails out, at which point it should give more information.

If it doesn’t I’m not quite sure how to troubleshoot at this point. I’m not seeing the issue on my side.

Can you also, in the browser try going to:

http://<yourfogserver/fog/service/Pre_Stage1.php?mac=<macofhost>&type=down

It should fail. The problem has nothing to do with the line you’re seeing reporting in the log (that’s just where the issue occurs, but it’s before that point that it’s caused.)

jmeyer

@Tom-Elliott

Here is what i see in debug.
It loop on check in endlessly (I let it run for more than one hour yesterday).

http://fogserver/fog/service/Pre_Stage1.php?mac=4c:cc:6a:32:e5:4b&type=down

Brings a black page.

I have run top from ssh on host during debug task when it try to check in.
I see the line coming and leaving.

curl -Lks --data sysuuid=6ec79d68-0c89-11e6-9a1e-1543db3a1300&mac=4c:cc:6a:32:e5:4b&type=down http://10.121.84.8/fog/service/Pre_Stage1.php

This apache log when I try to acces to http://fogserver/fog/service/Pre_Stage1.php?mac=4c:cc:6a:32:e5:4b&type=down
(10.121.84.13 is my computer).

[Fri Feb 28 15:03:15.432341 2020] [proxy_fcgi:error] [pid 474] [client 10.121.84.13:52235] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 1069056 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'
[Fri Feb 28 15:03:16.827698 2020] [proxy_fcgi:error] [pid 709] [client 10.121.84.13:52222] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 909312 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'

Sebastian Roth

@jmeyer said in [FOG 1.6] “Attempting to check in… Failed”:

/var/www/html/fog/lib/fog/fogbase.class.php on line 483

@Tom-Elliott You may have a look at this. To me it doesn’t make sense it would exhaust memory at this part of the code.

Tom Elliott

@Sebastian-Roth And I’m not seeing the same issue. I go to the page on my server and it works just fine. So I’m really not sure whats going on.

@jmeyer HOw many hosts do you have?

On one of the “clean” systems you have, can you add only one host and it’s image information? Then go to that URL in the browser? Does that return anything?

This seems to be like it’s processing too many hosts at once.

jmeyer

@Tom-Elliott

I have 290 host.
Delete host from GUI also bring the memory exhausted error.
What is strange is that import works fine.

I had to do this to run the test :

MariaDB [fog]> delete from hosts;
Query OK, 289 rows affected (0.155 sec)

I tried with only one host but I still can’t access the page.

Sebastian Roth

@jmeyer Would it be possible you share your CSV hosts (and maybe groups) export with us (private message to Tom and me)? This way we can try to replicate the issue.

jmeyer

@Sebastian-Roth Here are the 2 files :
Export Hosts FOG Project.csv
Export Groups FOG Project.csv

Maybe you also need installed apps version or a full db backup ?

After a week server was saying this and I had to force reboot.

jmeyer

@Sebastian-Roth
I have done this :

root@fogserver:/home/adminfog/fogproject# git checkout dev-branch
Extraction des fichiers: 100% (1029/1029), fait.
La branche 'dev-branch' est paramétrée pour suivre la branche distante 'dev-branch' depuis 'origin'.
Basculement sur la nouvelle branche 'dev-branch'
root@fogserver:/home/adminfog/fogproject# git pull
Déjà à jour.
root@fogserver:/home/adminfog/fogproject# git log -1
commit bf60833b45eb0f34452ded883b0eef87c6b2a0b1 (HEAD -> dev-branch, origin/dev-branch)
Author: Sebastian Roth <sebaroth@gmx.de>
Date:   Sun Mar 8 19:09:53 2020 +0100

    Prevent FOGMulticastManager from completing a task too early when the first host has checked out and fix typo in killTask function call
root@fogserver:/home/adminfog/fogproject# cd bin/
root@fogserver:/home/adminfog/fogproject/bin# ./installfog.sh

It has installed version 1.5.8.15 that is working fine.

Sebastian Roth

@jmeyer Switching between the two development versions dev-branch and working-1.6 is not recommended!! They are very different (not just the GUI but also in the database) and I am very sure you will run into issues sooner or later unless you have done a clean re-install of dev-branch.

Thanks for the CSV data. We will try to replicate the issue and see why it would cause memory exhaustion.

jmeyer

@Sebastian-Roth
Oops…
working-1.6 hasn’t the SQL fix and it fail during install.
And even if I end installing it, server looks to crash after a while.

What should I do ?

Sebastian Roth

@jmeyer Sorry, I just found the time to also add the SQL installer fixes to working-1.6 as well. I will work on trying to replicate the issue described here later on this week.

jmeyer

@Sebastian-Roth
I go back to working-1.6 and during install mysql process usage rise to 70%+ making server nearly not responding.

I have moved db to start a fresh one but I have the error message “Unable to update schema” when installing it.
I end only with this in db directory :

root@fogserver:/var/lib/mysql# ls fog -l
total 580
-rw-rw---- 1 mysql mysql    67 mars  13 17:07 db.opt
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 groupMembers.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 groups.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 history.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 hosts.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 images.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 schemaVersion.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 supportedOS.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 tasks.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 users.ibd

[FOG 1.6] “Attempting to check in… Failed”

245

12.0k

17.3k

155.2k