[FOG 1.6] “Attempting to check in… Failed”

[jmeyer]

@Sebastian-Roth I just see that my OS partition is full. I have fix it before.

Each time I have the check in error, i have the memory exhausted error.
Endless error… I have set -1 as limit and it’s same…

[Thu Feb 27 10:38:35.348972 2020] [proxy_fcgi:error] [pid 30368] [client 10.121.86.78:36270] AH01071: Got error 'PHP message: PHP Fatal error:  Out of memory (allocated 4118974464) (tried to allocate 20480 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 483\n'

[Sebastian Roth]

@jmeyer Are you on the very latest FOG working-1.6 version? Probably not as there is constantly being worked on and new things committed. Please update to the very latest and see if it’s fixed.

I have looked through the current code and it just doesn’t make sense to error out on line 483 in fogbase.class.php on the latest code in working-1.6. Update and try again. Either it’s gone or you’ll probably see a different line of code in the error message.

[jmeyer]

@Sebastian-Roth

Fresh install of version 1.5.8.964 on Debian 10 bring the same message.
10.121.84.110 is the host with “check in failed”.
Other one is a powered registered host with fog client.

[Thu Feb 27 16:44:51.107987 2020] [proxy_fcgi:error] [pid 4573] [client 10.121.84.13:55316] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/db/pdodb.class.php on line 617'
[Thu Feb 27 16:44:58.220066 2020] [proxy_fcgi:error] [pid 4296] [client 10.121.86.110:45384] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 483'
[Thu Feb 27 16:45:07.058804 2020] [proxy_fcgi:error] [pid 8835] [client 10.121.84.43:62335] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 962560 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'

[Sebastian Roth]

@jmeyer Fresh install using an empty DB or importing an existing DB?

@Tom-Elliott Probably something you need to look into.

[jmeyer]

@Sebastian-Roth

It’s an empty db.
Only hosts and groups have been imported by csv. I have manually added images.
I have deleted my testing host and add it again in gui (name and mac) to remove any imported data.
I have run hardware inventory without problem.

I also notice that php-fpm7.3 use 100% of cpu when FOG do task such tell computer to start from lan, load pxe and just before check in failed. Between these, CPU go back to less than 10% use.

[Tom Elliott]

As this is handled from the inits, there is no log available for you to “get”.

If you can run the task in Debug, when this errors out can you get us a photo of the exact error in context? (I know the check in’s happen every 10 seconds until it fully fails out, at which point it should give more information.

If it doesn’t I’m not quite sure how to troubleshoot at this point. I’m not seeing the issue on my side.

Can you also, in the browser try going to:

http://<yourfogserver/fog/service/Pre_Stage1.php?mac=<macofhost>&type=down

It should fail. The problem has nothing to do with the line you’re seeing reporting in the log (that’s just where the issue occurs, but it’s before that point that it’s caused.)

[jmeyer]

@Tom-Elliott

Here is what i see in debug.
It loop on check in endlessly (I let it run for more than one hour yesterday).

http://fogserver/fog/service/Pre_Stage1.php?mac=4c:cc:6a:32:e5:4b&type=down

Brings a black page.

I have run top from ssh on host during debug task when it try to check in.
I see the line coming and leaving.

curl -Lks --data sysuuid=6ec79d68-0c89-11e6-9a1e-1543db3a1300&mac=4c:cc:6a:32:e5:4b&type=down http://10.121.84.8/fog/service/Pre_Stage1.php

This apache log when I try to acces to http://fogserver/fog/service/Pre_Stage1.php?mac=4c:cc:6a:32:e5:4b&type=down
(10.121.84.13 is my computer).

[Fri Feb 28 15:03:15.432341 2020] [proxy_fcgi:error] [pid 474] [client 10.121.84.13:52235] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 1069056 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'
[Fri Feb 28 15:03:16.827698 2020] [proxy_fcgi:error] [pid 709] [client 10.121.84.13:52222] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 909312 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'

[Sebastian Roth]

@jmeyer said in [FOG 1.6] “Attempting to check in… Failed”:

/var/www/html/fog/lib/fog/fogbase.class.php on line 483

@Tom-Elliott You may have a look at this. To me it doesn’t make sense it would exhaust memory at this part of the code.

[Tom Elliott]

@Sebastian-Roth And I’m not seeing the same issue. I go to the page on my server and it works just fine. So I’m really not sure whats going on.

@jmeyer HOw many hosts do you have?

On one of the “clean” systems you have, can you add only one host and it’s image information? Then go to that URL in the browser? Does that return anything?

This seems to be like it’s processing too many hosts at once.

[jmeyer]

@Tom-Elliott

I have 290 host.
Delete host from GUI also bring the memory exhausted error.
What is strange is that import works fine.

I had to do this to run the test :

MariaDB [fog]> delete from hosts;
Query OK, 289 rows affected (0.155 sec)

I tried with only one host but I still can’t access the page.

[Sebastian Roth]

@jmeyer Would it be possible you share your CSV hosts (and maybe groups) export with us (private message to Tom and me)? This way we can try to replicate the issue.

[jmeyer]

@Sebastian-Roth Here are the 2 files :
Export Hosts FOG Project.csv
Export Groups FOG Project.csv

Maybe you also need installed apps version or a full db backup ?

After a week server was saying this and I had to force reboot.

[jmeyer]

@Sebastian-Roth
I have done this :

root@fogserver:/home/adminfog/fogproject# git checkout dev-branch
Extraction des fichiers: 100% (1029/1029), fait.
La branche 'dev-branch' est paramétrée pour suivre la branche distante 'dev-branch' depuis 'origin'.
Basculement sur la nouvelle branche 'dev-branch'
root@fogserver:/home/adminfog/fogproject# git pull
Déjà à jour.
root@fogserver:/home/adminfog/fogproject# git log -1
commit bf60833b45eb0f34452ded883b0eef87c6b2a0b1 (HEAD -> dev-branch, origin/dev-branch)
Author: Sebastian Roth <sebaroth@gmx.de>
Date:   Sun Mar 8 19:09:53 2020 +0100

    Prevent FOGMulticastManager from completing a task too early when the first host has checked out and fix typo in killTask function call
root@fogserver:/home/adminfog/fogproject# cd bin/
root@fogserver:/home/adminfog/fogproject/bin# ./installfog.sh

It has installed version 1.5.8.15 that is working fine.

[Sebastian Roth]

@jmeyer Switching between the two development versions dev-branch and working-1.6 is not recommended!! They are very different (not just the GUI but also in the database) and I am very sure you will run into issues sooner or later unless you have done a clean re-install of dev-branch.

Thanks for the CSV data. We will try to replicate the issue and see why it would cause memory exhaustion.

[jmeyer]

@Sebastian-Roth
Oops…
working-1.6 hasn’t the SQL fix and it fail during install.
And even if I end installing it, server looks to crash after a while.

What should I do ?

[Sebastian Roth]

@jmeyer Sorry, I just found the time to also add the SQL installer fixes to working-1.6 as well. I will work on trying to replicate the issue described here later on this week.

[jmeyer]

@Sebastian-Roth
I go back to working-1.6 and during install mysql process usage rise to 70%+ making server nearly not responding.

I have moved db to start a fresh one but I have the error message “Unable to update schema” when installing it.
I end only with this in db directory :

root@fogserver:/var/lib/mysql# ls fog -l
total 580
-rw-rw---- 1 mysql mysql    67 mars  13 17:07 db.opt
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 groupMembers.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 groups.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 history.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 hosts.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 images.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 schemaVersion.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 supportedOS.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 tasks.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 users.ibd

[Sebastian Roth]

@jmeyer said in [FOG 1.6] “Attempting to check in… Failed”:

I have moved db to start a fresh one but I have the error message “Unable to update schema” when installing it.

There should be more information in that error message when something goes wrong on shema update.

What Linux OS do you use?

I will work on this stuff on the weekend.

[Sebastian Roth]

@jmeyer I am not exactly sure what’s going on. I did a clean fresh install of FOG from working-1.6, schema update and importing your CSVs worked like a charm (though it took some seconds for both on my test VM).

I can capture and deploy images perfectly fine. Don’t see high CPU load or failed checkins on imaging tasks.

So my guess is that your hosts with fog-client installed are killing the webserver/PHP/DB thus causing you different issues.

Let’s try to work through this:

• Setup firewall rules to lock out most of the fog-client requests until we have it all back up:

sudo -i
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -s 10.121.84.13 -j ACCEPT
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Make sure the IP address used is the one of the PC you work on.

• Connect to the database via command line client and drop the current FOG DB:

shell> mysql -u root -p
...
mysql> DROP DATABASE fog;
mysql quit

• Re-run the FOG installer on the console. Now this will ask you about disabling the firewall. Say no here so the rules from above stay in place!
• Now when you get to the schema update this time I would be fairly sure it’ll work right away.
• Login to the web UI -> FOG Configuration -> FOG Settings -> search for FOG_CLIENT_CHECKIN_TIME and increase that setting from 60 to 300 for now.
• I would expect the web UI to work just fine and don’t cause much load on the server. Go ahead and import your hosts and groups from CSV as well as create image definitions.
• As the last step we have to bring back the clients. I would start by adding iptables rules to allow for a few clients to connect at a time. Though this depends on how your network is structured and how many clients you have. So this is just an example of commands you will probably adjust to suite your situation:

iptables -A INPUT -p tcp --dport 80 -s 10.121.84.0/25 -j ACCEPT

• Allow for the fog-client software to talk to the server and pull the new longer checkin time set above (check fog.log in a couple of those machines). Then go for the next block of clients.

iptables -A INPUT -p tcp --dport 80 -s 10.121.84.128/25 -j ACCEPT

• And so on. You can allow for bigger blocks of clients using different address and netmask. It depends on your network. If this seems to be too much to go step by step there is another way. Just drop the iptables rules altogether and have all the fog-clients fire at your server at once again. It will take a while but should calm down by itself sooner or later.

iptables -P INPUT ACCEPT
iptables -F INPUT

[Sebastian Roth]

@jmeyer I am marking this solved as we have not heard back on this. Feel free to update this topic if you are still working on this and need help.