[FOG 1.6] “Attempting to check in… Failed”

jmeyer

Here is what i see in debug.
It loop on check in endlessly (I let it run for more than one hour yesterday).

http://fogserver/fog/service/Pre_Stage1.php?mac=4c:cc:6a:32:e5:4b&type=down

Brings a black page.

I have run top from ssh on host during debug task when it try to check in.
I see the line coming and leaving.

curl -Lks --data sysuuid=6ec79d68-0c89-11e6-9a1e-1543db3a1300&mac=4c:cc:6a:32:e5:4b&type=down http://10.121.84.8/fog/service/Pre_Stage1.php

This apache log when I try to acces to http://fogserver/fog/service/Pre_Stage1.php?mac=4c:cc:6a:32:e5:4b&type=down
(10.121.84.13 is my computer).

[Fri Feb 28 15:03:15.432341 2020] [proxy_fcgi:error] [pid 474] [client 10.121.84.13:52235] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 1069056 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'
[Fri Feb 28 15:03:16.827698 2020] [proxy_fcgi:error] [pid 709] [client 10.121.84.13:52222] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 909312 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'

Sebastian Roth

@jmeyer said in [FOG 1.6] “Attempting to check in… Failed”:

/var/www/html/fog/lib/fog/fogbase.class.php on line 483

@Tom-Elliott You may have a look at this. To me it doesn’t make sense it would exhaust memory at this part of the code.

Tom Elliott

@Sebastian-Roth And I’m not seeing the same issue. I go to the page on my server and it works just fine. So I’m really not sure whats going on.

@jmeyer HOw many hosts do you have?

On one of the “clean” systems you have, can you add only one host and it’s image information? Then go to that URL in the browser? Does that return anything?

This seems to be like it’s processing too many hosts at once.

jmeyer

@Tom-Elliott

I have 290 host.
Delete host from GUI also bring the memory exhausted error.
What is strange is that import works fine.

I had to do this to run the test :

MariaDB [fog]> delete from hosts;
Query OK, 289 rows affected (0.155 sec)

I tried with only one host but I still can’t access the page.

Sebastian Roth

@jmeyer Would it be possible you share your CSV hosts (and maybe groups) export with us (private message to Tom and me)? This way we can try to replicate the issue.

jmeyer

@Sebastian-Roth Here are the 2 files :
Export Hosts FOG Project.csv
Export Groups FOG Project.csv

Maybe you also need installed apps version or a full db backup ?

After a week server was saying this and I had to force reboot.

jmeyer

@Sebastian-Roth
I have done this :

root@fogserver:/home/adminfog/fogproject# git checkout dev-branch
Extraction des fichiers: 100% (1029/1029), fait.
La branche 'dev-branch' est paramétrée pour suivre la branche distante 'dev-branch' depuis 'origin'.
Basculement sur la nouvelle branche 'dev-branch'
root@fogserver:/home/adminfog/fogproject# git pull
Déjà à jour.
root@fogserver:/home/adminfog/fogproject# git log -1
commit bf60833b45eb0f34452ded883b0eef87c6b2a0b1 (HEAD -> dev-branch, origin/dev-branch)
Author: Sebastian Roth <sebaroth@gmx.de>
Date:   Sun Mar 8 19:09:53 2020 +0100

    Prevent FOGMulticastManager from completing a task too early when the first host has checked out and fix typo in killTask function call
root@fogserver:/home/adminfog/fogproject# cd bin/
root@fogserver:/home/adminfog/fogproject/bin# ./installfog.sh

It has installed version 1.5.8.15 that is working fine.

Sebastian Roth

@jmeyer Switching between the two development versions dev-branch and working-1.6 is not recommended!! They are very different (not just the GUI but also in the database) and I am very sure you will run into issues sooner or later unless you have done a clean re-install of dev-branch.

Thanks for the CSV data. We will try to replicate the issue and see why it would cause memory exhaustion.

jmeyer

@Sebastian-Roth
Oops…
working-1.6 hasn’t the SQL fix and it fail during install.
And even if I end installing it, server looks to crash after a while.

What should I do ?

Sebastian Roth

@jmeyer Sorry, I just found the time to also add the SQL installer fixes to working-1.6 as well. I will work on trying to replicate the issue described here later on this week.

jmeyer

@Sebastian-Roth
I go back to working-1.6 and during install mysql process usage rise to 70%+ making server nearly not responding.

I have moved db to start a fresh one but I have the error message “Unable to update schema” when installing it.
I end only with this in db directory :

root@fogserver:/var/lib/mysql# ls fog -l
total 580
-rw-rw---- 1 mysql mysql    67 mars  13 17:07 db.opt
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 groupMembers.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 groups.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 history.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 hosts.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 images.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 schemaVersion.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 supportedOS.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 tasks.ibd
-rw-rw---- 1 mysql mysql 65536 mars  13 17:07 users.ibd

Sebastian Roth

@jmeyer said in [FOG 1.6] “Attempting to check in… Failed”:

I have moved db to start a fresh one but I have the error message “Unable to update schema” when installing it.

There should be more information in that error message when something goes wrong on shema update.

What Linux OS do you use?

I will work on this stuff on the weekend.

Sebastian Roth

@jmeyer I am not exactly sure what’s going on. I did a clean fresh install of FOG from working-1.6, schema update and importing your CSVs worked like a charm (though it took some seconds for both on my test VM).

I can capture and deploy images perfectly fine. Don’t see high CPU load or failed checkins on imaging tasks.

So my guess is that your hosts with fog-client installed are killing the webserver/PHP/DB thus causing you different issues.

Let’s try to work through this:

Setup firewall rules to lock out most of the fog-client requests until we have it all back up:

sudo -i
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -s 10.121.84.13 -j ACCEPT
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Make sure the IP address used is the one of the PC you work on.

Connect to the database via command line client and drop the current FOG DB:

shell> mysql -u root -p
...
mysql> DROP DATABASE fog;
mysql quit

Re-run the FOG installer on the console. Now this will ask you about disabling the firewall. Say no here so the rules from above stay in place!
Now when you get to the schema update this time I would be fairly sure it’ll work right away.
Login to the web UI -> FOG Configuration -> FOG Settings -> search for FOG_CLIENT_CHECKIN_TIME and increase that setting from 60 to 300 for now.
I would expect the web UI to work just fine and don’t cause much load on the server. Go ahead and import your hosts and groups from CSV as well as create image definitions.
As the last step we have to bring back the clients. I would start by adding iptables rules to allow for a few clients to connect at a time. Though this depends on how your network is structured and how many clients you have. So this is just an example of commands you will probably adjust to suite your situation:

iptables -A INPUT -p tcp --dport 80 -s 10.121.84.0/25 -j ACCEPT

Allow for the fog-client software to talk to the server and pull the new longer checkin time set above (check fog.log in a couple of those machines). Then go for the next block of clients.

iptables -A INPUT -p tcp --dport 80 -s 10.121.84.128/25 -j ACCEPT

And so on. You can allow for bigger blocks of clients using different address and netmask. It depends on your network. If this seems to be too much to go step by step there is another way. Just drop the iptables rules altogether and have all the fog-clients fire at your server at once again. It will take a while but should calm down by itself sooner or later.

iptables -P INPUT ACCEPT
iptables -F INPUT

Sebastian Roth

@jmeyer I am marking this solved as we have not heard back on this. Feel free to update this topic if you are still working on this and need help.

[FOG 1.6] “Attempting to check in… Failed”

232

11.6k

17.1k

154.6k