[FOG 1.6] “Attempting to check in… Failed”



  • Hello again,

    When i try to deploy, it loops on “Attempting to check in …… Failed”.
    Where can I get log about this part to have more information of the problem ?

    Thank you.


  • Developer

    @jmeyer I am marking this solved as we have not heard back on this. Feel free to update this topic if you are still working on this and need help.


  • Developer

    @jmeyer I am not exactly sure what’s going on. I did a clean fresh install of FOG from working-1.6, schema update and importing your CSVs worked like a charm (though it took some seconds for both on my test VM).

    I can capture and deploy images perfectly fine. Don’t see high CPU load or failed checkins on imaging tasks.

    So my guess is that your hosts with fog-client installed are killing the webserver/PHP/DB thus causing you different issues.

    Let’s try to work through this:

    • Setup firewall rules to lock out most of the fog-client requests until we have it all back up:
    sudo -i
    iptables -A INPUT -i lo -j ACCEPT
    iptables -A INPUT -p tcp --dport 22 -j ACCEPT
    iptables -A INPUT -p tcp --dport 80 -s 10.121.84.13 -j ACCEPT
    iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
    

    Make sure the IP address used is the one of the PC you work on.

    • Connect to the database via command line client and drop the current FOG DB:
    shell> mysql -u root -p
    ...
    mysql> DROP DATABASE fog;
    mysql quit
    
    • Re-run the FOG installer on the console. Now this will ask you about disabling the firewall. Say no here so the rules from above stay in place!
    • Now when you get to the schema update this time I would be fairly sure it’ll work right away.
    • Login to the web UI -> FOG Configuration -> FOG Settings -> search for FOG_CLIENT_CHECKIN_TIME and increase that setting from 60 to 300 for now.
    • I would expect the web UI to work just fine and don’t cause much load on the server. Go ahead and import your hosts and groups from CSV as well as create image definitions.
    • As the last step we have to bring back the clients. I would start by adding iptables rules to allow for a few clients to connect at a time. Though this depends on how your network is structured and how many clients you have. So this is just an example of commands you will probably adjust to suite your situation:
    iptables -A INPUT -p tcp --dport 80 -s 10.121.84.0/25 -j ACCEPT
    
    • Allow for the fog-client software to talk to the server and pull the new longer checkin time set above (check fog.log in a couple of those machines). Then go for the next block of clients.
    iptables -A INPUT -p tcp --dport 80 -s 10.121.84.128/25 -j ACCEPT
    
    • And so on. You can allow for bigger blocks of clients using different address and netmask. It depends on your network. If this seems to be too much to go step by step there is another way. Just drop the iptables rules altogether and have all the fog-clients fire at your server at once again. It will take a while but should calm down by itself sooner or later.
    iptables -P INPUT ACCEPT
    iptables -F INPUT
    

  • Developer

    @jmeyer said in [FOG 1.6] “Attempting to check in… Failed”:

    I have moved db to start a fresh one but I have the error message “Unable to update schema” when installing it.

    There should be more information in that error message when something goes wrong on shema update.

    What Linux OS do you use?

    I will work on this stuff on the weekend.



  • @Sebastian-Roth
    I go back to working-1.6 and during install mysql process usage rise to 70%+ making server nearly not responding.

    I have moved db to start a fresh one but I have the error message “Unable to update schema” when installing it.
    I end only with this in db directory :

    root@fogserver:/var/lib/mysql# ls fog -l
    total 580
    -rw-rw---- 1 mysql mysql    67 mars  13 17:07 db.opt
    -rw-rw---- 1 mysql mysql 65536 mars  13 17:07 groupMembers.ibd
    -rw-rw---- 1 mysql mysql 65536 mars  13 17:07 groups.ibd
    -rw-rw---- 1 mysql mysql 65536 mars  13 17:07 history.ibd
    -rw-rw---- 1 mysql mysql 65536 mars  13 17:07 hosts.ibd
    -rw-rw---- 1 mysql mysql 65536 mars  13 17:07 images.ibd
    -rw-rw---- 1 mysql mysql 65536 mars  13 17:07 schemaVersion.ibd
    -rw-rw---- 1 mysql mysql 65536 mars  13 17:07 supportedOS.ibd
    -rw-rw---- 1 mysql mysql 65536 mars  13 17:07 tasks.ibd
    -rw-rw---- 1 mysql mysql 65536 mars  13 17:07 users.ibd
    

  • Developer

    @jmeyer Sorry, I just found the time to also add the SQL installer fixes to working-1.6 as well. I will work on trying to replicate the issue described here later on this week.



  • @Sebastian-Roth
    Oops…
    working-1.6 hasn’t the SQL fix and it fail during install.
    And even if I end installing it, server looks to crash after a while.

    What should I do ?


  • Developer

    @jmeyer Switching between the two development versions dev-branch and working-1.6 is not recommended!! They are very different (not just the GUI but also in the database) and I am very sure you will run into issues sooner or later unless you have done a clean re-install of dev-branch.

    Thanks for the CSV data. We will try to replicate the issue and see why it would cause memory exhaustion.



  • @Sebastian-Roth
    I have done this :

    root@fogserver:/home/adminfog/fogproject# git checkout dev-branch
    Extraction des fichiers: 100% (1029/1029), fait.
    La branche 'dev-branch' est paramétrée pour suivre la branche distante 'dev-branch' depuis 'origin'.
    Basculement sur la nouvelle branche 'dev-branch'
    root@fogserver:/home/adminfog/fogproject# git pull
    Déjà à jour.
    root@fogserver:/home/adminfog/fogproject# git log -1
    commit bf60833b45eb0f34452ded883b0eef87c6b2a0b1 (HEAD -> dev-branch, origin/dev-branch)
    Author: Sebastian Roth <sebaroth@gmx.de>
    Date:   Sun Mar 8 19:09:53 2020 +0100
    
        Prevent FOGMulticastManager from completing a task too early when the first host has checked out and fix typo in killTask function call
    root@fogserver:/home/adminfog/fogproject# cd bin/
    root@fogserver:/home/adminfog/fogproject/bin# ./installfog.sh
    
    

    It has installed version 1.5.8.15 that is working fine.



  • @Sebastian-Roth Here are the 2 files :
    Export Hosts FOG Project.csv
    Export Groups FOG Project.csv

    Maybe you also need installed apps version or a full db backup ?

    After a week server was saying this and I had to force reboot.

    IMG_20200309_090137.jpg


  • Developer

    @jmeyer Would it be possible you share your CSV hosts (and maybe groups) export with us (private message to Tom and me)? This way we can try to replicate the issue.



  • @Tom-Elliott

    I have 290 host.
    Delete host from GUI also bring the memory exhausted error.
    What is strange is that import works fine.

    I had to do this to run the test :

    MariaDB [fog]> delete from hosts;
    Query OK, 289 rows affected (0.155 sec)
    

    I tried with only one host but I still can’t access the page.


  • Senior Developer

    @Sebastian-Roth And I’m not seeing the same issue. I go to the page on my server and it works just fine. So I’m really not sure whats going on.

    @jmeyer HOw many hosts do you have?

    On one of the “clean” systems you have, can you add only one host and it’s image information? Then go to that URL in the browser? Does that return anything?

    This seems to be like it’s processing too many hosts at once.


  • Developer

    @jmeyer said in [FOG 1.6] “Attempting to check in… Failed”:

    /var/www/html/fog/lib/fog/fogbase.class.php on line 483

    @Tom-Elliott You may have a look at this. To me it doesn’t make sense it would exhaust memory at this part of the code.



  • @Tom-Elliott

    Here is what i see in debug.
    It loop on check in endlessly (I let it run for more than one hour yesterday).
    IMG_20200228_143101.jpg

    http://fogserver/fog/service/Pre_Stage1.php?mac=4c:cc:6a:32:e5:4b&type=down

    Brings a black page.

    I have run top from ssh on host during debug task when it try to check in.
    I see the line coming and leaving.

    curl -Lks --data sysuuid=6ec79d68-0c89-11e6-9a1e-1543db3a1300&mac=4c:cc:6a:32:e5:4b&type=down http://10.121.84.8/fog/service/Pre_Stage1.php
    

    This apache log when I try to acces to http://fogserver/fog/service/Pre_Stage1.php?mac=4c:cc:6a:32:e5:4b&type=down
    (10.121.84.13 is my computer).

    [Fri Feb 28 15:03:15.432341 2020] [proxy_fcgi:error] [pid 474] [client 10.121.84.13:52235] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 1069056 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'
    [Fri Feb 28 15:03:16.827698 2020] [proxy_fcgi:error] [pid 709] [client 10.121.84.13:52222] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 909312 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'
    

  • Senior Developer

    As this is handled from the inits, there is no log available for you to “get”.

    If you can run the task in Debug, when this errors out can you get us a photo of the exact error in context? (I know the check in’s happen every 10 seconds until it fully fails out, at which point it should give more information.

    If it doesn’t I’m not quite sure how to troubleshoot at this point. I’m not seeing the issue on my side.

    Can you also, in the browser try going to:

    http://<yourfogserver/fog/service/Pre_Stage1.php?mac=<macofhost>&type=down

    It should fail. The problem has nothing to do with the line you’re seeing reporting in the log (that’s just where the issue occurs, but it’s before that point that it’s caused.)



  • @Sebastian-Roth

    It’s an empty db.
    Only hosts and groups have been imported by csv. I have manually added images.
    I have deleted my testing host and add it again in gui (name and mac) to remove any imported data.
    I have run hardware inventory without problem.

    I also notice that php-fpm7.3 use 100% of cpu when FOG do task such tell computer to start from lan, load pxe and just before check in failed. Between these, CPU go back to less than 10% use.

    8423c5e3-c486-4564-a993-f5610eeb978d-image.png


  • Developer

    @jmeyer Fresh install using an empty DB or importing an existing DB?

    @Tom-Elliott Probably something you need to look into.



  • @Sebastian-Roth

    Fresh install of version 1.5.8.964 on Debian 10 bring the same message.
    10.121.84.110 is the host with “check in failed”.
    Other one is a powered registered host with fog client.

    [Thu Feb 27 16:44:51.107987 2020] [proxy_fcgi:error] [pid 4573] [client 10.121.84.13:55316] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/db/pdodb.class.php on line 617'
    [Thu Feb 27 16:44:58.220066 2020] [proxy_fcgi:error] [pid 4296] [client 10.121.86.110:45384] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 483'
    [Thu Feb 27 16:45:07.058804 2020] [proxy_fcgi:error] [pid 8835] [client 10.121.84.43:62335] AH01071: Got error 'PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 962560 bytes) in /var/www/html/fog/lib/fog/fogbase.class.php on line 2382'
    

  • Developer

    @jmeyer Are you on the very latest FOG working-1.6 version? Probably not as there is constantly being worked on and new things committed. Please update to the very latest and see if it’s fixed.

    I have looked through the current code and it just doesn’t make sense to error out on line 483 in fogbase.class.php on the latest code in working-1.6. Update and try again. Either it’s gone or you’ll probably see a different line of code in the error message.


Log in to reply
 

302
Online

7.0k
Users

14.2k
Topics

134.4k
Posts