• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Fog 1.6.0.14 Master Storage Node not handing off to Slave Storage Nodes

    Scheduled Pinned Locked Moved Unsolved FOG Problems
    28 Posts 2 Posters 292 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      FlareImp
      last edited by

      After updating dev-branch 1.6.0.188 with the fixes you listed above. I noticed that my replication logs on all nodes (Master and Slave) are reading everything as:

      [11-18-26 4:14:16 pm]  * We are group ID: 1. We are group name: default
      [11-18-26 4:14:16 pm]  * We are node ID: 1. We are node name: DefaultMember
      [11-18-26 4:14:16 pm]  * Attempting to perform Group -> Group image replication.
      [11-18-26 4:14:16 pm]  | There are no images available!.
      
      

      Nothing has changed in my settings so everything is still a member of its own induvial locations.

      1 Reply Last reply Reply Quote 0
      • Tom ElliottT
        Tom Elliott @FlareImp
        last edited by

        @FlareImp Thanks for checking all of that. With those settings, the two hosts should have moved, and I cannot make my lab hold them back. So I need FOG to tell us why.

        Until now, when a host could not move, FOG wrote nothing about the reason. I added a log line for that case: https://github.com/FOGProject/fogproject/pull/2108. It names the node the host waits on, its task count and Max Clients, and each other node in the group with the reason it was passed over.

        When it is in dev-branch:

        • Run updatefog.sh --branch dev-branch on the main server.
        • Deploy the same 12 hosts again.
        • If any host waits, post the output of this, from the main server:
        grep "FOG storage node overflow" /var/log/apache2/error.log | tail -5
        
        • Also post the Storage Node column from the Active Tasks page for the hosts that wait.

        If a host waits and the grep finds nothing, that is useful too. It means FOG did not see the node as over its Max Clients at PXE boot.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        1 Reply Last reply Reply Quote 0
        • Tom ElliottT
          Tom Elliott @FlareImp
          last edited by

          @FlareImp Thanks. That line means the replicator on that server thinks it is node 1. On 11-13, after the fix in your other thread, your nodes showed their own IDs, for example node 13 TVNE-Fog. Nothing between 1.6.0.181 and 1.6.0.188 changed how a server finds its own node.

          One detail stands out. On 11-13, node 1 was named TV-Fog-Main. Now the log calls node 1 “DefaultMember”, in group “default”. Those are the names FOG gives the first node and group when it creates them.

          Please post:

          • From Storage Node Management: the name, IP address, storage group and Is Master setting of node 1. If no node is named DefaultMember, say so.
          • Which server you read each log on, and how: the Log Viewer page, or the file /opt/fog/log/fogreplicator.log on that server.
          • On one storage node that shows node ID 1, the output of:
          hostname; hostname -I; grep -v '^#' /etc/hosts
          head -20 /opt/fog/log/fogreplicator.log
          

          Your log says group “default” has no images, so these runs copy nothing. As a precaution until we know why, stop the replicators on the storage nodes again, but not on the main server. On 11-13, a node that acted as the main server deleted files on the other nodes:

          sudo systemctl stop FOGImageReplicator FOGSnapinReplicator
          

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

          F 1 Reply Last reply Reply Quote 0
          • F
            FlareImp @Tom Elliott
            last edited by FlareImp

            @Tom-Elliott There is not a default group. The main fog server is TV-Fog-Main and its in a storage group A(Default) by itself. I’m reading the logs from /opt/fog/log/fogreplicator.log on each storage node.

            head -20 /opt/fog/log/fogreplicator.log
            TVMS-Fog
            10.229.9.50
            127.0.0.1 localhost
            127.0.1.1 tvms-fog
            
            ::1     ip6-localhost ip6-loopback
            fe00::0 ip6-localnet
            ff00::0 ip6-mcastprefix
            ff02::1 ip6-allnodes
            ff02::2 ip6-allrouters
            [11-17-26 8:08:06 pm] ===== FOG 1.6.0.181 -- ImageReplicator starting =====
            [11-17-26 8:08:06 pm] Interface Ready with IP Address: 10.229.9.50
            [11-17-26 8:08:06 pm] Interface Ready with IP Address: 127.0.0.1
            [11-17-26 8:08:06 pm] Interface Ready with IP Address: 127.0.1.1
            [11-17-26 8:08:06 pm] Interface Ready with IP Address: TVMS-Fog
            [11-17-26 8:08:06 pm]  * Starting ImageReplicator Service
            [11-17-26 8:08:06 pm]  * Checking for new items every 600 seconds
            [11-17-26 8:08:06 pm]  * Starting service loop
            [11-17-26 8:08:06 pm]  * Starting Image Replication.
            [11-17-26 8:08:06 pm]  * We are group ID: 1. We are group name: default
            [11-17-26 8:08:06 pm]  * We are node ID: 1. We are node name: DefaultMember
            [11-17-26 8:08:06 pm]  * Attempting to perform Group -> Group image replication.
            [11-17-26 8:08:06 pm]  | There are no images available!.
            [11-17-26 8:31:31 pm] ===== FOG 1.6.0.181 -- ImageReplicator starting =====
            [11-17-26 8:31:32 pm] Interface Ready with IP Address: 10.229.9.50
            [11-17-26 8:31:32 pm] Interface Ready with IP Address: 127.0.0.1
            [11-17-26 8:31:32 pm] Interface Ready with IP Address: 127.0.1.1
            [11-17-26 8:31:32 pm] Interface Ready with IP Address: TVMS-Fog
            [11-17-26 8:31:32 pm]  * Starting ImageReplicator Service
            [11-17-26 8:31:32 pm]  * Checking for new items every 600 seconds
            
            Tom ElliottT 1 Reply Last reply Reply Quote 0
            • Tom ElliottT
              Tom Elliott @FlareImp
              last edited by

              @FlareImp Thanks, that shows it. Your real database has no group “default” and no node “DefaultMember”. So TVMS-Fog is not reading your main server’s database. It reads a database on TVMS-Fog itself. The FOG installer fills a new database with that group and node, and gives the node the server’s own IP. So TVMS-Fog matches node 1 in its own database, and that database has no images.

              A storage node must use the main server’s database. Please run this on TVMS-Fog and post the output. It shows no passwords:

              grep -E '^(installtype|snmysqlhost|snmysqluser)=' /opt/fog/.fogsettings
              grep DATABASE_HOST /var/www/html/fog/commons/config.class.php
              

              On a storage node, installtype should be ‘S’, and snmysqlhost and DATABASE_HOST should be your main server’s IP. If they are not, that server was installed or updated as a full server. Please run the same two commands on your other storage nodes too.

              Keep the replicators on the storage nodes stopped until this is fixed.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              F 1 Reply Last reply Reply Quote 0
              • F
                FlareImp @Tom Elliott
                last edited by FlareImp

                @Tom-Elliott I figured it out and it is totally my bad. Two of my sites i forgot to delete the old answer files so when I set them up they got the information for the old server. after deleting the fog settings and going back through the setup they are working correctly. Apologies for the extra work Tom.

                Tom ElliottT 1 Reply Last reply Reply Quote 0
                • Tom ElliottT
                  Tom Elliott @FlareImp
                  last edited by

                  @FlareImp No problem at all, thanks for tracking it down. Now that those two servers point at the main server’s database, you can start their replicators again:

                  sudo systemctl enable --now FOGImageReplicator FOGSnapinReplicator
                  

                  After one cycle, /opt/fog/log/fogreplicator.log on each should show its own node and group, not node 1.

                  For the two hosts that waited in the queue: the next time you deploy that lab, post the FOG storage node overflow lines from /var/log/apache2/error.log if any host waits.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  F 1 Reply Last reply Reply Quote 0
                  • F
                    FlareImp @Tom Elliott
                    last edited by

                    @Tom-Elliott The fog replication log on both is showing the correct information. Hopefully tomorrow morning I can post about the queuing issues.

                    1 Reply Last reply Reply Quote 1
                    • 1
                    • 2
                    • 2 / 2
                    • First post
                      Last post

                    54

                    Online

                    12.8k

                    Users

                    17.7k

                    Topics

                    157.5k

                    Posts
                    Copyright © 2012-2026 FOG Project