• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Fog 1.6.0.14 Master Storage Node not handing off to Slave Storage Nodes

    Scheduled Pinned Locked Moved Unsolved FOG Problems
    11 Posts 2 Posters 187 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      FlareImp
      last edited by

      At one of my slave locations I just imaged a 32 PC lab and they all pulled from the Master Storage Node. I have two other Storage Nodes in the group but neither of them got any tasks. Everything stayed with the Master Storage Node. Once the Que was full they just waited in line rather than get the image from one of the other two nodes that had no clients. I even tried coping the Plugin folder to the nodes but that didn’t change anything.

      Fog 1.6.0.14
      Ubuntu 26.04.1

      Tom ElliottT 2 Replies Last reply Reply Quote 0
      • Tom ElliottT
        Tom Elliott @FlareImp
        last edited by

        @FlareImp Thank you. This is a bug in 1.6, and it explains what you see.

        What happens: FOG picks a storage node for each host when you create the task. It should pick the node with the lowest load. In 1.6 the load value was always empty, so every task went to the first node in the group, which is your master. A host also stays on the node it got at task creation. When that node is full, the host waits in line there, even when another node in the group is idle.

        The fix: https://github.com/FOGProject/fogproject/pull/2082. It does two things:

        • Task creation picks the least-loaded node in the group.
        • At boot, a host whose node is full moves to the node in its group with the most free slots. A capture never moves, because it must upload to the master.

        It goes to dev-branch first, and reaches stable in the next stable release.

        Check your location settings first. Copying the plugin folder does not change this, but your location can. If the location for those hosts has a Storage Node set, every host in that location goes to that one node. That is by design, and the fix does not change it. To spread the hosts across the group, open the location, clear the Storage Node, and keep the Storage Group set.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        F 2 Replies Last reply Reply Quote 0
        • Tom ElliottT
          Tom Elliott @FlareImp
          last edited by

          @FlareImp One correction to my last post. With the fix, a location’s Storage Node is where its hosts start, not a hard limit. When that node is full, a waiting host moves to a free node in the location’s own storage group. It never leaves that group, so the image still comes from your site. You can leave the location’s Storage Node set.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

          1 Reply Last reply Reply Quote 0
          • F
            FlareImp @Tom Elliott
            last edited by

            @Tom-Elliott Okay, thank you for the information.

            1 Reply Last reply Reply Quote 0
            • F
              FlareImp @Tom Elliott
              last edited by FlareImp

              @Tom-Elliott I just imaged a lab of 30. At my remote location, there is one Master node and two slave nodes for a count of 30. In the task page it shows 10 systems at each node but on the Partclone display on the clients they are all showing just the single Master node. I also confirmed this by watching the bandwidth on all 3 nodes. node 1 is maxed, nodes 2 and 3 are not doing anything.

              Fog Dev Branch 1.6.0.167

              Tom ElliottT 1 Reply Last reply Reply Quote 0
              • Tom ElliottT
                Tom Elliott @FlareImp
                last edited by

                @FlareImp Thank you, that test found it. The task list and the clients disagreed, and the clients were right.

                What happens: a client learns its storage node once, at PXE boot, and mounts that node. My earlier fix moved a waiting host to a free node later, at check-in. That changed the task list, but not the node the client had already mounted. Your location sends its hosts to the master, so all 30 imaged from the master while the task list showed 10 on each node.

                The fix: https://github.com/FOGProject/fogproject/pull/2103. The move now happens at PXE boot. When a host boots and its node already has more tasks than its Max Clients, it gets the node in the same storage group with the most free slots. The client mounts that node, and the task list shows the same node. The check-in move is removed. Captures and multicast still use the master.

                With your setup, hosts move off the master as they boot until it holds 10, and the other two nodes take 10 each. Hosts that PXE boot at the same instant can still land on the same node. Those extra hosts wait in line there.

                It goes to dev-branch first. After it merges, update the main server and the storage nodes with updatefog.sh --branch dev-branch, as before, and deploy the lab again.

                Until then: clear the Storage Node on that location and keep the Storage Group set. Then each host gets the least-loaded node in the group at PXE boot.

                Please let me know what the task list and the node bandwidth show on the next run.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                F 2 Replies Last reply Reply Quote 0
                • F
                  FlareImp @Tom Elliott
                  last edited by

                  @Tom-Elliott Thank you for this, I will give it a try. School has started so I won’t be able to give an update until tomorrow morning.

                  1 Reply Last reply Reply Quote 0
                  • F
                    FlareImp @Tom Elliott
                    last edited by

                    @Tom-Elliott after upgrading to the latest dev branch, when you pxe a host it shows as not being registered even though it showed up correctly on the previous version of FOG. If you check the mac on any of the hosts, nothing is selected as primary and after making a choice and refreshing the screen it goes back to nothing selected.

                    Tom ElliottT 1 Reply Last reply Reply Quote 0
                    • Tom ElliottT
                      Tom Elliott @FlareImp
                      last edited by

                      @FlareImp Thank you. This is a different problem from the storage nodes. Nothing that finds a host by its MAC, or saves the primary MAC, changed between 1.6.0.167 and the current dev-branch. My lab runs the current dev-branch, and its hosts boot as registered and keep their primary MAC. So I need to see what your server has.

                      Please post these:

                      1. The version on the main server: open http://<your-fog-server>/fog/service/getversion.php.
                      2. The MAC flags in the database. On the main server, run:
                        sudo mysql fog -e "SELECT hmPrimary, hmPending, COUNT(*) FROM hostMAC GROUP BY hmPrimary, hmPending; SHOW COLUMNS FROM hostMAC;"
                      3. The web server error log right after you set a primary MAC and refresh. Mark a primary on one host, then post the last 30 lines of the PHP error log. On Debian or Ubuntu with Apache, that is /var/log/apache2/error.log. On Rocky, Alma or RHEL, it is /var/log/php-fpm/www-error.log.
                      4. Did the installer finish on the main server, with no error at the end?

                      When you save a primary MAC, FOG first clears the primary flag on all of the host’s MACs, then sets the one you chose. You see nothing selected after a refresh, so the second step fails on your server. The error log should say why.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      F 1 Reply Last reply Reply Quote 0
                      • F
                        FlareImp @Tom Elliott
                        last edited by

                        @Tom-Elliott

                        1. Main and all nodes are on version 1.6.0.181

                        administrator@tv-fog-main:~$ sudo mysql fog -e "SELECT hmPrimary, hmPending, COUNT(*) FROM hostMAC GROUP BY hmPrimary, hmPending; SHOW COLUMNS FROM hostMAC;" [sudo: authenticate] Password: +-----------+-----------+----------+ | hmPrimary | hmPending | COUNT(*) | +-----------+-----------+----------+ | 0 | 0 | 3 | | 1 | 0 | 1295 | +-----------+-----------+----------+ +-----------------+-------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-----------------+-------------+------+-----+---------+----------------+ | hmID | int(11) | NO | PRI | NULL | auto_increment | | hmHostID | int(11) | NO | MUL | NULL | | | hmMAC | varchar(59) | NO | UNI | NULL | | | hmDesc | longtext | NO | | '' | | | hmPrimary | tinyint(1) | NO | | 0 | | | hmPending | tinyint(1) | NO | | 0 | | | hmIgnoreClient | tinyint(1) | NO | | 0 | | | hmIgnoreImaging | tinyint(1) | NO | | 0 | | +-----------------+-------------+------+-----+---------+----------------+

                        [Wed Nov 18 00:50:12.685862 2026] [mpm_prefork:notice] [pid 15288:tid 15288] AH00163: Apache/2.4.66 (Ubuntu) OpenSSL/3.5.5 configured -- resuming normal operations [Wed Nov 18 00:50:12.685895 2026] [core:notice] [pid 15288:tid 15288] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND' [Wed Nov 18 07:25:19.794937 2026] [proxy_fcgi:error] [pid 207505:tid 207505] [client 10.220.99.11:63745] AH01071: Got error 'PHP message: FOG storage node availability: the group has no enabled master node. Group Adamsville Storage Group (id 2); node id 4 chosen anyway. The probe is a reachability test, not a test of the image share, so tasking continues -- if the task fails, check that this node is reachable.', referer: https://tv-fog-main.tvschools.local/fog/management/index.php?node=host&sub=list

                        1. I don’t recall seeing any errors.
                        Tom ElliottT 1 Reply Last reply Reply Quote 0
                        • Tom ElliottT
                          Tom Elliott @FlareImp
                          last edited by

                          @FlareImp Thank you, that helps. Your database is correct: 1295 MACs are marked primary. So the flag is lost after FOG reads it. My lab runs the same code and shows the primary MAC, so I need to see what your server sends back.

                          Please post these:

                          1. Sign in to the FOG UI. Open a host that shows no primary MAC, and note the id= number in its address bar. Then open this address in the same browser, with that number at the end, and post what the page shows:
                            https://tv-fog-main.tvschools.local/fog/management/index.php?node=host&sub=getMacaddressesList&id=<id>
                          2. Open this address, with one of that host’s MACs, and post what the page shows. It is the boot script the host gets at PXE boot:
                            https://tv-fog-main.tvschools.local/fog/service/ipxe/boot.php?mac=<mac>
                          3. The PHP and database versions, and whether the update upgraded them:
                            php -v; mysql --version; grep -E " (upgrade|install) .*(php|mysql|mariadb)" /var/log/dpkg.log | tail -20

                          One separate item from your log: “the group has no enabled master node” for Adamsville Storage Group (id 2). Every storage group needs one enabled master node. Captures upload to the master, and replication copies images from it. Open that storage group’s nodes and set one of them as the master.

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          1 Reply Last reply Reply Quote 0
                          • 1 / 1
                          • First post
                            Last post

                          45

                          Online

                          12.8k

                          Users

                          17.7k

                          Topics

                          157.5k

                          Posts
                          Copyright © 2012-2026 FOG Project