Fog 1.6.0.14 Master Storage Node not handing off to Slave Storage Nodes
-
At one of my slave locations I just imaged a 32 PC lab and they all pulled from the Master Storage Node. I have two other Storage Nodes in the group but neither of them got any tasks. Everything stayed with the Master Storage Node. Once the Que was full they just waited in line rather than get the image from one of the other two nodes that had no clients. I even tried coping the Plugin folder to the nodes but that didn’t change anything.
Fog 1.6.0.14
Ubuntu 26.04.1 -
@FlareImp Thank you. This is a bug in 1.6, and it explains what you see.
What happens: FOG picks a storage node for each host when you create the task. It should pick the node with the lowest load. In 1.6 the load value was always empty, so every task went to the first node in the group, which is your master. A host also stays on the node it got at task creation. When that node is full, the host waits in line there, even when another node in the group is idle.
The fix: https://github.com/FOGProject/fogproject/pull/2082. It does two things:
- Task creation picks the least-loaded node in the group.
- At boot, a host whose node is full moves to the node in its group with the most free slots. A capture never moves, because it must upload to the master.
It goes to dev-branch first, and reaches stable in the next stable release.
Check your location settings first. Copying the plugin folder does not change this, but your location can. If the location for those hosts has a Storage Node set, every host in that location goes to that one node. That is by design, and the fix does not change it. To spread the hosts across the group, open the location, clear the Storage Node, and keep the Storage Group set.
-
@FlareImp One correction to my last post. With the fix, a location’s Storage Node is where its hosts start, not a hard limit. When that node is full, a waiting host moves to a free node in the location’s own storage group. It never leaves that group, so the image still comes from your site. You can leave the location’s Storage Node set.
-
@Tom-Elliott Okay, thank you for the information.
-
@Tom-Elliott I just imaged a lab of 30. At my remote location, there is one Master node and two slave nodes for a count of 30. In the task page it shows 10 systems at each node but on the Partclone display on the clients they are all showing just the single Master node. I also confirmed this by watching the bandwidth on all 3 nodes. node 1 is maxed, nodes 2 and 3 are not doing anything.
Fog Dev Branch 1.6.0.167
-
@FlareImp Thank you, that test found it. The task list and the clients disagreed, and the clients were right.
What happens: a client learns its storage node once, at PXE boot, and mounts that node. My earlier fix moved a waiting host to a free node later, at check-in. That changed the task list, but not the node the client had already mounted. Your location sends its hosts to the master, so all 30 imaged from the master while the task list showed 10 on each node.
The fix: https://github.com/FOGProject/fogproject/pull/2103. The move now happens at PXE boot. When a host boots and its node already has more tasks than its Max Clients, it gets the node in the same storage group with the most free slots. The client mounts that node, and the task list shows the same node. The check-in move is removed. Captures and multicast still use the master.
With your setup, hosts move off the master as they boot until it holds 10, and the other two nodes take 10 each. Hosts that PXE boot at the same instant can still land on the same node. Those extra hosts wait in line there.
It goes to dev-branch first. After it merges, update the main server and the storage nodes with
updatefog.sh --branch dev-branch, as before, and deploy the lab again.Until then: clear the Storage Node on that location and keep the Storage Group set. Then each host gets the least-loaded node in the group at PXE boot.
Please let me know what the task list and the node bandwidth show on the next run.
-
@Tom-Elliott Thank you for this, I will give it a try. School has started so I won’t be able to give an update until tomorrow morning.
-
@Tom-Elliott after upgrading to the latest dev branch, when you pxe a host it shows as not being registered even though it showed up correctly on the previous version of FOG. If you check the mac on any of the hosts, nothing is selected as primary and after making a choice and refreshing the screen it goes back to nothing selected.