Fog 1.6 Multiple Sites, Location not working as intended

[FlareImp]

@Tom-Elliott My Apologies, when I say site, I’m referring to the location where my fog VMs are at, not what is within the Fog WebGUI.

Open the host, go to its Location tab. Is it set to the Site B location?
All 31 Computers are set to “Site B”

Open that location. Which storage group and which storage node does it show?
The Storage Group is “SiteB” and the Storage Node is “SiteB-Test”

Is the image listed in the Site B storage group (Image > Storage Groups tab)?
The Image is associated with “Site A” and “Site B”

[FlareImp]

When I deploy the image it will show the correct storage node under tasks up until the point of actual deployment, then it switches back to the default location of the Main Fog Server.

[Tom Elliott]

@FlareImp Thank you, that detail found it. This is a bug in the 1.6 installer, not your setup.

What happens: a storage node install does not download the plugins, so your Site B node has no location plugin in its web root. Your Site B clients PXE boot from that node, so its boot.php answers them. Without the plugin, it falls back to the image’s primary storage group and changes the task to that node. That is the switch you see when the deploy starts.

The fix: https://github.com/FOGProject/fogproject/pull/2050. It is on dev-branch now and reaches stable in the next stable release.

You have two ways to get going before then.

Option 1: copy the plugins to the node (stays on 1.6.0.14). Run this on the Site B node as root:

rsync -a root@<main-server-ip>:/var/www/html/fog/lib/plugins/ /var/www/html/fog/lib/plugins/

Then deploy again. A reinstall of the storage node removes the copy, so repeat it after each storage node install until you are on the fixed release.

Option 2: move to dev-branch (tests the real fix). Update the main server FIRST, then the storage node. Do not update only the storage node: dev-branch has a newer database schema than 1.6.0.14, and a storage node never updates the schema. Until the main server updates it, the node sends every PXE boot to the schema update page instead of the boot menu.

On the main server, then on the Site B node, from your fogproject checkout:

cd /path/to/fogproject/bin
sudo ./updatefog.sh --branch dev-branch

On the main server, answer Y when the installer asks to update the database schema. Finish the main server before you start the node. --branch is a one-off: the next plain updatefog.sh run goes back to the stable channel.

Either way, please let me know if the Site B deploy now pulls from SiteB-Test.

[FlareImp]

@Tom-Elliott Option 1 and doing a reboot of the site B node has taken care of the issue. Thanks for your awesomness!

I have come across another problem though. Would you like a new thread started or just add it here?

[Tom Elliott]

@FlareImp What’s the issue?

[FlareImp]

@Tom-Elliott I assigned an image to site B and it starts to replicate but then after a time it will stop coping and the folder will disappear from the /images directory on the storage node. Then once the replication service kicks off again it starts all over again. I didn’t notice this before because I manually ran rsync to mirror what was in the images directory on the main fog server. I have to remote site B set to throttle at 200mb since I don’t want to saturate the pipe.

[Tom Elliott]

@FlareImp Glad option 1 worked.

The replication restart is a bug too. Each replication cycle is supposed to skip an image whose copy is still running. That check looked in the wrong place, so it never skipped. The next cycle saw the half-copied file on Site B as the wrong size, deleted it, stopped the copy, and started it again. The image replicator runs every 600 seconds by default. Any image that takes longer than that to copy over your throttled link never finishes.

The fix: https://github.com/FOGProject/fogproject/pull/2051.

Until it is in a release, make the cycle longer than your biggest image takes to copy. On the main server, go to FOG Configuration > FOG Settings > FOG Linux Service Sleep Times, and raise IMAGEREPSLEEPTIME (in seconds). For example, if your largest image takes about 2 hours to copy at 200 Mbit, set it to 10800.

One question so I can be sure this covers everything: when the copy stops, is the image folder on Site B really gone, or is it there but empty or missing files? The bug above deletes files, not the folder. If the folder itself disappears, please post the lines from /opt/fog/log/fogreplicator.log on the main server around that time.

[FlareImp]

@Tom-Elliott That took care of the replication issue. The folder itself doesn’t vanish. here is the log from my attempt yesterday. “Win11_25H2_Smart26” is the image and TVNE-Fog-Test is the site B fog node.

[11-02-26 4:59:14 pm] | Deleting remote file: win11/7450 AIO-WIN10-A10-YHP57.CAB
chmod: Access failed: 550 SITE CHMOD command failed. (./d1p3.img)
[11-02-26 4:59:16 pm] | Drivers_OptiPlex7450AIO: No need to sync win11/Video.zip file to TVNE-Fog-Test
[11-02-26 4:59:17 pm] | Drivers_OptiPlex7450AIO: No need to sync WLAN-Tri-Valley.xml file to TVNE-Fog-Test
[11-02-26 4:59:17 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file “/opt/fog/log/fogreplicator.Drivers_OptiPlex7450AIO.transfer.TVNE-Fog-Test.log”;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;set net:limit-tot>
[11-02-26 4:59:17 pm] * Started sync for Image Drivers_OptiPlex7450AIO - Resource id #1100
[11-02-26 4:59:17 pm] * Not syncing Image between groups
[11-02-26 4:59:17 pm] | Image Name: Win10Lgcy22H2
[11-02-26 4:59:17 pm] | There are no members to sync to
[11-02-26 4:59:17 pm] * Found Image to transfer to 2 groups
[11-02-26 4:59:17 pm] | Image Name: Win11_25H2_Base
[11-02-26 4:59:19 pm] | Win11_25H2_Base: No need to sync d1.fixed_size_partitions file to TVNE-Fog-Test
[11-02-26 4:59:19 pm] | Win11_25H2_Base: No need to sync d1.mbr file to TVNE-Fog-Test
[11-02-26 4:59:20 pm] | Win11_25H2_Base: No need to sync d1.minimum.partitions file to TVNE-Fog-Test
[11-02-26 4:59:21 pm] | Win11_25H2_Base: No need to sync d1.original.fstypes file to TVNE-Fog-Test
[11-02-26 4:59:22 pm] | Win11_25H2_Base: No need to sync d1.original.swapuuids file to TVNE-Fog-Test
[11-02-26 4:59:23 pm] | Win11_25H2_Base: No need to sync d1.partitions file to TVNE-Fog-Test
[11-02-26 4:59:24 pm] | Win11_25H2_Base: No need to sync d1.shrunken.partitions file to TVNE-Fog-Test
[11-02-26 4:59:25 pm] | Win11_25H2_Base: No need to sync d1p1.img file to TVNE-Fog-Test
[11-02-26 4:59:26 pm] | Win11_25H2_Base: No need to sync d1p2.img file to TVNE-Fog-Test
[11-02-26 4:59:27 pm] | Win11_25H2_Base: No need to sync d1p3.img file to TVNE-Fog-Test
[11-02-26 4:59:28 pm] | Win11_25H2_Base: No need to sync d1p4.img file to TVNE-Fog-Test
[11-02-26 4:59:28 pm] * All files synced for this item
[11-02-26 4:59:28 pm] * Found Image to transfer to 2 groups
[11-02-26 4:59:28 pm] | Image Name: Win11_25H2_Smart26
[11-02-26 4:59:29 pm] | Win11_25H2_Smart26: No need to sync d1.fixed_size_partitions file to TVNE-Fog-Test
[11-02-26 4:59:30 pm] | Win11_25H2_Smart26: No need to sync d1.mbr file to TVNE-Fog-Test
[11-02-26 4:59:31 pm] | Win11_25H2_Smart26: No need to sync d1.minimum.partitions file to TVNE-Fog-Test
[11-02-26 4:59:32 pm] | Win11_25H2_Smart26: No need to sync d1.original.fstypes file to TVNE-Fog-Test
[11-02-26 4:59:33 pm] | Win11_25H2_Smart26: No need to sync d1.original.swapuuids file to TVNE-Fog-Test
[11-02-26 4:59:34 pm] | Win11_25H2_Smart26: No need to sync d1.partitions file to TVNE-Fog-Test
[11-02-26 4:59:35 pm] | Win11_25H2_Smart26: No need to sync d1.shrunken.partitions file to TVNE-Fog-Test
[11-02-26 4:59:36 pm] | Win11_25H2_Smart26: No need to sync d1p1.img file to TVNE-Fog-Test
[11-02-26 4:59:37 pm] | Win11_25H2_Smart26: No need to sync d1p2.img file to TVNE-Fog-Test
[11-02-26 4:59:37 pm] # Win11_25H2_Smart26: File does not exist d1p3.img(TVNE-Fog-Test)
[11-02-26 4:59:38 pm] | Win11_25H2_Smart26: No need to sync d1p4.img file to TVNE-Fog-Test
[11-02-26 4:59:38 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file “/opt/fog/log/fogreplicator.Win11_25H2_Smart26.transfer.TVNE-Fog-Test.log”;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;set net:limit-total-ra>
[11-02-26 4:59:38 pm] * Started sync for Image Win11_25H2_Smart26 - Resource id #1365
[11-02-26 4:59:38 pm] * Attempting to perform Group -> Nodes image replication.
[11-02-26 4:59:38 pm] * Not syncing Image between nodes
[11-02-26 4:59:38 pm] | Image Name: Apps
[11-02-26 4:59:38 pm] | There are no members to sync to
[11-02-26 4:59:38 pm] * Not syncing Image between nodes
[11-02-26 4:59:38 pm] | Image Name: Drivers
[11-02-26 4:59:38 pm] | There are no members to sync to
[11-02-26 4:59:38 pm] * Not syncing Image between nodes
[11-02-26 4:59:38 pm] | Image Name: Drivers_OptiPlex7450AIO
[11-02-26 4:59:38 pm] | There are no members to sync to
[11-02-26 4:59:38 pm] * Not syncing Image between nodes
[11-02-26 4:59:38 pm] | Image Name: Win10Lgcy22H2
[11-02-26 4:59:38 pm] | There are no members to sync to
[11-02-26 4:59:38 pm] * Not syncing Image between nodes
[11-02-26 4:59:38 pm] | Image Name: Win11_25H2_Base
[11-02-26 4:59:38 pm] | There are no members to sync to
[11-02-26 4:59:38 pm] * Not syncing Image between nodes
[11-02-26 4:59:38 pm] | Image Name: Win11_25H2_Smart26
[11-02-26 4:59:38 pm] | There are no members to sync to
[11-02-26 4:59:38 pm] | Sync finished - Resource id #1035

[Tom Elliott]

@FlareImp So we’re good to go then?

[FlareImp]

@Tom-Elliott All good! Thank you and keep up the awesome work!