FOG Node Propagation issues....FTP? Faulty Install?

[danieln]

Hello,

OS: Debian 10
FOG Version: 1.5.9-RC2

I installed an additional node to my FOG Cluster today and it doesn’t appear to be propagating images correctly. I currently only have the Master Node, Node 1, and now the (new) Node 2.

Here is a snippet of my log file:

[02-06-23 8:46:44 pm]   # VACA Pearson - 840 G3: No need to sync d1.original.fstypes (Node 1)
[02-06-23 8:46:44 pm]   # VACA Pearson - 840 G3: No need to sync d1.original.swapuuids (Node 1)
[02-06-23 8:46:44 pm]   # VACA Pearson - 840 G3: No need to sync d1.partitions (Node 1)
[02-06-23 8:46:45 pm]   # VACA Pearson - 840 G3: No need to sync d1p1.img (Node 1)
[02-06-23 8:46:47 pm]   # VACA Pearson - 840 G3: No need to sync d1p2.img (Node 1)
[02-06-23 8:46:49 pm]   # VACA Pearson - 840 G3: No need to sync d1p3.img (Node 1)
[02-06-23 8:46:49 pm]  * All files synced for this item.
[02-06-23 8:46:50 pm]   # VACA Pearson - 840 G3: No need to sync d1.fixed_size_partitions (Node 2)
[02-06-23 8:46:50 pm]   # VACA Pearson - 840 G3: No need to sync d1.mbr (Node 2)
[02-06-23 8:46:50 pm]   # VACA Pearson - 840 G3: No need to sync d1.minimum.partitions (Node 2)
[02-06-23 8:46:50 pm]   # VACA Pearson - 840 G3: No need to sync d1.original.fstypes (Node 2)
[02-06-23 8:46:50 pm]   # VACA Pearson - 840 G3: No need to sync d1.original.swapuuids (Node 2)
[02-06-23 8:46:50 pm]   # VACA Pearson - 840 G3: No need to sync d1.partitions (Node 2)
[02-06-23 8:46:51 pm]   # VACA Pearson - 840 G3: No need to sync d1p1.img (Node 2)
[02-06-23 8:46:51 pm]   # VACA Pearson - 840 G3: File size mismatch - d1p2.img: 11539496891 != 2360515784
[02-06-23 8:46:51 pm]   # VACA Pearson - 840 G3: Deleting remote file d1p2.img
[02-06-23 8:46:52 pm]   # VACA Pearson - 840 G3: No need to sync d1p3.img (Node 2)
[02-06-23 8:46:52 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogreplicator.VACA Pearson - 840 G3.transfer.Node 2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/VACAPearson-840G3" "/images/VACAPearson-840G3"; exit' -u fogproject,[Protected] 10.15.0.4
[02-06-23 8:46:52 pm]  | Started sync for Image VACA Pearson - 840 G3 - Resource id #4469
[02-06-23 8:46:52 pm]  * Found Image to transfer to 2 nodes
[02-06-23 8:46:52 pm]  | Image Name: Visit by GES v4.20.81 - 840 G3
[02-06-23 8:46:53 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.fixed_size_partitions (Node 1)
[02-06-23 8:46:53 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.grub.mbr (Node 1)
[02-06-23 8:46:53 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.mbr (Node 1)
[02-06-23 8:46:53 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.minimum.partitions (Node 1)
[02-06-23 8:46:53 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.original.fstypes (Node 1)
[02-06-23 8:46:53 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.original.swapuuids (Node 1)
[02-06-23 8:46:53 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.partitions (Node 1)
[02-06-23 8:46:54 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1p1.img (Node 1)
[02-06-23 8:46:56 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1p2.img (Node 1)
[02-06-23 8:46:58 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1p3.img (Node 1)
[02-06-23 8:46:58 pm]  * All files synced for this item.
[02-06-23 8:46:58 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.fixed_size_partitions (Node 2)
[02-06-23 8:46:58 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.grub.mbr (Node 2)
[02-06-23 8:46:58 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.mbr (Node 2)
[02-06-23 8:46:58 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.minimum.partitions (Node 2)
[02-06-23 8:46:58 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.original.fstypes (Node 2)
[02-06-23 8:46:59 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.original.swapuuids (Node 2)
[02-06-23 8:46:59 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1.partitions (Node 2)
[02-06-23 8:46:59 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1p1.img (Node 2)
[02-06-23 8:46:59 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1p2.img (Node 2)
[02-06-23 8:47:00 pm]   # Visit by GES v4.20.81 - 840 G3: No need to sync d1p3.img (Node 2)
[02-06-23 8:47:00 pm]  * All files synced for this item.
[02-06-23 8:47:00 pm]  * Found Image to transfer to 2 nodes
[02-06-23 8:47:00 pm]  | Image Name: Visit GES Check-In v5.1.82 - 840 G3
[02-06-23 8:47:00 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.fixed_size_partitions (Node 1)
[02-06-23 8:47:00 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.grub.mbr (Node 1)
[02-06-23 8:47:00 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.mbr (Node 1)
[02-06-23 8:47:00 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.minimum.partitions (Node 1)
[02-06-23 8:47:00 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.original.fstypes (Node 1)
[02-06-23 8:47:00 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.original.swapuuids (Node 1)
[02-06-23 8:47:01 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.partitions (Node 1)
[02-06-23 8:47:01 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1p1.img (Node 1)
[02-06-23 8:47:04 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1p2.img (Node 1)
[02-06-23 8:47:06 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1p3.img (Node 1)
[02-06-23 8:47:06 pm]  * All files synced for this item.
[02-06-23 8:47:06 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.fixed_size_partitions (Node 2)
[02-06-23 8:47:06 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.grub.mbr (Node 2)
[02-06-23 8:47:06 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.mbr (Node 2)
[02-06-23 8:47:06 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.minimum.partitions (Node 2)
[02-06-23 8:47:06 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.original.fstypes (Node 2)
[02-06-23 8:47:06 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.original.swapuuids (Node 2)
[02-06-23 8:47:06 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1.partitions (Node 2)
[02-06-23 8:47:06 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1p1.img (Node 2)
[02-06-23 8:47:07 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1p2.img (Node 2)
[02-06-23 8:47:07 pm]   # Visit GES Check-In v5.1.82 - 840 G3: No need to sync d1p3.img (Node 2)
[02-06-23 8:47:07 pm]  * All files synced for this item.
[02-06-23 8:47:07 pm]  * Found Image to transfer to 2 nodes
[02-06-23 8:47:07 pm]  | Image Name: Widex USA - 840 G3
[02-06-23 8:47:09 pm]   # Widex USA - 840 G3: No need to sync d1.fixed_size_partitions (Node 1)
[02-06-23 8:47:09 pm]   # Widex USA - 840 G3: No need to sync d1.mbr (Node 1)
[02-06-23 8:47:09 pm]   # Widex USA - 840 G3: No need to sync d1.minimum.partitions (Node 1)
[02-06-23 8:47:09 pm]   # Widex USA - 840 G3: No need to sync d1.original.fstypes (Node 1)
[02-06-23 8:47:09 pm]   # Widex USA - 840 G3: No need to sync d1.original.swapuuids (Node 1)
[02-06-23 8:47:09 pm]   # Widex USA - 840 G3: No need to sync d1.partitions (Node 1)
[02-06-23 8:47:11 pm]   # Widex USA - 840 G3: No need to sync d1p1.img (Node 1)
[02-06-23 8:47:21 pm]   # Widex USA - 840 G3: No need to sync d1p2.img (Node 1)
[02-06-23 8:47:33 pm]   # Widex USA - 840 G3: No need to sync d1p3.img (Node 1)
[02-06-23 8:47:33 pm]  * All files synced for this item.
[02-06-23 8:47:33 pm]   # Widex USA - 840 G3: No need to sync d1.fixed_size_partitions (Node 2)
[02-06-23 8:47:33 pm]   # Widex USA - 840 G3: No need to sync d1.mbr (Node 2)
[02-06-23 8:47:33 pm]   # Widex USA - 840 G3: No need to sync d1.minimum.partitions (Node 2)
[02-06-23 8:47:33 pm]   # Widex USA - 840 G3: No need to sync d1.original.fstypes (Node 2)
[02-06-23 8:47:33 pm]   # Widex USA - 840 G3: No need to sync d1.original.swapuuids (Node 2)
[02-06-23 8:47:33 pm]   # Widex USA - 840 G3: No need to sync d1.partitions (Node 2)
[02-06-23 8:47:34 pm]   # Widex USA - 840 G3: No need to sync d1p1.img (Node 2)
[02-06-23 8:47:34 pm]   # Widex USA - 840 G3: File size mismatch - d1p2.img: 22242172849 != 5294692168
[02-06-23 8:47:34 pm]   # Widex USA - 840 G3: Deleting remote file d1p2.img
[02-06-23 8:47:36 pm]   # Widex USA - 840 G3: No need to sync d1p3.img (Node 2)
[02-06-23 8:47:36 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogreplicator.Widex USA - 840 G3.transfer.Node 2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/WidexUSA-840G3" "/images/WidexUSA-840G3"; exit' -u fogproject,[Protected] 10.15.0.4

Here it appears to be syncing the images between the two nodes, but it has the CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogreplicator.<image name>.transfer.Node 2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/<image name>" "/images/<image name>"; exit' -u fogproject,[Protected] 10.15.0.4 text after the last line it tries to sync on Node 2.

Here is another snippet of the same log file later in the log (check the timestamp):

[02-06-23 9:33:30 pm]  | Image Name: ! Dell 7480
[02-06-23 9:33:30 pm]   # ! Dell 7480: No need to sync d1.fixed_size_partitions (Node 1)
[02-06-23 9:33:30 pm]   # ! Dell 7480: No need to sync d1.mbr (Node 1)
[02-06-23 9:33:30 pm]   # ! Dell 7480: No need to sync d1.minimum.partitions (Node 1)
[02-06-23 9:33:30 pm]   # ! Dell 7480: No need to sync d1.original.fstypes (Node 1)
[02-06-23 9:33:30 pm]   # ! Dell 7480: No need to sync d1.original.swapuuids (Node 1)
[02-06-23 9:33:30 pm]   # ! Dell 7480: No need to sync d1.partitions (Node 1)
[02-06-23 9:33:31 pm]   # ! Dell 7480: No need to sync d1p1.img (Node 1)
[02-06-23 9:33:44 pm]   # ! Dell 7480: No need to sync d1p2.img (Node 1)
[02-06-23 9:33:44 pm]  * All files synced for this item.
[02-06-23 9:33:44 pm] | Replication already running with PID: 2531
[02-06-23 9:33:44 pm]  * Found Image to transfer to 2 nodes
[02-06-23 9:33:44 pm]  | Image Name: ! Dell E5450
[02-06-23 9:33:44 pm]   # ! Dell E5450: No need to sync d1.fixed_size_partitions (Node 1)
[02-06-23 9:33:44 pm]   # ! Dell E5450: No need to sync d1.mbr (Node 1)
[02-06-23 9:33:44 pm]   # ! Dell E5450: No need to sync d1.minimum.partitions (Node 1)
[02-06-23 9:33:44 pm]   # ! Dell E5450: No need to sync d1.original.fstypes (Node 1)
[02-06-23 9:33:44 pm]   # ! Dell E5450: No need to sync d1.original.swapuuids (Node 1)
[02-06-23 9:33:44 pm]   # ! Dell E5450: No need to sync d1.partitions (Node 1)
[02-06-23 9:33:44 pm]   # ! Dell E5450: No need to sync d1p1.img (Node 1)
[02-06-23 9:33:55 pm]   # ! Dell E5450: No need to sync d1p2.img (Node 1)
[02-06-23 9:33:55 pm]  * All files synced for this item.
[02-06-23 9:33:55 pm] | Replication already running with PID: 2548
[02-06-23 9:33:55 pm]  * Found Image to transfer to 2 nodes
[02-06-23 9:33:55 pm]  | Image Name: ! Dell E5480
[02-06-23 9:33:56 pm]   # ! Dell E5480: No need to sync d1.fixed_size_partitions (Node 1)
[02-06-23 9:33:56 pm]   # ! Dell E5480: No need to sync d1.mbr (Node 1)
[02-06-23 9:33:56 pm]   # ! Dell E5480: No need to sync d1.minimum.partitions (Node 1)
[02-06-23 9:33:56 pm]   # ! Dell E5480: No need to sync d1.original.fstypes (Node 1)
[02-06-23 9:33:56 pm]   # ! Dell E5480: No need to sync d1.original.swapuuids (Node 1)
[02-06-23 9:33:56 pm]   # ! Dell E5480: No need to sync d1.partitions (Node 1)
[02-06-23 9:33:56 pm]   # ! Dell E5480: No need to sync d1p1.img (Node 1)
[02-06-23 9:34:16 pm]   # ! Dell E5480: No need to sync d1p2.img (Node 1)
[02-06-23 9:34:16 pm]  * All files synced for this item.
[02-06-23 9:34:16 pm] | Replication already running with PID: 2585
[02-06-23 9:34:16 pm]  * Found Image to transfer to 2 nodes
[02-06-23 9:34:16 pm]  | Image Name: ! Dell E7470
[02-06-23 9:34:17 pm]   # ! Dell E7470: No need to sync d1.fixed_size_partitions (Node 1)
[02-06-23 9:34:17 pm]   # ! Dell E7470: No need to sync d1.mbr (Node 1)
[02-06-23 9:34:17 pm]   # ! Dell E7470: No need to sync d1.minimum.partitions (Node 1)
[02-06-23 9:34:17 pm]   # ! Dell E7470: No need to sync d1.original.fstypes (Node 1)
[02-06-23 9:34:17 pm]   # ! Dell E7470: No need to sync d1.original.swapuuids (Node 1)
[02-06-23 9:34:17 pm]   # ! Dell E7470: No need to sync d1.partitions (Node 1)
[02-06-23 9:34:17 pm]   # ! Dell E7470: No need to sync d1p1.img (Node 1)
[02-06-23 9:34:32 pm]   # ! Dell E7470: No need to sync d1p2.img (Node 1)
[02-06-23 9:34:32 pm]  * All files synced for this item.
[02-06-23 9:34:32 pm] | Replication already running with PID: 2603
[02-06-23 9:34:32 pm]  * Found Image to transfer to 2 nodes
[02-06-23 9:34:32 pm]  | Image Name: ! HP Elitebook 840 G3
[02-06-23 9:34:33 pm]   # ! HP Elitebook 840 G3: No need to sync d1.fixed_size_partitions (Node 1)
[02-06-23 9:34:33 pm]   # ! HP Elitebook 840 G3: No need to sync d1.mbr (Node 1)

Here it’s only syncing to Node 1. It just stops syncing to Node 2. Weird, right?

I did see on a previous log there was a Type: 1024, File: /var/www/fog/lib/fog/fogftp.class.php, Line: 219, Message: FTP connection failed, which has not popped up again…but that doesn’t sound good!

I will say that I had issues during the installation of this node if you reference this post but I just reinstalled and it worked and didn’t think anything of it.

Also, the FOG dashboard is running REALLY slow when I try to load it. If I disable the new node that I just installed it seems to run quicker. Does that make sense to anybody? Does it need to have a solid FTP connection or be able to smoothly communicate to all nodes in order to load the FOG Dashboard? I’ve seen that before where the FOG dash is slow and just shutting off one of the problematic nodes seems to make it run a lot faster. Just curious.

Any help or insight would be supremely appreciated. I feel like I may just need to cut my losses and just try to wipe the machine and reinstall it as a new node again, but i just wanted to run it by some of the folks here to see if there are any ideas…

Thanks in advance!

[danieln]

Hey everyone,

Just wanted to issue an update on this as it’s still marked as “unsolved” and I’ve done a bit of tinkering and learning here.

I ended up just cutting my losses and wiping the external storage node, reinstalling Debian, and cloning the FOG repo again to do a fresh install. In the process of doing this I noticed that on my storage node page on the dashboard, I already had a storage node set under the same IP address of the node that I was adding, but it was just disabled on the FOG dashboard, so it appears that there was an IP conflict that may have been causing some of the issues I was seeing. Maybe not though(?).

At any rate, I removed the old node and replaced it with the new one via the normal installation methods and everything has been going smoothly for the past few days.

Feel free to mark this one as “Solved”
Thanks,

[Sebastian Roth]

@danieln Great to see you figured this one out yourself. Would have been quite a challenge for us to help find that issue.