FOG UEFI image sync with storage node seems to be looping\failing on UUID info.

BardWood · Nov 27, 2017, 6:03 PM

Greetings all,

I’ve checked the SN and it does have data. I’m not really sure if this is specifically related to EFI images but I’ve had many issues with UUID and the efi images. This seems to be another instance of UUID weirdness. I’ve let this sync run for ~5 days but it never seems to finish. We are talking about roughly 70GB over a 10mb link (VPLS) so it’s possible that it’s still not complete. Anyway, here’s a ton of replication log output:

[11-27-17 10:53:21 pm] | X1CG2-UEFI-FC-V1: No need to sync d1.partitions file to dub-imgstore-s
[11-27-17 10:53:13 pm] | X1CG2-UEFI-FC-V1: No need to sync d1.original.uuids file to dub-imgstore-s
[11-27-17 10:53:03 pm] * Deleting remote file: /images/X1CG2-UEFI-FC-V1/
[11-27-17 10:53:03 pm] | Files do not match.
[11-27-17 10:53:03 pm] | 0 0 /images/X1CG2-UEFI-FC-V1/d1.original.swapuuids ftp://fog:Ec9m%2FQHVbCJkFwUGMkqCEFvxxpldVFGfZnW49xVuZWU%3D@172.18.40.2/images/X1CG2-UEFI-FC-V1/
[11-27-17 10:52:56 pm] | X1CG2-UEFI-FC-V1: No need to sync d1.original.fstypes file to dub-imgstore-s
[11-27-17 10:52:47 pm] | X1CG2-UEFI-FC-V1: No need to sync d1.minimum.partitions file to dub-imgstore-s
[11-27-17 10:52:38 pm] | X1CG2-UEFI-FC-V1: No need to sync d1.mbr file to dub-imgstore-s
[11-27-17 10:52:25 pm] | X1CG2-UEFI-FC-V1: No need to sync d1.fixed_size_partitions file to dub-imgstore-s
[11-27-17 10:52:14 pm] | Image Name: X1CG2-UEFI-FC-V1
[11-27-17 10:52:14 pm] * Found Image to transfer to 1 node
[11-27-17 10:52:14 pm] * Started sync for Image O7010-WIN10-LEGACY
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.O7010-WIN10-LEGACY.transfer.dub-imgstore-s.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/O7010-WIN10-LEGACY" "/images/O7010-WIN10-LEGACY"; exit' -u fog,[Protected] 172.18.40.2
[11-27-17 10:52:14 pm] | CMD:
[11-27-17 10:52:14 pm] * Starting Sync Actions
[11-27-17 10:52:14 pm] * Deleting remote file: /images/O7010-WIN10-LEGACY/d1p2.img
[11-27-17 10:52:14 pm] | Files do not match.
[11-27-17 10:52:14 pm] | 11381549962 0 /images/O7010-WIN10-LEGACY/d1p2.img ftp://fog:Ec9m%2FQHVbCJkFwUGMkqCEFvxxpldVFGfZnW49xVuZWU%3D@172.18.40.2/images/O7010-WIN10-LEGACY/d1p2.img
[11-27-17 10:52:08 pm] | O7010-WIN10-LEGACY: No need to sync d1p1.img file to dub-imgstore-s
[11-27-17 10:46:20 pm] | O7010-WIN10-LEGACY: No need to sync d1.partitions file to dub-imgstore-s
[11-27-17 10:46:10 pm] * Deleting remote file: /images/O7010-WIN10-LEGACY/
[11-27-17 10:46:10 pm] | Files do not match.
[11-27-17 10:46:10 pm] | 0 0 /images/O7010-WIN10-LEGACY/d1.original.swapuuids ftp://fog:Ec9m%2FQHVbCJkFwUGMkqCEFvxxpldVFGfZnW49xVuZWU%3D@172.18.40.2/images/O7010-WIN10-LEGACY/
[11-27-17 10:46:03 pm] | O7010-WIN10-LEGACY: No need to sync d1.original.fstypes file to dub-imgstore-s
[11-27-17 10:45:54 pm] | O7010-WIN10-LEGACY: No need to sync d1.minimum.partitions file to dub-imgstore-s
[11-27-17 10:45:45 pm] | O7010-WIN10-LEGACY: No need to sync d1.mbr file to dub-imgstore-s
[11-27-17 10:45:32 pm] | O7010-WIN10-LEGACY: No need to sync d1.fixed_size_partitions file to dub-imgstore-s
[11-27-17 10:45:21 pm] | Image Name: O7010-WIN10-LEGACY
[11-27-17 10:45:21 pm] * Found Image to transfer to 1 node
[11-27-17 10:45:21 pm] * Attempting to perform Group -> Nodes image replication.
[11-27-17 10:45:21 pm] * Started sync for Image X1CG5-UEFI-FC-V1
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.X1CG5-UEFI-FC-V1.transfer.SF-MASTER.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/X1CG5-UEFI-FC" "/images/X1CG5-UEFI-FC"; exit' -u fog,[Protected] 172.16.40.33
[11-27-17 10:45:21 pm] | CMD:
[11-27-17 10:45:21 pm] * Starting Sync Actions
[11-27-17 10:45:21 pm] * Deleting remote file: /images/X1CG5-UEFI-FC/
[11-27-17 10:45:21 pm] | Files do not match.
[11-27-17 10:45:21 pm] | 14512943391 0 /images/X1CG5-UEFI-FC/d1p4.img cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:21 pm] * Deleting remote file: /images/X1CG5-UEFI-FC/
[11-27-17 10:45:21 pm] | Files do not match.
[11-27-17 10:45:21 pm] | 6259444 0 /images/X1CG5-UEFI-FC/d1p3.img cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:20 pm] * Deleting remote file: /images/X1CG5-UEFI-FC/
[11-27-17 10:45:20 pm] | Files do not match.
[11-27-17 10:45:20 pm] | 12902901 0 /images/X1CG5-UEFI-FC/d1p2.img cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:20 pm] * Deleting remote file: /images/X1CG5-UEFI-FC/
[11-27-17 10:45:20 pm] | Files do not match.
[11-27-17 10:45:20 pm] | 357253418 0 /images/X1CG5-UEFI-FC/d1p1.img cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:20 pm] * Deleting remote file: /images/X1CG5-UEFI-FC/
[11-27-17 10:45:20 pm] | Files do not match.
[11-27-17 10:45:20 pm] | 886 0 /images/X1CG5-UEFI-FC/d1.partitions cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:20 pm] | X1CG5-UEFI-FC-V1: No need to sync d1.original.swapuuids file to SF-MASTER
[11-27-17 10:45:19 pm] * Deleting remote file: /images/X1CG5-UEFI-FC/
[11-27-17 10:45:19 pm] | Files do not match.
[11-27-17 10:45:19 pm] | 20 0 /images/X1CG5-UEFI-FC/d1.original.fstypes cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:19 pm] * Deleting remote file: /images/X1CG5-UEFI-FC/
[11-27-17 10:45:19 pm] | Files do not match.
[11-27-17 10:45:19 pm] | 886 0 /images/X1CG5-UEFI-FC/d1.minimum.partitions cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:19 pm] * Deleting remote file: /images/X1CG5-UEFI-FC/
[11-27-17 10:45:19 pm] | Files do not match.
[11-27-17 10:45:19 pm] | 1048576 0 /images/X1CG5-UEFI-FC/d1.mbr cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:18 pm] * Deleting remote file: /images/X1CG5-UEFI-FC/
[11-27-17 10:45:18 pm] | Files do not match.
[11-27-17 10:45:18 pm] | 7 0 /images/X1CG5-UEFI-FC/d1.fixed_size_partitions cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:17 pm] | Image Name: X1CG5-UEFI-FC-V1
[11-27-17 10:45:17 pm] * Found Image to transfer to 2 groups
[11-27-17 10:45:17 pm] * Started sync for Image X1CG4-UEFI-FC-V1
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.X1CG4-UEFI-FC-V1.transfer.SF-MASTER.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/X1CG4-UEFI-FC-V1" "/images/X1CG4-UEFI-FC-V1"; exit' -u fog,[Protected] 172.16.40.33
[11-27-17 10:45:17 pm] | CMD:
[11-27-17 10:45:17 pm] * Starting Sync Actions
[11-27-17 10:45:17 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:17 pm] | Files do not match.
[11-27-17 10:45:17 pm] | 14832263898 0 /images/X1CG4-UEFI-FC-V1/d1p4.img cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:17 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:17 pm] | Files do not match.
[11-27-17 10:45:17 pm] | 6259444 0 /images/X1CG4-UEFI-FC-V1/d1p3.img cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:17 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:17 pm] | Files do not match.
[11-27-17 10:45:17 pm] | 12903254 0 /images/X1CG4-UEFI-FC-V1/d1p2.img cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:16 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:16 pm] | Files do not match.
[11-27-17 10:45:16 pm] | 357256473 0 /images/X1CG4-UEFI-FC-V1/d1p1.img cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:16 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:16 pm] | Files do not match.
[11-27-17 10:45:16 pm] | 861 0 /images/X1CG4-UEFI-FC-V1/d1.partitions cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:16 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:16 pm] | Files do not match.
[11-27-17 10:45:16 pm] | 283 0 /images/X1CG4-UEFI-FC-V1/d1.original.uuids cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:16 pm] | X1CG4-UEFI-FC-V1: No need to sync d1.original.swapuuids file to SF-MASTER
[11-27-17 10:45:15 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:15 pm] | Files do not match.
[11-27-17 10:45:15 pm] | 15 0 /images/X1CG4-UEFI-FC-V1/d1.original.fstypes cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:15 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:15 pm] | Files do not match.
[11-27-17 10:45:15 pm] | 861 0 /images/X1CG4-UEFI-FC-V1/d1.minimum.partitions cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:15 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:15 pm] | Files do not match.
[11-27-17 10:45:15 pm] | 1048576 0 /images/X1CG4-UEFI-FC-V1/d1.mbr cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
[11-27-17 10:45:14 pm] * Deleting remote file: /images/X1CG4-UEFI-FC-V1/
[11-27-17 10:45:14 pm] | Files do not match.
[11-27-17 10:45:14 pm] | 7 0 /images/X1CG4-UEFI-FC-V1/d1.fixed_size_partitions cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e

Mod edited to use codebox

JGallo · Nov 27, 2017, 10:47 PM

@bardwood

Is it possible that the replication hasn’t finished prior to the next image replication check occurs? That happened to me and I had to increase the IMAGEREPSLEEPTIME for storage nodes on networks with slow connections.

Wayne Workman · Nov 27, 2017, 5:50 PM

Just wanted to drop a line and say that the image type doesn’t have any bearing on how FOG Image Replication works.

Wayne Workman · Nov 28, 2017, 6:12 PM

@bardwood said in FOG UEFI image sync with storage node seems to be looping\failing on UUID info.:

70GB over a 10mb link

And that’s a large image over a tiny link… it could be possible that the sync isn’t finished… but it may also be having problems.

Basics first:

Please check free space on the destination node with the command df -h Look for partitions that are full. Do not assume your huge disks are not full, please go run the command and verify there are no full partitions.
Next, is the destination node’s hard disk failing? Maybe run a disk diagnostic tool to doublecheck.
Third, is the source node’s hard disk failing? Run a test and find out.

Wayne Workman · Nov 27, 2017, 6:12 PM

@jgallo said in FOG UEFI image sync with storage node seems to be looping\failing on UUID info.:

Is it possible that the replication hasn’t finished prior to the next image replication check occurs?

No. I found that bug years ago, I worked with @tom-elliott to resolve it. He changed the replication so that it is aware of lftp processes that it started in the past, and will skip a file that is still in progress from a previous iteration.

BardWood · Nov 28, 2017, 5:45 PM

@wayne-workman

No disk full or disk error issues:

[root@dub-imgstore-s images]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_sfimgstores-LogVol01
                      198G   66G  122G  35% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sda1             190M  164M   17M  91% /boot

The disk on MASTER/SN are fine but there is more to the story. When I was capturing these images, all the disk info would get captured but it would error on ‘d1.original.swapuuids’. I found an old forum post saying to delete these if this error happened (paraphrasing). I deleted these, the image capture completed, did a few imaging tests using this image, and I thought I was good to go. What I suspect is happening, is that somewhere there’s a file manifest which lists the file(s) I deleted. The file isn’t there, so when it does the remote check it seems there is a mismatch. It tries to sync but can’t sync that file (because it doesn’t exist) so it starts over again. What I suspect I need to do, is somehow edit whatever constitutes a manifest.

Sebastian Roth · Nov 28, 2017, 7:26 PM

@BardWood Can you post a link to the thread that mentioned you should delete that d1.original.swapuuids file?

BardWood · Nov 28, 2017, 7:36 PM

@sebastian-roth Searching… It was something Tom posted a while back (but I’ve seen the post within the last 30 days).

BardWood · Nov 28, 2017, 8:44 PM

@sebastian-roth I’m not seeing it (but I haven’t given up!). In the meantime, is there a graceful way to stop the sync? Or edit something which tells it to ignore any missing files? The issue is that I know the images are ‘good’ (meaning deployable in the current state) because I’ve deployed them to relevant hardware.

Sebastian Roth · Dec 2, 2017, 5:36 AM

@bardwood said in FOG UEFI image sync with storage node seems to be looping\failing on UUID info.:

When I was capturing these images, all the disk info would get captured but it would error on ‘d1.original.swapuuids’.

We need to know the exact error - otherwise it’s just guess work going the wrong way.

Though I am not an expert on the replication logic yet I think the issue could be that you’ve deleted that file on the master. Please move that d1.original.swapuuids on your storage node out of the way (so you still have a backup copy of it just in case): mv /images/X1CG2-UEFI-FC-V1/d1.original.swapuuids /root

Then see if replication stops looping.

Replication is a very tricky thing to get right and so messing with it (deleting files by hand) makes it even harder.

FOG UEFI image sync with storage node seems to be looping\failing on UUID info.

177

12.0k

17.3k

155.2k