FOGImageReplicator runs in a loop

kek

We recently deployed a new storage node, it works fine except for the replicator service. At first it runs fine but before the files got fully replicated it seems that they’re got deleted and got replicated again, and again… In the logs are PHP Warnings about the fclose() function.

Oct 19 19:49:22 srv020 FOGImageReplicator[14732]: Transferring file `d1p2.img'
Oct 19 19:49:36 srv020 FOGImageReplicator[14732]: Transferring file `d1p4.img'
Oct 19 20:00:33 srv020 FOGImageReplicator[14732]: Finished transfer `d1p1.img' (486.0 KiB/s)
Oct 19 20:00:33 srv020 FOGImageReplicator[14732]: PHP Warning:  fclose() expects parameter 1 to be resource, array given in /var/www/fog/lib/service/fogservice.class.php on line 995
Oct 19 20:00:33 srv020 FOGImageReplicator[14732]: PHP Warning:  fclose(): supplied resource is not a valid stream resource in /var/www/fog/lib/service/fogservice.class.php on line 997
Oct 19 20:04:23 srv020 FOGImageReplicator[14732]: Finished transfer `d1p2.img' (613.3 KiB/s)
Oct 19 20:04:23 srv020 FOGImageReplicator[14732]: PHP Warning:  fclose() expects parameter 1 to be resource, array given in /var/www/fog/lib/service/fogservice.class.php on line 995
Oct 19 20:04:23 srv020 FOGImageReplicator[14732]: PHP Warning:  fclose(): supplied resource is not a valid stream resource in /var/www/fog/lib/service/fogservice.class.php on line 997
Oct 19 20:09:19 srv020 FOGImageReplicator[14732]: Transferring file `d1p1.img'
Oct 19 20:09:27 srv020 FOGImageReplicator[14732]: Transferring file `d1p2.img'
Oct 19 21:27:36 srv020 FOGImageReplicator[14732]: Finished transfer `d1p1.img' (457.1 KiB/s)
Oct 19 21:27:37 srv020 FOGImageReplicator[14732]: PHP Warning:  fclose() expects parameter 1 to be resource, array given in /var/www/fog/lib/service/fogservice.class.php on line 995
Oct 19 21:27:37 srv020 FOGImageReplicator[14732]: PHP Warning:  fclose(): supplied resource is not a valid stream resource in /var/www/fog/lib/service/fogservice.class.php on line 997
Oct 19 21:29:33 srv020 FOGImageReplicator[14732]: Transferring file `d1p1.img'
Oct 19 21:33:55 srv020 FOGImageReplicator[14732]: Finished transfer `d1p2.img' (337.8 KiB/s)
Oct 19 21:33:55 srv020 FOGImageReplicator[14732]: PHP Warning:  fclose() expects parameter 1 to be resource, array given in /var/www/fog/lib/service/fogservice.class.php on line 995
Oct 19 21:33:55 srv020 FOGImageReplicator[14732]: PHP Warning:  fclose(): supplied resource is not a valid stream resource in /var/www/fog/lib/service/fogservice.class.php on line 997

Thanks for the help in advance.

Sebastian Roth

@kek From the lines numbers we see in the error messages I would guess you are using FOG 1.5.6, right? Since then there have been three new releases and numerous fixes in the code. I would ask you to update to the latest release (1.5.9 as of now) and see if you still get the same behavior.

Beside that I might mention that from FOG 1.5.4 to 1.5.5 we have changed the replication checksums used to compare the files. Depending on the version you use on the storage node that might play a role as well.

kek

Sorry, i provided the wrong log:

[10-20-20 6:51:04 pm] * Starting Image Replication.
[10-20-20 6:51:04 pm] * We are group ID: 1. We are group name: Company
[10-20-20 6:51:04 pm] * We are node ID: 1. We are node name: Location
[10-20-20 6:51:04 pm] * Attempting to perform Group -> Group image replication.
[10-20-20 6:51:04 pm] | Replicating postdownloadscripts
[10-20-20 6:51:04 pm] * Found Image to transfer to 1 node
[10-20-20 6:51:04 pm] | File Name: postdownloadscripts
[10-20-20 6:51:06 pm] # postdownloadscripts: No need to sync fog.postdownload (LocationID2)
[10-20-20 6:51:06 pm] * All files synced for this item.
[10-20-20 6:51:06 pm] | Replicating postinitscripts
[10-20-20 6:51:07 pm] * Found Image to transfer to 1 node
[10-20-20 6:51:07 pm] | File Name: dev/postinitscripts
[10-20-20 6:51:07 pm] | Location2 does not appear to be online.
[10-20-20 6:51:07 pm] * Not syncing Image between groups
[10-20-20 6:51:07 pm] | Image Name: I1
[10-20-20 6:51:07 pm] | There are no other members to sync to.
[10-20-20 6:51:07 pm] * Not syncing Image between groups
[10-20-20 6:51:07 pm] | Image Name: I2
[10-20-20 6:51:07 pm] | There are no other members to sync to.
[10-20-20 6:51:07 pm] * Not syncing Image between groups
[10-20-20 6:51:07 pm] | Image Name: I3
[10-20-20 6:51:07 pm] | There are no other members to sync to.
[10-20-20 6:51:07 pm] * Not syncing Image between groups
[10-20-20 6:51:07 pm] | Image Name: I4
[10-20-20 6:51:07 pm] | There are no other members to sync to.
[10-20-20 6:51:07 pm] * Not syncing Image between groups
[10-20-20 6:51:07 pm] | Image Name: I5
[10-20-20 6:51:07 pm] | There are no other members to sync to.
[10-20-20 6:51:07 pm] * Not syncing Image between groups
[10-20-20 6:51:07 pm] | Image Name: I6
[10-20-20 6:51:07 pm] | There are no other members to sync to.
[10-20-20 6:51:07 pm] * Attempting to perform Group -> Nodes image replication.
[10-20-20 6:51:07 pm] * Found Image to transfer to 1 node
[10-20-20 6:51:07 pm] | Image Name: I1
[10-20-20 6:51:10 pm] # I1: No need to sync I1.000 (Location2)
[10-20-20 6:51:10 pm] * All files synced for this item.
[10-20-20 6:51:10 pm] * Found Image to transfer to 1 node
[10-20-20 6:51:10 pm] | Image Name: I2
[10-20-20 6:51:13 pm] # I2: No need to sync I2.000 (Location2)
[10-20-20 6:51:14 pm] * All files synced for this item.
[10-20-20 6:51:14 pm] * Found Image to transfer to 1 node
[10-20-20 6:51:14 pm] | Image Name: I3
[10-20-20 6:51:15 pm] # I3: No need to sync d1.fixed_size_partitions (Location2)
[10-20-20 6:51:17 pm] # I3: No need to sync d1.has_grub (Location2)
[10-20-20 6:51:17 pm] # I3: No need to sync d1.mbr (Location2)
[10-20-20 6:51:19 pm] # I3: No need to sync d1.minimum.partitions (Location2)
[10-20-20 6:51:20 pm] # I3: No need to sync d1.original.fstypes (Location2)
[10-20-20 6:51:21 pm] # I3: No need to sync d1.original.swapuuids (Location2)
[10-20-20 6:51:22 pm] # I3: No need to sync d1.partitions (Location2)
[10-20-20 6:51:22 pm] # I3: File size mismatch - d1p1.img: -2040571859 != 2254395437
[10-20-20 6:51:22 pm] # I3: Deleting remote file d1p1.img
[10-20-20 6:51:23 pm] * Starting Sync Actions
[10-20-20 6:51:23 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogreplicator.I3.transfer.Location2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/I3" "/images/I3"; exit' -u fogproject,[Protected] xx.xx.xx.xx
[10-20-20 6:51:23 pm] | Started sync for Image I3 - Resource id #4294

Is there a way to not upgrade, because we customized a few files…
It would be perfect if we could just replace the files.
BTW: I updated the file /var/www/html/fog/lib/service/fogservice.class.php, still not working…
FOG Master Server Version: 1.5.7
FOG Storage Node Version: 1.5.8

Thanks for the Help.

Sebastian Roth

@kek said in FOGImageReplicator runs in a loop:

File size mismatch - d1p1.img: -2040571859 != 2254395437

Do you have a 32 bit build of PHP installed on your FOG master server? What Linux OS is installed? Manually installed PHP e.g. from external repos?

Is there a way to not upgrade, because we customized a few files…
It would be perfect if we could just replace the files.

For how long do you want to keep back on updating? Asking for things to get fixed but only updating partially is not playing nicely. You better find a good way to keep track of your customization to be able to merge this with new code or you might even share this and ask for it to be added to the official code.

kek

@Sebastian-Roth said in FOGImageReplicator runs in a loop:

Do you have a 32 bit build of PHP installed on your FOG master server? What Linux OS is installed? Manually installed PHP e.g. from external repos?

Yes back when the server was installed, the wrong ISO was picked because someone putted it in the x64 folder… It’s Debian 10 Buster (4.19.0-6-686-pae). PHP was installed via the foginstaller.

For how long do you want to keep back on updating? Asking for things to get fixed but only updating partially is not playing nicely. You better find a good way to keep track of your customization to be able to merge this with new code or you might even share this and ask for it to be added to the official code.

On our company we don’t update our linux infrastructure very often, and have much undocumented things, but thats an internal problem, that needs to be changed (and will hopefully in some time).

Most of the customisation is also very company-specific.

I temporarily set the filesequal variable to true to stop the loop. Code:

$filesequal = true;
                        if ($filesequal != true) {
                            $allsynced = false;
                            self::outall(sprintf('  # %s: %s %s', $name, _('Deleting remote file'), $filename));
                            self::$FOGFTP->delete($remotefilename);
                        } else {
                            self::outall(sprintf(
                                '  # %s: %s %s (%s)',
                                $name,
                                _('No need to sync. Temporarily turned off deletion...'),
                                $filename,
                                $nodename
                            ));
                            continue;
                        }

Sebastian Roth

@kek Some time back we switched from using Linux du shell exec command to PHP’s filesize() function. If you are keen you can switch back to use du. See here: https://github.com/FOGProject/fogproject/commit/11bcc4092307b0aade315af02107516a353dd403#diff-743f393a621db0938336e35d59e7f62d3a29a7d9d50783ab5925c9083c7fb60c

Please make sure you read and understand the comments below the code. The code used escapeshellarg() in the getFilesize() while it was being escaped earlier already… So I would imagine you don’t want to use that particular line 2305 from that code.

Also be aware the getFilesize() is being called from several other places in the code. So you might mess up a few things playing with this part of the code.

kek

@Sebastian-Roth Thank you very much for your help! After you pointed me to the right file, i found a very easy and good working solution.

/var/www/html/fog/lib/fog/fogbase.class.php:

Changed line 2305 from:

$size = filesize($file);

to:

$size = exec("stat -c %s ".$file);

I found this nice solution over here:

stackoverflow

Reverted my changes that i showed in the last post, restarted FOGImageReplicator service, and it works perfect!

We know that this can’t work for forever, and Updates are also nice and good, so we need to change a few internal things, and also reinstall this Server with x64 Debian.

Thanks for your great support. Keep up this very very nice project!

FOGImageReplicator runs in a loop

172

12.0k

17.3k

155.2k