FOG server not replicating drivers correctly to secondary node

[THEMCV]

Servers

• FOG Version: 1.5.0 RC-6
• OS: Debian 8.6

Description

Happy Tuesday everybody.

I’m running into an issue with my FOG server not wanting to replicate over drivers. Now I have them added in the location plugin and the images seem to replicate over fine, but when it gets to to drivers (there’s an “image” named driver sync that points to /images/drivers) the fogreplicator.log spews out this:

16502282312 0 /images/drivers/Win7
[08-08-17 4:25:26 pm]  | Files do not match.
[08-08-17 4:25:26 pm]  * Deleting remote file: /images/drivers/
[08-08-17 4:25:27 pm]  | 90356653
90360749
90364845
401204936
401209032
401213128
2002704
69884
80810
53612
86018
83558
85226
82794
83848
72664
59682
81948
86682

So on and so forth of these numbers until it get to

953470108
5425995480 0 /images/drivers/Win8.1
[08-08-17 4:25:27 pm]  | Files do not match.
[08-08-17 4:25:27 pm]  * Deleting remote file: /images/drivers/
[08-08-17 4:25:27 pm]  * Starting Sync Actions
[08-08-17 4:25:27 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Driver Sync.transfer.Michigan.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:lim$
[08-08-17 4:25:27 pm]  * Started sync for Image Driver Sync

Checking the /opt/fog/log/fogreplicator.Driver Sync.transfer.Michigan.log makes it look like it has been replicating okay.

Mirroring directory `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK'
Mirroring directory `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/IVB'
Mirroring directory `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/IVB/win32'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/IVB/win32'
Mirroring directory `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/IVB/x64'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/IVB/x64'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/IVB'
Mirroring directory `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/SNB'
Mirroring directory `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/SNB/win32'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/SNB/win32'
Mirroring directory `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/SNB/x64'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/SNB/x64'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK/SNB'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf/MediaSDK'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64/kit47932.inf'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00/Windows7-x64'
Finished mirror `Win7/XPS L412Z/x64/video/76RR4_A00-00'
Mirroring directory `Win7/XPS L412Z/x64/video/WXGWV_A00-00'
Mirroring directory `Win7/XPS L412Z/x64/video/WXGWV_A00-00/Production'
Mirroring directory `Win7/XPS L412Z/x64/video/WXGWV_A00-00/Production/Windows7-x86'
Finished mirror `Win7/XPS L412Z/x64/video/WXGWV_A00-00/Production/Windows7-x86'
Finished mirror `Win7/XPS L412Z/x64/video/WXGWV_A00-00/Production'
Finished mirror `Win7/XPS L412Z/x64/video/WXGWV_A00-00'
Finished mirror `Win7/XPS L412Z/x64/video'
Finished mirror `Win7/XPS L412Z/x64'
Finished mirror `Win7/XPS L412Z'
Finished mirror `Win7'

Any help on why the drivers are not syncing over correctly would be appreciated.

Thanks!

[george1421]

it looks like its syncing to me. Do you mean to tell me that on the storage node you don’t have a /images/drivers directory?

[THEMCV]

@george1421 It has the directory, but even after 24 hours it has not copied over the contents of /images/drivers.

One log says it’s doing it, the other seems to say that it copied over bad and is deleting the copied files.

[george1421]

@themcv It is replicating the images OK, just the drivers are failing?
Did you create a fake image definition for the /drivers directory on the master node?

[THEMCV]

@george1421 That’s the way it seems.

Yes I did. Doesn’t seem to want to work still.

[george1421]

@themcv OK one comment first, make sure you protect these drivers to ensure they don’t get accidentally overwritten.

So the next steps…

1. Go to the remote storage node configuration on the FOG server’s web gui.
2. Collect and document the user management ID and password from the web gui.
3. From the master node ftp to the storage node in question using the management user ID and password collected from step 2.
4. Change to the /images directory using the ftp client.
5. Create a directory in /images on the storage node (we are testing permissions)
6. put a local test file from the master node to the remote storage node in that new directory.
7. rm that file you put to the remote storage node.

If you can test the mechanics of the replication then we will move it over to a bug in 1.5.0. But test the mechanics first.

[Tom Elliott]

@george1421 This wouldn’t be a bug. The very same mechanism that handles images (which you’ve already pointed out) handles the drivers (particularly when you consider the drivers is also an image too). So while there is a problem, it’s highly unlikely this is a problem being caused by fog, at least not directly.

[THEMCV]

@george1421 If I did that right, then all worked okay for me when doing that. Made /images/testdir on the remote, used put testfile into that directory then deleted it.

Seemed okay doing that (i just want to make sure I did exactly how you asked! )

[george1421]

@themcv On the receiving storage node (assuming its a linux fog server) can you look in /var/logs there should be a log file for the ftp service. Check to see if there is any useful error messages when the drivers try to replicate.

The images replicate like they should, the drivers and images use the same replication engine.

The ftp process works.

Have we check the permissions on the master FOG server to ensure the directory and file permissions for /images/drivers match /images/<image_name>?

If Tom is saying its not a bug, then it has to be something simple we’re missing.

[THEMCV]

@george1421 I didn’t see anything in there related to ftp unfortunately.

I agree, this has to be something little that we’re missing.

I chmod 777’d /images/drivers to be sure.

[george1421]

@themcv chmod on both source and destination /images/drivers folders?

If that is the case and you restart the fog image replicator service with no joy then we will need to get the @Developers involved for additional debugging steps.

[THEMCV]

@george1421 Just upgraded to RC-7. I will restart both of the FOG servers just in case. @Tom-Elliott do you have time to remote in this week by any chance?

[Tom Elliott]

I’ve remoted in we are checking a change to see if the replication of sub sub sub folder stuff will work appropriately. Right now it’s in the middle of file 1.6million (I don’t really know but it’s a lot of them).

At least it appears to be checking the files iterative now.
Once it completes the first check, I’ll restart the service and hopefully it will work. It will take a while of course :(.

[Sebastian Roth]

@THEMCV Was this actually solved. Sounds like it but I am not absolutely sure.

[THEMCV]

@sebastian-roth Not yet. He setup a check to see if the servers were online, but right now the check doesn’t work so it doesn’t replicate. Still having issues, but for now I’m using rsync to send them.

[Sebastian Roth]

@THEMCV You might want to upgrade to RC9 and see if this is working for you.

[THEMCV]

@sebastian-roth Doesn’t look like it. Doing an rsync for the time being and that seems to work okay for now.

[Tom Elliott]

@themcv Is it possible there’s a firewall blocking ping checks to port 21? All the change I made does is check that the server is available and responding on port 21. (or whatever your ftp port is defined within /var/www/fog/lib/fog/fogftp.class.php (Usually 21))

[THEMCV]

@tom-elliott Anything is possible where I work. : )

I will see if that could be the issue and let you know.