Continuous Fog Storage Node Replication problem

[Wayne Workman]

@mpsadmin So in the replication log, it should print the command the replicator is using for each of these replication activities.

You can stop the replicator like this generally:

systemctl stop FOGImageReplicator

And then try to manually run the command from the master node. See what happens as it runs. You will need to input the correct password in the command.

Later, you can turn replication back on like this:

systemctl start FOGImageReplicator

[MPSAdmin]

@wayne-workman
Also, the fogreplicator…transfer.(synologyname).log shows the following:
(edited for brevity…it just flips the same two files back and forth)

2018-02-19 13:55:42 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 24.3K/s
2018-02-19 14:05:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 17.7K/s
2018-02-19 14:05:44 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 4.4K/s
2018-02-19 14:15:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 19.1K/s
2018-02-19 14:15:43 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 8.1K/s
2018-02-19 14:25:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 3.6K/s
2018-02-19 14:25:43 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 24.3K/s
2018-02-19 14:35:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 5.7K/s
2018-02-19 14:35:44 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 4.6K/s
2018-02-19 14:45:43 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 5.5K/s
2018-02-19 14:45:43 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 24.3K/s
2018-02-19 14:55:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 5.1K/s
2018-02-19 14:55:44 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 4.2K/s
2018-02-19 15:05:43 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 2.8K/s
2018-02-19 15:05:44 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 24.3K/s

[MPSAdmin]

@wayne-workman
I know how to start/stop the service, thanks for that. The problem is that I can’t stop the service fast enough before it starts writing to one of the files again. It is literally continuous replication…over and over again. See below:

2018-02-19 12:35:11 /images/G3-0118.1/d1.fixed_size_partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.fixed_size_partitions 0-5 156b/s
2018-02-19 12:35:11 /images/G3-0118.1/d1.mbr -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.mbr 0-1048576 15.15M/s
2018-02-19 12:35:11 /images/G3-0118.1/d1.minimum.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.minimum.partitions 0-330 6.7K/s
2018-02-19 12:35:12 /images/G3-0118.1/d1.original.fstypes -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.fstypes 0-40 1.9K/s
2018-02-19 12:35:12 /images/G3-0118.1/d1.original.swapuuids -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.swapuuids 0-0
2018-02-19 12:35:12 /images/G3-0118.1/d1.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.partitions 0-330 21.5K/s
2018-02-19 12:35:41 /images/G3-0118.1/d1.fixed_size_partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.fixed_size_partitions 0-5 455b/s
2018-02-19 12:35:41 /images/G3-0118.1/d1.mbr -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.mbr 0-1048576 14.29M/s
2018-02-19 12:35:41 /images/G3-0118.1/d1.minimum.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.minimum.partitions 0-330 40.3K/s
2018-02-19 12:35:41 /images/G3-0118.1/d1.original.fstypes -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.fstypes 0-40 5.6K/s
2018-02-19 12:35:41 /images/G3-0118.1/d1.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.partitions 0-330 46.0K/s
2018-02-19 12:36:08 /images/G3-0118.1/d1p1.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p1.img 0-364722448 12.87M/s
2018-02-19 13:32:10 /images/G3-0118.1/d1p2.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p2.img 0-44072908270 12.50M/s
2018-02-19 13:48:51 /images/G3-0118.1/d1p3.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p3.img 0-13123585448 12.50M/s
2018-02-19 13:48:53 /images/G3-0118.1/d1p4.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p4.img 0-34532488 17.37M/s
2018-02-19 13:56:11 /images/G3-0118.1/d1.fixed_size_partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.fixed_size_partitions 0-5 500b/s
2018-02-19 13:56:11 /images/G3-0118.1/d1.mbr -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.mbr 0-1048576 19.23M/s
2018-02-19 13:56:11 /images/G3-0118.1/d1.minimum.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.minimum.partitions 0-330 46.0K/s
2018-02-19 13:56:11 /images/G3-0118.1/d1.original.fstypes -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.fstypes 0-40 4.9K/s
2018-02-19 13:56:11 /images/G3-0118.1/d1.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.partitions 0-330 46.0K/s
2018-02-19 13:56:45 /images/G3-0118.1/d1p1.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p1.img 0-364722448 10.35M/s
2018-02-19 14:52:47 /images/G3-0118.1/d1p2.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p2.img 0-44072908270 12.50M/s
2018-02-19 15:09:28 /images/G3-0118.1/d1p3.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p3.img 0-13123585448 12.50M/s
2018-02-19 15:09:30 /images/G3-0118.1/d1p4.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p4.img 0-34532488 12.55M/s
2018-02-19 15:16:06 /images/G3-0118.1/d1.fixed_size_partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.fixed_size_partitions 0-5 455b/s
2018-02-19 15:16:06 /images/G3-0118.1/d1.mbr -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.mbr 0-1048576 18.87M/s
2018-02-19 15:16:06 /images/G3-0118.1/d1.minimum.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.minimum.partitions 0-330 46.0K/s
2018-02-19 15:16:06 /images/G3-0118.1/d1.original.fstypes -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.fstypes 0-40 5.6K/s
2018-02-19 15:16:06 /images/G3-0118.1/d1.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.partitions 0-330 46.0K/s
2018-02-19 15:16:33 /images/G3-0118.1/d1p1.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p1.img 0-364722448 12.84M/s

[Wayne Workman]

@mpsadmin The idea was to stop the service - wait for all the replicators to stop - then try to execute the LFTP command manually so you can observe any error messages or undesired behavior.

[MPSAdmin]

@wayne-workman
missed that part, sorry! I’ll take a shot.

[MPSAdmin]

ok… I noticed something else… the portions I deleted from the originall fogreplicator.log are html pages from the synology. Is there a reason that the fog server is pulling html files from the synology?

[Wayne Workman]

@mpsadmin said in Continuous Fog Storage Node Replication problem:

Is there a reason that the fog server is pulling html files from the synology?

It could be that the FTP Root is not set correctly for the synology. See this section: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP#FTP_Path There are notes in there specifically about Synology devices. Use the troubleshooting higher in the article to determine what the FTP Root path is.

[george1421]

Here is my POC tutorial to setup a synology nas as a FOG storage node: https://forums.fogproject.org/topic/9430/synology-nas-as-fog-storage-node

You need to watch directory permission on the synology NAS as well as nfs share permissions.

[MPSAdmin]

Still doing the same thing. Rolling right through each file over and over again. I changed the rights to the folders as recommended.

[Wayne Workman]

@mpsadmin But did you check the FTP root by logging in via FTP?

[MPSAdmin]

@george1421
Hi George. I had hoped that you would chime in as I followed your fantastic storage node document when I set this up.
in just noticed that the ‘images’ shared folder didn’t have rights for the fog user.

[MPSAdmin]

@wayne-workman
yes I did. I logged in via ftp and was able to browse with no issues.

[george1421]

@mpsadmin So are the files actually making it to the NAS? The only reason why the fog server would keep replicating is because the check sums would not match so it will send the file over again. Possibly date/time mismatch might trigger the replication again, but I think they are focusing on the check sum as the key to replicate.

[Wayne Workman]

@mpsadmin said in Continuous Fog Storage Node Replication problem:

I logged in via ftp and was able to browse with no issues.

That’s not what I am referring to - I would like for you to FTP into the NAS and figure out where you land - i.e. the FTP Path directory.

• Is the directory where you land the FTP Path that is set for this node?
• What is your landing directory?

[MPSAdmin]

@wayne-workman Apologies for not providing clearer info. The paths for the storage node are configured within fog as follows:

The ftp paths are as follows:

The files being replicated are within the G3-0118.1 folder, as listed below:

the source folder on the fog server is the /images folder, as shown below.

Additionally, I have verified that both the synololgy and the fog server are using the same ntp source as the rest of the devices on the network. (a windows domain controller). After changing the ntp settings, i stopped the fog replication service, deleted the files from the synology folder, and restarted the replication service.

Thanks. I’ll keep digging. Most likely it is something simple that I have done.

[MPSAdmin]

@george1421 yes, the files are making it to the NAS. Fog is replicating at full speed, up to the configured bandwidth limit. Once the files are replicated, the process begins anew.

[MPSAdmin]

Also, just checked with the d1.mbr file… md5 checksums match.

040d5ad57af1942104ab788d3b12778e fogserver
040d5ad57af1942104ab788d3b12778e synology web console

d1p1.img…checksums match here too.
79d8b1a77b8111418481c121c246501b fog server
79d8b1a77b8111418481c121c246501b synology web console

[george1421]

@mpsadmin OK then we might need an opinion from one of the @developers to give us an idea why the NAS is not reporting what the FOG replicator expects.

If I remember right there is 2 or 3 replicator logs. One is the main replicator log, one is for the storage node, and there may be a log file for each image being replicated. I don’t have access to my production FOG server at the moment to confirm. But I thought the logs were pretty descriptive to why it was replicating.

[george1421]

@mpsadmin I have my fog-pi server with me that has FOG 1.4.4 on it. I’m looking at the code and it appears to rely on the file byte count being the same to indicate if the file needs to be replicated or not. I’m not a programmer so this object oriented php is a bit greek to me. But so far it looks like byte count is the trigger. Now this is the byte count as reported by FTP session. Also the log writes to /opt/fog/log look fairly descriptive to what exactly the replicator is doing.

[Wayne Workman]

@mpsadmin Did you use the FTP option when logging in via WinSCP? I ask because I want to be sure. Photo: