• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Continuous Fog Storage Node Replication problem

    Scheduled Pinned Locked Moved
    FOG Problems
    4
    32
    3.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      MPSAdmin
      last edited by

      Hi all! Let me first say how awesome Fog is and thank you for making it so amazing.

      I am having an issue with replication between my primary FOG node (running centos 7) and a remote storage node, running on a synology system. I have followed the synology storage node guides and have everything set up at the remote site. It all works perfectly except that the main node continually replicates the image files to the secondary node. Not sure why. Any ideas?

      Thanks!

      Wayne WorkmanW 1 Reply Last reply Reply Quote 0
      • Wayne WorkmanW
        Wayne Workman @MPSAdmin
        last edited by

        @mpsadmin said in Continuous Fog Storage Node Replication problem:

        Any ideas?

        Check the space, maybe it’s running out? Also, you should check the master node’s replication log. It’s in here generally: /var/log/fog/

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
        Daily Clean Installation Results:
        https://fogtesting.fogproject.us/
        FOG Reporting:
        https://fog-external-reporting-results.fogproject.us/

        M 1 Reply Last reply Reply Quote 0
        • M
          MPSAdmin
          last edited by Tom Elliott

          Here are the abbreviated contents of the fogreplicator.log:

          [02-19-18 7:55:43 pm]  | Files do not match.
[02-19-18 7:55:43 pm]  * Deleting remote file: /images/dev/postinitscripts/fog.postinit
[02-19-18 7:55:43 pm]  * Starting Sync Actions
[02-19-18 7:55:43 pm]  | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.spg-syn1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-rate 0:13107200; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/dev/postinitscripts" "/images/dev/postinitscripts"; exit' -u foguser,[Protected] 10.1.0.133
[02-19-18 7:55:43 pm]  * Started sync for Image dev/postinitscripts
[02-19-18 7:55:43 pm]  * Not syncing Image between groups
[02-19-18 7:55:43 pm]  | Image Name: G3-0118.1
[02-19-18 7:55:43 pm]  | There are no other members to sync to.
[02-19-18 7:55:43 pm]  * Attempting to perform Group -> Nodes image replication.
[02-19-18 7:55:43 pm]  * Found Image to transfer to 1 node
[02-19-18 7:55:43 pm]  | Image Name: G3-0118.1
[02-19-18 7:55:43 pm] | Replication already running with PID: 9840
[02-19-18 8:05:40 pm]  * Starting Image Replication.
[02-19-18 8:05:40 pm]  * We are group ID: 1. We are group name: default
[02-19-18 8:05:40 pm]  * We are node ID: 1. We are node name: DefaultMember
[02-19-18 8:05:40 pm]  * Attempting to perform Group -> Group image replication.
[02-19-18 8:05:40 pm]  | Replicating postdownloadscripts
[02-19-18 8:05:40 pm]  * Found Image to transfer to 1 node
[02-19-18 8:05:40 pm]  | File Name: postdownloadscripts
[02-19-18 8:05:41 pm]  | 235 235 /images/postdownloadscripts/fog.postdownload <!DOCTYPE html>

....(file contents here)....
[02-19-18 8:05:41 pm]  | Files do not match.
[02-19-18 8:05:41 pm]  * Deleting remote file: /images/postdownloadscripts/fog.postdownload
[02-19-18 8:05:42 pm]  * Starting Sync Actions
[02-19-18 8:05:42 pm]  | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.spg-syn1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-rate 0:13107200; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u foguser,[Protected] 10.1.0.133
[02-19-18 8:05:42 pm]  * Started sync for Image postdownloadscripts
[02-19-18 8:05:42 pm]  | Replicating postinitscripts
[02-19-18 8:05:42 pm]  * Found Image to transfer to 1 node
[02-19-18 8:05:42 pm]  | File Name: dev/postinitscripts
[02-19-18 8:05:43 pm]  | 249 249 /images/dev/postinitscripts/fog.postinit <!DOCTYPE html>

....(file contents printed here)....

[02-19-18 8:05:43 pm]  | Files do not match.
[02-19-18 8:05:43 pm]  * Deleting remote file: /images/dev/postinitscripts/fog.postinit
[02-19-18 8:05:43 pm]  * Starting Sync Actions
[02-19-18 8:05:43 pm]  | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.spg-syn1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-rate 0:13107200; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/dev/postinitscripts" "/images/dev/postinitscripts"; exit' -u foguser,[Protected] 10.1.0.133
[02-19-18 8:05:43 pm]  * Started sync for Image dev/postinitscripts
[02-19-18 8:05:43 pm]  * Not syncing Image between groups
[02-19-18 8:05:43 pm]  | Image Name: G3-0118.1
[02-19-18 8:05:43 pm]  | There are no other members to sync to.
[02-19-18 8:05:43 pm]  * Attempting to perform Group -> Nodes image replication.
[02-19-18 8:05:43 pm]  * Found Image to transfer to 1 node
[02-19-18 8:05:43 pm]  | Image Name: G3-0118.1
[02-19-18 8:05:43 pm] | Replication already running with PID: 9840
          Wayne WorkmanW 1 Reply Last reply Reply Quote 0
          • M
            MPSAdmin
            last edited by

            Thanks for the quick reply.
            Plenty of space… (9TB free)

            1 Reply Last reply Reply Quote 0
            • Wayne WorkmanW
              Wayne Workman @MPSAdmin
              last edited by Wayne Workman

              @mpsadmin So in the replication log, it should print the command the replicator is using for each of these replication activities.

              You can stop the replicator like this generally:

              systemctl stop FOGImageReplicator

              And then try to manually run the command from the master node. See what happens as it runs. You will need to input the correct password in the command.

              Later, you can turn replication back on like this:

              systemctl start FOGImageReplicator

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
              Daily Clean Installation Results:
              https://fogtesting.fogproject.us/
              FOG Reporting:
              https://fog-external-reporting-results.fogproject.us/

              M 1 Reply Last reply Reply Quote 0
              • M
                MPSAdmin @Wayne Workman
                last edited by

                @wayne-workman
                Also, the fogreplicator…transfer.(synologyname).log shows the following:
                (edited for brevity…it just flips the same two files back and forth)

                2018-02-19 13:55:42 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 24.3K/s
                2018-02-19 14:05:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 17.7K/s
                2018-02-19 14:05:44 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 4.4K/s
                2018-02-19 14:15:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 19.1K/s
                2018-02-19 14:15:43 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 8.1K/s
                2018-02-19 14:25:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 3.6K/s
                2018-02-19 14:25:43 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 24.3K/s
                2018-02-19 14:35:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 5.7K/s
                2018-02-19 14:35:44 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 4.6K/s
                2018-02-19 14:45:43 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 5.5K/s
                2018-02-19 14:45:43 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 24.3K/s
                2018-02-19 14:55:42 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 5.1K/s
                2018-02-19 14:55:44 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 4.2K/s
                2018-02-19 15:05:43 /images/postdownloadscripts/fog.postdownload -> ftp://foguser@10.1.0.133/images/postdownloadscripts/fog.postdownload 0-235 2.8K/s
                2018-02-19 15:05:44 /images/dev/postinitscripts/fog.postinit -> ftp://foguser@10.1.0.133/images/dev/postinitscripts/fog.postinit 0-249 24.3K/s

                1 Reply Last reply Reply Quote 0
                • M
                  MPSAdmin @Wayne Workman
                  last edited by

                  @wayne-workman
                  I know how to start/stop the service, thanks for that. The problem is that I can’t stop the service fast enough before it starts writing to one of the files again. It is literally continuous replication…over and over again. See below:

                  2018-02-19 12:35:11 /images/G3-0118.1/d1.fixed_size_partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.fixed_size_partitions 0-5 156b/s
                  2018-02-19 12:35:11 /images/G3-0118.1/d1.mbr -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.mbr 0-1048576 15.15M/s
                  2018-02-19 12:35:11 /images/G3-0118.1/d1.minimum.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.minimum.partitions 0-330 6.7K/s
                  2018-02-19 12:35:12 /images/G3-0118.1/d1.original.fstypes -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.fstypes 0-40 1.9K/s
                  2018-02-19 12:35:12 /images/G3-0118.1/d1.original.swapuuids -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.swapuuids 0-0
                  2018-02-19 12:35:12 /images/G3-0118.1/d1.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.partitions 0-330 21.5K/s
                  2018-02-19 12:35:41 /images/G3-0118.1/d1.fixed_size_partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.fixed_size_partitions 0-5 455b/s
                  2018-02-19 12:35:41 /images/G3-0118.1/d1.mbr -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.mbr 0-1048576 14.29M/s
                  2018-02-19 12:35:41 /images/G3-0118.1/d1.minimum.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.minimum.partitions 0-330 40.3K/s
                  2018-02-19 12:35:41 /images/G3-0118.1/d1.original.fstypes -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.fstypes 0-40 5.6K/s
                  2018-02-19 12:35:41 /images/G3-0118.1/d1.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.partitions 0-330 46.0K/s
                  2018-02-19 12:36:08 /images/G3-0118.1/d1p1.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p1.img 0-364722448 12.87M/s
                  2018-02-19 13:32:10 /images/G3-0118.1/d1p2.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p2.img 0-44072908270 12.50M/s
                  2018-02-19 13:48:51 /images/G3-0118.1/d1p3.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p3.img 0-13123585448 12.50M/s
                  2018-02-19 13:48:53 /images/G3-0118.1/d1p4.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p4.img 0-34532488 17.37M/s
                  2018-02-19 13:56:11 /images/G3-0118.1/d1.fixed_size_partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.fixed_size_partitions 0-5 500b/s
                  2018-02-19 13:56:11 /images/G3-0118.1/d1.mbr -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.mbr 0-1048576 19.23M/s
                  2018-02-19 13:56:11 /images/G3-0118.1/d1.minimum.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.minimum.partitions 0-330 46.0K/s
                  2018-02-19 13:56:11 /images/G3-0118.1/d1.original.fstypes -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.fstypes 0-40 4.9K/s
                  2018-02-19 13:56:11 /images/G3-0118.1/d1.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.partitions 0-330 46.0K/s
                  2018-02-19 13:56:45 /images/G3-0118.1/d1p1.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p1.img 0-364722448 10.35M/s
                  2018-02-19 14:52:47 /images/G3-0118.1/d1p2.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p2.img 0-44072908270 12.50M/s
                  2018-02-19 15:09:28 /images/G3-0118.1/d1p3.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p3.img 0-13123585448 12.50M/s
                  2018-02-19 15:09:30 /images/G3-0118.1/d1p4.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p4.img 0-34532488 12.55M/s
                  2018-02-19 15:16:06 /images/G3-0118.1/d1.fixed_size_partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.fixed_size_partitions 0-5 455b/s
                  2018-02-19 15:16:06 /images/G3-0118.1/d1.mbr -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.mbr 0-1048576 18.87M/s
                  2018-02-19 15:16:06 /images/G3-0118.1/d1.minimum.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.minimum.partitions 0-330 46.0K/s
                  2018-02-19 15:16:06 /images/G3-0118.1/d1.original.fstypes -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.original.fstypes 0-40 5.6K/s
                  2018-02-19 15:16:06 /images/G3-0118.1/d1.partitions -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1.partitions 0-330 46.0K/s
                  2018-02-19 15:16:33 /images/G3-0118.1/d1p1.img -> ftp://foguser@10.1.0.133/images/G3-0118.1/d1p1.img 0-364722448 12.84M/s

                  Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                  • Wayne WorkmanW
                    Wayne Workman @MPSAdmin
                    last edited by

                    @mpsadmin The idea was to stop the service - wait for all the replicators to stop - then try to execute the LFTP command manually so you can observe any error messages or undesired behavior.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                    Daily Clean Installation Results:
                    https://fogtesting.fogproject.us/
                    FOG Reporting:
                    https://fog-external-reporting-results.fogproject.us/

                    M 1 Reply Last reply Reply Quote 0
                    • M
                      MPSAdmin @Wayne Workman
                      last edited by

                      @wayne-workman
                      missed that part, sorry! I’ll take a shot.

                      1 Reply Last reply Reply Quote 0
                      • M
                        MPSAdmin
                        last edited by

                        ok… I noticed something else… the portions I deleted from the originall fogreplicator.log are html pages from the synology. Is there a reason that the fog server is pulling html files from the synology?

                        Wayne WorkmanW 2 Replies Last reply Reply Quote 0
                        • Wayne WorkmanW
                          Wayne Workman @MPSAdmin
                          last edited by

                          @mpsadmin said in Continuous Fog Storage Node Replication problem:

                          Is there a reason that the fog server is pulling html files from the synology?

                          It could be that the FTP Root is not set correctly for the synology. See this section: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP#FTP_Path There are notes in there specifically about Synology devices. Use the troubleshooting higher in the article to determine what the FTP Root path is.

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                          Daily Clean Installation Results:
                          https://fogtesting.fogproject.us/
                          FOG Reporting:
                          https://fog-external-reporting-results.fogproject.us/

                          1 Reply Last reply Reply Quote 0
                          • george1421G
                            george1421 Moderator
                            last edited by

                            Here is my POC tutorial to setup a synology nas as a FOG storage node: https://forums.fogproject.org/topic/9430/synology-nas-as-fog-storage-node

                            You need to watch directory permission on the synology NAS as well as nfs share permissions.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                            M 1 Reply Last reply Reply Quote 1
                            • M
                              MPSAdmin
                              last edited by

                              Still doing the same thing. Rolling right through each file over and over again. I changed the rights to the folders as recommended.

                              Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                              • Wayne WorkmanW
                                Wayne Workman @MPSAdmin
                                last edited by

                                @mpsadmin But did you check the FTP root by logging in via FTP?

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                Daily Clean Installation Results:
                                https://fogtesting.fogproject.us/
                                FOG Reporting:
                                https://fog-external-reporting-results.fogproject.us/

                                M 1 Reply Last reply Reply Quote 0
                                • M
                                  MPSAdmin @george1421
                                  last edited by

                                  @george1421
                                  Hi George. I had hoped that you would chime in as I followed your fantastic storage node document when I set this up.
                                  in just noticed that the ‘images’ shared folder didn’t have rights for the fog user.

                                  george1421G 1 Reply Last reply Reply Quote 0
                                  • M
                                    MPSAdmin @Wayne Workman
                                    last edited by

                                    @wayne-workman
                                    yes I did. I logged in via ftp and was able to browse with no issues.

                                    Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                                    • george1421G
                                      george1421 Moderator @MPSAdmin
                                      last edited by george1421

                                      @mpsadmin So are the files actually making it to the NAS? The only reason why the fog server would keep replicating is because the check sums would not match so it will send the file over again. Possibly date/time mismatch might trigger the replication again, but I think they are focusing on the check sum as the key to replicate.

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                      M 1 Reply Last reply Reply Quote 0
                                      • Wayne WorkmanW
                                        Wayne Workman @MPSAdmin
                                        last edited by

                                        @mpsadmin said in Continuous Fog Storage Node Replication problem:

                                        I logged in via ftp and was able to browse with no issues.

                                        That’s not what I am referring to - I would like for you to FTP into the NAS and figure out where you land - i.e. the FTP Path directory.

                                        • Is the directory where you land the FTP Path that is set for this node?
                                        • What is your landing directory?

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                        Daily Clean Installation Results:
                                        https://fogtesting.fogproject.us/
                                        FOG Reporting:
                                        https://fog-external-reporting-results.fogproject.us/

                                        M 1 Reply Last reply Reply Quote 0
                                        • M
                                          MPSAdmin @Wayne Workman
                                          last edited by

                                          @wayne-workman Apologies for not providing clearer info. The paths for the storage node are configured within fog as follows:
                                          0_1519158734380_c31cc54a-37a2-40d7-aa84-1ab1f8a791b6-image.png

                                          The ftp paths are as follows:
                                          0_1519158795996_966faeda-2106-434a-b85e-3dcbc6c308f2-image.png

                                          The files being replicated are within the G3-0118.1 folder, as listed below:
                                          0_1519158844529_806dca24-017a-4760-ac1f-1448da4e5ba3-image.png

                                          the source folder on the fog server is the /images folder, as shown below.
                                          0_1519158945084_de15469e-5417-453b-937a-52daf3e9da66-image.png

                                          Additionally, I have verified that both the synololgy and the fog server are using the same ntp source as the rest of the devices on the network. (a windows domain controller). After changing the ntp settings, i stopped the fog replication service, deleted the files from the synology folder, and restarted the replication service.

                                          Thanks. I’ll keep digging. Most likely it is something simple that I have done.

                                          Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                                          • M
                                            MPSAdmin @george1421
                                            last edited by

                                            @george1421 yes, the files are making it to the NAS. Fog is replicating at full speed, up to the configured bandwidth limit. Once the files are replicated, the process begins anew.

                                            1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post

                                            290

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project