• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Image replication between nodes keeps restarting

    Scheduled Pinned Locked Moved Unsolved
    FOG Problems
    5
    19
    3.5k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      AndersHoeg
      last edited by

      I’ve got two servers running 1.4.0.
      The master storage node is located in Denmark and the other storage node is located in Norway.
      The replication works, but every time it has replicated an image, it just deletes the image on the server in Norway and starts over once again. It has been doing this for about a week, any ideas?

      1 Reply Last reply Reply Quote 0
      • george1421G
        george1421 Moderator
        last edited by

        This is an interesting issue.

        The fog replicator compares the md5sum of a certain number of bytes of each file before it decides if they are different. If the md5sum key is not the same then it makes the decision that the storage node is out of sync and starts replicating the image again.

        The fog replicator logs are in /opt/fog/logs does the master replication log or the replication log for the norway server give you an idea why it just keeps replicating?

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        1 Reply Last reply Reply Quote 0
        • A
          AndersHoeg
          last edited by

          Hmmm maybe it is trying to replicate to often and that’s the issue.
          Any clue where I can see/change how often it tries to replicate an image?

          Wayne WorkmanW 1 Reply Last reply Reply Quote 0
          • Wayne WorkmanW
            Wayne Workman @AndersHoeg
            last edited by Wayne Workman

            @AndersHoeg said in Image replication between nodes keeps restarting:

            Any clue where I can see/change how often it tries to replicate an image?

            FOG Web Interface -> FOG Configuration -> FOG Settings -> FOG Linux Service Sleep Times -> IMAGEREPSLEEPTIME

            FOG Replication logs are also viewable in the web interface:
            FOG Web Interface -> FOG Configuration -> Log Viewer -> Image Replicator

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
            Daily Clean Installation Results:
            https://fogtesting.fogproject.us/
            FOG Reporting:
            https://fog-external-reporting-results.fogproject.us/

            A J 2 Replies Last reply Reply Quote 0
            • A
              AndersHoeg @Wayne Workman
              last edited by

              @Wayne-Workman Thank you so much! It worked, it sync the images and don’t delete them over and over again.

              1 Reply Last reply Reply Quote 0
              • J
                JGallo @Wayne Workman
                last edited by JGallo

                @wayne-workman

                I know this is a really old post and I apologize if I should have started a new topic. I’m experiencing the same symptoms after only accidentally discovering the replication keeps deleting my images from the nodes. By increasing the value of the FOG Linux Service Sleep Times - IMAGEREPSLEEPTIME, will this stop the image from being deleted?

                Basically would I need to set the value to a point beyond the time it takes FOG to finish replicating? If the value is less than the time it takes FOG to finish replicating, would this explain why the images keeps being deleted and replication restarting?

                Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                • Wayne WorkmanW
                  Wayne Workman @JGallo
                  last edited by Wayne Workman

                  @jgallo If the image replicator is deleting the files if it’s not completed by the next iteration - this is a bug. I thought this was fixed completely somewhere between 1.2 and 1.3… I guess the issue has cropped up again. So yeah, just to see if this is the case, set your IMAGEREPSLEEPTIME to something huge and see if it’s fixed. If that fixes it, then that’s bug confirmed.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                  Daily Clean Installation Results:
                  https://fogtesting.fogproject.us/
                  FOG Reporting:
                  https://fog-external-reporting-results.fogproject.us/

                  J 1 Reply Last reply Reply Quote 0
                  • J
                    JGallo @Wayne Workman
                    last edited by

                    @wayne-workman

                    I increased it to 10800 which is 3 hours just for the hell of it. Seems to work fine on 1.5.0 RC7 unless replication takes longer than that LOL but issue I ran into was when I try to go and upload an image I get some weird DHCP issues. So I went ahead and went to RC9 and then replication does’t work. I came across a post regarding same issue and suggestion was to go back to RC7. So I’m in a bind and not sure what to do. I’m going to start from scratch and add each node and tail the fogreplicator.log file and see if the same errors occur. I’m getting a (storage node) server does not appear to be online. This only occurs in RC9 and as soon as I go back to RC7 replication works fine.

                    Tom ElliottT 2 Replies Last reply Reply Quote 0
                    • Tom ElliottT
                      Tom Elliott @JGallo
                      last edited by

                      @jgallo the only reason I can think rc9 is broken is the available checks in place. However this would lead me to think it’s a firewall issue. You c an fix this by commenting the new lines in the file. That is the only thing that would be causing the issue from what I can see.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      1 Reply Last reply Reply Quote 0
                      • Tom ElliottT
                        Tom Elliott @JGallo
                        last edited by

                        @jgallo specifically lines 533 - 549 of /var/www/fog/lib/service/fogservice.class.php

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        J 1 Reply Last reply Reply Quote 0
                        • J
                          JGallo @Tom Elliott
                          last edited by

                          @tom-elliott

                          Thank you Tom. I went ahead and made the changes. Worked as soon as a reboot occurred.

                          1 Reply Last reply Reply Quote 0
                          • Tom ElliottT
                            Tom Elliott
                            last edited by

                            Mind installing working branch and see if it is working for you now? I looked over the available code and apparently I made one mistake on a variable for the “domain” element being checked. This variable was not defined anywhere. So i have now setup what should be accurate and just need something to test off of.

                            git checkout working
                            git pull
                            cd bin
                            ./installfog.sh -y
                            

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                            J 3 Replies Last reply Reply Quote 0
                            • Tom ElliottT
                              Tom Elliott
                              last edited by

                              To see what the change I made is see here:
                              https://github.com/FOGProject/fogproject/commit/a0eca190b1f85e4ea7c7586f683f81fc947bb81f

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              J 1 Reply Last reply Reply Quote 0
                              • J
                                JGallo @Tom Elliott
                                last edited by

                                @tom-elliott

                                I don’t mind at all. I will install working branch once this image that is currently replicating is complete. Once I install working branch, I will upload an image and check if replication i working.

                                1 Reply Last reply Reply Quote 0
                                • J
                                  JGallo @Tom Elliott
                                  last edited by

                                  @tom-elliott

                                  I uploaded an image and tried to manually start the replication. I did a tail on the fogreplicator.log file and it seemed like it wasn’t working. So I went to lunch and came back. Looks like it eventually worked. I see the image replicated to my only storage node right now. There are some other weird things occurring but I will create separate post as it has to do with my custom fog.man.reg being copied. Thank you for your help.

                                  1 Reply Last reply Reply Quote 0
                                  • J
                                    JGallo @Tom Elliott
                                    last edited by

                                    @tom-elliott

                                    Added another storage node and replication seems to operate normally. I still have the linux sleeptimer at 10800 seconds so I think we are good for now. My question is do I stay on the working branch or do I go back to RC9? I’m not too familiar when changes are made to RC’s and working branches.

                                    Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                                    • Wayne WorkmanW
                                      Wayne Workman @JGallo
                                      last edited by

                                      @jgallo Stay on the working branch until the next RC.

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                      Daily Clean Installation Results:
                                      https://fogtesting.fogproject.us/
                                      FOG Reporting:
                                      https://fog-external-reporting-results.fogproject.us/

                                      1 Reply Last reply Reply Quote 0
                                      • J
                                        JGallo @Tom Elliott
                                        last edited by

                                        @tom-elliott

                                        So today I got a chance to start adding even more storage nodes. The FOG Server is on the working branch as well as the storage node. Once I’m finished installing the storage node, I go to the console and make changes like enable graph and change the storage node name. I also place a value in the Replication Bandwidth because it won’t let me use 0. So when I tail the fogreplicator.log file, it states a

                                        *Type: 2, File: /var/www/fog/lib/fog/fogftp.class.php, Line: 463, Message: ftp_login(): Login incorrect., Host: 10.225.100.21, Username: fog

                                        Now when I ran the installer, I did type the storage node password from FOG correctly. I know this because the IP shows up under storage. This is awkward because my first two storage nodes worked but were on RC9. Should I keep the storage nodes at RC9 or put the storage nodes on working branch?

                                        Tom ElliottT 1 Reply Last reply Reply Quote 0
                                        • Tom ElliottT
                                          Tom Elliott @JGallo
                                          last edited by

                                          @jgallo I’d put everything on the same particularly as it addresses replication directly. The ftp error however is not because of the password you add when installing a new node. That password is what’s created for the node automatically. Sometimes, however, it’s possible it creates unusable characters.

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                          1 Reply Last reply Reply Quote 0
                                          • 1 / 1
                                          • First post
                                            Last post

                                          196

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project