• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

Image replication between nodes keeps restarting

Scheduled Pinned Locked Moved Unsolved
FOG Problems
5
19
3.5k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • A
    AndersHoeg
    last edited by May 31, 2017, 10:44 AM

    I’ve got two servers running 1.4.0.
    The master storage node is located in Denmark and the other storage node is located in Norway.
    The replication works, but every time it has replicated an image, it just deletes the image on the server in Norway and starts over once again. It has been doing this for about a week, any ideas?

    1 Reply Last reply Reply Quote 0
    • G
      george1421 Moderator
      last edited by May 31, 2017, 11:21 AM

      This is an interesting issue.

      The fog replicator compares the md5sum of a certain number of bytes of each file before it decides if they are different. If the md5sum key is not the same then it makes the decision that the storage node is out of sync and starts replicating the image again.

      The fog replicator logs are in /opt/fog/logs does the master replication log or the replication log for the norway server give you an idea why it just keeps replicating?

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

      1 Reply Last reply Reply Quote 0
      • A
        AndersHoeg
        last edited by May 31, 2017, 11:31 AM

        Hmmm maybe it is trying to replicate to often and that’s the issue.
        Any clue where I can see/change how often it tries to replicate an image?

        W 1 Reply Last reply Jun 1, 2017, 1:44 AM Reply Quote 0
        • W
          Wayne Workman @AndersHoeg
          last edited by Wayne Workman May 31, 2017, 7:46 PM Jun 1, 2017, 1:44 AM

          @AndersHoeg said in Image replication between nodes keeps restarting:

          Any clue where I can see/change how often it tries to replicate an image?

          FOG Web Interface -> FOG Configuration -> FOG Settings -> FOG Linux Service Sleep Times -> IMAGEREPSLEEPTIME

          FOG Replication logs are also viewable in the web interface:
          FOG Web Interface -> FOG Configuration -> Log Viewer -> Image Replicator

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
          Daily Clean Installation Results:
          https://fogtesting.fogproject.us/
          FOG Reporting:
          https://fog-external-reporting-results.fogproject.us/

          A J 2 Replies Last reply Jun 1, 2017, 10:09 AM Reply Quote 0
          • A
            AndersHoeg @Wayne Workman
            last edited by Jun 1, 2017, 10:09 AM

            @Wayne-Workman Thank you so much! It worked, it sync the images and don’t delete them over and over again.

            1 Reply Last reply Reply Quote 0
            • J
              JGallo @Wayne Workman
              last edited by JGallo Sep 14, 2017, 9:44 AM Sep 14, 2017, 3:29 PM

              @wayne-workman

              I know this is a really old post and I apologize if I should have started a new topic. I’m experiencing the same symptoms after only accidentally discovering the replication keeps deleting my images from the nodes. By increasing the value of the FOG Linux Service Sleep Times - IMAGEREPSLEEPTIME, will this stop the image from being deleted?

              Basically would I need to set the value to a point beyond the time it takes FOG to finish replicating? If the value is less than the time it takes FOG to finish replicating, would this explain why the images keeps being deleted and replication restarting?

              W 1 Reply Last reply Sep 14, 2017, 7:52 PM Reply Quote 0
              • W
                Wayne Workman @JGallo
                last edited by Wayne Workman Sep 14, 2017, 1:56 PM Sep 14, 2017, 7:52 PM

                @jgallo If the image replicator is deleting the files if it’s not completed by the next iteration - this is a bug. I thought this was fixed completely somewhere between 1.2 and 1.3… I guess the issue has cropped up again. So yeah, just to see if this is the case, set your IMAGEREPSLEEPTIME to something huge and see if it’s fixed. If that fixes it, then that’s bug confirmed.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                Daily Clean Installation Results:
                https://fogtesting.fogproject.us/
                FOG Reporting:
                https://fog-external-reporting-results.fogproject.us/

                J 1 Reply Last reply Sep 14, 2017, 8:28 PM Reply Quote 0
                • J
                  JGallo @Wayne Workman
                  last edited by Sep 14, 2017, 8:28 PM

                  @wayne-workman

                  I increased it to 10800 which is 3 hours just for the hell of it. Seems to work fine on 1.5.0 RC7 unless replication takes longer than that LOL but issue I ran into was when I try to go and upload an image I get some weird DHCP issues. So I went ahead and went to RC9 and then replication does’t work. I came across a post regarding same issue and suggestion was to go back to RC7. So I’m in a bind and not sure what to do. I’m going to start from scratch and add each node and tail the fogreplicator.log file and see if the same errors occur. I’m getting a (storage node) server does not appear to be online. This only occurs in RC9 and as soon as I go back to RC7 replication works fine.

                  T 2 Replies Last reply Sep 14, 2017, 10:43 PM Reply Quote 0
                  • T
                    Tom Elliott @JGallo
                    last edited by Sep 14, 2017, 10:43 PM

                    @jgallo the only reason I can think rc9 is broken is the available checks in place. However this would lead me to think it’s a firewall issue. You c an fix this by commenting the new lines in the file. That is the only thing that would be causing the issue from what I can see.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    1 Reply Last reply Reply Quote 0
                    • T
                      Tom Elliott @JGallo
                      last edited by Sep 14, 2017, 10:48 PM

                      @jgallo specifically lines 533 - 549 of /var/www/fog/lib/service/fogservice.class.php

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      J 1 Reply Last reply Sep 15, 2017, 3:57 PM Reply Quote 0
                      • J
                        JGallo @Tom Elliott
                        last edited by Sep 15, 2017, 3:57 PM

                        @tom-elliott

                        Thank you Tom. I went ahead and made the changes. Worked as soon as a reboot occurred.

                        1 Reply Last reply Reply Quote 0
                        • T
                          Tom Elliott
                          last edited by Sep 15, 2017, 5:01 PM

                          Mind installing working branch and see if it is working for you now? I looked over the available code and apparently I made one mistake on a variable for the “domain” element being checked. This variable was not defined anywhere. So i have now setup what should be accurate and just need something to test off of.

                          git checkout working
                          git pull
                          cd bin
                          ./installfog.sh -y
                          

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          J 3 Replies Last reply Sep 15, 2017, 5:30 PM Reply Quote 0
                          • T
                            Tom Elliott
                            last edited by Sep 15, 2017, 5:03 PM

                            To see what the change I made is see here:
                            https://github.com/FOGProject/fogproject/commit/a0eca190b1f85e4ea7c7586f683f81fc947bb81f

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                            J 1 Reply Last reply Sep 18, 2017, 10:55 PM Reply Quote 0
                            • J
                              JGallo @Tom Elliott
                              last edited by Sep 15, 2017, 5:30 PM

                              @tom-elliott

                              I don’t mind at all. I will install working branch once this image that is currently replicating is complete. Once I install working branch, I will upload an image and check if replication i working.

                              1 Reply Last reply Reply Quote 0
                              • J
                                JGallo @Tom Elliott
                                last edited by Sep 15, 2017, 7:29 PM

                                @tom-elliott

                                I uploaded an image and tried to manually start the replication. I did a tail on the fogreplicator.log file and it seemed like it wasn’t working. So I went to lunch and came back. Looks like it eventually worked. I see the image replicated to my only storage node right now. There are some other weird things occurring but I will create separate post as it has to do with my custom fog.man.reg being copied. Thank you for your help.

                                1 Reply Last reply Reply Quote 0
                                • J
                                  JGallo @Tom Elliott
                                  last edited by Sep 15, 2017, 10:21 PM

                                  @tom-elliott

                                  Added another storage node and replication seems to operate normally. I still have the linux sleeptimer at 10800 seconds so I think we are good for now. My question is do I stay on the working branch or do I go back to RC9? I’m not too familiar when changes are made to RC’s and working branches.

                                  W 1 Reply Last reply Sep 15, 2017, 10:27 PM Reply Quote 0
                                  • W
                                    Wayne Workman @JGallo
                                    last edited by Sep 15, 2017, 10:27 PM

                                    @jgallo Stay on the working branch until the next RC.

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                    Daily Clean Installation Results:
                                    https://fogtesting.fogproject.us/
                                    FOG Reporting:
                                    https://fog-external-reporting-results.fogproject.us/

                                    1 Reply Last reply Reply Quote 0
                                    • J
                                      JGallo @Tom Elliott
                                      last edited by Sep 18, 2017, 10:55 PM

                                      @tom-elliott

                                      So today I got a chance to start adding even more storage nodes. The FOG Server is on the working branch as well as the storage node. Once I’m finished installing the storage node, I go to the console and make changes like enable graph and change the storage node name. I also place a value in the Replication Bandwidth because it won’t let me use 0. So when I tail the fogreplicator.log file, it states a

                                      *Type: 2, File: /var/www/fog/lib/fog/fogftp.class.php, Line: 463, Message: ftp_login(): Login incorrect., Host: 10.225.100.21, Username: fog

                                      Now when I ran the installer, I did type the storage node password from FOG correctly. I know this because the IP shows up under storage. This is awkward because my first two storage nodes worked but were on RC9. Should I keep the storage nodes at RC9 or put the storage nodes on working branch?

                                      T 1 Reply Last reply Sep 18, 2017, 11:50 PM Reply Quote 0
                                      • T
                                        Tom Elliott @JGallo
                                        last edited by Sep 18, 2017, 11:50 PM

                                        @jgallo I’d put everything on the same particularly as it addresses replication directly. The ftp error however is not because of the password you add when installing a new node. That password is what’s created for the node automatically. Sometimes, however, it’s possible it creates unusable characters.

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                        1 Reply Last reply Reply Quote 0
                                        • 1 / 1
                                        • First post
                                          Last post

                                        140

                                        Online

                                        12.0k

                                        Users

                                        17.3k

                                        Topics

                                        155.2k

                                        Posts
                                        Copyright © 2012-2024 FOG Project