• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Replication oddity after moving Master node

    Scheduled Pinned Locked Moved
    FOG Problems
    3
    15
    1.2k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      Kiweegie @Kiweegie
      last edited by

      So I just added 2 more Storage nodes to the equation. On both replication didn’t kick in til I’d initiated a manual rsync from Master to storage node. After doing that (again for one image only) the other 3 images all synced OK.

      Wondering if there needs to be some syncing of SSH keys or similar to pemit the servers to talk to each other and thats not being handled automatically? Just a theory…

      regards Tom

      1 Reply Last reply Reply Quote 0
      • S
        Sebastian Roth Moderator
        last edited by

        @kiweegie said in Replication oddity after moving Master node:

        Wondering if there needs to be some syncing of SSH keys or similar to pemit the servers to talk to each other and thats not being handled automatically? Just a theory…

        Not that I know of! Sounds strange that you need to manually sync one before it does the other ones for you.

        @Tom-Elliott Would you have an idea?

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        Tom ElliottT 1 Reply Last reply Reply Quote 0
        • Tom ElliottT
          Tom Elliott @Sebastian Roth
          last edited by

          @sebastian-roth I’m not aware of anything special with this. This doesn’t make much sense to me. We don’t use SSH at all for replicating data. LFTP is used. So Maybe selinux was causing issues that rsync corrected for? I don’t know other wise.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

          1 Reply Last reply Reply Quote 0
          • K
            Kiweegie @Kiweegie
            last edited by

            @Tom-Elliott @Sebastian-Roth

            Morning gents, just revisting this as I have some more findings. The replication issue seems to be limited to the Master node not replicating images listed for Storage Group 2 for which a DCSTORAGE is the master.

            Using the previously sent image of the setup, images capture to DCMASTER and it replicates fine to storage nodes with the same storage group set Storage Group 1

            It is not being captured or replicating to DCSTORAGE however which is the master storage node for the second storage group.

            I’ve checked the wiki re replication and it states there that:

            Images always capture to the primary groups master storage node - so the image which is set as Storage Group 2 should capture to the master node for that storage group DCSTORAGE but is actually capturing to the Master Node DCMASTER.

            So it would seem that that image is not seeing that DCSTORAGE is its master storage node even though the image(s) are defined as such.

            I’ve confirmed also that DCSTORAGE is set as the Master node for Storage Group 2

            Anything else I could me missing here?

            regards Tom

            K Tom ElliottT 2 Replies Last reply Reply Quote 0
            • K
              Kiweegie @Kiweegie
              last edited by

              @kiweegie said in Replication oddity after moving Master node:

              @Tom-Elliott @Sebastian-Roth

              Morning gents, just revisting this as I have some more findings. The replication issue seems to be limited to the Master node not replicating images listed for Storage Group 2 for which a DCSTORAGE is the master.

              Using the previously sent image of the setup, images capture to DCMASTER and it replicates fine to storage nodes with the same storage group set Storage Group 1

              It is not being captured or replicating to DCSTORAGE however which is the master storage node for the second storage group.

              I’ve checked the wiki re replication and it states there that:

              Images always capture to the primary groups master storage node - so the image which is set as Storage Group 2 should capture to the master node for that storage group DCSTORAGE but is actually capturing to the Master Node DCMASTER.

              So it would seem that that image is not seeing that DCSTORAGE is its master storage node even though the image(s) are defined as such.

              I’ve confirmed also that DCSTORAGE is set as the Master node for Storage Group 2

              Anything else I could me missing here?

              regards Tom

              Additional note - when image is captured (to Master node) it sets ownership as the local unix user:root not fogproject. Not sure if this has bearing on things?

              1 Reply Last reply Reply Quote 0
              • Tom ElliottT
                Tom Elliott @Kiweegie
                last edited by

                @kiweegie So the way replication works for Storage groups:

                1. An image can be defined to multiple storage groups.
                2. An image can be told which storage group is the image’s Primary Group

                What this does:

                The image’s Primary Storage group will then replicate to the other storage groups master nodes that are defined for that image.

                If I’m to draw a picture of your environment:

                So this image is configured to have Storage Group 1 and Storage Group 2, but what it sounds like, too me, is this image has Storage Group 1 set as the Primary group. Can you go to the image and validate that Storage Group 2 is defined as this image’s Primary Storage group?

                Can you also get the output from mariadb/mysql of:

                SELECT imageID,imageName,ngID,ngName
                FROM imageGroupAssoc
                LEFT OUTER JOIN images ON igaImageID = imageID
                LEFT OUTER JOIN nfsGroups ON igaStorageGroupID = ngID
                WHERE imageName = 'nameofimage';

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                K 1 Reply Last reply Reply Quote 0
                • K
                  Kiweegie @Tom Elliott
                  last edited by Sebastian Roth

                  @tom-elliott said in Replication oddity after moving Master node:

                  SELECT imageID,imageName,ngID,ngName
                  FROM imageGroupAssoc
                  LEFT OUTER JOIN images ON igaImageID = imageID
                  LEFT OUTER JOIN nfsGroups ON igaStorageGroupID = ngID
                  WHERE imageName = ‘nameofimage’;

                  Thanks for getting in touch Tom.

                  The whole both storage groups “can” apply to an image was throwing me I confess. I actually added a brand new storage group “Storage Group 3” for the sake of the earlier example and deleted Storage group 2.

                  I’ve set only Storage Group 3 on the image(s) I want to sync from the DCSTORAGE node and made sure it’s set as master

                  Output of SQL above is as follows:

                  Database changed
                  MariaDB [fog]> SELECT imageID,imageName,ngID,ngName
                      -> FROM imageGroupAssoc
                      -> LEFT OUTER JOIN images ON igaImageID = imageID
                      -> LEFT OUTER JOIN nfsGroups ON igaStorageGroupID = ngID
                      -> WHERE imageName = 'My_image';
                  +---------+------------+------+-----------------+
                  | imageID | imageName  | ngID | ngName          |
                  +---------+------------+------+-----------------+
                  |       9 | My_image   |    4 | Storage Group 3 |
                  +---------+------------+------+-----------------+
                  1 row in set (0.01 sec)
                  

                  On the assumption this is correct (now) do I need to restart any services or servers themselves to have them pick up on this change and start syncing?

                  regards Tom

                  Tom ElliottT 2 Replies Last reply Reply Quote 0
                  • Tom ElliottT
                    Tom Elliott @Kiweegie
                    last edited by

                    @kiweegie You shouldn’t need to, but it wouldn’t hurt.

                    I haven’t played in the replicator code in quite some time though.

                    You can restart the replication services via:

                    for i in FOG{Image,Snapin}Replicator.service; do systemctl restart $i;done

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    1 Reply Last reply Reply Quote 0
                    • Tom ElliottT
                      Tom Elliott @Kiweegie
                      last edited by Sebastian Roth

                      @kiweegie Based no what I see in the SQL, “My_image” is only set for Storage Group 3. So if My_image is not on DCSTORAGE, it won’t get replicated.

                      So what you should see in teh output there would be something like:

                      +---------+-----------+------+-----------------+
                      | imageID | imageName | ngID | ngName          |
                      +---------+-----------+------+-----------------+
                      |       9 | My_image  |    4 | Storage Group 3 |
                      |       9 | My_image  |    3 | DCSTORAGE       |
                      +---------+-----------+------+-----------------+
                      

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      K 1 Reply Last reply Reply Quote 0
                      • K
                        Kiweegie @Tom Elliott
                        last edited by

                        @tom-elliott So to clarify that last point Tom, the SQL should reflect under ngName both the Storage group name AND the Storage node name? DCSTORAGE in this case is the name of the Storage node master server for the Storage Group 3 storage group.

                        I’ve compared to the other image(s) which are only hosted on (and are being replicated from) the Master node and they only reflect the main storage group under ngName.

                        I’ve made note of the replicator service restart command, cheers. I went a bit neanderthal and just rebooted all my servers just now to see if that kicked things into life.

                        Checking the Image Replicator logs under FOG Configuration > Log Viewer it only references DCMASTER and the 2 nodes in its storage group. Does replication from the Storage master of the other storage group show in these logs? Or do we need to check local log files on the server itself?

                        cheers Tom

                        1 Reply Last reply Reply Quote 0
                        • 1 / 1
                        • First post
                          Last post

                        240

                        Online

                        12.1k

                        Users

                        17.3k

                        Topics

                        155.4k

                        Posts
                        Copyright © 2012-2024 FOG Project