• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    master node sending data to storage node.

    Scheduled Pinned Locked Moved Solved
    FOG Problems
    4
    24
    3.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mwolfe60
      last edited by

      Our fog master node started sending large amounts of data to one of the storage node. This started 2 days ago. No changes have been made to either of the nodes. What could of lead to this behavior? I can get more info but right now I’m not at work.

      Thanks

      1 Reply Last reply Reply Quote 0
      • george1421G
        george1421 Moderator
        last edited by

        What version of fog are you using?
        Do you know what kind of data is it sending?
        Have you reviewed the FOG image replicator logs to see if the replicator is stuck on something?
        Is your storage node a full fog server or is it a nas device?

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        1 Reply Last reply Reply Quote 1
        • M
          mwolfe60
          last edited by

          We are currently using Fog 1.5.4
          no we do not know what data
          I’m not sure where the image replicator logs are located to review.
          the storage node is a full fog server and not a NAS device.

          I should also mention that today we attempted to update the server 1.5.5 and the update failed.
          It is running on an Ubuntu Linux server that had problems in the past with auto updates filling up the root file system.

          1 Reply Last reply Reply Quote 0
          • S
            Sebastian Roth Moderator
            last edited by

            @mwolfe60 Hope you have both nodes still on 1.5.4! Please make sure you read and understand the release notes for 1.5.5 - https://news.fogproject.org/fog-1-5-5-officially-released/

            IMPORTANT NOTICE:
            Nodes being on different versions (1.5.4 vs. 1.5.5) will replicate images over and over again as some of the hashing code needed to be changed. Therefore we advise you to update all nodes in one go! Please make sure you stop replication on the master first systemctl stop FOGImageReplicator; systemctl stop FOGSnapinReplicator, then update the storage node(s) and then update master node as a last step.

            Now to your points:

            I’m not sure where the image replicator logs are located to review.

            Take a look at /opt/fog/log/fogreplicator.log on your FOG server.

            I should also mention that today we attempted to update the server 1.5.5 and the update failed.

            We need more details to be able to help. At what stage exactly did it fail? Please post the exact output you get. Either as text or take a picture and post here.

            … in the past with auto updates filling up the root file system.

            Have you made sure there is sufficient space on both nodes?? Run df -h and post output here.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            M 1 Reply Last reply Reply Quote 0
            • M
              mwolfe60 @Sebastian Roth
              last edited by

              @Sebastian-Roth results from df -h

              df -h
              Filesystem Size Used Avail Use% Mounted on
              udev 3.9G 0 3.9G 0% /dev
              tmpfs 799M 8.8M 790M 2% /run
              /dev/mapper/FOGMTNA–vg-root 786G 570G 181G 76% /
              tmpfs 3.9G 0 3.9G 0% /dev/shm
              tmpfs 5.0M 0 5.0M 0% /run/lock
              tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
              /dev/sda1 472M 160M 289M 36% /boot
              tmpfs 799M 0 799M 0% /run/user/1000

              snippet of the error updating

              • Skipping package: php7.1-mysql…(Already Installed)
              • Skipping package: php-gettext…(Already Installed)
              • Installing package: sysv-rc-conf…Failed! (Will try later)
              • Skipping package: tar…(Already Installed)
              • Skipping package: tftpd-hpa…(Already Installed)
              • Skipping package: tftp-hpa…(Already Installed)
              • Skipping package: unzip…(Already Installed)
              • Skipping package: vsftpd…(Already Installed)
              • Skipping package: wget…(Already Installed)
              • Skipping package: xinetd…(Already Installed)
              • Skipping package: zlib1g…(Already Installed)
              • Updating packages as needed…OK
              • Installing now everything is updated…Failed!

              I think I want to move this fog server onto a new server. This server was not setup by me and has some things that are not idea. The images directory is not its own mount point.

              1 Reply Last reply Reply Quote 0
              • S
                Sebastian Roth Moderator
                last edited by

                @mwolfe60 Ok, disk space does not seem to be your issue here. It’s something about the package manager not being able to update packages.

                Please run apt-get update ; apt-get upgrade and I reckon you’ll run into an issue there. Please post the full output you get here. We’ll probably be able to help.

                I think I want to move this fog server onto a new server. This server was not setup by me and has some things that are not idea. The images directory is not its own mount point.

                Well that’s up to you. Quiet often it’s not too bad of an idea to start from scratch but make sure you have a backup copy of all your images read our wiki article: https://wiki.fogproject.org/wiki/index.php?title=Migrate_FOG

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                M 1 Reply Last reply Reply Quote 0
                • S
                  Sebastian Roth Moderator
                  last edited by

                  @mwolfe60 By the way, what about your initial request? Did you take a look at the replication log? Please post here so we can help on this issue.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  1 Reply Last reply Reply Quote 0
                  • M
                    mwolfe60 @Sebastian Roth
                    last edited by

                    @Sebastian-Roth putty.log

                    Apt-get update ran fine.

                    I attached the output from apt-get upgrade

                    When I looked at the Replicator logs it was trying to copy images over. But this was after we updated the storage node to 1.5.5 so we don’t know what it was sending prior to that.

                    I have read the wiki on migrating when i attempted do the migration. The web interface would not load after running through the SSL migration part. Prior to that it would load just fine.

                    Thanks for the help. I’m new to fog and rusty on linux.

                    1 Reply Last reply Reply Quote 0
                    • S
                      Sebastian Roth Moderator
                      last edited by

                      @mwolfe60 Seems like something corrupted your mysql package. But there is a solution: https://askubuntu.com/questions/643251/having-trouble-installing-and-removing-mysql-in-ubuntu

                      Simply create the file /etc/mysql/my.cnf.fallback with the following content: https://www.apt-browse.org/browse/ubuntu/xenial-security/main/all/mysql-common/5.7.25-0ubuntu0.16.04.2/file/etc/mysql/my.cnf.fallback

                      After that again run apt-get upgrade

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      M 1 Reply Last reply Reply Quote 0
                      • M
                        mwolfe60 @Sebastian Roth
                        last edited by

                        @Sebastian-Roth
                        I successfully ran apt-get upgrade after creating the my.cnf.fallback file and updated fog to 1.5.5. Unfortunately the data started replicating to the storage node. It looked like its was replicating all the images across our site to site link to the storage node. I let this run for the evening and checked the for replication log and found that it was all synced.

                        This shows that all files are synced but it is still transmitting about 100Mb/s of data.

                        c9a0bd4f-30ef-4b8f-b274-33d17e4296a6-image.png

                        1 Reply Last reply Reply Quote 0
                        • S
                          Sebastian Roth Moderator
                          last edited by

                          @mwolfe60 Are you sure ALL nodes are updated to 1.5.5??

                          Do you use snapins? We have a bug in 1.5.5 where snapin replication is not working properly and will re-transfer over and over. This is fixed and will be in the next release coming very soon!

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          M 1 Reply Last reply Reply Quote 0
                          • M
                            mwolfe60 @Sebastian Roth
                            last edited by

                            @Sebastian-Roth The only snapin ( I’m assuming plugins are what they are called in the web console) is location which I removed. It still looks like its replicating images to the storage node.

                            1 Reply Last reply Reply Quote 0
                            • S
                              Sebastian Roth Moderator
                              last edited by

                              @mwolfe60 Snapins is not Plugins!

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              1 Reply Last reply Reply Quote 0
                              • M
                                mwolfe60
                                last edited by

                                Then we do not use snapins. I let the replication run all day Saturday and it never stopped. I stopped before it fill the storage node completely.

                                Tom ElliottT 1 Reply Last reply Reply Quote 0
                                • Tom ElliottT
                                  Tom Elliott @mwolfe60
                                  last edited by

                                  @mwolfe60 Did you update all the storage nodes in your environment to be at 1.5.5? If not, It’d be highly recommended that you do. We’re aware of a bug in replication for Snapins, but this does not sound like to be your case. There is a release note explaining the reasoning for needing all nodes to be at the same version.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  1 Reply Last reply Reply Quote 0
                                  • M
                                    mwolfe60
                                    last edited by

                                    Yes all nodes are running 1.5.5

                                    1 Reply Last reply Reply Quote 0
                                    • M
                                      mwolfe60
                                      last edited by

                                      are there any logs that I could provide that would help figure out what is going on here. I’m going to need the server as I have about 70 computers to image in the coming months. I would like to get this one fixed or stand up a new server whichever is going to be the easiest solution.

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        Sebastian Roth Moderator
                                        last edited by Sebastian Roth

                                        @mwolfe60 said:

                                        are there any logs that I could provide that would help figure out what is going on here.

                                        Yes, simply post the full fogreplication.log! If you fear to share this with the whole world you can upload to a private share and send me a link or password protect it and send me the PW. Whatever you like.

                                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                        M 1 Reply Last reply Reply Quote 0
                                        • M
                                          mwolfe60 @Sebastian Roth
                                          last edited by

                                          @Sebastian-Roth I sent the link in a chat.

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            Sebastian Roth Moderator
                                            last edited by Sebastian Roth

                                            @mwolfe60 Thanks for the log. Unfortunately there is only one loop of the replicator in the log and therefore we don’t really see what it’s doing further on. Need more of the log!

                                            What I see in the log so far is that only one single file is not matching (file size different) and it tries to re-transfer that. My guess is that this will take a fair amount of time to transfer as it is roughly 45 GB in size. What speed is the link between your two nodes??

                                            As well, can you post the contents of /opt/fog/log/fogreplicator.Automation_TB_CF54.transfer.Greenville.log or send me a link again?

                                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                            M 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post

                                            161

                                            Online

                                            12.1k

                                            Users

                                            17.3k

                                            Topics

                                            155.3k

                                            Posts
                                            Copyright © 2012-2024 FOG Project