• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

Replication Issue

Scheduled Pinned Locked Moved Solved
FOG Problems
5
59
10.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    mronh
    last edited by mronh Sep 24, 2018, 10:27 AM Sep 24, 2018, 4:13 PM

    Hello there, Im having a problem with the replication in FOG 1.5.4, I have a node storage on and working perfectly until im upgraded both server and node versions

    the replication never ends… always make one or more parts of images “not matching” as show below

    “[09-24-18 3:56:34 pm] | Sala 204_207 Anexo I: No need to sync d1p2.img.106 file to 2 - Storage (X.X.210.208)
    [09-24-18 3:56:31 pm] | Deleting remote file: /images/Sala204_207AnexoI/d1p2.img.105
    [09-24-18 3:56:31 pm] | Files do not match on server: 2 - Storage (X.X.210.208)
    [09-24-18 3:56:28 pm] | Sala 204_207 Anexo I: No need to sync d1p2.img.104 file to 2 - Storage (X.X.210.208)”

    but the thing is: the source image was never changed since its creation… =/ and anytime I need to deploy a PC with images from this node the windows BSOD, from the ServerNode works fine…

    Any clues?
    thanks in advance, sorry by my english, not my native language.

    W 1 Reply Last reply Sep 24, 2018, 11:44 PM Reply Quote 0
    • W
      Wayne Workman @mronh
      last edited by Sep 24, 2018, 11:44 PM

      @mronh On the storage node, check the free space. use this: df -h and look for partitions that are 99% or 100% used.

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
      Daily Clean Installation Results:
      https://fogtesting.fogproject.us/
      FOG Reporting:
      https://fog-external-reporting-results.fogproject.us/

      1 Reply Last reply Reply Quote 0
      • S
        Sebastian Roth Moderator
        last edited by Sebastian Roth Sep 24, 2018, 11:33 PM Sep 25, 2018, 5:32 AM

        @mronh I am sorry but this is a known issue: https://forums.fogproject.org/topic/12439/fog-1-5-4-replication-issue

        Tom is onto it but we’re extremely busy at the moment and can’t push out a new version quickly. As you see in the other thread you can try updating to the latest working branch. It should be fixed there already.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        M 2 Replies Last reply Sep 25, 2018, 2:21 PM Reply Quote 0
        • M
          mronh @Sebastian Roth
          last edited by Sep 25, 2018, 2:21 PM

          @Sebastian-Roth OK Thanks for the info 🙂

          1 Reply Last reply Reply Quote 0
          • S
            Sebastian Roth Moderator
            last edited by Sep 25, 2018, 2:42 PM

            @mronh Are you interested in trying the working branch?

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            M 3 Replies Last reply Sep 26, 2018, 1:23 PM Reply Quote 0
            • M
              mronh
              last edited by Sep 25, 2018, 7:36 PM

              @Wayne-Workman not this at all, my storage had an 3 Tb HDD, just 1 TB used… but the clue above is a north to guide…

              1 Reply Last reply Reply Quote 0
              • M
                mronh @Sebastian Roth
                last edited by Sep 26, 2018, 1:23 PM

                @Sebastian-Roth, sorry by the delay of response… yes I can try the working branch,

                I’ll make the git pull and return with the result

                1 Reply Last reply Reply Quote 0
                • M
                  mronh @Sebastian Roth
                  last edited by Sep 26, 2018, 1:28 PM

                  @Sebastian-Roth as you are one of the developer, let me ask what is the “business rule” of the replication…

                  its a cron in the server side making checksums of the parts comparing with the cron in the node side, if mismatch delete and "lftp -e "?

                  1 Reply Last reply Reply Quote 0
                  • M
                    mronh @Sebastian Roth
                    last edited by mronh Sep 26, 2018, 8:57 AM Sep 26, 2018, 1:41 PM

                    @Sebastian-Roth make the git pull in the server and the node side…

                    now happens this with the sync

                    “Type: 8, File: /var/www/fog/lib/service/fogservice.class.php, Line: 484, Message: Undefined variable: itemType”

                    I’ll make some deploy to test the images

                    edit: now the ftp server hangs due the max-conections reaches
                    mirror: d1p5.img: Erro fatal: max-retries exceeded (421 There are too many connections from your internet address.)

                    1 Reply Last reply Reply Quote 0
                    • M
                      mronh @Sebastian Roth
                      last edited by Sep 27, 2018, 2:51 PM

                      @Sebastian-Roth
                      No good, get the working branch, and still keeping replicating image parts thats already replicate… and worse… deploys made with images in the storage node make corrupted windows inits (goes to the restoration screen)

                      now I disabled the storage node… thinking of rollback to 1.5.3 (or 1.5.2)

                      1 Reply Last reply Reply Quote 0
                      • S
                        Sebastian Roth Moderator
                        last edited by Sep 27, 2018, 4:02 PM

                        @mronh I am sorry! Hope that I can figure this one out for you. I’ll try to look into it over the weekend. Though I am not as much a PHP wizz than Tom is.

                        Can you please post your logs from the latest try?

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        1 Reply Last reply Reply Quote 0
                        • S
                          Sebastian Roth Moderator
                          last edited by Sep 30, 2018, 4:19 PM

                          @mronh I just installed fresh clean test nodes to see if I can replicate your issue. For me it’s working perfectly fine. Not saying that our code is perfect but at least with my test setup I cannot see an issue. Replication works fine, only copying files over that actually have changed.

                          So again I need to ask you to provide more logs: /var/log/fog/fogreplicator.log and /opt/fog/log/fogreplicator.repl.transfer.x.x.x.x.log for that particular storage node.

                          I am wondering if the replication just cannot finish the copy process in time and therefore the image is corrupted because it never gets transferred all the way?! This would explain the BSODs as well.

                          Please check the sleep time (FOG configuration -> FOG settings -> FOG Linux Service Sleep Times -> IMAGEREPSLEEPTIME). Is it set to the default 600 seconds? Would that be enough to fully sync the image within your network? As far as I know Tom has worked on the issue where one replication run is not finished yet when the next comes along. I think he’s fixed that but I am not sure exactly. You can try increasing the sleep time to something like 3600 (every hour) and see if that fixes your issue… Please let us know.

                          How fast is the link between master node and storage node?

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          M 1 Reply Last reply Oct 5, 2018, 1:51 PM Reply Quote 0
                          • W
                            Wayne Workman
                            last edited by Wayne Workman Sep 30, 2018, 3:33 PM Sep 30, 2018, 9:33 PM

                            @Sebastian-Roth is right, the logs will tell us more. Also, Tom did fix the issue of the replication sleep cycle maybe two years ago. if lftp processes are still running, no more get started. They have timeouts so no fears of infinite hanging processes.

                            Also I’m just going to say that most replication issues I’ve seen people have are normally due to configuration problems. Like a bad fogstorage username & pass, or MySQL remote access being disabled, or firewall on the OS blocking the needed ports, or bad FTP username/pass for the storage node.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                            Daily Clean Installation Results:
                            https://fogtesting.fogproject.us/
                            FOG Reporting:
                            https://fog-external-reporting-results.fogproject.us/

                            1 Reply Last reply Reply Quote 0
                            • M
                              mronh @Sebastian Roth
                              last edited by Oct 5, 2018, 1:51 PM

                              @Sebastian-Roth my apologies by the delay in the answer, I had to rollback to 1.5.2 to get back on the business here ( my boss giving me headaches…)

                              about ur questions:

                              IMAGEREPSLEEPTIME was 600, put 3600. i’ll try again ( but my net here is a giga conection… a LAN in fact, server and storage in the same switch giga port)

                              this weekend i’ll update fog and test it again

                              cheers

                              1 Reply Last reply Reply Quote 0
                              • S
                                Sebastian Roth Moderator
                                last edited by Oct 5, 2018, 2:44 PM

                                @mronh Keeping my fingers crossed it’ll work better this time. Make sure you post full logs here in case you run into issues again.

                                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                M 1 Reply Last reply Oct 9, 2018, 12:40 PM Reply Quote 0
                                • M
                                  mronh @Sebastian Roth
                                  last edited by Oct 9, 2018, 12:40 PM

                                  @Sebastian-Roth

                                  Hi there, the problem still, sadly… as asked, bellow the logs ( I guess about the time spend X speed of the lan but guessing)

                                  cheers 🙂

                                  1_1539088564934_fogreplicator.Sala209RebootRX.transfer.2 - Storage (YYY.YY.210.208).log

                                  0_1539088564933_fogreplicator.log

                                  M 1 Reply Last reply Oct 9, 2018, 1:13 PM Reply Quote 0
                                  • M
                                    mronh @mronh
                                    last edited by mronh Oct 9, 2018, 7:17 AM Oct 9, 2018, 1:13 PM

                                    @mronh what im doing now to try to bypass this: set the sleeptime to 7200, and set a bandwitch limit of the replication of the server and storage node to 10000 Kbps ( maybe, just maybe, lftpd set a default value if its not set in the parms, and pull it down limit)

                                    while this, i’ll seek for some bug in my infra structure…

                                    cheers

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      Sebastian Roth Moderator
                                      last edited by Oct 10, 2018, 10:40 PM

                                      @mronh Please take a look at the apache and php-fpm logs on the storage node (see my signature on where to find those).

                                      The master node asks the storage about file size and hash values of the files. I have a feeling that something is going wrong with this check on your servers. And therefore the master node often thinks files are not equal on both nodes and re-transfers.

                                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                      M 1 Reply Last reply Oct 16, 2018, 4:12 PM Reply Quote 0
                                      • M
                                        mronh @Sebastian Roth
                                        last edited by mronh Oct 16, 2018, 10:16 AM Oct 16, 2018, 4:12 PM

                                        @Sebastian-Roth Hi man, sorry by the new delay ( end of the year put some wood in the fire here =/ )

                                        I’ll attach the log u need, i let the storage node inactive till we find the bug/whatever

                                        thanks in advance dudes
                                        0_1539706517590_php7.1-fpm.log

                                        0_1539706561486_php7.1-fpm.2.log

                                        M 1 Reply Last reply Oct 16, 2018, 4:23 PM Reply Quote 0
                                        • M
                                          mronh @mronh
                                          last edited by Sebastian Roth Oct 16, 2018, 11:27 AM Oct 16, 2018, 4:23 PM

                                          @mronh apache error log pasted here cause has “entity error” when tried to upload (1200 lines of it)

                                          [Tue Oct 09 00:06:05.071128 2018] [mpm_prefork:notice] [pid 4797] AH00163: Apache/2.4.34 (Ubuntu) OpenSSL/1.1.0h configured -- resuming normal operations
[Tue Oct 09 00:06:05.071170 2018] [core:notice] [pid 4797] AH00094: Command line: '/usr/sbin/apache2'
[Tue Oct 09 10:03:55.253531 2018] [proxy_fcgi:error] [pid 16424] [client YYY.YY.211.13:60152] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Tue Oct 09 10:03:56.483605 2018] [proxy_fcgi:error] [pid 16420] [client YYY.YY.211.13:60160] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Tue Oct 09 10:03:57.679877 2018] [proxy_fcgi:error] [pid 16422] [client YYY.YY.211.13:60168] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Tue Oct 09 10:03:58.862615 2018] [proxy_fcgi:error] [pid 16421] [client YYY.YY.211.13:60192] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Tue Oct 09 10:04:00.075334 2018] [proxy_fcgi:error] [pid 16423] [client YYY.YY.211.13:60208] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Tue Oct 09 10:04:01.293366 2018] [proxy_fcgi:error] [pid 16424] [client YYY.YY.211.13:60224] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Tue Oct 09 10:04:02.526186 2018] [proxy_fcgi:error] [pid 16420] [client YYY.YY.211.13:60240] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Tue Oct 09 10:04:03.757965 2018] [proxy_fcgi:error] [pid 16422] [client YYY.YY.211.13:60248] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Tue Oct 09 10 ........

......... [Tue Oct 09 13:50:11.661757 2018] [proxy_fcgi:error] [pid 2761] [client YYY.YY.211.13:32864] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Tue Oct 09 13:50:12.895096 2018] [proxy_fcgi:error] [pid 1109] [client YYY.YY.211.13:32872] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice:  A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n'
[Wed Oct 10 00:06:37.231462 2018] [mpm_prefork:notice] [pid 4797] AH00171: Graceful restart requested, doing restart
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
                                          1 Reply Last reply Reply Quote 0
                                          • 1
                                          • 2
                                          • 3
                                          • 2 / 3
                                          2 / 3
                                          • First post
                                            6/59
                                            Last post

                                          204

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project