• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    storage node sha512sum at 100% CPU/HDD usage

    Scheduled Pinned Locked Moved Solved
    Bug Reports
    5
    31
    4.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Tom ElliottT
      Tom Elliott
      last edited by

      This would seem to indicate, to me, a potential hdd problem?

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      Wayne WorkmanW 1 Reply Last reply Reply Quote 1
      • Wayne WorkmanW
        Wayne Workman @Tom Elliott
        last edited by

        @tom-elliott Higly likely.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
        Daily Clean Installation Results:
        https://fogtesting.fogproject.us/
        FOG Reporting:
        https://fog-external-reporting-results.fogproject.us/

        1 Reply Last reply Reply Quote 0
        • M
          mp12
          last edited by

          Did a fsck and nothing suspicious.
          Machine is running on ESXi. Did a check of the vmdk aswell.
          The harddisc where the VM is located also has no errors.

          I am doing a resync of images/snapins to the storagenode.

          Any ideas?

          Tom ElliottT 1 Reply Last reply Reply Quote 0
          • Tom ElliottT
            Tom Elliott @mp12
            last edited by

            @mp12 this still seems odd. As Wayne stat d fog only checks the first and last meg of the file for hashing. Unless the file is smaller than this. Do you do driver type things? This could be a part of the problem if there’s a lot of small sized files to be checked.

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            M 1 Reply Last reply Reply Quote 0
            • M
              mp12 @Tom Elliott
              last edited by

              @tom-elliott where can I check hashes to compare between master and node? Are they saved somewhere?
              Any logfiles if something is going wrong between master and node?

              1 Reply Last reply Reply Quote 0
              • S
                Sebastian Roth Moderator
                last edited by

                @mp12 Mind running ps ax -o ppid=,pid=,args | grep sha and posting the result here?

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                M 1 Reply Last reply Reply Quote 1
                • M
                  mp12 @Sebastian Roth
                  last edited by

                  @sebastian-roth

                   4029 18135 sh -c sha512sum '/images/pbu_win7_19042018/d1p2.img' | awk '{print $1}'
18135 18136 sha512sum /images/pbu_win7_19042018/d1p2.img
18496 18506 sh -c sha512sum '/images/sojus_win10_09012018/d1p2.img' | awk '{print $1}'
18506 18507 sha512sum /images/sojus_win10_09012018/d1p2.img
                  1 Reply Last reply Reply Quote 0
                  • S
                    Sebastian Roth Moderator
                    last edited by Sebastian Roth

                    @mp12 As Tom already said it should only calculate a checksum for 10 MB of the file. Please run

                    du -b /images/pbu_win7_19042018/d1p2.img /images/sojus_win10_09012018/d1p2.img
ls -al /images/pbu_win7_19042018/d1p2.img /images/sojus_win10_09012018/d1p2.img

                    I assume the files are larger than 10 MB?!

                    Possibly the du command is somehow broken on your system?!

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    M 1 Reply Last reply Reply Quote 0
                    • M
                      mp12 @Sebastian Roth
                      last edited by mp12

                      @sebastian-roth here the results

                      du --version
du (GNU coreutils) 8.2

                      sha512sum --version
sha512sum (GNU coreutils) 8.21

                      du -b /images/pbu_win7_19042018/d1p2.img /images/sojus_win10_09012018/d1p2.img

                      91071384023     /images/pbu_win7_19042018/d1p2.img
90351730508     /images/sojus_win10_09012018/d1p2.img

                      ls -al /images/pbu_win7_19042018/d1p2.img /images/sojus_win10_09012018/d1p2.img

                      -rwxr-xr-x 1 fog fog 91071384023 Apr 25 11:05 /images/pbu_win7_19042018/d1p2.img
-rwxr-xr-x 1 fog fog 90351730508 Apr 25 11:24 /images/sojus_win10_09012018/d1p2.img

                      After waiting a while today (I started the node in the morning) now there are about 5 sha512sum processes 🙂
                      These sha512sum are blocking the website on the master. Timeout 504 😞
                      Will shutdown the node overnight.

                      31667  3364 sh -c sha512sum '/images/sojus_win10_09012018/d1p2.img' | awk '{print $1}'
 3364  3365 sha512sum /images/sojus_win10_09012018/d1p2.img
26029  4053 sh -c sha512sum '/images/sojus_win10_09012018/d1p2.img' | awk '{print $1}'
 4053  4054 sha512sum /images/sojus_win10_09012018/d1p2.img
 3462  4295 sh -c sha512sum '/images/sojus_win7_20042018/d1p2.img' | awk '{print $1}'
 4295  4296 sha512sum /images/sojus_win7_20042018/d1p2.img
22534  4548 sh -c sha512sum '/images/sojus_win7_20042018/d1p2.img' | awk '{print $1}'
 4548  4549 sha512sum /images/sojus_win7_20042018/d1p2.img
14808  5149 sh -c sha512sum '/images/sojus_win7_26042018/d1p2.img' | awk '{print $1}'
 5149  5150 sha512sum /images/sojus_win7_26042018/d1p2.img
                      1 Reply Last reply Reply Quote 0
                      • S
                        Sebastian Roth Moderator
                        last edited by

                        @Tom-Elliott Do you have an idea why the size check wouldn’t work?

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        Tom ElliottT 1 Reply Last reply Reply Quote 0
                        • Tom ElliottT
                          Tom Elliott @Sebastian Roth
                          last edited by

                          @sebastian-roth if I remember correctly if the node only has ftp, say the node is a nas, it will attempt via ftp to get the hash

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          1 Reply Last reply Reply Quote 1
                          • M
                            mp12
                            last edited by

                            Here is a copy from the fogreplicator.log. Maybe this helps

                            [04-30-18 1:25:42 pm] 
==================================
=== ==== ===== ====
=== ========= == === == ===
=== ======== ==== == ==== ===
=== ======== ==== == =========
=== ==== ==== == =========
=== ======== ==== == === ===
=== ======== ==== == ==== ===
=== ========= == === == ===
=== ========== ===== ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[04-30-18 1:25:42 pm] Interface Ready with IP Address: 127.0.0.1
[04-30-18 1:25:42 pm] Interface Ready with IP Address: 127.0.1.1
[04-30-18 1:25:42 pm] Interface Ready with IP Address: IPADRESS
[04-30-18 1:25:42 pm] Interface Ready with IP Address: HOSTNAME
[04-30-18 1:25:42 pm] * Starting ImageReplicator Service
[04-30-18 1:25:42 pm] * Checking for new items every 600 seconds
[04-30-18 1:25:42 pm] * Starting service loop
[04-30-18 1:25:42 pm] * Starting Image Replication.
[04-30-18 1:25:42 pm] * We are group ID: 1. We are group name: default
[04-30-18 1:25:42 pm] * We are node ID: 1. We are node name: fog
[04-30-18 1:25:42 pm] * Attempting to perform Group -> Group image replication.
[04-30-18 1:25:42 pm] | Replicating postdownloadscripts
[04-30-18 1:25:42 pm] * Found Image to transfer to 1 node
[04-30-18 1:25:42 pm] | File Name: postdownloadscripts
[04-30-18 1:25:43 pm] | postdownloadscripts: No need to sync fog.postdownload file to fognode
[04-30-18 1:25:43 pm] | 233 0 /images/postdownloadscripts/fog.postdownload ftp://XXX@IPADRESS/images/postdownloadscripts/fog.postdownload
[04-30-18 1:25:43 pm] | Files do not match.
[04-30-18 1:25:43 pm] * Deleting remote file: /images/postdownloadscripts/fog.postdownload
[04-30-18 1:25:43 pm] * Starting Sync Actions
[04-30-18 1:25:43 pm] | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.fognode.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] IPADRESS
[04-30-18 1:25:43 pm] * Started sync for Image postdownloadscripts
[04-30-18 1:25:43 pm] | Replicating postinitscripts
[04-30-18 1:25:43 pm] * Found Image to transfer to 1 node
[04-30-18 1:25:43 pm] | File Name: dev/postinitscripts
[04-30-18 1:25:43 pm] | dev/postinitscripts: No need to sync fog.postinit file to fognode
[04-30-18 1:25:44 pm] | 249 0 /images/dev/postinitscripts/fog.postinit ftp://XXX@IPADRESS/images/dev/postinitscripts/fog.postinit
[04-30-18 1:25:44 pm] | Files do not match.
[04-30-18 1:25:44 pm] * Deleting remote file: /images/dev/postinitscripts/fog.postinit
[04-30-18 1:25:44 pm] * Starting Sync Actions
[04-30-18 1:25:44 pm] | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.fognode.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/dev/postinitscripts" "/images/dev/postinitscripts"; exit' -u fog,[Protected] IPADRESS
[04-30-18 1:25:44 pm] * Started sync for Image dev/postinitscripts
[04-30-18 1:25:44 pm] * Not syncing Image between groups
[04-30-18 1:25:44 pm] | Image Name: Dell_Default_Win10
[04-30-18 1:25:44 pm] | There are no other members to sync to.
[04-30-18 1:25:44 pm] * Not syncing Image between groups
[04-30-18 1:25:44 pm] | Image Name: pbu_win7_19042018
[04-30-18 1:25:44 pm] | There are no other members to sync to.
[04-30-18 1:25:44 pm] * Not syncing Image between groups
[04-30-18 1:25:44 pm] | Image Name: sojus_win10_09012018
[04-30-18 1:25:44 pm] | There are no other members to sync to.
[04-30-18 1:25:44 pm] * Not syncing Image between groups
[04-30-18 1:25:44 pm] | Image Name: sojus_win7_20042018
[04-30-18 1:25:44 pm] | There are no other members to sync to.
[04-30-18 1:25:44 pm] * Not syncing Image between groups
[04-30-18 1:25:44 pm] | Image Name: sojus_win7_26042018
[04-30-18 1:25:44 pm] | There are no other members to sync to.
[04-30-18 1:25:44 pm] * Attempting to perform Group -> Nodes image replication.
[04-30-18 1:25:44 pm] * Found Image to transfer to 1 node
[04-30-18 1:25:44 pm] | Image Name: Dell_Default_Win10
[04-30-18 1:25:44 pm] | Dell_Default_Win10: No need to sync d1.fixed_size_partitions file to fognode
[04-30-18 1:25:44 pm] | 5 0 /images/Dell_Win10/d1.fixed_size_partitions ftp://XXX@IPADRESS/images/Dell_Win10/d1.fixed_size_partitions
[04-30-18 1:25:44 pm] | Files do not match.
[04-30-18 1:25:44 pm] * Deleting remote file: /images/Dell_Win10/d1.fixed_size_partitions
[04-30-18 1:25:44 pm] | 1048576 0 /images/Dell_Win10/d1.mbr ftp://XXX@IPADRESS/images/Dell_Win10/d1.mbr
[04-30-18 1:25:44 pm] | Files do not match.
[04-30-18 1:25:44 pm] * Deleting remote file: /images/Dell_Win10/d1.mbr
[04-30-18 1:25:44 pm] | 1048576 0 /images/Dell_Win10/d1.mbr ftp://XXX@IPADRESS/images/Dell_Win10/d1.mbr
[04-30-18 1:25:44 pm] | Files do not match.
[04-30-18 1:25:44 pm] * Deleting remote file: /images/Dell_Win10/d1.mbr
[04-30-18 1:25:44 pm] | 839 0 /images/Dell_Win10/d1.minimum.partitions ftp://XXX@IPADRESS/images/Dell_Win10/d1.minimum.partitions
[04-30-18 1:25:44 pm] | Files do not match.
[04-30-18 1:25:44 pm] * Deleting remote file: /images/Dell_Win10/d1.minimum.partitions
[04-30-18 1:25:45 pm] | 839 0 /images/Dell_Win10/d1.minimum.partitions ftp://XXX@IPADRESS/images/Dell_Win10/d1.minimum.partitions
[04-30-18 1:25:45 pm] | Files do not match.
[04-30-18 1:25:45 pm] * Deleting remote file: /images/Dell_Win10/d1.minimum.partitions
[04-30-18 1:25:45 pm] | 30 0 /images/Dell_Win10/d1.original.fstypes ftp://XXX@IPADRESS/images/Dell_Win10/d1.original.fstypes
[04-30-18 1:25:45 pm] | Files do not match.
[04-30-18 1:25:45 pm] * Deleting remote file: /images/Dell_Win10/d1.original.fstypes
[04-30-18 1:25:45 pm] | 30 0 /images/Dell_Win10/d1.original.fstypes ftp://XXX@IPADRESS/images/Dell_Win10/d1.original.fstypes
[04-30-18 1:25:45 pm] | Files do not match.
[04-30-18 1:25:45 pm] * Deleting remote file: /images/Dell_Win10/d1.original.fstypes
[04-30-18 1:25:45 pm] | Dell_Default_Win10: No need to sync d1.original.swapuuids file to fognode
[04-30-18 1:25:45 pm] | Dell_Default_Win10: No need to sync d1.original.swapuuids file to fognode
[04-30-18 1:25:45 pm] | 283 0 /images/Dell_Win10/d1.original.uuids ftp://XXX@IPADRESS/images/Dell_Win10/d1.original.uuids
[04-30-18 1:25:45 pm] | Files do not match.
[04-30-18 1:25:45 pm] * Deleting remote file: /images/Dell_Win10/d1.original.uuids
[04-30-18 1:25:45 pm] | 283 0 /images/Dell_Win10/d1.original.uuids ftp://XXX@IPADRESS/images/Dell_Win10/d1.original.uuids
[04-30-18 1:25:45 pm] | Files do not match.
[04-30-18 1:25:45 pm] * Deleting remote file: /images/Dell_Win10/d1.original.uuids
[04-30-18 1:25:45 pm] | 833 0 /images/Dell_Win10/d1.partitions ftp://XXX@IPADRESS/images/Dell_Win10/d1.partitions
[04-30-18 1:25:45 pm] | Files do not match.
[04-30-18 1:25:45 pm] * Deleting remote file: /images/Dell_Win10/d1.partitions
[04-30-18 1:25:46 pm] | 833 0 /images/Dell_Win10/d1.partitions ftp://XXX@IPADRESS/images/Dell_Win10/d1.partitions
[04-30-18 1:25:46 pm] | Files do not match.
[04-30-18 1:25:46 pm] * Deleting remote file: /images/Dell_Win10/d1.partitions
[04-30-18 1:25:46 pm] | 10829637 0 /images/Dell_Win10/d1p1.img ftp://XXX@IPADRESS/images/Dell_Win10/d1p1.img
[04-30-18 1:25:46 pm] | Files do not match.
[04-30-18 1:25:46 pm] * Deleting remote file: /images/Dell_Win10/d1p1.img
[04-30-18 1:25:46 pm] | 10829637 0 /images/Dell_Win10/d1p1.img ftp://XXX@IPADRESS/images/Dell_Win10/d1p1.img
[04-30-18 1:25:46 pm] | Files do not match.
[04-30-18 1:25:46 pm] * Deleting remote file: /images/Dell_Win10/d1p1.img
[04-30-18 1:25:46 pm] | 1503016 0 /images/Dell_Win10/d1p2.img ftp://XXX@IPADRESS/images/Dell_Win10/d1p2.img
[04-30-18 1:25:46 pm] | Files do not match.
[04-30-18 1:25:46 pm] * Deleting remote file: /images/Dell_Win10/d1p2.img
[04-30-18 1:25:46 pm] | 1503016 0 /images/Dell_Win10/d1p2.img ftp://XXX@IPADRESS/images/Dell_Win10/d1p2.img
[04-30-18 1:25:46 pm] | Files do not match.
[04-30-18 1:25:46 pm] * Deleting remote file: /images/Dell_Win10/d1p2.img
[04-30-18 1:33:11 pm] | 11452366679 0 /images/Dell_Win10/d1p3.img ftp://XXX@IPADRESS/images/Dell_Win10/d1p3.img
[04-30-18 1:33:11 pm] | Files do not match.
[04-30-18 1:33:11 pm] * Deleting remote file: /images/Dell_Win10/d1p3.img
[04-30-18 1:33:44 pm] | 11452366679 0 /images/Dell_Win10/d1p3.img ftp://XXX@IPADRESS/images/Dell_Win10/d1p3.img
[04-30-18 1:33:44 pm] | Files do not match.
[04-30-18 1:33:44 pm] * Deleting remote file: /images/Dell_Win10/d1p3.img
[04-30-18 1:33:47 pm] | 400428853 0 /images/Dell_Win10/d1p4.img ftp://XXX@IPADRESS/images/Dell_Win10/d1p4.img
[04-30-18 1:33:47 pm] | Files do not match.
[04-30-18 1:33:47 pm] * Deleting remote file: /images/Dell_Win10/d1p4.img
[04-30-18 1:33:48 pm] | 400428853 0 /images/Dell_Win10/d1p4.img ftp://XXX@IPADRESS/images/Dell_Win10/d1p4.img
[04-30-18 1:33:48 pm] | Files do not match.
[04-30-18 1:33:48 pm] * Deleting remote file: /images/Dell_Win10/d1p4.img
[04-30-18 1:33:48 pm] * Starting Sync Actions
[04-30-18 1:33:48 pm] | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Dell_Default_Win10.transfer.fognode.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Dell_Win10" "/images/Dell_Win10"; exit' -u fog,[Protected] IPADRESS
[04-30-18 1:33:48 pm] * Started sync for Image Dell_Default_Win10
[04-30-18 1:33:48 pm] * Found Image to transfer to 1 node
[04-30-18 1:33:48 pm] | Image Name: pbu_win7_19042018
[04-30-18 1:33:49 pm] | pbu_win7_19042018: No need to sync d1.fixed_size_partitions file to fognode
[04-30-18 1:33:49 pm] | 3 0 /images/pbu_win7_19042018/d1.fixed_size_partitions ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1.fixed_size_partitions
[04-30-18 1:33:49 pm] | Files do not match.
[04-30-18 1:33:49 pm] * Deleting remote file: /images/pbu_win7_19042018/d1.fixed_size_partitions
[04-30-18 1:33:49 pm] | 1048576 0 /images/pbu_win7_19042018/d1.mbr ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1.mbr
[04-30-18 1:33:49 pm] | Files do not match.
[04-30-18 1:33:49 pm] * Deleting remote file: /images/pbu_win7_19042018/d1.mbr
[04-30-18 1:33:49 pm] | 1048576 0 /images/pbu_win7_19042018/d1.mbr ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1.mbr
[04-30-18 1:33:49 pm] | Files do not match.
[04-30-18 1:33:49 pm] * Deleting remote file: /images/pbu_win7_19042018/d1.mbr
[04-30-18 1:33:49 pm] | 190 0 /images/pbu_win7_19042018/d1.minimum.partitions ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1.minimum.partitions
[04-30-18 1:33:49 pm] | Files do not match.
[04-30-18 1:33:49 pm] * Deleting remote file: /images/pbu_win7_19042018/d1.minimum.partitions
[04-30-18 1:33:50 pm] | 190 0 /images/pbu_win7_19042018/d1.minimum.partitions ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1.minimum.partitions
[04-30-18 1:33:50 pm] | Files do not match.
[04-30-18 1:33:50 pm] * Deleting remote file: /images/pbu_win7_19042018/d1.minimum.partitions
[04-30-18 1:33:50 pm] | 15 0 /images/pbu_win7_19042018/d1.original.fstypes ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1.original.fstypes
[04-30-18 1:33:50 pm] | Files do not match.
[04-30-18 1:33:50 pm] * Deleting remote file: /images/pbu_win7_19042018/d1.original.fstypes
[04-30-18 1:33:50 pm] | 15 0 /images/pbu_win7_19042018/d1.original.fstypes ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1.original.fstypes
[04-30-18 1:33:50 pm] | Files do not match.
[04-30-18 1:33:50 pm] * Deleting remote file: /images/pbu_win7_19042018/d1.original.fstypes
[04-30-18 1:33:50 pm] | pbu_win7_19042018: No need to sync d1.original.swapuuids file to fognode
[04-30-18 1:33:50 pm] | pbu_win7_19042018: No need to sync d1.original.swapuuids file to fognode
[04-30-18 1:33:50 pm] | 190 0 /images/pbu_win7_19042018/d1.partitions ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1.partitions
[04-30-18 1:33:50 pm] | Files do not match.
[04-30-18 1:33:50 pm] * Deleting remote file: /images/pbu_win7_19042018/d1.partitions
[04-30-18 1:33:50 pm] | 190 0 /images/pbu_win7_19042018/d1.partitions ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1.partitions
[04-30-18 1:33:50 pm] | Files do not match.
[04-30-18 1:33:50 pm] * Deleting remote file: /images/pbu_win7_19042018/d1.partitions
[04-30-18 1:33:51 pm] | 7931434 0 /images/pbu_win7_19042018/d1p1.img ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1p1.img
[04-30-18 1:33:51 pm] | Files do not match.
[04-30-18 1:33:51 pm] * Deleting remote file: /images/pbu_win7_19042018/d1p1.img
[04-30-18 1:33:51 pm] | 7931434 0 /images/pbu_win7_19042018/d1p1.img ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1p1.img
[04-30-18 1:33:51 pm] | Files do not match.
[04-30-18 1:33:51 pm] * Deleting remote file: /images/pbu_win7_19042018/d1p1.img
[04-30-18 1:38:51 pm] | 91071384023 0 /images/pbu_win7_19042018/d1p2.img ftp://XXX@IPADRESS/images/pbu_win7_19042018/d1p2.img
[04-30-18 1:38:51 pm] | Files do not match.
[04-30-18 1:38:51 pm] * Deleting remote file: /images/pbu_win7_19042018/d1p2.img

                            The last entry “/images/pbu_win7_19042018/d1p2.img” is now being checked by sha512sum.

                            Tom ElliottT 1 Reply Last reply Reply Quote 0
                            • Tom ElliottT
                              Tom Elliott @mp12
                              last edited by

                              @mp12 based on what I’m seeing the hash is working properly it’s just not finding the right file size. Unless you have two nodes being replicated to?

                              The lines showing the file name with the numbers is the file sizes it’s seeing. First one is the masternodes and the second is the remote nodes. All seem to show the file size of the remote as being 0.

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              1 Reply Last reply Reply Quote 0
                              • Tom ElliottT
                                Tom Elliott
                                last edited by

                                I guess I need to see / understand how your nodes are setup. Are they all regular storage nodes (installed just like a server/node would be) or are they nas or other type of custom layout.

                                Knowing this information will certainly help me find the problem and possibly a solution if they are normal install nodes.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                M 1 Reply Last reply Reply Quote 0
                                • M
                                  mp12 @Tom Elliott
                                  last edited by

                                  @tom-elliott just normal installed. One master server and one storage node. Maybe this is the mistake? Some time ago we had the location plugin enabled. But the problem existed before, I guess. We had a secound storage node some time ago, which I removed from the group. Both (all) running as VM.

                                  1 Reply Last reply Reply Quote 0
                                  • Tom ElliottT
                                    Tom Elliott
                                    last edited by

                                    I did a bit of work on the fogservice in regards to getting the replication tested more appropriately. I just wanted to give you a heads up that I hadn’t forgotten about this, just been kind of busy.

                                    If you would be so kind as to install the github working branch and see if this helps out? I did a bit of work toward getting the filesize and hashes a bit more correctly as it seemed things were in a double check before (which would explain maxing out the sha512sum too).

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                    1 Reply Last reply Reply Quote 0
                                    • M
                                      mp12
                                      last edited by

                                      Thanks @Tom-Elliott. I switched to working. Deleted all images on fog node to see if replication is working properly. I will report back tomorrow.

                                      1 Reply Last reply Reply Quote 0
                                      • M
                                        mp12
                                        last edited by

                                        The load from sha512sum process has decreased to 80% on the node. But now the master is also involved checking hash with high load at around 80%. Smaller files seem to be synced/checked properly. Bigger files obvisly do not match?

                                        Inside the fogreplication.log I am getting messages like this:

                                        [05-17-18 3:38:59 pm]  | Image Name: xxx_win7_26042018
[05-17-18 3:39:00 pm]  | xxx_win7_26042018: No need to sync d1.fixed_size_partitions file to node
[05-17-18 3:39:00 pm]  | xxx_win7_26042018: No need to sync d1.mbr file to node
[05-17-18 3:39:00 pm]  | xxx_win7_26042018: No need to sync d1.minimum.partitions file to node
[05-17-18 3:39:00 pm]  | xxx_win7_26042018: No need to sync d1.original.fstypes file to node
[05-17-18 3:39:01 pm]  | xxx_win7_26042018: No need to sync d1.original.swapuuids file to node
[05-17-18 3:39:01 pm]  | xxx_win7_26042018: No need to sync d1.partitions file to node
[05-17-18 3:39:01 pm]  | xxx_win7_26042018: No need to sync d1p1.img file to node
[05-17-18 3:51:16 pm]  | Files do not match on server: node
[05-17-18 3:51:16 pm]  | Deleting remote file: /images/xxx_win7_26042018/d1p2.img
[05-17-18 3:51:16 pm]  * Starting Sync Actions
[05-17-18 3:51:16 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.xxx_win7_26042018.transfer.node.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/xxx_win7_26042018" "/images/xxx_win7_26042018"; exit' -u fog,[Protected] x.x.x.x

                                        Last entry /opt/fog/log/fogreplicator.xxx_win7_26042018.transfer.node.log is in the past. I don’t think that the d1p2.img file was copied to the node again. I checked the Bandwidthmonitor for 30 minutes. Nothing higher than 3Mbps.

                                        2018-05-17 10:12:37 /images/xxx_win7_26042018/d1p2.img -> ftp://xxx@x.x.x.x/%2Fimages/xxx_win7_26042018/d1p2.img 0-101176680804 27.78 MiB/s

                                        So I think that the sync/check process now only has problems with larger files.
                                        Size of d1p2.img is around 95G.

                                        Tom ElliottT 1 Reply Last reply Reply Quote 0
                                        • Tom ElliottT
                                          Tom Elliott @mp12
                                          last edited by

                                          @mp12 I’m not seeing the same issue. I suppose it could be how large the files are but that still seems unlikely. Is it possible the Lftp command is stuck itself? On the transferring node you can see running lftp processes via ps -ef|grep lftp and I’m going to guess there is none running. Maybe disk space on the receiving node is out?

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                          M 1 Reply Last reply Reply Quote 0
                                          • M
                                            mp12 @Tom Elliott
                                            last edited by mp12

                                            @tom-elliott there is no lftp running. I am having one default image that passes the check normally. The filed1p3.img is around 11G.

                                            [05-18-18 7:29:19 am] | Image Name: xxx_Default_Win10
[05-18-18 7:29:19 am] | xxx_Default_Win10: No need to sync d1.fixed_size_partitions file to node
[05-18-18 7:29:19 am] | xxx_Default_Win10: No need to sync d1.mbr file to node
[05-18-18 7:29:20 am] | xxx_Default_Win10: No need to sync d1.minimum.partitions file to node
[05-18-18 7:29:20 am] | xxx_Default_Win10: No need to sync d1.original.fstypes file to node
[05-18-18 7:29:20 am] | xxx_Default_Win10: No need to sync d1.original.swapuuids file to node
[05-18-18 7:29:20 am] | xxx_Default_Win10: No need to sync d1.original.uuids file to node
[05-18-18 7:29:20 am] | xxx_Default_Win10: No need to sync d1.partitions file to node
[05-18-18 7:29:21 am] | xxx_Default_Win10: No need to sync d1p1.img file to node
[05-18-18 7:29:21 am] | xxx_Default_Win10: No need to sync d1p2.img file to node
[05-18-18 7:30:49 am] | xxx_Default_Win10: No need to sync d1p3.img file to node
[05-18-18 7:30:52 am] | xxx_Default_Win10: No need to sync d1p4.img file to node
[05-18-18 7:30:52 am] * All files synced for this item.

                                            Free disk space around 70G. I will move one image to free up some space and reboot both maschines.

                                            Tom ElliottT 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 2 / 2
                                            • First post
                                              Last post

                                            260

                                            Online

                                            11.6k

                                            Users

                                            17.1k

                                            Topics

                                            154.6k

                                            Posts
                                            Copyright © 2012-2024 FOG Project