Can't image from storage node



  • master node/server: fog 1.5.0-RC-10
    os: ubuntu 14.04 LTS

    storage node: fog 1.5.0-RC-10
    os: ubuntu 14.04 LTS

    both server hardware and software are identical.

    master server images perfectly fine and have been using it all year long. Just recently added the storage node and it replicated all the images fine according to the logs. Then when trying to image two computer, the 2nd computer one fail with an error. It connects to the storage node, pulls the image and does the calculating bitmap and right when it gets to d1p3.img file it says “read Error: no such file or directory”

    I check the folder and the file is there. I believe it’s a file integrity issue.

    pigz: skipping: <stdin>: corrupt -- invalid deflate data (invalid code lengths set)  pigz: abort: internal threads error```
    

    I get that error in the partsclone window on computer 2.
    then a warning message shows
    read ERROR:No such file or directory
    args Passed: /images/LABGEN/d1p3.img* /dev/sda3

    then I get the ntfs volume failed error but that’s because it didn’t image sda3 correctly.

    i reboot computer two and set it to image by itself so it can talk to the master node and it images perfectly on the same selected image.

    I saw this thread: https://forums.fogproject.org/topic/6318/image-file-integrity/6 but cannot understand its solution. maybe its a tool to replicate and do a checksum after?

    thanks.


  • Developer

    @nrg Well then it might be the storage node having an issue - disk or RAM!?!



  • @sebastian-roth said in Can’t image from storage node:

    memtest

    did a badblock test on all three sda partition on the storage node. no errors.

    administrator@31-FOGNODE1:~$ sudo badblocks -v /dev/sda1
    Checking blocks 0 to 524287
    Checking for bad blocks (read-only test): done                                                       
    Pass completed, 0 bad blocks found. (0/0/0 errors)
    administrator@31-FOGNODE1:~$ sudo badblocks -v /dev/sda2
    Checking blocks 0 to 483710975
    Checking for bad blocks (read-only test): done
    Pass completed, 0 bad blocks found. (0/0/0 errors)
    administrator@31-FOGNODE1:~$ sudo badblocks -v /dev/sda3
    Checking blocks 0 to 4149247
    Checking for bad blocks (read-only test): done
    Pass completed, 0 bad blocks found. (0/0/0 errors)
    

    btw, i have my fog storage node running Ubuntu EFI installed. could that be an issue?
    also, i couldn’t find memtest in the grub menu. i also tried putting it back and installing it without success. guess im SOL.

    don’t think there’s anything wrong with the client I tried to image from the storage node. Reason, I was able to image from the master node just fine.

    everyone, thank you for all your efforts =)

    edit: I ran memtest86 and there is no errors. pass perfectly fine. so no hard drive issue or memory issue… dont know whats wrong =\


  • Developer

    @nrg So if it’s clearly not the image files (checksums identical, log ok) I can only think of the client machine being the issue.

    Looking at the pictures again I find the message “No such file or directory” to have caused some confusion. It kind of sounds as if the file d1p3.img is missing. But this is not the case. What FOG does is pipe d1pX.img files through a temporary FIFO device that will extract the content on the fly so partclone can use the data straight away. When this temporary FIFO extraction fails partclone would crash with “No such file or directory” as the FIFO isn’t there anymore.

    So the question is, why does the temporary FIFO process pigz abort with “corrupted – invalid deflate data (invalid code lengths set)”? I’d run badblock test and memtest on both the FOG storage node and the client.



  • @sebastian-roth

    master node:

    822a5ed907cb41c30631dc6c160f243f  /images/LABGEN/d1.fixed_size_partitions
    dc9ea3b81f67be37d7d63b297dec1941  /images/LABGEN/d1.mbr
    69d95b8f7a25b7cb62095f3cd358e55c  /images/LABGEN/d1.minimum.partitions
    b293627989626a35e2d6631747b45faf  /images/LABGEN/d1.original.fstypes
    d41d8cd98f00b204e9800998ecf8427e  /images/LABGEN/d1.original.swapuuids
    72c1e188a1f4c3594d7712c41657d227  /images/LABGEN/d1.original.uuids
    2c72b745cb040e72a8f0de0cabc18120  /images/LABGEN/d1p1.img
    9fa1331ab13e0b4f0798289b26752109  /images/LABGEN/d1p2.img
    2021c4a6186077ef7e52320ff7718ef5  /images/LABGEN/d1p3.img
    1fd435efeeb55203f8277a883ff1c17c  /images/LABGEN/d1.partitions
    

    Storage Node:

    822a5ed907cb41c30631dc6c160f243f  /images/LABGEN/d1.fixed_size_partitions
    dc9ea3b81f67be37d7d63b297dec1941  /images/LABGEN/d1.mbr
    69d95b8f7a25b7cb62095f3cd358e55c  /images/LABGEN/d1.minimum.partitions
    b293627989626a35e2d6631747b45faf  /images/LABGEN/d1.original.fstypes
    d41d8cd98f00b204e9800998ecf8427e  /images/LABGEN/d1.original.swapuuids
    72c1e188a1f4c3594d7712c41657d227  /images/LABGEN/d1.original.uuids
    2c72b745cb040e72a8f0de0cabc18120  /images/LABGEN/d1p1.img
    9fa1331ab13e0b4f0798289b26752109  /images/LABGEN/d1p2.img
    2021c4a6186077ef7e52320ff7718ef5  /images/LABGEN/d1p3.img
    1fd435efeeb55203f8277a883ff1c17c  /images/LABGEN/d1.partitions
    

    The logs look fine.
    [12-01-17 8:10:21 pm] | LABGEN: No need to sync d1p3.img file to 31-FOGNODE1
    [12-01-17 8:10:21 pm] | LABGEN: No need to sync d1p3.img file to 31-FOGNODE1
    [12-01-17 8:10:21 pm] | LABGEN: No need to sync d1p2.img file to 31-FOGNODE1
    [12-01-17 8:10:21 pm] | LABGEN: No need to sync d1p2.img file to 31-FOGNODE1
    [12-01-17 8:10:20 pm] | LABGEN: No need to sync d1p1.img file to 31-FOGNODE1
    [12-01-17 8:10:20 pm] | LABGEN: No need to sync d1p1.img file to 31-FOGNODE1
    [12-01-17 8:10:20 pm] | LABGEN: No need to sync d1.partitions file to 31-FOGNODE1
    [12-01-17 8:10:20 pm] | LABGEN: No need to sync d1.partitions file to 31-FOGNODE1
    [12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.uuids file to 31-FOGNODE1
    [12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.uuids file to 31-FOGNODE1
    [12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.swapuuids file to 31-FOGNODE1
    [12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.swapuuids file to 31-FOGNODE1
    [12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.fstypes file to 31-FOGNODE1
    [12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.fstypes file to 31-FOGNODE1
    [12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.minimum.partitions file to 31-FOGNODE1
    [12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.minimum.partitions file to 31-FOGNODE1
    [12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.mbr file to 31-FOGNODE1
    [12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.mbr file to 31-FOGNODE1
    [12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.fixed_size_partitions file to 31-FOGNODE1
    [12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.fixed_size_partitions file to 31-FOGNODE1
    [12-01-17 8:10:17 pm] | Image Name: LABGEN

    for the LABGEN image, it was first captured Jun 15th 2016 on the master node and it works fine.
    the LABGEN image was replicated on the storage node on Nov 9th 2017. The reason why you see Dec 1st on the two files on the storage node is because I manually deleted d1p3.img on the storage node and had the masternode replicate the file again. I knew that it runs the replicator service every 30mins? so if the server did not see the file, it would replicate itself again. that explains the date.

    NOTE: WTF? why does the logs show duplicated lines? bug?


  • Developer

    @nrg Just do make sure… What do you get when running md5sum /images/LABGEN/* on both system (this will take a while to calculate, couple of minutes…).

    As well please check the replication log in the web UI and post here. I am wondering about the timestamps of the files on the storage node. Most are Nov 9th but d1p3.img and d1.fixed_size_partitions are Dez 1st. Maybe those are being replicated over and over again and you seem to run into that when trying to image.



  • This is from the storage node 10.31.1.16:

    total 18148960
    drwxrwxrwx  2 fog fog         4096 Dec  1 09:07 .
    drwxrwxrwx 15 fog root        4096 Nov  9 15:14 ..
    -rwxrwxrwx  1 fog fog            4 Dec  1 09:07 d1.fixed_size_partitions
    -rwxrwxrwx  1 fog fog      1048576 Nov  9 15:14 d1.mbr
    -rwxrwxrwx  1 fog fog          629 Nov  9 15:14 d1.minimum.partitions
    -rwxrwxrwx  1 fog fog           15 Nov  9 15:14 d1.original.fstypes
    -rwxrwxrwx  1 fog fog            0 Nov  9 15:14 d1.original.swapuuids
    -rwxrwxrwx  1 fog fog          215 Nov  9 15:14 d1.original.uuids
    -rwxrwxrwx  1 fog fog     11614954 Nov  9 15:14 d1p1.img
    -rwxrwxrwx  1 fog fog      2171231 Nov  9 15:14 d1p2.img
    -rwxrwxrwx  1 fog fog  18569660631 Dec  1 09:10 d1p3.img
    -rwxrwxrwx  1 fog fog          629 Nov  9 15:14 d1.partitions
    

    this is from the master node 10.31.1.15:

    total 18148960
    drwxrwxrwx  2 fog root        4096 Jun 15  2016 .
    drwxrwxrwx 15 fog root        4096 Aug 17 12:41 ..
    -rwxrwxrwx  1 fog root           4 Jun 15  2016 d1.fixed_size_partitions
    -rwxrwxrwx  1 fog root     1048576 Jun 15  2016 d1.mbr
    -rwxrwxrwx  1 fog root         629 Jun 15  2016 d1.minimum.partitions
    -rwxrwxrwx  1 fog root          15 Jun 15  2016 d1.original.fstypes
    -rwxrwxrwx  1 fog root           0 Jun 15  2016 d1.original.swapuuids
    -rwxrwxrwx  1 fog root         215 Jun 15  2016 d1.original.uuids
    -rwxrwxrwx  1 fog root    11614954 Jun 15  2016 d1p1.img
    -rwxrwxrwx  1 fog root     2171231 Jun 15  2016 d1p2.img
    -rwxrwxrwx  1 fog root 18569660631 Jun 15  2016 d1p3.img
    -rwxrwxrwx  1 fog root         629 Jun 15  2016 d1.partitions
    
    

    You can see the date change of d1p3 on the storage node because I deleted the file and had fog replicate it again today. replicating works, mysql and ftp works.

    I believe fog is designed to jump to the storage node on the 2nd computer when imaging.
    reading through the wiki, it says that computer 1 = MasterNode, computer 2 = Storage node.
    I can see the IP of the storage node on the 2nd computer in the partsclone screen.
    0_1512152787122_error_1.png
    0_1512153744637_error_2.png

    kernal parameter
    0_1512153651684_error_3.jpg


  • Moderator

    On the storage node can you run the following command from the linux console.

    ls -la /images/LABGEN and post the results here?

    One additional question…
    How are you directing the clients to use the storage node vs the master node? When imaging fails it should print out the kernel parameters, confirm that you only see the ip address of the storage node in there for storageip or just use a mobile phone and grab a screen shot of the kernel parameters and post the image here and we can take a look at the results.


Log in to reply
 

385
Online

39.4k
Users

11.1k
Topics

105.3k
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.