Can't image from storage node

[nrg]

master node/server: fog 1.5.0-RC-10
os: ubuntu 14.04 LTS

storage node: fog 1.5.0-RC-10
os: ubuntu 14.04 LTS

both server hardware and software are identical.

master server images perfectly fine and have been using it all year long. Just recently added the storage node and it replicated all the images fine according to the logs. Then when trying to image two computer, the 2nd computer one fail with an error. It connects to the storage node, pulls the image and does the calculating bitmap and right when it gets to d1p3.img file it says “read Error: no such file or directory”

I check the folder and the file is there. I believe it’s a file integrity issue.

pigz: skipping: <stdin>: corrupt -- invalid deflate data (invalid code lengths set)  pigz: abort: internal threads error```

I get that error in the partsclone window on computer 2.
then a warning message shows
read ERROR:No such file or directory
args Passed: /images/LABGEN/d1p3.img* /dev/sda3

then I get the ntfs volume failed error but that’s because it didn’t image sda3 correctly.

i reboot computer two and set it to image by itself so it can talk to the master node and it images perfectly on the same selected image.

I saw this thread: https://forums.fogproject.org/topic/6318/image-file-integrity/6 but cannot understand its solution. maybe its a tool to replicate and do a checksum after?

thanks.

[george1421]

On the storage node can you run the following command from the linux console.

ls -la /images/LABGEN and post the results here?

One additional question…
How are you directing the clients to use the storage node vs the master node? When imaging fails it should print out the kernel parameters, confirm that you only see the ip address of the storage node in there for storageip or just use a mobile phone and grab a screen shot of the kernel parameters and post the image here and we can take a look at the results.

[nrg]

This is from the storage node 10.31.1.16:

total 18148960
drwxrwxrwx  2 fog fog         4096 Dec  1 09:07 .
drwxrwxrwx 15 fog root        4096 Nov  9 15:14 ..
-rwxrwxrwx  1 fog fog            4 Dec  1 09:07 d1.fixed_size_partitions
-rwxrwxrwx  1 fog fog      1048576 Nov  9 15:14 d1.mbr
-rwxrwxrwx  1 fog fog          629 Nov  9 15:14 d1.minimum.partitions
-rwxrwxrwx  1 fog fog           15 Nov  9 15:14 d1.original.fstypes
-rwxrwxrwx  1 fog fog            0 Nov  9 15:14 d1.original.swapuuids
-rwxrwxrwx  1 fog fog          215 Nov  9 15:14 d1.original.uuids
-rwxrwxrwx  1 fog fog     11614954 Nov  9 15:14 d1p1.img
-rwxrwxrwx  1 fog fog      2171231 Nov  9 15:14 d1p2.img
-rwxrwxrwx  1 fog fog  18569660631 Dec  1 09:10 d1p3.img
-rwxrwxrwx  1 fog fog          629 Nov  9 15:14 d1.partitions

this is from the master node 10.31.1.15:

total 18148960
drwxrwxrwx  2 fog root        4096 Jun 15  2016 .
drwxrwxrwx 15 fog root        4096 Aug 17 12:41 ..
-rwxrwxrwx  1 fog root           4 Jun 15  2016 d1.fixed_size_partitions
-rwxrwxrwx  1 fog root     1048576 Jun 15  2016 d1.mbr
-rwxrwxrwx  1 fog root         629 Jun 15  2016 d1.minimum.partitions
-rwxrwxrwx  1 fog root          15 Jun 15  2016 d1.original.fstypes
-rwxrwxrwx  1 fog root           0 Jun 15  2016 d1.original.swapuuids
-rwxrwxrwx  1 fog root         215 Jun 15  2016 d1.original.uuids
-rwxrwxrwx  1 fog root    11614954 Jun 15  2016 d1p1.img
-rwxrwxrwx  1 fog root     2171231 Jun 15  2016 d1p2.img
-rwxrwxrwx  1 fog root 18569660631 Jun 15  2016 d1p3.img
-rwxrwxrwx  1 fog root         629 Jun 15  2016 d1.partitions

You can see the date change of d1p3 on the storage node because I deleted the file and had fog replicate it again today. replicating works, mysql and ftp works.

I believe fog is designed to jump to the storage node on the 2nd computer when imaging.
reading through the wiki, it says that computer 1 = MasterNode, computer 2 = Storage node.
I can see the IP of the storage node on the 2nd computer in the partsclone screen.

kernal parameter

[Sebastian Roth]

@nrg Just do make sure… What do you get when running md5sum /images/LABGEN/* on both system (this will take a while to calculate, couple of minutes…).

As well please check the replication log in the web UI and post here. I am wondering about the timestamps of the files on the storage node. Most are Nov 9th but d1p3.img and d1.fixed_size_partitions are Dez 1st. Maybe those are being replicated over and over again and you seem to run into that when trying to image.

[nrg]

@sebastian-roth

master node:

822a5ed907cb41c30631dc6c160f243f  /images/LABGEN/d1.fixed_size_partitions
dc9ea3b81f67be37d7d63b297dec1941  /images/LABGEN/d1.mbr
69d95b8f7a25b7cb62095f3cd358e55c  /images/LABGEN/d1.minimum.partitions
b293627989626a35e2d6631747b45faf  /images/LABGEN/d1.original.fstypes
d41d8cd98f00b204e9800998ecf8427e  /images/LABGEN/d1.original.swapuuids
72c1e188a1f4c3594d7712c41657d227  /images/LABGEN/d1.original.uuids
2c72b745cb040e72a8f0de0cabc18120  /images/LABGEN/d1p1.img
9fa1331ab13e0b4f0798289b26752109  /images/LABGEN/d1p2.img
2021c4a6186077ef7e52320ff7718ef5  /images/LABGEN/d1p3.img
1fd435efeeb55203f8277a883ff1c17c  /images/LABGEN/d1.partitions

Storage Node:

822a5ed907cb41c30631dc6c160f243f  /images/LABGEN/d1.fixed_size_partitions
dc9ea3b81f67be37d7d63b297dec1941  /images/LABGEN/d1.mbr
69d95b8f7a25b7cb62095f3cd358e55c  /images/LABGEN/d1.minimum.partitions
b293627989626a35e2d6631747b45faf  /images/LABGEN/d1.original.fstypes
d41d8cd98f00b204e9800998ecf8427e  /images/LABGEN/d1.original.swapuuids
72c1e188a1f4c3594d7712c41657d227  /images/LABGEN/d1.original.uuids
2c72b745cb040e72a8f0de0cabc18120  /images/LABGEN/d1p1.img
9fa1331ab13e0b4f0798289b26752109  /images/LABGEN/d1p2.img
2021c4a6186077ef7e52320ff7718ef5  /images/LABGEN/d1p3.img
1fd435efeeb55203f8277a883ff1c17c  /images/LABGEN/d1.partitions

The logs look fine.
[12-01-17 8:10:21 pm] | LABGEN: No need to sync d1p3.img file to 31-FOGNODE1
[12-01-17 8:10:21 pm] | LABGEN: No need to sync d1p3.img file to 31-FOGNODE1
[12-01-17 8:10:21 pm] | LABGEN: No need to sync d1p2.img file to 31-FOGNODE1
[12-01-17 8:10:21 pm] | LABGEN: No need to sync d1p2.img file to 31-FOGNODE1
[12-01-17 8:10:20 pm] | LABGEN: No need to sync d1p1.img file to 31-FOGNODE1
[12-01-17 8:10:20 pm] | LABGEN: No need to sync d1p1.img file to 31-FOGNODE1
[12-01-17 8:10:20 pm] | LABGEN: No need to sync d1.partitions file to 31-FOGNODE1
[12-01-17 8:10:20 pm] | LABGEN: No need to sync d1.partitions file to 31-FOGNODE1
[12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.uuids file to 31-FOGNODE1
[12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.uuids file to 31-FOGNODE1
[12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.swapuuids file to 31-FOGNODE1
[12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.swapuuids file to 31-FOGNODE1
[12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.fstypes file to 31-FOGNODE1
[12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.original.fstypes file to 31-FOGNODE1
[12-01-17 8:10:19 pm] | LABGEN: No need to sync d1.minimum.partitions file to 31-FOGNODE1
[12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.minimum.partitions file to 31-FOGNODE1
[12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.mbr file to 31-FOGNODE1
[12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.mbr file to 31-FOGNODE1
[12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.fixed_size_partitions file to 31-FOGNODE1
[12-01-17 8:10:18 pm] | LABGEN: No need to sync d1.fixed_size_partitions file to 31-FOGNODE1
[12-01-17 8:10:17 pm] | Image Name: LABGEN

for the LABGEN image, it was first captured Jun 15th 2016 on the master node and it works fine.
the LABGEN image was replicated on the storage node on Nov 9th 2017. The reason why you see Dec 1st on the two files on the storage node is because I manually deleted d1p3.img on the storage node and had the masternode replicate the file again. I knew that it runs the replicator service every 30mins? so if the server did not see the file, it would replicate itself again. that explains the date.

NOTE: WTF? why does the logs show duplicated lines? bug?

[Sebastian Roth]

@nrg So if it’s clearly not the image files (checksums identical, log ok) I can only think of the client machine being the issue.

Looking at the pictures again I find the message “No such file or directory” to have caused some confusion. It kind of sounds as if the file d1p3.img is missing. But this is not the case. What FOG does is pipe d1pX.img files through a temporary FIFO device that will extract the content on the fly so partclone can use the data straight away. When this temporary FIFO extraction fails partclone would crash with “No such file or directory” as the FIFO isn’t there anymore.

So the question is, why does the temporary FIFO process pigz abort with “corrupted – invalid deflate data (invalid code lengths set)”? I’d run badblock test and memtest on both the FOG storage node and the client.

[nrg]

@sebastian-roth said in Can’t image from storage node:

memtest

did a badblock test on all three sda partition on the storage node. no errors.

administrator@31-FOGNODE1:~$ sudo badblocks -v /dev/sda1
Checking blocks 0 to 524287
Checking for bad blocks (read-only test): done                                                       
Pass completed, 0 bad blocks found. (0/0/0 errors)
administrator@31-FOGNODE1:~$ sudo badblocks -v /dev/sda2
Checking blocks 0 to 483710975
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found. (0/0/0 errors)
administrator@31-FOGNODE1:~$ sudo badblocks -v /dev/sda3
Checking blocks 0 to 4149247
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found. (0/0/0 errors)

btw, i have my fog storage node running Ubuntu EFI installed. could that be an issue?
also, i couldn’t find memtest in the grub menu. i also tried putting it back and installing it without success. guess im SOL.

don’t think there’s anything wrong with the client I tried to image from the storage node. Reason, I was able to image from the master node just fine.

everyone, thank you for all your efforts

edit: I ran memtest86 and there is no errors. pass perfectly fine. so no hard drive issue or memory issue… dont know whats wrong =\

[Sebastian Roth]

@nrg Well then it might be the storage node having an issue - disk or RAM!?!

[nrg]

Re: Can’t image from storage node
Not sure what happened but it seems to be working now. Well imaging from the storage node works when I tried it last week in another group of computers from a separate image that’s stored on the server. I saw the storage node IP on the client computer when imaging.
I have not touched or updated neither the main or node servers.
I have not tried to image from LABGEN from storage node. so 50% working from the main issue. Will follow up or capture a new iamge for LABGEN image soon.

thank you everyone!