Deployment stuck in a loop, never finishes imaging?
@salted_cashews IN the d1.fixed_size_partitions, can you add
:3to the file? And see if that helps at all?
I’m grasping at straws and just guessing at what may be the issue. I don’t know what is the issue right now, of course, so I don’t expect any miracles here. But just testing a theory.
No problem, here’s the output:
administrator@FOG-DHCP:/images$ ls -lhart /images/PPS_v9.0R2_dev_CentOS/ total 30G -rwxrwxrwx 1 root root 5 May 2 07:27 d1.fixed_size_partitions -rwxrwxrwx 1 root root 368 May 2 07:27 d1.partitions -rwxrwxrwx 1 root root 48 May 2 07:29 d1.original.fstypes -rwxrwxrwx 1 root root 0 May 2 07:29 d1.has_grub -rwxrwxrwx 1 root root 1.0M May 2 07:29 d1.mbr -rwxrwxrwx 1 root root 368 May 2 07:29 d1.minimum.partitions -rwxrwxrwx 1 root root 512 May 2 07:29 d1p5.ebr -rwxrwxrwx 1 root root 173M May 2 07:29 d1p1.img -rwxrwxrwx 1 root root 29G May 2 08:12 d1p2.img -rwxrwxrwx 1 root root 512 May 2 08:16 d1p4.ebr -rwxrwxrwx 1 root root 47 May 2 08:16 d1.original.swapuuids -rwxrwxrwx 1 root root 1.1G May 2 08:16 d1p3.img drwxrwxrwx 2 root root 4.0K May 13 12:29 . -rw-rw-r-- 1 administrator administrator 90M May 13 12:29 d1p1_extracted.dat drwxrwxrwx 40 fog root 4.0K May 13 13:32 ..
This certainly is a weird occurrence, I’ve captured and deployed a few images since then so I know nothing has been permanently botched. I know getting that fsck error usually means a corrupt file system correct?
@salted_cashews The - ultimately shouldn’t matter in the context of the imaging process.
We pipe the image into a FIFO (First In First Out) file called /tmp/pigz, then we perform the decompression and write to disk from that filename. So no, it shouldn’t matter how the image is named. In particular, howeve,r is the fact that the image keeps saying no such file or directory called d1p3.img*. The * is in the case of “split” file formatting. (d1p3.img.001, d1p3.img.002, etc…) but should work regardless of the file as long as the file exists.
Looking through the thread, it appears each time asking for information that the file name’s are varying. I’m sure you have the files though, exact comparisons would be great.
In the case of the image named PPS_v9.0R2_dev_CentOS, You have d1p1.img and d1p2.img. So why is it failing on d1p3.img?
The scripting uses the same formatting for each image file. So when it comes to partition 1 it’s running the arg passed as /image/PPS_v9.0R2_dev_CentOS/d1p1.img* as well as d1p2.img* for partition 2. So it’s really confusing to me that it keeps failing on that same file.
Can you get us files inside of /images/PPS_v9.0R2_dev_CentOS/?
ls -lhart /images/PPS_v9.0R2_dev_CentOS/
I’m sure this is redundant and all, but just trying to get the full scope.
Here’s the result after renaming using a “_”. We have other images using “-” and “.”, would this be best practice to avoid even if they still work fine (haven’t had this issue until now)?
Image failed to restore and exited with exit code 1 (writeImage) Info: Partclone v0.2.89 http://partclone.org Starting to restore image (-) to device (/dev/sda3) note: Storage Location 10.10.100.252:/images/, Image name PPS_v9.0R2_dev_CentOS we need memory: 208468 bytes image head 4160, bitmap 200208, crc 4100 bytes Calculating bitmap... Please wait... get device size 53687091200 by ioctl BLKGETSIZE64, done! File system: EXTFS Device size: 6.6 GB = 1601624 Blocks Space in use: 4.5 GB = 1097323 Blocks Free Space: 2.1 GB = 504301 Blocks Block size: 4096 Byte read ERROR:No such file or directory Args Passed: /images/PPS_v9.0R2_dev_CentOS/d1p3.img* /dev/sda3 # Will continue in 1 minute # * Press [Enter] key to continue * Resizing extfs volume (/dev/sda3).................Failed * Press [Enter] key to continue # An error has been detected! # Could not check before resize (expandPartition) Info: /dev/sda3: Inode 393217 has an invalid extent (logical block 0, invalid physical block 1609504, len 26) /dev/sda3: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options) Args Passed: /dev/sda3 :4:5 Kernel variables and settings: loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 web=10.10.100.252/fog/ consoleblank=0 rootfstype=ext4 mac=60:45:cb:a2:86:eb ftp=10.10.100.252 storage=10.10.100.252:/images/ storageip=10.10.100.252 osid=50 irqpoll hostname=Kaibab-20 chkdsk=0 img=PPS_v9.0R2_dev_CentOS imgType=n imgPartitionType=all imgid=333 imgFormat=5 PIGZ_COMP=-19 hostearly=1 isdebug=yes type=down```
@Tom-Elliott Renamed, giving it another shot.
@salted_cashews I think so, yes. Rename the image to something without a - maybe?
@Tom-Elliott Interesting catch, would just renaming and re-associating via the Web GUI work to fix this?
I believe the problem is coming from the second
Particularly in the naming of the image, it appears to be doing:
zstdmt -d /images/PPS_v9.0R2Then get’s a second
So it’s literally, I think, doing:
zstdmt -d ev_CentOS/d1p1.img
Does this make sense?
I think the
-in the image name is causing issues parsing into the zstdmt command. The reason it doesn’t impact the Ciara_CentOS-BASEmk3 is because, likely, there is no argument for
-Bis it just uses it like a normal string.
Maybe we need to add some quoting to the scripting?
zstdmt -d /images/Ciara_CentOS-BASEmk3/d1p1.img -o /images/Ciara_CentOS-BASEmk3/d1p1_extracted.dat /images/Ciara_CentOS-BASEmk3/d1p1.img: 241217197 bytes
No error this time, the error I received running it manually is the same error that displays during the task as well (on the partclone progress screen).
Filesystem Size Used Avail Use% Mounted on udev 2.9G 0 2.9G 0% /dev tmpfs 597M 61M 537M 11% /run /dev/mapper/FOG--DHCP--vg-root 24G 7.7G 15G 36% / tmpfs 3.0G 0 3.0G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.0G 0 3.0G 0% /sys/fs/cgroup /dev/sdb1 1.8T 1.4T 334G 81% /images /dev/sda1 472M 108M 341M 24% /boot tmpfs 597M 0 597M 0% /run/user/1000
@salted_cashews Make sure you have enough space on your disk:
Now as a test, please do the same with another image file:
zstdmt -d /images/PPS_v9.0R2-dev_CentOS/d1p1.img -o /images/PPS_v9.0R2-dev_CentOS/d1p1_extracted.dat
From my point of view the manual extraction test should give you an error if the image file is fine.
-dev_CentOS/d1p3.img : 4201 MB... -dev_CentOS/d1p3.img : Read error (39) : premature end
@salted_cashews Possibly this version of
filedoes not detect Zstd compressed files. Please try to manually extract the image to see if that works properly:
zstdmt -d /images/PPS_v9.0R2-dev_CentOS/d1p3.img -o /images/PPS_v9.0R2-dev_CentOS/d1p3_extracted.dat
See if that triggers an error or not.
Hint: You might need to install package
zsdton your FOG server.
/images/PPS_v9.0R2-dev_CentOS/d1p1.img: data /images/PPS_v9.0R2-dev_CentOS/d1p2.img: data /images/PPS_v9.0R2-dev_CentOS/d1p3.img: data /images/PPS_v9.0R2-dev_CentOS/d1p4.ebr: DOS/MBR boot sector; partition 1 : ID=0x82, start-CHS (0x1bf,247,57), end-CHS (0x2d9,99,10), startsector 8192, 20971520 sectors, extended partition table (last) /images/PPS_v9.0R2-dev_CentOS/d1p5.ebr: data
@salted_cashews Please run the same for all image files in that directory:
@salted_cashews Please run the following command on your FOG server:
Post output here.
@Sebastian-Roth Thank you sir, as far as the logs are concerned this is what they report:
Partclone v0.2.89 http://partclone.org Starting to restore image (-) to device (/dev/sda3) note: Storage Location 10.10.100.252:/images/, Image name PPS_v9.0R2-dev_CentOS we need memory: 208468 bytes image head 4160, bitmap 200208, crc 4100 bytes Calculating bitmap... Please wait... get device size 53687091200 by ioctl BLKGETSIZE64, done! File system: EXTFS Device size: 6.6 GB = 1601624 Blocks Space in use: 4.5 GB = 1097323 Blocks Free Space: 2.1 GB = 504301 Blocks Block size: 4096 Byte read ERROR:No such file or directory
Following this I get a bunch of errors about “inode” something or other, and then the eventual reboot.
Though I have not tested this myself lately it should still work I reckon.
@Sebastian-Roth To my dismay the host was just “nuked” this morning. On the bright side I’m testing another deploy debug and I’m SSHd into the guy. Is it possible to have the root password set via
passwdby default on a deploy/capture? I’d love to just be able to jump in like this at-will.
Let me see if I can trace back exactly what we did.