How to manually upload an existing image

[dolf]

Say I have a few images made using CloneZilla. The folder structure looks like this:

user@myserver:~/partimag$ tree -ash
.
├── [4.0K]  image-name
│   ├── [ 677]  blkdev.list
│   ├── [ 249]  blkid.list
│   ├── [6.3K]  clonezilla-img
│   ├── [ 159]  dev-fs.list
│   ├── [   4]  disk
│   ├── [ 20K]  Info-dmi.txt
│   ├── [ 15K]  Info-lshw.txt
│   ├── [2.0K]  Info-lspci.txt
│   ├── [ 216]  Info-packages.txt
│   ├── [  90]  Info-saved-by-cmd.txt
│   ├── [  10]  parts
│   ├── [  33]  sda1.info
│   ├── [8.8M]  sda1.ntfs-ptcl-img.gz.aa
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.aa
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ab
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ac
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ad
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ae
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.af
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ag
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ah
│   ├── [1.5G]  sda2.ntfs-ptcl-img.gz.ai
│   ├── [  37]  sda-chs.sf
│   ├── [1024K]  sda-hidden-data-after-mbr
│   ├── [ 512]  sda-mbr
│   ├── [ 333]  sda-pt.parted
│   ├── [ 295]  sda-pt.parted.compact
│   └── [ 190]  sda-pt.sf
etc...

FOG Images, however, look like this:

user@myserver:/images$ tree
.
├── an-old-image
│   ├── d1.fixed_size_partitions
│   ├── d1.original.fstypes
│   ├── d1.original.partitions
│   ├── d1.original.swapuuids
│   ├── rec.img.000
│   └── sys.img.000
├── a-newer-image
│   ├── d1.fixed_size_partitions
│   ├── d1.mbr
│   ├── d1.minimum.partitions
│   ├── d1.original.fstypes
│   ├── d1.original.swapuuids
│   ├── d1p1.img
│   ├── d1p2.img
│   └── d1.partitions
etc...

Could I somehow import the CloneZilla images into FOG, without deploying and recapturing the image?

[Wayne Workman]

Oh boy… While I’m sure it’s possible, someone is going to have to become familiar with how CloneZilla stores images to do it. To be honest - it’d probably be easier to stick with FOG and get it working and spread what you learn so the Devs can consider it and so others can read it. But I’ll try to point you in the right direction.

Just glancing at the files, the 3.8GB files are obviously gun-zipped chunks. Those will need decompressed into one big image file. You’ll need to mount that via NTFS-3G in a Linux system, and then manually run the Partclone command that would take the image and compress it. This of course is just for partitions 1 and 2.

sda-mbr is obviously the MBR, and its 512 bytes. That can just be renamed to d1.mbr and then placed. The files parts sda1.info and parts sound important, you should post the contents of those.

And, 1,024 bytes worth of info after the MBR but before partition 1 sounds interesting, the sda-hidden-data-after-mbr file. That could honestly be the missing piece for your other thread: https://forums.fogproject.org/topic/8059/pc-unbootable-after-capture-fails/14

Additional reading on the subject of manually manipulating images for use with FOG: https://forums.fogproject.org/topic/7432/mount-and-extract-files-from-images?page=1

[dolf]

CloneZilla files, excluding the large ones.

[Wayne Workman]

@dolf Ok, I had a quick look through those - seems very simple, I like it.

disk - appears to be a list of all disks. this one just says sda
sda-pt.sf, sda-pt.parted.compact, and sda-pt.parted appear to be similar to an fdisk -l output.
sda-mbr is clearly a copy of the mbr.
sda-chs.sf is information about the disk’s cylinders, heads, and sectors.
sda1.info is just saying that’s the boot partition.
parts is a list of partitions. This one has just sda1 and sda2.
info-saved-by-cmd.txt appears to be output from a command CZ uses.
info-packages.txt is a list of exact packages that build of CZ used.
the info files are info about the computer it was pulled from.
dev-fs.list is a list of partitions and their format.
clonezilla-img is literally the output that CZ gave during the image it looks.
blkid.list is simply the output of blkid
blkdev.list I assume is the output of a package called blkdev

Really, to convert this I think all you need is the information from the sda-pt.sf file. This would be a non-resizable image in fog. Use an existing non-resizable image as a template to help guide you with putting the numbers in the right spots. Change sda-mbr to d1.mbr and then you need to uncompress and mount the image, and then capture+compress with partclone and place all files where they should be, assign permissions, and probably fail a few times before it’s right.

Of course the extra info between the mbr and partitions is what’s really interesting. It could just be meaningless gibberish in unused space, but maybe not…

[dolf]

WOW thanks for the advice. I’ll try that soon.

I think the 1MB space between the partitions is just for alignment.

[Wayne Workman]

@dolf I’d urge you to try a non-resizable image in fog first. See if it works. If all the computers are identical they will all have the same size HDD. It’ll either work or not. If it works, that’s how you do it for this model.

[dolf]

OK so while the imaging runs, I’m reading code.

CloneZilla saves the partition like this (from the log file clonezilla-img

partclone.ntfs -z 10485760 -N -L /var/log/partclone.log -c -s /dev/sda2 --output - | pigz -c --fast -b 1024 -p 16 --rsyncable | split -a 2 -b 4096MB - /home/partimag/2016-07-12-21-img-eelabtest/sda2.ntfs-ptcl-img.gz. 2> /tmp/split_error.TMVF3J

Fog does it like this (from savePartition() in src/buildroot/package/fog/scripts/usr/share/fog/lib/funcs.sh)

partclone.$fstype -fsck-src-part -c -s $part -O $fifoname -N -f 1

I’m not really sure how the FIFO buffer works, but it’s called /tmp/pigz1, so I guess it’s the same as piping the output of partclone through pigz, which is exactly what CloneZilla does. Therefore, maybe this would be sufficient to “convert from CloneZilla format to Fog format”:

cat sda2.ntfs-ptcl-img.gz.* > d1p2.img

In other words, no recompression is needed. Am I missing something here?

[dolf]

I created a new image in the Fog web interface, copied the d1.* files from a very similar image, and replaced d1p*.img with concatenated versions of the CloneZilla images. Currently deploying, and partclone seems happy thus far. Just wondering whether the resize will work…

By the way:
Sorry for the delayed responses. It typically takes me 45 minutes to deploy a 65GB image. Without compression the shortest time should be around ((65×1024)MB ÷ (100÷8)MB/s) ÷ 60s = 88 minutes. There’s a 100MB switch somewhere between here and the server

[dolf]

It works! The minimum partition size is probably wrong, but I will only deploy to larger drives in any case.

[Wayne Workman]

@dolf Wow awesome man, way to go! I had serious doubts about this being easy!

Can you please lay out details on all your steps to help people in the future?

[Quazz]

@dolf Could also be a CAT 5 cable somewhere.

[dolf]

Feel free to put this in the wiki if you think it’s worthy.

I’m working with a Windows 7 installation, where sda1 is a small boot partition and sda2 is the large partition called C:.

1. Use GParted (from your favourite linux disc or GParted Live) to resize sda2 to a minimum.
2. Use CloneZilla to capture the disk. I used beginner mode with the savedisk option. I transferred it to /home/user/czimg/… on the FOG server over ssh, but you could use a USB HDD or any other method.
3. Create a new resizable image on the FOG web interface. Note the image location.
4. Create the location specified in the previous step, e.g. mkdir /images/fogimg
5. Do magic:

cp /home/user/czimg/sda-pt.sf /images/fogimg/d1.minimum.partitions
cp /home/user/czimg/sda-pt.sf /images/fogimg/d1.partitions
cp /home/user/czimg/sda-mbr /images/fogimg/d1.mbr
echo "1" > /images/fogimg/d1.fixed_size_partitions
echo "/dev/sda2 ntfs" > /images/fogimg/d1.original.fstypes
cp /home/user/czimg/sda1.ntfs-ptcl-img.gz.aa /images/fogimg/d1p1.img
cat /home/user/czimg/sda2.ntfs-ptcl-img.gz.* > /images/fogimg/d1p2.img

The last command will take the longest. If you want to see something happening, install pipe viewer and replace the last command with:

cat /home/user/czimg/sda2.ntfs-ptcl-img.gz.* | pv > /images/fogimg/d1p2.img

Have fun

[Tom Elliott]

@dolf Maybe, with this addition, I can create some code that can do this automagically? May not be a while but I think it could be very useful. Of course I don’t know if gunzip will work if say the CloneZilla image is not already compressed.

[Wayne Workman]

@Tom-Elliott Or if something besides beginner mode is used in CloneZilla

[dolf]

If the CloneZilla image is not compressed, it will not have the .gz extention (easy to check for), and the last command becomes:

cat /home/user/czimg/sda2.ntfs-ptcl-img.* | pigz -stdout > /images/fogimg/d1p2.img

If CloneZilla was invoked using dd, we can concatenate and pipe to partclone:

cat /home/user/czimg/sda2.ntfs-dd-img.* | partclone.$fstype -fsck-src-part -c -s - -O - | pigz -c > /images/fogimg/d1p2.img

etc… There might be a few more cases to consider.