Best posts made by dolf

dolf

OK so while the imaging runs, I’m reading code.

CloneZilla saves the partition like this (from the log file clonezilla-img

partclone.ntfs -z 10485760 -N -L /var/log/partclone.log -c -s /dev/sda2 --output - | pigz -c --fast -b 1024 -p 16 --rsyncable | split -a 2 -b 4096MB - /home/partimag/2016-07-12-21-img-eelabtest/sda2.ntfs-ptcl-img.gz. 2> /tmp/split_error.TMVF3J

Fog does it like this (from savePartition() in src/buildroot/package/fog/scripts/usr/share/fog/lib/funcs.sh)

partclone.$fstype -fsck-src-part -c -s $part -O $fifoname -N -f 1

I’m not really sure how the FIFO buffer works, but it’s called /tmp/pigz1, so I guess it’s the same as piping the output of partclone through pigz, which is exactly what CloneZilla does. Therefore, maybe this would be sufficient to “convert from CloneZilla format to Fog format”:

cat sda2.ntfs-ptcl-img.gz.* > d1p2.img

In other words, no recompression is needed. Am I missing something here?

dolf

I’ve seen both of these things happen in recent trunks. It depends whether you

“Delete all selected items”
or
Edit the image and click “delete”

dolf

@Tom-Elliott you genius

Yes, I was copying files which I captured manually. Your three suggestions, to

temporarily disable replication,
not create the image definition,
or
use the dev folder

all work, and I’m not sure why I didn’t think of that… #facepalm

dolf

It works! The minimum partition size is probably wrong, but I will only deploy to larger drives in any case.

dolf

Feel free to put this in the wiki if you think it’s worthy.

I’m working with a Windows 7 installation, where sda1 is a small boot partition and sda2 is the large partition called C:.

Use GParted (from your favourite linux disc or GParted Live) to resize sda2 to a minimum.
Use CloneZilla to capture the disk. I used beginner mode with the savedisk option. I transferred it to /home/user/czimg/… on the FOG server over ssh, but you could use a USB HDD or any other method.
Create a new resizable image on the FOG web interface. Note the image location.
Create the location specified in the previous step, e.g. mkdir /images/fogimg
Do magic:

cp /home/user/czimg/sda-pt.sf /images/fogimg/d1.minimum.partitions
cp /home/user/czimg/sda-pt.sf /images/fogimg/d1.partitions
cp /home/user/czimg/sda-mbr /images/fogimg/d1.mbr
echo "1" > /images/fogimg/d1.fixed_size_partitions
echo "/dev/sda2 ntfs" > /images/fogimg/d1.original.fstypes
cp /home/user/czimg/sda1.ntfs-ptcl-img.gz.aa /images/fogimg/d1p1.img
cat /home/user/czimg/sda2.ntfs-ptcl-img.gz.* > /images/fogimg/d1p2.img

The last command will take the longest. If you want to see something happening, install pipe viewer and replace the last command with:

cat /home/user/czimg/sda2.ntfs-ptcl-img.gz.* | pv > /images/fogimg/d1p2.img

Have fun

dolf

@Wayne-Workman I didn’t, I’m just speculating. One possible use case might be if you have an SSD for dev to upload faster and a big slow HDD for the long term storage.

dolf

Not in South Africa, though

dolf

Say I have a few images made using CloneZilla. The folder structure looks like this:

user@myserver:~/partimag$ tree -ash
.
├── [4.0K]  image-name
│   ├── [ 677]  blkdev.list
│   ├── [ 249]  blkid.list
│   ├── [6.3K]  clonezilla-img
│   ├── [ 159]  dev-fs.list
│   ├── [   4]  disk
│   ├── [ 20K]  Info-dmi.txt
│   ├── [ 15K]  Info-lshw.txt
│   ├── [2.0K]  Info-lspci.txt
│   ├── [ 216]  Info-packages.txt
│   ├── [  90]  Info-saved-by-cmd.txt
│   ├── [  10]  parts
│   ├── [  33]  sda1.info
│   ├── [8.8M]  sda1.ntfs-ptcl-img.gz.aa
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.aa
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ab
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ac
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ad
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ae
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.af
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ag
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ah
│   ├── [1.5G]  sda2.ntfs-ptcl-img.gz.ai
│   ├── [  37]  sda-chs.sf
│   ├── [1024K]  sda-hidden-data-after-mbr
│   ├── [ 512]  sda-mbr
│   ├── [ 333]  sda-pt.parted
│   ├── [ 295]  sda-pt.parted.compact
│   └── [ 190]  sda-pt.sf
etc...

FOG Images, however, look like this:

user@myserver:/images$ tree
.
├── an-old-image
│   ├── d1.fixed_size_partitions
│   ├── d1.original.fstypes
│   ├── d1.original.partitions
│   ├── d1.original.swapuuids
│   ├── rec.img.000
│   └── sys.img.000
├── a-newer-image
│   ├── d1.fixed_size_partitions
│   ├── d1.mbr
│   ├── d1.minimum.partitions
│   ├── d1.original.fstypes
│   ├── d1.original.swapuuids
│   ├── d1p1.img
│   ├── d1p2.img
│   └── d1.partitions
etc...

Could I somehow import the CloneZilla images into FOG, without deploying and recapturing the image?

dolf

I’m still experiencing this problem. Running trunk these days. However, I have found another workaround to upload resizable images:

Capture a non-resizable image
Make a resizable image, and copy the files manually from the non-resizable image
Create these files manually:
- d1.fixed_size_partitions
  Just contains “:1”
- d1.minimum.partitions
  I make the minimum size of the resizable partition quite a bit larger than the uncompressed size of d1p2.img
  Can check this with gzip -l d1p2.img
- d1.original.fstypes
  /dev/sda2 ntfs
- d1.original.swapuuids
  empty
Deploy

It works. Therefore it seems that resizing the partition before and after capturing with partclone.ntfs is not necessary, even for resizable images.

dolf

Using the latest FOG server (8540) and client (0.11.3). One storage group with two nodes: the main server and one slave.

On the client, in C:\fog.log:

------------------------------------------------------------------------------
---------------------------------SnapinClient---------------------------------
------------------------------------------------------------------------------
 2016/07/13 09:18 PM Client-Info Client Version: 0.11.3
 2016/07/13 09:18 PM Client-Info Client OS:      Windows
 2016/07/13 09:18 PM Client-Info Server Version: 8540
 2016/07/13 09:18 PM Middleware::Response Success
 2016/07/13 09:18 PM SnapinClient Snapin Found:
 2016/07/13 09:18 PM SnapinClient     ID: -1
 2016/07/13 09:18 PM SnapinClient     Name: 
 2016/07/13 09:18 PM SnapinClient     Created: -1
 2016/07/13 09:18 PM SnapinClient     Action: 
 2016/07/13 09:18 PM SnapinClient     Hide: False
 2016/07/13 09:18 PM SnapinClient     TimeOut: -1
 2016/07/13 09:18 PM SnapinClient     RunWith: 
 2016/07/13 09:18 PM SnapinClient     RunWithArgs: 
 2016/07/13 09:18 PM SnapinClient     File: 
 2016/07/13 09:18 PM SnapinClient     Args: 
 2016/07/13 09:18 PM SnapinClient ERROR: Snapin hash does not exist
------------------------------------------------------------------------------

The ID is NOT -1 in the database. In my experience, error messages relating to FOG are usually confusing or misinforming, which is unfortunate. However, after reading this, I started investigating:

http://mymainserver/fog/status/getsnapinhash.php?filepath=/opt/fog/snapins/res1360x768.ps1 returns a hash, but http://mystoragenode/fog/status/getsnapinhash.php?filepath=/opt/fog/snapins/res1360x768.ps1 returns 0.

There were no files in /opt/fog/snapins/ on the storage node. The Snapin Replicator log reported:

[07-13-16 10:18:36 pm] * Starting Sync Actions
[07-13-16 10:18:36 pm] | CMD:
			lftp -e 'set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c -R -i res1360x768.ps1 --ignore-time -vvv --exclude 'dev/' --exclude 'ssl/' --exclude 'CA/' --delete-first /opt/fog/snapins /opt/fog/snapins; exit' -u fog,[Protected] ip.ip.ip.ip
[07-13-16 10:18:36 pm] * Started sync for Snapin Resolution 1360x768
mirror: Access failed: 553 Could not create file. (res1360x768.ps1)

Then I read this and tried:

# chown -R fog:root /opt/fog/snapins

So it works now.

BUT: This was on a fresh install of the latest trunk on a brand new Debian Jessie machine, following the installation instructions and “best practices” (sudo -i and what not) as best I could. So maybe that chown command should go in the installer?

dolf

CloneZilla files, excluding the large ones.

dolf

@Sebastian-Roth said in PC unbootable after capture fails:

To find out what exactly caused this we need the exact error message from when it happens (resize…?)!

See this message, as well as @Tom-Elliott’s reply on it.

The take-home message is still: GParted stops before destroying the disk. FOG should do the same.

dolf

Yep, it works now.

Just for fun: This is the command I use to pull random data from a remote host (and monitor the speed):

$ ssh user@remote "cat /dev/urandom" | pv > /dev/null

dolf

WOW thanks for the advice. I’ll try that soon.

I think the 1MB space between the partitions is just for alignment.

dolf

Hi Sebastian,

Would you help us write the texts and find a good way to place them in the web UI so people definitely see them but don’t get annoyed?

Although I am a developer, I’m not great with UX design. But I would think that some kind of info icon next to the drop-down list with a popup saying something like “Read about the image types here: [insert wiki link]” would suffice. It would draw the attention of anyone who is confused about the options, but is unobtrusive enough not to annoy anyone else.

I’ve seen this somewhere, and wanted to make a screenshot, but can’t find it, so I made a mockup:

This could be used in all kinds of places in the UI. You could even color it red, green or yellow depending on the importance or level of danger. These are just suggestions, and I’m not trying to do your job for you. If you want a pull request for this, I’ll need to figure out where the source code is, first. I’m not acquainted with your source code at all.

The system restore actually worked. It seems that /dev/sda5 (the recovery partition) was still intact, and the “HP recovery” simply formatted /dev/sda3 and rebuilt the entire OS. I think the reason it failed before was because /dev/sda3 was not properly sized to fill the partition, which is what I fixed by rewriting the boot sector with testdisk.

I won’t waste you time with this anymore, and I’ll make sure to take backups with Clonezilla and/or the Single disk (not resizable) option in FOG.

dolf

I created a new image in the Fog web interface, copied the d1.* files from a very similar image, and replaced d1p*.img with concatenated versions of the CloneZilla images. Currently deploying, and partclone seems happy thus far. Just wondering whether the resize will work…

By the way:
Sorry for the delayed responses. It typically takes me 45 minutes to deploy a 65GB image. Without compression the shortest time should be around ((65×1024)MB ÷ (100÷8)MB/s) ÷ 60s = 88 minutes. There’s a 100MB switch somewhere between here and the server

dolf

That looks amazing!

I think linking as much of the UI to the wiki would benefit everyone, even the technical writers. This is a great start. I think when linking a page to the Wiki, that wiki page becomes somewhat more “canonical”, and it shows users where to start, which is sometimes a challenge with Wikis (in comparison to other forms of documentation like an old-school manual).

dolf

If the CloneZilla image is not compressed, it will not have the .gz extention (easy to check for), and the last command becomes:

cat /home/user/czimg/sda2.ntfs-ptcl-img.* | pigz -stdout > /images/fogimg/d1p2.img

If CloneZilla was invoked using dd, we can concatenate and pipe to partclone:

cat /home/user/czimg/sda2.ntfs-dd-img.* | partclone.$fstype -fsck-src-part -c -s - -O - | pigz -c > /images/fogimg/d1p2.img

etc… There might be a few more cases to consider.