OK so while the imaging runs, I’m reading code.

CloneZilla saves the partition like this (from the log file clonezilla-img

partclone.ntfs -z 10485760 -N -L /var/log/partclone.log -c -s /dev/sda2 --output - | pigz -c --fast -b 1024 -p 16 --rsyncable | split -a 2 -b 4096MB - /home/partimag/2016-07-12-21-img-eelabtest/sda2.ntfs-ptcl-img.gz. 2> /tmp/split_error.TMVF3J

Fog does it like this (from savePartition() in src/buildroot/package/fog/scripts/usr/share/fog/lib/funcs.sh)

partclone.$fstype -fsck-src-part -c -s $part -O $fifoname -N -f 1

I’m not really sure how the FIFO buffer works, but it’s called /tmp/pigz1, so I guess it’s the same as piping the output of partclone through pigz, which is exactly what CloneZilla does. Therefore, maybe this would be sufficient to “convert from CloneZilla format to Fog format”:

cat sda2.ntfs-ptcl-img.gz.* > d1p2.img

In other words, no recompression is needed. Am I missing something here?

I’ve seen both of these things happen in recent trunks. It depends whether you

1. “Delete all selected items”
   or
2. Edit the image and click “delete”

@Tom-Elliott you genius

Yes, I was copying files which I captured manually. Your three suggestions, to

• temporarily disable replication,
• not create the image definition,
  or
• use the dev folder

all work, and I’m not sure why I didn’t think of that… #facepalm

It works! The minimum partition size is probably wrong, but I will only deploy to larger drives in any case.

Feel free to put this in the wiki if you think it’s worthy.

I’m working with a Windows 7 installation, where sda1 is a small boot partition and sda2 is the large partition called C:.

1. Use GParted (from your favourite linux disc or GParted Live) to resize sda2 to a minimum.
2. Use CloneZilla to capture the disk. I used beginner mode with the savedisk option. I transferred it to /home/user/czimg/… on the FOG server over ssh, but you could use a USB HDD or any other method.
3. Create a new resizable image on the FOG web interface. Note the image location.
4. Create the location specified in the previous step, e.g. mkdir /images/fogimg
5. Do magic:

cp /home/user/czimg/sda-pt.sf /images/fogimg/d1.minimum.partitions
cp /home/user/czimg/sda-pt.sf /images/fogimg/d1.partitions
cp /home/user/czimg/sda-mbr /images/fogimg/d1.mbr
echo "1" > /images/fogimg/d1.fixed_size_partitions
echo "/dev/sda2 ntfs" > /images/fogimg/d1.original.fstypes
cp /home/user/czimg/sda1.ntfs-ptcl-img.gz.aa /images/fogimg/d1p1.img
cat /home/user/czimg/sda2.ntfs-ptcl-img.gz.* > /images/fogimg/d1p2.img

The last command will take the longest. If you want to see something happening, install pipe viewer and replace the last command with:

cat /home/user/czimg/sda2.ntfs-ptcl-img.gz.* | pv > /images/fogimg/d1p2.img

Have fun

@Wayne-Workman I didn’t, I’m just speculating. One possible use case might be if you have an SSD for dev to upload faster and a big slow HDD for the long term storage.

Not in South Africa, though

Say I have a few images made using CloneZilla. The folder structure looks like this:

user@myserver:~/partimag$ tree -ash
.
├── [4.0K]  image-name
│   ├── [ 677]  blkdev.list
│   ├── [ 249]  blkid.list
│   ├── [6.3K]  clonezilla-img
│   ├── [ 159]  dev-fs.list
│   ├── [   4]  disk
│   ├── [ 20K]  Info-dmi.txt
│   ├── [ 15K]  Info-lshw.txt
│   ├── [2.0K]  Info-lspci.txt
│   ├── [ 216]  Info-packages.txt
│   ├── [  90]  Info-saved-by-cmd.txt
│   ├── [  10]  parts
│   ├── [  33]  sda1.info
│   ├── [8.8M]  sda1.ntfs-ptcl-img.gz.aa
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.aa
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ab
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ac
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ad
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ae
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.af
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ag
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ah
│   ├── [1.5G]  sda2.ntfs-ptcl-img.gz.ai
│   ├── [  37]  sda-chs.sf
│   ├── [1024K]  sda-hidden-data-after-mbr
│   ├── [ 512]  sda-mbr
│   ├── [ 333]  sda-pt.parted
│   ├── [ 295]  sda-pt.parted.compact
│   └── [ 190]  sda-pt.sf
etc...

FOG Images, however, look like this:

user@myserver:/images$ tree
.
├── an-old-image
│   ├── d1.fixed_size_partitions
│   ├── d1.original.fstypes
│   ├── d1.original.partitions
│   ├── d1.original.swapuuids
│   ├── rec.img.000
│   └── sys.img.000
├── a-newer-image
│   ├── d1.fixed_size_partitions
│   ├── d1.mbr
│   ├── d1.minimum.partitions
│   ├── d1.original.fstypes
│   ├── d1.original.swapuuids
│   ├── d1p1.img
│   ├── d1p2.img
│   └── d1.partitions
etc...

Could I somehow import the CloneZilla images into FOG, without deploying and recapturing the image?

I’m still experiencing this problem. Running trunk these days. However, I have found another workaround to upload resizable images:

• Capture a non-resizable image
• Make a resizable image, and copy the files manually from the non-resizable image
• Create these files manually:
  • d1.fixed_size_partitions
    Just contains “:1”
  • d1.minimum.partitions
    I make the minimum size of the resizable partition quite a bit larger than the uncompressed size of d1p2.img
    Can check this with gzip -l d1p2.img
  • d1.original.fstypes
    /dev/sda2 ntfs
  • d1.original.swapuuids
    empty
• Deploy

It works. Therefore it seems that resizing the partition before and after capturing with partclone.ntfs is not necessary, even for resizable images.

Using the latest FOG server (8540) and client (0.11.3). One storage group with two nodes: the main server and one slave.

On the client, in C:\fog.log:

------------------------------------------------------------------------------
---------------------------------SnapinClient---------------------------------
------------------------------------------------------------------------------
 2016/07/13 09:18 PM Client-Info Client Version: 0.11.3
 2016/07/13 09:18 PM Client-Info Client OS:      Windows
 2016/07/13 09:18 PM Client-Info Server Version: 8540
 2016/07/13 09:18 PM Middleware::Response Success
 2016/07/13 09:18 PM SnapinClient Snapin Found:
 2016/07/13 09:18 PM SnapinClient     ID: -1
 2016/07/13 09:18 PM SnapinClient     Name: 
 2016/07/13 09:18 PM SnapinClient     Created: -1
 2016/07/13 09:18 PM SnapinClient     Action: 
 2016/07/13 09:18 PM SnapinClient     Hide: False
 2016/07/13 09:18 PM SnapinClient     TimeOut: -1
 2016/07/13 09:18 PM SnapinClient     RunWith: 
 2016/07/13 09:18 PM SnapinClient     RunWithArgs: 
 2016/07/13 09:18 PM SnapinClient     File: 
 2016/07/13 09:18 PM SnapinClient     Args: 
 2016/07/13 09:18 PM SnapinClient ERROR: Snapin hash does not exist
------------------------------------------------------------------------------

The ID is NOT -1 in the database. In my experience, error messages relating to FOG are usually confusing or misinforming, which is unfortunate. However, after reading this, I started investigating:

http://mymainserver/fog/status/getsnapinhash.php?filepath=/opt/fog/snapins/res1360x768.ps1 returns a hash, but http://mystoragenode/fog/status/getsnapinhash.php?filepath=/opt/fog/snapins/res1360x768.ps1 returns 0.

There were no files in /opt/fog/snapins/ on the storage node. The Snapin Replicator log reported:

[07-13-16 10:18:36 pm] * Starting Sync Actions
[07-13-16 10:18:36 pm] | CMD:
			lftp -e 'set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c -R -i res1360x768.ps1 --ignore-time -vvv --exclude 'dev/' --exclude 'ssl/' --exclude 'CA/' --delete-first /opt/fog/snapins /opt/fog/snapins; exit' -u fog,[Protected] ip.ip.ip.ip
[07-13-16 10:18:36 pm] * Started sync for Snapin Resolution 1360x768
mirror: Access failed: 553 Could not create file. (res1360x768.ps1)

Then I read this and tried:

# chown -R fog:root /opt/fog/snapins

So it works now.

BUT: This was on a fresh install of the latest trunk on a brand new Debian Jessie machine, following the installation instructions and “best practices” (sudo -i and what not) as best I could. So maybe that chown command should go in the installer?

That looks amazing!

I think linking as much of the UI to the wiki would benefit everyone, even the technical writers. This is a great start. I think when linking a page to the Wiki, that wiki page becomes somewhat more “canonical”, and it shows users where to start, which is sometimes a challenge with Wikis (in comparison to other forms of documentation like an old-school manual).

Hi @Junkhacker,

You don’t have to convince me. I get that now.

However, I have read many Fog Forum posts and Fog Wiki pages, and have not come across any such ‘encouragement’. That’s why I suggested that it be made clear in the UI – it would be a form of self-documentation.

I just saw this on Amazon Web Services, in a table header. It’s similar to what I envisaged next to the dropdown:

Hi Sebastian,

Would you help us write the texts and find a good way to place them in the web UI so people definitely see them but don’t get annoyed?

Although I am a developer, I’m not great with UX design. But I would think that some kind of info icon next to the drop-down list with a popup saying something like “Read about the image types here: [insert wiki link]” would suffice. It would draw the attention of anyone who is confused about the options, but is unobtrusive enough not to annoy anyone else.

I’ve seen this somewhere, and wanted to make a screenshot, but can’t find it, so I made a mockup:

This could be used in all kinds of places in the UI. You could even color it red, green or yellow depending on the importance or level of danger. These are just suggestions, and I’m not trying to do your job for you. If you want a pull request for this, I’ll need to figure out where the source code is, first. I’m not acquainted with your source code at all.

---

The system restore actually worked. It seems that /dev/sda5 (the recovery partition) was still intact, and the “HP recovery” simply formatted /dev/sda3 and rebuilt the entire OS. I think the reason it failed before was because /dev/sda3 was not properly sized to fill the partition, which is what I fixed by rewriting the boot sector with testdisk.

I won’t waste you time with this anymore, and I’ll make sure to take backups with Clonezilla and/or the Single disk (not resizable) option in FOG.

Thanks for the quick feedback! It’s always a pleasure to be on this forum.

@Sebastian-Roth said in NTFS partitions corrupt after capturing resizable image:

Why do you choose resizable if you intend to make a backup copy of this laptop? This backup is supposed to go back to the same machine with the same disk and there shouldn’t be a need to use resizable image type for that reason!

Because it’s the default option, and when in doubt, default options are usually safe. In this case it’s not. I understand why this was a bad choice. Lesson learnt.

The non-resizable images type don’t take up more space for storing the image on your FOG server either.

Good to know!

May I suggest that there shoud be some help text or a link to the wiki in the GUI that warns users about the pros and cons of different types of images?

Did you disable Windows 10 fast boot and properly shutdown before you started the capture?

I always shut down from the command line before capturing. Is that sufficient? As far as I understand that disables fast boot for the next boot. That would be the same as “method 2” in the link you referred to, but a tad less hackish.

There is no way we can prevent users from all different kinds of situations. We try to make FOG with as much care as we can bit it’s like a swiss knife and if someone uses it the wrong way no one will blame the factory or engineers for having created a dangerous tool, right?!

This is true!! But maybe the default option should be the safe(-r) one?

Can you boot using a Windows 10 DVD and get into a CMD shell there and run chkdsk command? What output do you get from that? We need a picture!

Yes (Win10 flash drive made with Rufus, actually). This is how I ran chkdsk. I took a few pictures, but it reported so many errors that most of them got lost off-screen. It also unlinked a lot of files and threw them in lost+found.

---

Just before going home today (GMT+2 here), I tried something else. I restored again, from the Clonezilla backup I have, ran testdisk on /dev/sda3 (C:) and copied the backup bootsector over the normal bootsector. Then I rebooted into recovery and used the HP factory reset option instead of Windows’ “reset PC” option. It seemed to work, but it was still running when I left. I’ll post again tomorrow with results!

Relevant Q&A: https://unix.stackexchange.com/questions/349677/unsuccessfully-resized-ntfs-partition-using-gparted-now-it-has-enlarged-partiti

Here is the output of ntfsresize --info /dev/sda3 right after restoring the Clonezilla backup:

Hi there,

It’s a few years since I used FOG, but I’m back.

In short:

New HP 15-ra0xx laptop, that came with Win10 and MS Office pre-installed and activated. I decided it would be a good idea to back up the entire laptop using FOG before I make any changes. It failed, and now I have an brand new unbootable laptop, and the system recovery options won’t work, either.

In more detail:

• New laptop with Windows 10 and a 500G HDD.
• Running FOG 1.5.5 on my server.
• Created host and single disk resizable image.
• Started task to capture image the image, and went home.
• Had an issue with one of my storage nodes where the image path and the FTP path were not the same, which caused an infinite loop.
• The next morning, I came to work and saw this:
  
• Windows would not boot properly from this point on. I just get a black screen with a mouse cursor.
• I freaked out at this point and made a Clonezilla image before trying to fix anything.
• I tried various combinations of chkdsk, ntfsfix, ntfsresize, startup repair, and even the “Reset my PC” option, but nothing works.

This is an expensive issue for me… please help!

Epilogue

My previous experience with FOG also resulted in corrupted HDDs when capturing. It seems that FOG just isn’t careful enough with the source disk. You may recall this: https://forums.fogproject.org/topic/8059/pc-unbootable-after-capture-fails

@Wayne-Workman That worked! Thanks!

See this example: https://forums.fogproject.org/uploads/files/1468447170744-gparted_details_bad.htm

Thank you for looking at this issue! I will try out the new code and report back.

I updated FOG to RC15. Now the FOG clients do update themselves to 0.11.5, but only if I re-deploy the host (using the same image, not a new one).