NTFS partitions corrupt after capturing resizable image

[dolf]

Hi there,

It’s a few years since I used FOG, but I’m back.

In short:

New HP 15-ra0xx laptop, that came with Win10 and MS Office pre-installed and activated. I decided it would be a good idea to back up the entire laptop using FOG before I make any changes. It failed, and now I have an brand new unbootable laptop, and the system recovery options won’t work, either.

In more detail:

• New laptop with Windows 10 and a 500G HDD.
• Running FOG 1.5.5 on my server.
• Created host and single disk resizable image.
• Started task to capture image the image, and went home.
• Had an issue with one of my storage nodes where the image path and the FTP path were not the same, which caused an infinite loop.
• The next morning, I came to work and saw this:
  
• Windows would not boot properly from this point on. I just get a black screen with a mouse cursor.
• I freaked out at this point and made a Clonezilla image before trying to fix anything.
• I tried various combinations of chkdsk, ntfsfix, ntfsresize, startup repair, and even the “Reset my PC” option, but nothing works.

This is an expensive issue for me… please help!

Epilogue

My previous experience with FOG also resulted in corrupted HDDs when capturing. It seems that FOG just isn’t careful enough with the source disk. You may recall this: https://forums.fogproject.org/topic/8059/pc-unbootable-after-capture-fails

[dolf]

Here is the output of ntfsresize --info /dev/sda3 right after restoring the Clonezilla backup:

[Sebastian Roth]

@dolf said in NTFS partitions corrupt after capturing resizable image:

Created host and single disk resizable image.

Why do you choose resizable if you intend to make a backup copy of this laptop? This backup is supposed to go back to the same machine with the same disk and there shouldn’t be a need to use resizable image type for that reason!

This image type is “messing” with your partitions and therefore I’d not consider this as a “backup save” image type!!

The non-resizable images type don’t take up more space for storing the image on your FOG server either.

Did you disable Windows 10 fast boot and properly shutdown before you started the capture? https://wiki.fogproject.org/wiki/index.php?title=Windows_Dirty_Bit

My previous experience with FOG also resulted in corrupted HDDs when capturing. It seems that FOG just isn’t careful enough with the source disk.

There is no way we can prevent users from all different kinds of situations. We try to make FOG with as much care as we can bit it’s like a swiss knife and if someone uses it the wrong way no one will blame the factory or engineers for having created a dangerous tool, right?!

I am not sure if we are able to help. We will surely try but the situation is not easy with all the things that have gone wrong so far.

Can you boot using a Windows 10 DVD and get into a CMD shell there and run chkdsk command? What output do you get from that? We need a picture!

[dolf]

Thanks for the quick feedback! It’s always a pleasure to be on this forum.

@Sebastian-Roth said in NTFS partitions corrupt after capturing resizable image:

Why do you choose resizable if you intend to make a backup copy of this laptop? This backup is supposed to go back to the same machine with the same disk and there shouldn’t be a need to use resizable image type for that reason!

Because it’s the default option, and when in doubt, default options are usually safe. In this case it’s not. I understand why this was a bad choice. Lesson learnt.

The non-resizable images type don’t take up more space for storing the image on your FOG server either.

Good to know!

May I suggest that there shoud be some help text or a link to the wiki in the GUI that warns users about the pros and cons of different types of images?

Did you disable Windows 10 fast boot and properly shutdown before you started the capture?

I always shut down from the command line before capturing. Is that sufficient? As far as I understand that disables fast boot for the next boot. That would be the same as “method 2” in the link you referred to, but a tad less hackish.

There is no way we can prevent users from all different kinds of situations. We try to make FOG with as much care as we can bit it’s like a swiss knife and if someone uses it the wrong way no one will blame the factory or engineers for having created a dangerous tool, right?!

This is true!! But maybe the default option should be the safe(-r) one?

Can you boot using a Windows 10 DVD and get into a CMD shell there and run chkdsk command? What output do you get from that? We need a picture!

Yes (Win10 flash drive made with Rufus, actually). This is how I ran chkdsk. I took a few pictures, but it reported so many errors that most of them got lost off-screen. It also unlinked a lot of files and threw them in lost+found.

---

Just before going home today (GMT+2 here), I tried something else. I restored again, from the Clonezilla backup I have, ran testdisk on /dev/sda3 (C:) and copied the backup bootsector over the normal bootsector. Then I rebooted into recovery and used the HP factory reset option instead of Windows’ “reset PC” option. It seemed to work, but it was still running when I left. I’ll post again tomorrow with results!

Relevant Q&A: https://unix.stackexchange.com/questions/349677/unsuccessfully-resized-ntfs-partition-using-gparted-now-it-has-enlarged-partiti

[Sebastian Roth]

@dolf Great to see you read my reply as it was meant - a technical elaboration and not personal offense!

Why do you choose resizable if you intend to make a backup copy of this laptop? This backup is supposed to go back to the same machine with the same disk and there shouldn’t be a need to use resizable image type for that reason!

Because it’s the default option, and when in doubt, default options are usually safe. In this case it’s not. I understand why this was a bad choice. Lesson learnt.

This choice is years old, so not something we changed lately. It was even before I started to help working on this project, so I can just guess. I think it’s the most used image type because people want to be able to image to different machines and it is working in most cases. While you can use FOG for backing up machines it’s not the common aim of this software. Again, not saying it’s perfect but we try to make it as save as we possibly can.

May I suggest that there shoud be some help text or a link to the wiki in the GUI that warns users about the pros and cons of different types of images?

Would you help us write the texts and find a good way to place them in the web UI so people definitely see them but don’t get annoyed?

Did you disable Windows 10 fast boot and properly shutdown before you started the capture?

I always shut down from the command line before capturing. Is that sufficient? As far as I understand that disables fast boot for the next boot.

Yes, good you knew about this. I was under the impression that this might have been caused by a unclean filesystem because of fast boot that got somehow corrupted.

There is no way we can prevent users from all different kinds of situations. We try to make FOG with as much care as we can bit it’s like a swiss knife and if someone uses it the wrong way no one will blame the factory or engineers for having created a dangerous tool, right?!

This is true!! But maybe the default option should be the safe(-r) one?

What is the default option of a swiss knife? I suppose it’s opening the hinging blade. I think you are on the right track with saying, we should add help text for people to let them know about the pros and cons of the different choices.

I am sorry I can’t give you really good advice on the actual problem. Without having the machine in front of me I fear I could give instructions that could make things even worse. But let’s give it a try anyway. I suppose you have some parts of the image capture on the FOG server. Take a look at /images/dev/.... There should be a folder named by the MAC address of this computer. Take a look at that. Do you have files d1.mbr, d1.partitions and d1.minimum.partitions in that. Please post the contents of the later two here - they’re just text.

[dolf]

Hi Sebastian,

Would you help us write the texts and find a good way to place them in the web UI so people definitely see them but don’t get annoyed?

Although I am a developer, I’m not great with UX design. But I would think that some kind of info icon next to the drop-down list with a popup saying something like “Read about the image types here: [insert wiki link]” would suffice. It would draw the attention of anyone who is confused about the options, but is unobtrusive enough not to annoy anyone else.

I’ve seen this somewhere, and wanted to make a screenshot, but can’t find it, so I made a mockup:

This could be used in all kinds of places in the UI. You could even color it red, green or yellow depending on the importance or level of danger. These are just suggestions, and I’m not trying to do your job for you. If you want a pull request for this, I’ll need to figure out where the source code is, first. I’m not acquainted with your source code at all.

---

The system restore actually worked. It seems that /dev/sda5 (the recovery partition) was still intact, and the “HP recovery” simply formatted /dev/sda3 and rebuilt the entire OS. I think the reason it failed before was because /dev/sda3 was not properly sized to fill the partition, which is what I fixed by rewriting the boot sector with testdisk.

I won’t waste you time with this anymore, and I’ll make sure to take backups with Clonezilla and/or the Single disk (not resizable) option in FOG.

[dolf]

I just saw this on Amazon Web Services, in a table header. It’s similar to what I envisaged next to the dropdown:

[Junkhacker]

@dolf FOG is image deployment software, not backup software. resizable is the default because that’s the most useful and common type of image capture to be used for FOG’s intended purpose. While, yes, FOG can be used to backup systems, it is not it’s intended purpose and we have encouraged people for years to not use it as such.

[dolf]

Hi @Junkhacker,

You don’t have to convince me. I get that now.

However, I have read many Fog Forum posts and Fog Wiki pages, and have not come across any such ‘encouragement’. That’s why I suggested that it be made clear in the UI – it would be a form of self-documentation.

[Sebastian Roth]

@dolf Finally got around to look into this. Here is a screenshot of what it looks like right now. Any comments on the text or style?

[dolf]

That looks amazing!

I think linking as much of the UI to the wiki would benefit everyone, even the technical writers. This is a great start. I think when linking a page to the Wiki, that wiki page becomes somewhat more “canonical”, and it shows users where to start, which is sometimes a challenge with Wikis (in comparison to other forms of documentation like an old-school manual).

[Sebastian Roth]

Just pushed the change to dev-branch (reference).

@Tom-Elliott Would you please add this to working-1.6 as well?

[Tom Elliott]

@Sebastian-Roth https://github.com/FOGProject/fogproject/commit/d3a70dd417eb1d08b48a637e26ec8776f4409d29

I commented out the “style” info you put, and moved it to push info out to the left instead of the right so that the form field is still accessible when reading the information.