FOG resizes all partitions even System Reserved!

[Sebastian Roth]

@Quazz @Tom-Elliott I have done some testing on my own now and this is definitely unicode hell! Short story is I think we should not rely on those label checks anymore as they can go wrong so easily with non ASCII characters. Here you go with a bit of unicode fun in the client shell:

[Mon Dec 03 root@fogclient ~]# e=$(echo -ne '\xC3\xA9')
[Mon Dec 03 root@fogclient ~]# E=$(echo -ne '\xC3\x89')
[Mon Dec 03 root@fogclient ~]# label=$(echo -ne 'R\xC3\xA9serv\xC3\xA9_au_syst')
[Mon Dec 03 root@fogclient ~]# echo $e
é
[Mon Dec 03 root@fogclient ~]# echo $E
É
[Mon Dec 03 root@fogclient ~]# echo $label
Réservé_au_syst

Ok that’s for starters just to get the right characters set in variables as I can’t seem to enter those using my keyboard in a ssh session on a client (neither can I in the VM terminal). So I suppose bash and the underlaying libs are able to display unicode characters but it’s not fully supported anyhow.
Important: This is using the UTF-8 codes for é but there are other encoding standards like ISO-8859-1 through to ISO-8859-15 and many more that may encode the very same character with different codes. Or let me say it the other way round. If we read that label the returned string might be using different unicodes than we had used in the scripts although the characters look identical to our eyes it would still not match. So here comes the fun part:

[Mon Dec 03 root@fogclient ~]# if [[ $label =~ [Rr][Ee$E$e] ]]; then echo "JA"; fi
JA

So using the variables in the bash regex actually does work. But…

[Mon Dec 03 root@fogclient ~]# if [[ $label =~ [Rr][Ee$E$e][Ss] ]]; then echo "JA"; fi

What?!? I simply added [Ss] which should match, shouldn’t it? Ok let’s try to skip the special character for now.

[Mon Dec 03 root@fogclient ~]# if [[ $label =~ [Rr].[Ss] ]]; then echo "JA"; fi
[Mon Dec 03 root@fogclient ~]# if [[ $label =~ [Rr]..[Ss] ]]; then echo "JA"; fi
JA
[Mon Dec 03 root@fogclient ~]# if [[ $label =~ [Rr][Ee$E$e].[Ss] ]]; then echo "JA"; fi
JA

Crazy stuff. So this special character ends up being two characters when doing bash regex. I still have no idea what that extra character might be and how to find it other than using . as any character. I guess it stems from our buildroot bash only partially supporting UTF-8 unicode. Anyhow, this is how my new regex looks like:

[Mon Dec 03 root@fogclient ~]# if [[ $label =~ [Rr][Ee$E$e].[Ss][Ee][Rr][Vv][Ee$E$e].[Dd]? ]]; then echo "JA"; fi
JA

And exactly the same if we use grep instead:

[Mon Dec 03 root@fogclient ~]# echo $label | grep "[Rr][Ee$E$e][Ss][Ee][Rr][Vv][Ee$E$e]"
[Mon Dec 03 root@fogclient ~]# echo $label | grep "[Rr][Ee$E$e].[Ss][Ee][Rr][Vv][Ee$E$e].[Dd]*"
Réservé_au_syst

Same ugly hack I reckon. And please keep in mind that this could fail if some Windows installations were made using ISO-8859-1 code pages. So to sum it all up. Let’s move forward and not waste any more time to find the perfect regex matching all the labels out there.

We have started to gather information on that stuff and I think we should tackle it now and see if it works any better: https://github.com/FOGProject/fos/issues/18

[Quazz]

@Sebastian-Roth I’ll be glad to escape unicode hell, I do think focus on a language neutral solution is the way to go. I’m still unsure about the best direction to take.

So far, it seems we have two possible paths to explore:

File based detection:
Will check the mounted partition for specific files/folders, if they exist then we know what kind of partition we can expect (eg resizable partition if Windows folder is found)

However, I foresee some funky stuff being able to mess this up. Atypical installations might not be detected properly.

Partition flag based detection:
Will check the mounted partition for specific flags, if they exist then we know what kind of partition we can expect based on the set flags.

However, partition flags seem to have been somewhat inconsistently set as well as have changed between MBR and GPT somewhat. And of course users/software could theoretically alter these (although this will typically lead to issues with the system in question so less likely than with labels and marginally less likely than with files)

Our greatest advantage is that we only really need to think about NTFS partitions if we are simply replacing the old label system. This means we can work a lot more targetted regardless of the path taken.

I created and tested out the partition flag system which seems to work well on the systems I use, but of course I have no idea how it will behave outside of that.

[Sebastian Roth]

@Quazz Yes, I think those are the two options we have. I tend to go the file based way because this is something that people/software can’t fiddle with. I am fairly sure that no one will ever change the boot loader binary path/name or Windows installation dir name. There is no reason to do this. Sure there is not much reason to change the flags either but I feel like this can happen more easily and the difference is, that Windows will still work perfectly fine with altered flags - so people won’t notice until they clone with FOG.

Our greatest advantage is that we only really need to think about NTFS partitions if we are simply replacing the old label system. This means we can work a lot more targetted regardless of the path taken.

I was hoping to improve the whole set and also make detection if Linux partitions work better. But that might just be another step.

Essentially I want to distinguish between partition only being fixed in size (e.g. recovery as we don’t want to expand that) and those that need to be fixed in position and size (e.g. boot partitions). But that’s the bigger picture. I might have the time to work this out over Christmas. Don’t think I will get there before.

[Quazz]

@Sebastian-Roth I was looking into a file based approach but ran into a major speed bump.

It seems that on GPT builds of Windows 10 (at least on the latest official ISO release of Microsoft), the only folder (and there’s no files) on the first partition (which should be non-resizable) is ‘System Volume Information’ which is available on every NTFS partition…

Not sure how to resolve this at this point.

Got a working flag based approach available for now which should work fine for cases such as @maxcarpone has experienced.

edit: Even better news, had an OEM disk come in so I checked it out and it turns out they also use the ‘hidden’ flag which means we can mark them as non-resizable as well with this approach.

Of course as you pointed out, ideally they’d be ‘moveable’, but that’s a different issue I think, just fixing the partitions being resized that shouldn’t should do for now I think!

[Sebastian Roth]

@Quazz said:

It seems that on GPT builds of Windows 10 (at least on the latest official ISO release of Microsoft), the only folder (and there’s no files) on the first partition (which should be non-resizable) is ‘System Volume Information’ which is available on every NTFS partition…

What exactly is the first partition for I am wondering?!

Got a working flag based approach available for now which should work fine for cases such as @maxcarpone has experienced.

Great stuff. I would say let’s give it a try! Mind sending a pull request towards out fos git repo?

[Quazz]

@Sebastian-Roth https://github.com/FOGProject/fos/pull/22

[Sebastian Roth]

Merged the changes, should be building new inits now. @maxcarpone Will let you know as soon as they are available for download.

[maxcarpone]

@Sebastian-Roth Ok, I’ll be happy to try those new inits

[Sebastian Roth]

@maxcarpone Unfortunately our build server seems to play up. Has not started building yet and I am not sure why…

[Sebastian Roth]

@maxcarpone Sorry for the long deply again. Build server is fixed now and here you got the fresh new inits - on your FOG server run:

sudo -i
cd /var/www/html/fog/service/ipxe
mkdir initbackup
mv init.xz init_32.xz initbackup/
wget -O init.xz https://fogproject.org/inits/init.xz
wget -O init_32.xz https://fogproject.org/inits/init_32.xz

Now recapture the image and see if it’s any better.

[maxcarpone]

Hey guys!

It looks like it works so far. I tested with 2 fresh installations of W10 1803, one in MBR and the other one in UEFI and each capture has its own d1.fixed_size_partitions populated.

Content of these files :

• MBR => :1
  which is the partition labeled : Réservé au système

• UEFI => :2:3:1
  Which are Recover, System, MSR

I will do more experiments soon but I like what I saw ^^

Thanks for the good work

[imt_fog]

@Sebastian-Roth does it make sense to test these init files also with 1.5.5 stable? If yes, I’d be willing to test as I ran into the “resizing” issue just a few days ago having “System-Reserviert” as partition label on some of our Win10 Edu German master images. It took me days to find out that the boot loop indeed was not an IDE/AHCI ang buggy BIOS issue, which we had before on some point as well…

[Sebastian Roth]

@imt_fog Yeah, definitely give those a try. The ones you have in place with a 1.5.5 install are still using the label based detection which can cause issues on Windows installs with language other than English.

[imt_fog]

@Sebastian-Roth I am using the new inits, and right now, it seems there is no resizing at all.

If I redeploy an image of a MBR based system, it has exactly the same size as on the captured one. On the other hand, there is no “fixed partition” message on the screen, but also no resizing message.

Content of d1.fixed_size_partitions

:1:2

Content of d1.original.fstypes

/dev/sda2 ntfs

I’ll switch back to the original inits, but if there is anything I could provide to help debugging, please let me know! Or could there also be an issue because of our local naming policy naming “System”, thus being confused with the UEFI “System” partition of GPT based systems? I didn’t check the sources yet

[Sebastian Roth]

@imt_fog With the new inits we don’t check the labels anymore. But if you messed with the flags that could be an issue.

Did you capture and deploy using the new inits??

[imt_fog]

@Sebastian-Roth yep, both with the new ones.

I’ll do a debug run next week and check the messages, and I will check the flags as done in https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload#L109 - I did not “play” with the flags manually, but I will compare to a brand new Win10 MBR installation if the flags got mixed up.

Other question: could I just change the d1.fixed_size_partitions to

:1

as long as my source file system was “small enough”? I actually prepare the images on a virtual machine having the VHD slightly smaller than the smallest target system SSDs we have in use.

Normally, I prepare in the VM, deploy once to each target platforms, check the drivers and then capture each platform. Right now, we have 4 different models in our department (~30 to ~90 per model)

[Sebastian Roth]

@imt_fog said in FOG resizes all partitions even System Reserved!:

Other question: could I just change the d1.fixed_size_partitions to

:1

Yes! Then on deploy it would expand the second partition for you.

I can actually imagine the second partition to be marked as bootable and therefore being recognized as non-resizable

[Quazz]

@imt_fog If you boot the machine in question in debug mode, you should be able to run parted -l manually and check the flags.

I don’t think this is caused by such, however, but interested in seeing the results.

Worth noting that partitions that go through the checks but fail to resize get marked as fixed size as well (to prevent issues during deploy)