Move partitions on GPT layouts - need people to test

Sebastian Roth · Jul 11, 2021, 7:26 AM

UPDATE: This is now part the official inits!!

After several people noticed that Windows 10 2004 comes with a different partition layout that FOG is not able to shrink and deploy to a smaller size disk I started working on that code and adding the functionality to actually move the start sector of partitions. Up to now FOG never moved partitions because it would surely cause trouble as manipulating boot loader code in the MBR would also be needed and is way too error prone to do. With GPT partition layout on UEFI based systems partition start positions should not be “hardcoded” in bootloader code anymore.

Though this seems to work I would really like more people to test this before we actually add this to the official release!

Here are the steps I recommend when testing:

Make sure you have a working backup copy of your host used to capture the image from! Just in case something goes wrong when capturing with the new init.
Download the init file and put that in /var/www/html/fog/service/ipxe/ on your FOG server.
Create a new image definition for this test.
Decide which hosts you use for testing capture and deploy (the later having a smaller size disk to see if it works) with this new init, edit their hosts settings in the FOG web UI and set Host Init to init-201114.xz as well as the image definition you just created.
Schedule a capture task and pay attention to the boot process where it says bzImage..ok and init-201114.xz..ok to make sure it actually uses the new init file.
Schedule a deploy task for a machine with a smaller size disk and deploy to it. Again make sure it says init-201114.xz..ok when booting.
Please post here if you see issues as well as having success. Let us know what OS you have installed (Win 10 2004? Linux?) and the partition layout (contents of /images/NEWIMAGENAME/d1.partitions)!

References:
https://forums.fogproject.org/topic/14691/error-trying-to-restore-gpt-partition-when-deploying-image-to-smaller-drive-error-return-code-4
https://forums.fogproject.org/topic/13220/error-trying-to-restore-gpt-partition-deploying-an-image-to-smaller-disk
https://forums.fogproject.org/topic/14684/windows-8-1-corrupted-bcd
https://forums.fogproject.org/topic/15158/fog-failed-partition-issue
https://forums.fogproject.org/topic/15163/partitions-is-too-big-for-the-disk

Issue fixed later on:
https://forums.fogproject.org/topic/15363/issue-with-single-disk-resizable

Sebastian Roth · Dec 22, 2020, 7:00 AM

@testers Anyone of you still active in the forums?

JJ Fullmer · Jan 7, 2021, 3:31 PM

@sebastian-roth I hope to be building our 20H2 image the start of next year. So I can do some testing along with that for sure on this then.

JJ Fullmer · Jan 7, 2021, 3:31 PM

@Sebastian-Roth I just re-read through this as I’m starting work on our 20H2 image.
We actually purposely make our base image from a smaller disk already of around 64 GB and it usually shrinks down to around 20 GB when captured. Then we deploy to things with larger drives.

So just to be clear, I need to create the image from a new VM or physical machine that has a larger drive? i.e. expand the drive to something like 256 or 512 and then capture (where it will resize down to the usual 20ish GB or probably less for this test cause I’ll just install windows and that’s it) and then deploy to something with a 64 or 128 sized drive where it will resize.

Why are people creating images from things with drives larger than what they are deploying to?

Or is this testing a different type of image?

Sebastian Roth · Jan 7, 2021, 4:39 PM

@jj-fullmer said in Move partition in GPT layout - need people to test:

So just to be clear, I need to create the image from a new VM or physical machine that has a larger drive? i.e. expand the drive to something like 256 or 512 and then capture (where it will resize down to the usual 20ish GB or probably less for this test cause I’ll just install windows and that’s it) and then deploy to something with a 64 or 128 sized drive where it will resize.

That sounds like a reasonable test to me!

Why are people creating images from things with drives larger than what they are deploying to?

Probably because they use some hardware machine they have as master but don’t think about the disk size making a difference.

JJ Fullmer · Jan 7, 2021, 2:30 PM

@sebastian-roth I’m doing tests using just vms, so it’s possible that physical hardware could respond differently.
Here are my results.

I created an image successfully from a 128 GB VM, it was 10 GB in size after being resized (all I did was enter audit mode after installing windows and then captured)

I deployed it to a VM with a 60GB drive and had no issues.

I did confirm that the new init was used at boot for both. There was a bunch of extra output that happened to fast to catch on the capture before the gparted screens showed up, but everything worked fine.

Here is my d1.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2C465DF5-10FD-4883-B323-423E3D10FB2E, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7AFE394E-47A0-43FE-A06E-7AFD7286E200, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=   267168294, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=4A0FF43A-6E31-4AC9-8446-073E07C63AA8, name="Basic data partition"
/dev/nvme0n1p4 : start=   267409408, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=AEF97419-7B7D-4EC5-8C19-FF8C0217A965, attrs="RequiredPartition GUID:63"

Just to be safe, I will also test this in reverse with the init, Going to capture the 60 GB image and deploy it to the 128 GB VM, especially since that’s what I usually do.

Also, just for reference, here is the d1.partitions from our 1909 image. It looks like windows started putting the data partition at partition 3 instead of 4, kinda weird, that would make expanding disks for the main data drive more difficult, I wonder what the reasoning for that is.

[root@arrowfog Base-Stable]# cat d1.partitions
label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 134217694
sector-size: 512

/dev/sda1 : start=        2048, size=     1083392, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=75A55945-9352-4F30-A330-67878EC28FA1, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/sda2 : start=     1085440, size=      202752, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=D1336408-C5B3-446F-BE78-725587F7224D, name="EFI system partition", attrs="GUID:63"
/dev/sda3 : start=     1288192, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=C9F8E03C-928A-4233-9837-49CE510E39B0, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda4 : start=     1320960, size=   132894720, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=D7729B0D-A7B4-4E1F-A9F5-F06FAAAE6141, name="Basic data partition"

Sebastian Roth · Jan 7, 2021, 3:06 PM

@jj-fullmer Thanks for testing! Looking fine so far I would say.

it was 10 GB in size after being resized

You mean the image size shown in the FOG web UI or when you booted it up to Windows again? The later would really be an issue. It should not stay shrinked.

Here is my d1.partitions

Could you post the contents of d1.minimum.partitions as well?

JJ Fullmer · Jan 7, 2021, 10:13 PM

@sebastian-roth I did not see this before I overwrote the test image, so I don’t have the d1.minimum.partitions, but I still can for the 60 GB image if you want that just to see what the init wrote. I can re-capture the larger vm after I finish the test for smaller to larger and get that for you, I’ll just post it tomorrow.

The 10 GB was the fog gui, everything expanded as it should in windows.

JJ Fullmer · Jan 8, 2021, 9:00 AM

@sebastian-roth

The smaller to larger worked fine as well.
here are all the d1.partition and d1.minimum.partitions I have (will update this post tomorrow with the recaptured first image) so it’s all in one spot

1909 image

for reference of old windows partition table
This was NOT created with the new init, included this for comparison.

d1.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 134217694
sector-size: 512

/dev/sda1 : start=        2048, size=     1083392, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=75A55945-9352-4F30-A330-67878EC28FA1, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/sda2 : start=     1085440, size=      202752, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=D1336408-C5B3-446F-BE78-725587F7224D, name="EFI system partition", attrs="GUID:63"
/dev/sda3 : start=     1288192, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=C9F8E03C-928A-4233-9837-49CE510E39B0, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda4 : start=     1320960, size=   132894720, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=D7729B0D-A7B4-4E1F-A9F5-F06FAAAE6141, name="Basic data partition"

d1.minimum.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 134217694
sector-size: 512

/dev/sda1 : start=        2048, size=     1083392, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=75A55945-9352-4F30-A330-67878EC28FA1, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/sda2 : start=     1085440, size=      202752, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=D1336408-C5B3-446F-BE78-725587F7224D, name="EFI system partition", attrs="GUID:63"
/dev/sda3 : start=     1288192, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=C9F8E03C-928A-4233-9837-49CE510E39B0, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda4 : start=     1320960, size=    45462454, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=D7729B0D-A7B4-4E1F-A9F5-F06FAAAE6141, name="Basic data partition"

20H2 smaller image

partitions of image from 60 GB VM

d1.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 125829086
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2C465DF5-10FD-4883-B323-423E3D10FB2E, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7AFE394E-47A0-43FE-A06E-7AFD7286E200, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=   124567040, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=4A0FF43A-6E31-4AC9-8446-073E07C63AA8, name="Basic data partition"
/dev/nvme0n1p4 : start=   124806656, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=AEF97419-7B7D-4EC5-8C19-FF8C0217A965, attrs="RequiredPartition GUID:63"

d1.minimum.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 125829086
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2C465DF5-10FD-4883-B323-423E3D10FB2E, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7AFE394E-47A0-43FE-A06E-7AFD7286E200, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=    21331078, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=4A0FF43A-6E31-4AC9-8446-073E07C63AA8, name="Basic data partition"
/dev/nvme0n1p4 : start=    21571584, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=AEF97419-7B7D-4EC5-8C19-FF8C0217A965, attrs="RequiredPartition GUID:63"

20H2 Larger image

Partitions of image from 128 GB VM

d1.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2C465DF5-10FD-4883-B323-423E3D10FB2E, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7AFE394E-47A0-43FE-A06E-7AFD7286E200, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=   267173376, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=4A0FF43A-6E31-4AC9-8446-073E07C63AA8, name="Basic data partition"
/dev/nvme0n1p4 : start=   267412992, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=AEF97419-7B7D-4EC5-8C19-FF8C0217A965, attrs="RequiredPartition GUID:63"

d1.minimum.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2C465DF5-10FD-4883-B323-423E3D10FB2E, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7AFE394E-47A0-43FE-A06E-7AFD7286E200, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=    21351120, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=4A0FF43A-6E31-4AC9-8446-073E07C63AA8, name="Basic data partition"
/dev/nvme0n1p4 : start=    21592064, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=AEF97419-7B7D-4EC5-8C19-FF8C0217A965, attrs="RequiredPartition GUID:63"

JJ Fullmer · Jan 8, 2021, 2:53 PM

@sebastian-roth Do we also need to do some testing with this init on older versions of windows to be sure it doesn’t break anything there?
And Other OS’s too? Like maybe at least 1 linux distro

Sebastian Roth · Jan 8, 2021, 4:00 PM

@JJ-Fullmer Thanks for posting the other details as well. Looking all fine so far. We definitely want to do further tests.

Could you do one with Windows 7? Be sure to install in UEFI mode to get a GPT partition layout.

I can look into testing a Linux distro.

JJ Fullmer · Jan 8, 2021, 6:48 PM

@sebastian-roth I’m working on a windows 8.1 w/update image (since windows 7 isn’t supported anymore)
Will let you know the results

JJ Fullmer · Jan 8, 2021, 8:58 PM

@sebastian-roth

Windows 8.1

Image captured and deployed with new init
Did deploy test from larger 128 GB source VM and it had no problem deploying to 60 GB VM

d1.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      614400, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=6D92B4A9-E0B4-4DED-8C4C-E271AA36B06F, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=      616448, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2410B705-A7CB-4CE5-A393-8B3919C909D1, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      821248, size=      262144, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=58F556FE-7D3B-44EA-A846-C0848DB743CB, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1083392, size=   267350016, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=364F4B4F-4D49-4955-A5EB-9CDE151EEBF7, name="Basic data partition"

d1.minimum.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      614400, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=6D92B4A9-E0B4-4DED-8C4C-E271AA36B06F, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=      616448, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2410B705-A7CB-4CE5-A393-8B3919C909D1, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      821248, size=      262144, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=58F556FE-7D3B-44EA-A846-C0848DB743CB, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1083392, size=    17806622, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=364F4B4F-4D49-4955-A5EB-9CDE151EEBF7, name="Basic data partition"

btoffolon · Jan 15, 2021, 9:27 AM

Hi,

i try to capture an 80Gb disk with the last version of Windows 10 but when the capture begin i have the following error :

https://cloud.fairy-family.fr/index.php/s/skJP4Tkrd2WSoxp

did i forget something ?

Tom Elliott · Jan 15, 2021, 1:59 PM

@btoffolon What version of FOG are you using? The init error your seeing would seem to indicate you need to change the init ramdisk size from 127000 to 275000. This can be done from FOG Configuration Page -> FOG Settings.

If this is already set to 275000 then it would seem the full init wasn’t downloaded.

btoffolon · Jan 15, 2021, 2:48 PM

@tom-elliott I use fog 1.5.9, the ramdisk size it’s at 275000 i will try to download again the init file and try again

thanks for the answer

btoffolon · Jan 15, 2021, 3:51 PM

@tom-elliott i download again the file but i have the same issue

Sebastian Roth · Jan 15, 2021, 4:52 PM

@btoffolon That is kinda strange. I don’t think I have seen this error message before but it sounds like the init is corrupt or not fully loaded when booting. Though it doesn’t make sense to me as the original FOG 1.5.9 init works for you and other people tested this alpha stage init recently as well.

Please run those commands and post output here:

ls -al /var/www/html/fog/service/ipxe/init-201114.xz
ls -al /var/www/fog/service/ipxe/init-201114.xz
md5sum /var/www/html/fog/service/ipxe/init-201114.xz
md5sum /var/www/fog/service/ipxe/init-201114.xz

JJ Fullmer · Jan 15, 2021, 6:25 PM

@btoffolon Does this machine image fine with the original init?
You can also try a different bzImage. I’m using 5.618RT3 made by @george1421 found here https://forums.fogproject.org/topic/15017/lenovo-l13-yoga-maybe-the-usb-cable-is-bad-error-when-trying-to-register/18?_=1610731427661

I may have forgotten that I was using a different bzImage kernel when I started testing for this…

btoffolon · Jan 18, 2021, 1:08 PM

@sebastian-roth @JJ-Fullmer Thanks for your help, for unknonwn reason the file was deleted well i put it a third time and its work perfectly, i capture a 500Gb HDD disk and deploy in 120Gb SSD.

d1.partitions

fog@fog:/images/Salle-Technologie$ cat d1.partitions
label: gpt
label-id: 79FE4766-EDD2-4DD0-826C-6DD816A2E7C6
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 976773134
sector-size: 512

/dev/sda1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=5217DE87-4A45-4AC4-A4DD-0C8EB1C5CF5B, name="EFI system partition", attrs="GUID:63"
/dev/sda2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=15C5E73F-7855-4DA4-A464-4694B0E56628, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda3 : start=      239616, size=   975486976, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=E130C384-3F46-4D8F-AD25-B85FDB0127CD, name="Basic data partition"
/dev/sda4 : start=   975726592, size=     1046528, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=D3FA37D6-10A7-4303-976C-6D8F23C4258F, name="attrs=\x22RequiredPartition GUID:63", attrs="RequiredPartition GUID:63"

d1.minimum.partitions

fog@fog:/images/Salle-Technologie$ cat d1.minimum.partitions
label: gpt
label-id: 79FE4766-EDD2-4DD0-826C-6DD816A2E7C6
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 976773134
sector-size: 512

/dev/sda1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=5217DE87-4A45-4AC4-A4DD-0C8EB1C5CF5B, name="EFI system partition", attrs="GUID:63"
/dev/sda2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=15C5E73F-7855-4DA4-A464-4694B0E56628, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda3 : start=      239616, size=    72336566, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=E130C384-3F46-4D8F-AD25-B85FDB0127CD, name="Basic data partition"
/dev/sda4 : start=    72577024, size=     1046528, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=D3FA37D6-10A7-4303-976C-6D8F23C4258F, name="attrs=\x22RequiredPartition GUID:63", attrs="RequiredPartition GUID:63"```

Move partitions on GPT layouts - need people to test

1909 image

d1.partitions

d1.minimum.partitions

20H2 smaller image

d1.partitions

d1.minimum.partitions

20H2 Larger image

d1.partitions

d1.minimum.partitions

Windows 8.1

d1.partitions

d1.minimum.partitions

242

12.0k

17.3k

155.2k