Move partitions on GPT layouts - need people to test

  • Moderator

    UPDATE: This is now part the official inits!!

    After several people noticed that Windows 10 2004 comes with a different partition layout that FOG is not able to shrink and deploy to a smaller size disk I started working on that code and adding the functionality to actually move the start sector of partitions. Up to now FOG never moved partitions because it would surely cause trouble as manipulating boot loader code in the MBR would also be needed and is way too error prone to do. With GPT partition layout on UEFI based systems partition start positions should not be “hardcoded” in bootloader code anymore.

    Though this seems to work I would really like more people to test this before we actually add this to the official release!

    Here are the steps I recommend when testing:

    1. Make sure you have a working backup copy of your host used to capture the image from! Just in case something goes wrong when capturing with the new init.
    2. Download the init file and put that in /var/www/html/fog/service/ipxe/ on your FOG server.
    3. Create a new image definition for this test.
    4. Decide which hosts you use for testing capture and deploy (the later having a smaller size disk to see if it works) with this new init, edit their hosts settings in the FOG web UI and set Host Init to init-201114.xz as well as the image definition you just created.
    5. Schedule a capture task and pay attention to the boot process where it says bzImage..ok and init-201114.xz..ok to make sure it actually uses the new init file.
    6. Schedule a deploy task for a machine with a smaller size disk and deploy to it. Again make sure it says init-201114.xz..ok when booting.
    7. Please post here if you see issues as well as having success. Let us know what OS you have installed (Win 10 2004? Linux?) and the partition layout (contents of /images/NEWIMAGENAME/d1.partitions)!

    References:
    https://forums.fogproject.org/topic/14691/error-trying-to-restore-gpt-partition-when-deploying-image-to-smaller-drive-error-return-code-4
    https://forums.fogproject.org/topic/13220/error-trying-to-restore-gpt-partition-deploying-an-image-to-smaller-disk
    https://forums.fogproject.org/topic/14684/windows-8-1-corrupted-bcd
    https://forums.fogproject.org/topic/15158/fog-failed-partition-issue
    https://forums.fogproject.org/topic/15163/partitions-is-too-big-for-the-disk

    Issue fixed later on:
    https://forums.fogproject.org/topic/15363/issue-with-single-disk-resizable


  • @sebastian-roth it’s working thank you !!

  • Moderator

    @btoffolon You are pretty lucky as I am sure I have fixed that issue already in the latest test init. Please download from github, put that in /var/www/html/fog/service/ipxe/, rename the original init.xz, put the new file in place.

    Then you need to re-capture the image. Deploy should work then.


  • Hi,
    i have a bug when i tried to deploy. i don’t know if it’s link to the previous bug.
    Here the vidéo with the error.

    and the partition files :

    d1.partitions

    root@fog:/images/SalleTechnologieTest# cat d1.partitions
    label: gpt
    label-id: 79FE4766-EDD2-4DD0-826C-6DD816A2E7C6
    device: /dev/sda
    unit: sectors
    first-lba: 34
    last-lba: 500118158
    sector-size: 512
    
    /dev/sda1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=5217DE87-4A45-4AC4-A4DD-0C8EB1C5CF5B, name="EFI system partition", attrs="GUID:63"
    /dev/sda2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=15C5E73F-7855-4DA4-A464-4694B0E56628, name="Microsoft reserved partition", attrs="GUID:63"
    /dev/sda3 : start=      239616, size=   498831872, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=E130C384-3F46-4D8F-AD25-B85FDB0127CD, name="Basic data partition"
    /dev/sda4 : start=   499071488, size=     1046528, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=D3FA37D6-10A7-4303-976C-6D8F23C4258F, name="attrs=\x22RequiredPartition GUID:63", attrs="RequiredPartition GUID:63"
    

    d1.minimum.partitions

    root@fog:/images/SalleTechnologieTest# cat d1.minimum.partitions
    label: gpt
    label-id: 79FE4766-EDD2-4DD0-826C-6DD816A2E7C6
    device: /dev/sda
    unit: sectors
    first-lba: 34
    last-lba: 500118158
    sector-size: 512
    
    /dev/sda1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=5217DE87-4A45-4AC4-A4DD-0C8EB1C5CF5B, name="EFI system partition", attrs="GUID:63"
    /dev/sda2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=15C5E73F-7855-4DA4-A464-4694B0E56628, name="Microsoft reserved partition", attrs="GUID:63"
    /dev/sda3 : start=      239616, size=    85029512, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=E130C384-3F46-4D8F-AD25-B85FDB0127CD, name="Basic data partition"
    /dev/sda4 : start=   499071488, size=     1046528, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=D3FA37D6-10A7-4303-976C-6D8F23C4258F, name="attrs=\x22RequiredPartition GUID:63", attrs="RequiredPartition GUID:63"
    

    Can you help me ?

    Thank you


  • @sebastian-roth
    I tried this solution with Windows 21H1 the image capture and deploy works without any problems from a 1 TB NVMe SSD to the 500 GB SATA SSD.

    d1.minimum.partitions

    cat d1.minimum.partitions
    label: gpt
    label-id: 32D055B8-40FD-4110-9551-5794A08735E8
    device: /dev/nvme0n1
    unit: sectors
    first-lba: 34
    last-lba: 2000409230
    sector-size: 512
    
    /dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=28F8CABB-959B-4C15-986E-0F8640E4F0C7, name="EFI system partition", attrs="GUID:63"
    /dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=ED9F8F39-E265-428D-8BE1-C2B7E4304D52, name="Microsoft reserved partition", attrs="GUID:63"
    /dev/nvme0n1p3 : start=      239616, size=   291316922, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=05966690-06CD-4A69-BF88-376CC3EBFAC0, name="Basic data partition"
    /dev/nvme0n1p4 : start=   291556864, size=     5639168, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=49569B35-41CE-42F5-82CB-6F68BA36C239, attrs="RequiredPartition GUID:63"
    

    d1.partitions

    cat d1.partitions
    label: gpt
    label-id: 32D055B8-40FD-4110-9551-5794A08735E8
    device: /dev/nvme0n1
    unit: sectors
    first-lba: 34
    last-lba: 2000409230
    sector-size: 512
    
    /dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=28F8CABB-959B-4C15-986E-0F8640E4F0C7, name="EFI system partition", attrs="GUID:63"
    /dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=ED9F8F39-E265-428D-8BE1-C2B7E4304D52, name="Microsoft reserved partition", attrs="GUID:63"
    /dev/nvme0n1p3 : start=      239616, size=  1994530304, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=05966690-06CD-4A69-BF88-376CC3EBFAC0, name="Basic data partition"
    /dev/nvme0n1p4 : start=  1994769920, size=     5639168, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=49569B35-41CE-42F5-82CB-6F68BA36C239, attrs="RequiredPartition GUID:63"
    
  • Moderator

    @fry_p said in Move partitions on GPT layouts - need people to test:

    Just a quick question, is there a place where I can get this init without moving to the dev branch?

    Yes, you can manually download those: 64 bit and 32 bit

  • Moderator

    @sebastian-roth Just a quick question, is there a place where I can get this init without moving to the dev branch? I am very compelled to stay on the stable branch at this time. Are the inits on fogproject.org this version, or the old stable ones?


  • @sebastian-roth ok thank you, that explains everything.

    Most probably i’m experiencing the same problem of this topic
    https://forums.fogproject.org/topic/15455/unable-to-restore-gpt-partition-tables-on-one-image
    Since is the same laptop, same gen, same 5 partitions layout, Latest fog.
    I’ll continue there then.

    Thank you again, for everything

  • Moderator

    @clopezbelenguer said in Move partitions on GPT layouts - need people to test:

    I upgraded to the latest dev-branch and tried to capture and check d1.s hoping that the latest init downloaded by the installer where the working ones but aparently they aren’t or I’m doing something wrong.

    I removed the download link for init-201114 as the latest dev-branch inits actually have this feature included. Don’t think you’ve done something wrong here. Might just be a slightly different issue you are facing. Not sure though, need more information. I think it’s best if you open a new topic and post the contents of your d1.minimum.partitions and d1.partitions text files that you find in the images directory.

    In case they are updated in my instalation, Is it completely necessary to create a new image or can I capture and deploy again on a existing one??

    There is no need to create a new image definition in the web UI. That was just meant for the early days when people test this new feature in case there was a problem so they still had their old image at hand. BUT you need to re-capture the image because most of the magic happens on capture!


  • Hi,

    Sorry if i’m missing something but I’m experiencing the very one problem of having partitions at the end of the disk that make the image too big for smaller disk and i’m willing to try the beta init-201114 but I’m not entirely sure where can I get it.

    I upgraded to the latest dev-branch and tried to capture and check d1.s hoping that the latest init downloaded by the installer where the working ones but aparently they aren’t or I’m doing something wrong.
    In case they are updated in my instalation, Is it completely necessary to create a new image or can I capture and deploy again on a existing one??

    Thank you in advance and sorry to revive this old thread.


  • @sebastian-roth
    Yes Youare right.
    I focused on the "corrupted size vs. and did not see previous message.
    A scan disk on the master and it works.

    Thanks.

  • Moderator

    @lebrun78 said in Move partitions on GPT layouts - need people to test:

    Tried with your proposed modifications on a 1.5.9.63 testing server.
    I have got this error with an existing image on the server

    I don’t understand what you mean with this. In the picture we see a capture what is most probably failing due to an unclean NTFS filesystem (hard to read the text at the top of the picture but looks like it). I would not expect the init-201114.xz to cause this problem.

    Remove the image files on the server, it worked.

    ??


  • @sebastian-roth
    Hello Sebastian,
    Tried with your proposed modifications on a 1.5.9.63 testing server.
    I have got this error with an existing image on the server
    1616756682935.jpg

    Remove the image files on the server, it worked.

    Arnaud


  • @sebastian-roth Hi,

    The deploy seems to work. We will continue to deploy with fog and keep you inform of our bugs thanks for your help !

  • Moderator

    @btoffolon said in Move partitions on GPT layouts - need people to test:

    The capture worked, i capture from 240Gb SSD. I want to deploy it on 120Gb SSD do you want me to try it ?

    Thanks for posting the video. Definitely helpful to debug this. As far as I can tell from the video it was able to move sda4 forward. So I would expect d1.minimum.partitions to look different this time and it should deploy to a smaller size disk. So give it a try.

    If deploy fails then post the contents of d1.minimum.partitions here again.


  • Hi,

    @sebastian-roth I take a video. Like that you have everything.
    https://youtu.be/KzmZGP9T_To

    You can see in this picture that the init is lauching well :
    https://cloud.fairy-family.fr/index.php/s/DwCadbqr3SgKSCf

    The capture worked, i capture from 240Gb SSD. I want to deploy it on 120Gb SSD do you want me to try it ?

    Thanks for your help !

  • Moderator

    @btoffolon As well make sure it actually boots into init-201114.xz on PXE startup. When it loads the kernel and init you should see init-201114.xz...ok!

  • Moderator

    @btoffolon said in Move partitions on GPT layouts - need people to test:

    /dev/sda3 : start=      239616, size=    80978882, ...
    /dev/sda4 : start=   249022976, size=     1046528, ...
    

    Looks like FOS was not able to move the 4th partition forward as the start sector number is the same in both files.

    Create a new image definition for testing. Then assign this new image to the host you capture from and schedule a debug capture task. Boot it up and hit ENTER twice to get to the command shell. Now run the simple command fog to start capturing and step through the process. At some point you see it print the partition tables on screen. Take a picture of all that and post here. If unsure what to take a photo of, take some more, upload all pictures to a file share and post a link here.


  • Hi, i have a new issue when i deploy W10 2004 on a smaller SSD. i put an error screen and break the disk partitions. I can’t format it on windows (i don’t try on linux) there is now a protected GPT partition. Did i do something wrong ?

    Thanks for reading.

    This is the d1.partition :

    root@fog:/images# cat Salle-Technologie/d1.partitions 
    label: gpt
    label-id: 79FE4766-EDD2-4DD0-826C-6DD816A2E7C6
    device: /dev/sda
    unit: sectors
    first-lba: 34
    last-lba: 250069646
    sector-size: 512
    
    /dev/sda1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=5217DE87-4A45-4AC4-A4DD-0C8EB1C5CF5B, name="EFI system partition", attrs="GUID:63"
    /dev/sda2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=15C5E73F-7855-4DA4-A464-4694B0E56628, name="Microsoft reserved partition", attrs="GUID:63"
    /dev/sda3 : start=      239616, size=    80984324, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=E130C384-3F46-4D8F-AD25-B85FDB0127CD, name="Basic data partition"
    /dev/sda4 : start=   249022976, size=     1046528, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=D3FA37D6-10A7-4303-976C-6D8F23C4258F, name="attrs=\x22RequiredPartition GUID:63", attrs="RequiredPartition GUID:63"
    
    

    This is the d1.minimum.partition

    root@fog:/images# cat Salle-Technologie/d1.minimum.partitions 
    label: gpt
    label-id: 79FE4766-EDD2-4DD0-826C-6DD816A2E7C6
    device: /dev/sda
    unit: sectors
    first-lba: 34
    last-lba: 250069646
    sector-size: 512
    
    /dev/sda1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=5217DE87-4A45-4AC4-A4DD-0C8EB1C5CF5B, name="EFI system partition", attrs="GUID:63"
    /dev/sda2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=15C5E73F-7855-4DA4-A464-4694B0E56628, name="Microsoft reserved partition", attrs="GUID:63"
    /dev/sda3 : start=      239616, size=    80978882, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=E130C384-3F46-4D8F-AD25-B85FDB0127CD, name="Basic data partition"
    /dev/sda4 : start=   249022976, size=     1046528, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=D3FA37D6-10A7-4303-976C-6D8F23C4258F, name="attrs=\x22RequiredPartition GUID:63", attrs="RequiredPartition GUID:63"
    

    Files :

    ls -al /var/www/html/fog/service/ipxe/init-201114.xz 
    -rw-r--r-- 1 fogproject www-data 21180716 janv. 18 09:12 /var/www/html/fog/service/ipxe/init-201114.xz
    
    root@fog:/var/www/html/fog/service/ipxe# ls -al /var/www/fog/service/ipxe/init-201114.xz 
    -rw-r--r-- 1 fog fog 21180716 janv. 15 16:49 /var/www/fog/service/ipxe/init-201114.xz
    

    md5sum :

    root@fog:/var/www/html/fog/service/ipxe# md5sum /var/www/html/fog/service/ipxe/init-201114.xz 
    04b6d3ed8269f493c1552e4c28f2d708  /var/www/html/fog/service/ipxe/init-201114.xz
    
    root@fog:/var/www/html/fog/service/ipxe# md5sum /var/www/fog/service/ipxe/init-201114.xz 
    04b6d3ed8269f493c1552e4c28f2d708  /var/www/fog/service/ipxe/init-201114.xz
    
    
  • Moderator

    We have more evidence this is working even with non-standard partition layouts: https://forums.fogproject.org/topic/15213/clone-500gb-hdd-to-256gb-ssd

277
Online

8.9k
Users

15.6k
Topics

145.0k
Posts