Surface Go 4 incompatible

[JJ Fullmer]

@JJ-Fullmer No difference with a different image.
This is the error message

Here is the ls -l of that image

 ls -l /images/Base-Stable/
total 11098480
-rwxrwxr-x 1 fogproject root           6 Oct 12 07:34 d1.fixed_size_partitions
-rwxrwxr-x 1 fogproject root     1048576 Oct 12 07:35 d1.mbr
-rwxrwxr-x 1 fogproject root         872 Oct 12 07:35 d1.minimum.partitions
-rwxrwxr-x 1 fogproject root          20 Oct 12 07:34 d1.original.fstypes
-rwxrwxr-x 1 fogproject root           0 Oct 12 07:34 d1.original.swapuuids
-rwxrwxr-x 1 fogproject root    10722570 Oct 12 07:35 d1p1.img
-rwxrwxr-x 1 fogproject root         754 Oct 12 07:35 d1p2.img
-rwxrwxr-x 1 fogproject root 11352884934 Oct 12 07:40 d1p3.img
-rwxrwxr-x 1 fogproject root      153330 Oct 12 07:40 d1p4.img
-rwxrwxr-x 1 fogproject root         872 Oct 12 07:34 d1.partitions
-rwxrwxr-x 1 fogproject root         872 Oct 12 07:35 d1.shrunken.partitions

And the contents of d1.partitions

cat /images/Base-Stable/d1.partitions
label: gpt
label-id: 745A6813-735A-4940-840D-28D0E805850A
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 83886046
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=CFB2D745-2975-480D-BD56-EA18B65E03C5, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=57426B89-9DEE-4BDF-9CBF-E9452D985FD5, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=    57417925, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=857B1626-9DC1-44FD-8120-2AB922BBBD85, name="Basic data partition"
/dev/nvme0n1p4 : start=    82761728, size=     1120256, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=DE086091-5C6A-4D63-89E2-579581B9BD63, attrs="RequiredPartition GUID:63"

Maybe it’s because it’s captured as /dev/nvmeXX instead of /dev/sdaXX ? Just brainstorming there.

Doing an update to the latest dev version just to be sure this isn’t fixed in a recent patch.

[JJ Fullmer]

Made a pull request for including the ufs in the kernel config
https://github.com/FOGProject/fos/pull/78

Probably should get it fully working before doing that pull though

@SEBASTIAN-ROTH any thoughts?

[rodluz]

@JJ-Fullmer I’m currently working on new kernels and inits. Having issues with building partclone for the inits, but I’ll hopefully have some time this weekend to get it done.
I will include the ufs changes for the kernels.

[JJ Fullmer]

So more findings

Did a test of a different image that is based on /dev/sda instead of /dev/nvmeXnX

That image deployed with the modified kernel just fine (minus booting to the OS, but that was a windows server image, which I didn’t expect to work on a surface go).

But wait there’s more!

As I started playing around in a debug session I found that the UFS disk wants to have a sector size of 4096. This is the dump after manually creating the partitions to match the d1.minimum.partitions of the image

 sfdisk --dump /dev/sda
label: gpt
label-id: 745A6813-735A-4940-840D-28D0E805850A
device: /dev/sda
unit: sectors
first-lba: 6
last-lba: 31246330
sector-size: 4096


/dev/sda1 : start=         256, size=       25600, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=CFB2D745-2975-480D-BD56-EA18B65E03C5, name="EFI system partition", attrs="GUID:63"
/dev/sda2 : start=       25856, size=        4096, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=57426B89-9DEE-4BDF-9CBF-E9452D985FD5, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda3 : start=       29952, size=     7340032, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=857B1626-9DC1-44FD-8120-2AB922BBBD85, name="Basic data partition"
/dev/sda4 : start=     7369984, size=      140032, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=DE086091-5C6A-4D63-89E2-579581B9BD63, attrs="RequiredPartition GUID:63"

Whereas the header of d1.minimum.partitions in my normal base image is 512, though it seems like it might actually be 2048 based on math of partitionSectors/sector-size = size in MB (but I might be wrong about that formula, it’s been a minute since I’ve had to deal with sectors).

This is the contents of my d1.minimum.partitions from the image I want to deploy

label: gpt
label-id: 745A6813-735A-4940-840D-28D0E805850A
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 83886046
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=CFB2D745-2975-480D-BD56-EA18B65E03C5, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=57426B89-9DEE-4BDF-9CBF-E9452D985FD5, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=    57417925, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=857B1626-9DC1-44FD-8120-2AB922BBBD85, name="Basic data partition"
/dev/nvme0n1p4 : start=    82761728, size=     1120256, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=DE086091-5C6A-4D63-89E2-579581B9BD63, attrs="RequiredPartition GUID:63"

I tried what @Sebastian-Roth mentions in another post here https://forums.fogproject.org/topic/15608/imaging-from-large-drive-to-small-drive/4?=1700488459744 and what @Tom-Elliott mentions here https://forums.fogproject.org/topic/13004/error-trying-to-restore-gpt-partition-tables-on-multiple-dell-machines/13?=1700488480736

When I try to restore from the original file I get an error saying the last-lba is invalid/too high for the disk.

What’s weird, is I can use that syntax to restore the server based image, and that min parts file also has a higher lba, but it makes a little sense as the final sector is below the drive’s “31246330” last lba.

cat ../SVR-Base-Stable/d1.minimum.partitions
label: gpt
label-id: 0B6865A1-3053-4F45-9371-B73F033C7395
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 188743646
sector-size: 512

/dev/sda1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=3DF3DEBB-C856-4CC2-9263-EAF0AABF2DDF, name="EFI system partition", attrs="GUID:63"
/dev/sda2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=C9738054-91F0-4B3A-AFFA-A9C9ABCC9663, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda3 : start=      239616, size=    27175378, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=42364778-3F9E-4421-88C1-698974428C14, name="Basic data partition", attrs="GUID:63"
/dev/sda4 : start=    27415040, size=     1269760, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=CA25B771-91EC-40A3-B200-1360371A0AD8, attrs="RequiredPartition GUID:63"

So I made a backup of my original image mbr and patition files and edited the d1.minimum.partitions to look like this

label: gpt
label-id: 745A6813-735A-4940-840D-28D0E805850A
device: /dev/nvme0n1
unit: sectors
first-lba: 6
last-lba: 31246330
sector-size: 4096

/dev/nvme0n1p1 : start=        256, size=      25600, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=CFB2D745-2975-480D-BD56-EA18B65E03C5, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      25856, size=       4096, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=57426B89-9DEE-4BDF-9CBF-E9452D985FD5, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      29952, size=    7340032, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=857B1626-9DC1-44FD-8120-2AB922BBBD85, name="Basic data partition"
/dev/nvme0n1p4 : start=    7369984, size=     140032, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=DE086091-5C6A-4D63-89E2-579581B9BD63, attrs="RequiredPartition GUID:63"

Making it match the dump after making the partitions by MB and GB size manually.

After editing it this command worked without error

sfdisk /dev/sda < /images/Base-Stable/d1.minimum.partitions

So I made a new version of the d1.mbr
sgdisk -b /mnt/d1.mbr /dev/sda

And popped that on the server to test deploying with that. ON this test, I’m not editing the d1.partitions or d1.shrunken.partitions which may make it so it doesn’t expand properly, but we’ll see how it goes…

It got past the previous errors and is now imaging! woo hoo!
How to fix this permanently though, that’s a different question.

[JJ Fullmer]

The image did complete but it did not boot into Windows. Looks like I need to embed the storage controller driver for UFS.

Or it could be a 512b vs 4k block disk type issue.

[JJ Fullmer]

@Sebastian-Roth or @Tom-Elliott any chance you have some time to help me out with this issue?
It’s looking like its a 4k block issue.
I was trying to find a formulaic way to change the partitions files to line up to the 4k layout, like dividing by 8 and some other tweaking as needed but found it more complicated than expected.

Any ideas?

[JJ Fullmer]

It’s looking like I have to find a way to either capture the image or a separate image from a device that defaults to 4k logical sectors or a way to force the device to change sector size to 512 when deploying a 512e based image

I tried hdparm but got this error

hdparm --set-sector-size 512 --please-destroy-my-drive /dev/sda

/dev/sda:
READ_LOG_EXT(SECTOR_CONFIGURATION) failed: No such file or directory

I also tried capturing the image using the new kernel on the off chance that would make a difference, but no good.

I guess I will have to try making an image from a 4k native device since it’s not being changed?

[JJ Fullmer]

@JJ-Fullmer I captured an image from the surface go 4 with the new kernel that has UFS support and I was able to deploy that image with its 4k block based partition table without issue.
Of course that image can now not be deployed on 512e block based disks.

If I ever get the time, I may try testing manipulation of the parition blocks and using an efi tool in a fog post install script to make it so an image can be deployed to either but its probably unlikely to be possible.

For reference, here’s what the 4k block based partitions look like

d1.partitions

label: gpt
label-id: DEECB42F-8F01-49B9-85BE-8398A932E4E5
device: /dev/sda
unit: sectors
first-lba: 6
last-lba: 31246330
sector-size: 4096

/dev/sda1 : start=         256, size=       76800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=FE5BA595-3498-4D4A-B741-9EB5D84C1EA5, name="EFI system partition", attrs="GUID:63"
/dev/sda2 : start=       77056, size=        4096, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=752C5B50-96B7-4130-9789-9A02B05F8A75, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda3 : start=       81152, size=    31024716, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=C088525B-56B9-4D7E-A040-1C8F5A782ADC, name="Basic data partition"
/dev/sda4 : start=    31106048, size=      139776, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=E2B6791E-DC55-4039-A658-9007B5749FEE, attrs="RequiredPartition GUID:63"

d1.minimum.partitions

label: gpt
label-id: DEECB42F-8F01-49B9-85BE-8398A932E4E5
device: /dev/sda
unit: sectors
first-lba: 6
last-lba: 31246330
sector-size: 4096

/dev/sda1 : start=         256, size=       76800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=FE5BA595-3498-4D4A-B741-9EB5D84C1EA5, name="EFI system partition", attrs="GUID:63"
/dev/sda2 : start=       77056, size=        4096, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=752C5B50-96B7-4130-9789-9A02B05F8A75, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda3 : start=       81152, size=    31024716, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=C088525B-56B9-4D7E-A040-1C8F5A782ADC, name="Basic data partition"
/dev/sda4 : start=    31106048, size=      139776, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=E2B6791E-DC55-4039-A658-9007B5749FEE, attrs="RequiredPartition GUID:63"

Something odd is it doesn’t look like the main OS partition (sda3) shrunk, but it for sure did as it is only 10.56 GB on the server (compressed size) and deploys around 20 GB on the client, same size as the VM based 512e image.

[JJ Fullmer]

@JJ-Fullmer I just took another look as I’ve been just maintaining a separate 4kn image and realized that the disk isn’t expanding after imaging.
I’ll do a debug session tomorrow and report back.

[JJ Fullmer]

Did a debug session and ntfsresize with -c to check and --info shows partition 3 as resizable but it’s not being resized after imaging.

Running the image in deploy and seeing what it says after imaging and will see if there are any errors.

I fear this is going to be a 4kn drive alignment/resize issue.

[JJ Fullmer]

Some info from the debug session

cat /images/4KDisk-Base-Dev/d1.original.fstypes
/dev/vda3 ntfs

[Fri Jan 26 root@fogclient ~]# cat /images/4KDisk-Base-Dev/d1.partitions
label: gpt
label-id: 9865AAFC-B984-4860-ACF5-4D6F2513747D
device: /dev/vda
unit: sectors
first-lba: 6
last-lba: 16777210
sector-size: 4096

/dev/vda1 : start=         256, size=       76800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=7C4743B5-7150-4672-B521-7B537528D7E7, name="EFI system partition", attrs="GUID:63"
/dev/vda2 : start=       77056, size=        4096, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=F6216A84-0172-4445-B616-E36DFA20C731, name="Microsoft reserved partition", attrs="GUID:63"
/dev/vda3 : start=       81152, size=    16501760, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=B6DA06DC-A5D0-434C-A6FE-494A1EFB515E, name="Basic data partition"
/dev/vda4 : start=    16582912, size=      193792, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=7BB90563-4BB7-4281-98EA-3FF4BCF1FCA5, attrs="RequiredPartition GUID:63"

[Fri Jan 26 root@fogclient ~]# cat /images/4KDisk-Base-Dev/d1.minimum.partitions
label: gpt
label-id: 9865AAFC-B984-4860-ACF5-4D6F2513747D
device: /dev/vda
unit: sectors
first-lba: 6
last-lba: 16777210
sector-size: 4096

/dev/vda1 : start=         256, size=       76800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=7C4743B5-7150-4672-B521-7B537528D7E7, name="EFI system partition", attrs="GUID:63"
/dev/vda2 : start=       77056, size=        4096, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=F6216A84-0172-4445-B616-E36DFA20C731, name="Microsoft reserved partition", attrs="GUID:63"
/dev/vda3 : start=       81152, size=    16501760, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=B6DA06DC-A5D0-434C-A6FE-494A1EFB515E, name="Basic data partition"
/dev/vda4 : start=    16582912, size=      193792, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=7BB90563-4BB7-4281-98EA-3FF4BCF1FCA5, attrs="RequiredPartition GUID:63"

[Fri Jan 26 root@fogclient ~]# cat /images/4KDisk-Base-Dev/d1.fixed_size_partitions
1:2:4

[Fri Jan 26 root@fogclient ~]# cat /images/4KDisk-Base-Dev/d1.
d1.fixed_size_partitions  d1.minimum.partitions     d1.original.swapuuids     d1.shrunken.partitions
d1.mbr                    d1.original.fstypes       d1.partitions

[Fri Jan 26 root@fogclient ~]# cat /images/4KDisk-Base-Dev/d1.shrunken.partitions
label: gpt
label-id: 9865AAFC-B984-4860-ACF5-4D6F2513747D
device: /dev/vda
unit: sectors
first-lba: 6
last-lba: 16777210
sector-size: 4096

/dev/vda1 : start=         256, size=       76800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=7C4743B5-7150-4672-B521-7B537528D7E7, name="EFI system partition", attrs="GUID:63"
/dev/vda2 : start=       77056, size=        4096, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=F6216A84-0172-4445-B616-E36DFA20C731, name="Microsoft reserved partition", attrs="GUID:63"
/dev/vda3 : start=       81152, size=    16501760, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=B6DA06DC-A5D0-434C-A6FE-494A1EFB515E, name="Basic data partition"
/dev/vda4 : start=    16582912, size=      193792, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=7BB90563-4BB7-4281-98EA-3FF4BCF1FCA5, attrs="RequiredPartition GUID:63"

gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 31246336 sectors, 119.2 GiB
Model: KLUDG4UHGC-B0E1
Sector size (logical/physical): 4096/4096 bytes
Disk identifier (GUID): 9865AAFC-B984-4860-ACF5-4D6F2513747D
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 5
First usable sector is 6, last usable sector is 31246330
Partitions will be aligned on 256-sector boundaries
Total free space is 14469877 sectors (55.2 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1             256           77055   300.0 MiB   EF00  EFI system partition
   2           77056           81151   16.0 MiB    0C01  Microsoft reserved ...
   3           81152        16582911   62.9 GiB    0700  Basic data partition
   4        16582912        16776703   757.0 MiB   2700

It’s a 128 GB drive, the image was a 64 GB drive, I expected it to expand to 128 GB

ntfsresize info on parts 4 and 3

 ntfsresize --info /dev/sda4
ntfsresize v2022.10.3 (libntfs-3g)
Device name        : /dev/sda4
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 793772032 bytes (794 MB)
Current device size: 793772032 bytes (794 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use       : 14 MB (1.7%)
Collecting resizing constraints ...
You might resize at 13193216 bytes or 14 MB (freeing 780 MB).
Please make a test run using both the -n and -s options before real resizing!

[Fri Jan 26 root@fogclient ~]# ntfsresize --info /dev/sda3
ntfsresize v2022.10.3 (libntfs-3g)
Device name        : /dev/sda3
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 67591208960 bytes (67592 MB)
Current device size: 67591208960 bytes (67592 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use       : 24382 MB (36.1%)
Collecting resizing constraints ...
You might resize at 24381546496 bytes or 24382 MB (freeing 43210 MB).
Please make a test run using both the -n and -s options before real resizing!

[JJ Fullmer]

In the end I am maintaining a separate image, I was able to get management to let us buy a separate surface go 4 for maintaining the 4k disk image.
I found that bhyve based VMs can be set to 4k blocks but it was cumbersome to get it to boot to fog to capture the image at the end. And when that image was deployed, it did not expand on the surface go.