So more findings

Did a test of a different image that is based on /dev/sda instead of /dev/nvmeXnX

That image deployed with the modified kernel just fine (minus booting to the OS, but that was a windows server image, which I didn’t expect to work on a surface go).

But wait there’s more!

As I started playing around in a debug session I found that the UFS disk wants to have a sector size of 4096. This is the dump after manually creating the partitions to match the d1.minimum.partitions of the image

 sfdisk --dump /dev/sda
label: gpt
label-id: 745A6813-735A-4940-840D-28D0E805850A
device: /dev/sda
unit: sectors
first-lba: 6
last-lba: 31246330
sector-size: 4096


/dev/sda1 : start=         256, size=       25600, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=CFB2D745-2975-480D-BD56-EA18B65E03C5, name="EFI system partition", attrs="GUID:63"
/dev/sda2 : start=       25856, size=        4096, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=57426B89-9DEE-4BDF-9CBF-E9452D985FD5, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda3 : start=       29952, size=     7340032, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=857B1626-9DC1-44FD-8120-2AB922BBBD85, name="Basic data partition"
/dev/sda4 : start=     7369984, size=      140032, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=DE086091-5C6A-4D63-89E2-579581B9BD63, attrs="RequiredPartition GUID:63"

Whereas the header of d1.minimum.partitions in my normal base image is 512, though it seems like it might actually be 2048 based on math of partitionSectors/sector-size = size in MB (but I might be wrong about that formula, it’s been a minute since I’ve had to deal with sectors).

This is the contents of my d1.minimum.partitions from the image I want to deploy

label: gpt
label-id: 745A6813-735A-4940-840D-28D0E805850A
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 83886046
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=CFB2D745-2975-480D-BD56-EA18B65E03C5, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=57426B89-9DEE-4BDF-9CBF-E9452D985FD5, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=    57417925, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=857B1626-9DC1-44FD-8120-2AB922BBBD85, name="Basic data partition"
/dev/nvme0n1p4 : start=    82761728, size=     1120256, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=DE086091-5C6A-4D63-89E2-579581B9BD63, attrs="RequiredPartition GUID:63"

I tried what @Sebastian-Roth mentions in another post here https://forums.fogproject.org/topic/15608/imaging-from-large-drive-to-small-drive/4?=1700488459744 and what @Tom-Elliott mentions here https://forums.fogproject.org/topic/13004/error-trying-to-restore-gpt-partition-tables-on-multiple-dell-machines/13?=1700488480736

When I try to restore from the original file I get an error saying the last-lba is invalid/too high for the disk.

What’s weird, is I can use that syntax to restore the server based image, and that min parts file also has a higher lba, but it makes a little sense as the final sector is below the drive’s “31246330” last lba.

cat ../SVR-Base-Stable/d1.minimum.partitions
label: gpt
label-id: 0B6865A1-3053-4F45-9371-B73F033C7395
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 188743646
sector-size: 512

/dev/sda1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=3DF3DEBB-C856-4CC2-9263-EAF0AABF2DDF, name="EFI system partition", attrs="GUID:63"
/dev/sda2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=C9738054-91F0-4B3A-AFFA-A9C9ABCC9663, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda3 : start=      239616, size=    27175378, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=42364778-3F9E-4421-88C1-698974428C14, name="Basic data partition", attrs="GUID:63"
/dev/sda4 : start=    27415040, size=     1269760, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=CA25B771-91EC-40A3-B200-1360371A0AD8, attrs="RequiredPartition GUID:63"

So I made a backup of my original image mbr and patition files and edited the d1.minimum.partitions to look like this

label: gpt
label-id: 745A6813-735A-4940-840D-28D0E805850A
device: /dev/nvme0n1
unit: sectors
first-lba: 6
last-lba: 31246330
sector-size: 4096

/dev/nvme0n1p1 : start=        256, size=      25600, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=CFB2D745-2975-480D-BD56-EA18B65E03C5, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      25856, size=       4096, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=57426B89-9DEE-4BDF-9CBF-E9452D985FD5, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      29952, size=    7340032, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=857B1626-9DC1-44FD-8120-2AB922BBBD85, name="Basic data partition"
/dev/nvme0n1p4 : start=    7369984, size=     140032, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=DE086091-5C6A-4D63-89E2-579581B9BD63, attrs="RequiredPartition GUID:63"

Making it match the dump after making the partitions by MB and GB size manually.

After editing it this command worked without error

sfdisk /dev/sda < /images/Base-Stable/d1.minimum.partitions

So I made a new version of the d1.mbr
sgdisk -b /mnt/d1.mbr /dev/sda

And popped that on the server to test deploying with that. ON this test, I’m not editing the d1.partitions or d1.shrunken.partitions which may make it so it doesn’t expand properly, but we’ll see how it goes…

It got past the previous errors and is now imaging! woo hoo!
How to fix this permanently though, that’s a different question.

@matthewadams12 Why do you need the Fog Server to be offline?
Maybe there’s a solution where we can isolate just the part you want to isolate. i.e. if you are trying to remove internet access where images are stored you could setup a storage node and keep that offline after initial install. The storage node would also need packages and updates to fog, but you could capture them during updates on the main server and then sneakernet them to the storage node.

Storage nodes are just what I’m using as an example. If it’s not the images stored on the server you’re trying to isolate, then there could still be another solution. You could also firewall the server with your external firewall or the os firewall to only allow access to the fog github page for update downloads and then the repo sites that download the package updates?

Just thought I’d throw out some other possibilities.

@ksrvpvl said in ability to load a menu item on a specific machine as an immediate task:

@baovipboy156 I don’t understand where is this plugin? is there documentation?

There is documentation! https://docs.fogproject.org/en/latest/tasks
The documentation is still under construction a bit, but task info can be found there

Another minor bugfix release, I left a random ‘a’ at the end of a line.
Also added a switch to set-fogsnapins for running the repair-fogsnapinassociations before attempting to add new ones

Release notes: https://github.com/darksidemilk/FogApi/releases/tag/2311.6.4
PSGallery: https://www.powershellgallery.com/packages/FogApi/2311.6.4

@cul3r0

I have such a redirect

<VirtualHost *:80>
    <FilesMatch "\.php$">
        SetHandler "proxy:fcgi://127.0.0.1:9000/"
    </FilesMatch>
    KeepAlive Off
    ServerName ip.of.fog.server
    ServerAlias hostnameOfFogServer
    DocumentRoot /var/www/html/
    <Directory /var/www/html/fog/>
        DirectoryIndex index.php index.html index.htm
    </Directory>
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]
    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-d
    RewriteRule ^/fog/(.*)$ /fog/api/index.php [QSA,L]
</VirtualHost>

This is assuming you have your cert configured in ssl.conf too.
I’m running centos with httpd as that might make a difference if you’re running debian based apache

@processor Are we good to mark this as solved then?

Made a pull request for including the ufs in the kernel config
https://github.com/FOGProject/fos/pull/78

Probably should get it fully working before doing that pull though

@SEBASTIAN-ROTH any thoughts?

@lan The latest “public” version of fog is 1.5.10
The dev-branch version install is described in the latter sections of this doc
https://docs.fogproject.org/en/latest/install-fog-server

@JeremyBrown Are you still having this issue?
The fog log is saying that the host has multiple macs

If it’s still happening I will gladly help you use the fogApi powershell module to find the duplicate macs and how to fix it from there.

@JJ-Fullmer No difference with a different image.
This is the error message

Here is the ls -l of that image

 ls -l /images/Base-Stable/
total 11098480
-rwxrwxr-x 1 fogproject root           6 Oct 12 07:34 d1.fixed_size_partitions
-rwxrwxr-x 1 fogproject root     1048576 Oct 12 07:35 d1.mbr
-rwxrwxr-x 1 fogproject root         872 Oct 12 07:35 d1.minimum.partitions
-rwxrwxr-x 1 fogproject root          20 Oct 12 07:34 d1.original.fstypes
-rwxrwxr-x 1 fogproject root           0 Oct 12 07:34 d1.original.swapuuids
-rwxrwxr-x 1 fogproject root    10722570 Oct 12 07:35 d1p1.img
-rwxrwxr-x 1 fogproject root         754 Oct 12 07:35 d1p2.img
-rwxrwxr-x 1 fogproject root 11352884934 Oct 12 07:40 d1p3.img
-rwxrwxr-x 1 fogproject root      153330 Oct 12 07:40 d1p4.img
-rwxrwxr-x 1 fogproject root         872 Oct 12 07:34 d1.partitions
-rwxrwxr-x 1 fogproject root         872 Oct 12 07:35 d1.shrunken.partitions

And the contents of d1.partitions

cat /images/Base-Stable/d1.partitions
label: gpt
label-id: 745A6813-735A-4940-840D-28D0E805850A
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 83886046
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=CFB2D745-2975-480D-BD56-EA18B65E03C5, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=57426B89-9DEE-4BDF-9CBF-E9452D985FD5, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=    57417925, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=857B1626-9DC1-44FD-8120-2AB922BBBD85, name="Basic data partition"
/dev/nvme0n1p4 : start=    82761728, size=     1120256, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=DE086091-5C6A-4D63-89E2-579581B9BD63, attrs="RequiredPartition GUID:63"

Maybe it’s because it’s captured as /dev/nvmeXX instead of /dev/sdaXX ? Just brainstorming there.

Doing an update to the latest dev version just to be sure this isn’t fixed in a recent patch.

Found that enabling just the PCI option

PCI bus based UFS Controller support     

is all that is needed for the disk to show in FOS

On my first attempt to image though it gave errors regarding partition overlap and partition size. The disk shows in fog as 119 GB, my win 10 image (single disk resizable) is under 22 GB. Not sure how it isn’t expanding into the disk, there may be something else special needed.
Trying a different image.

Should also mention I’m using the latest available linux kernel version for the test 6.6.1

Some possible usb options found when searching UFS in kernel config.
Seems like we need the UFSCHD driver defined here https://www.kernel.org/doc/html/latest/scsi/ufs.html
One generic option for UFS was enabled at the top level.

Symbol: PHY_MTK_UFS [=n]                                                                                        │
  │ Type  : tristate                                                                                                │
  │ Defined at drivers/phy/mediatek/Kconfig:30                                                                      │
  │   Prompt: MediaTek UFS M-PHY driver                                                                             │
  │   Depends on: (ARCH_MEDIATEK || COMPILE_TEST [=n]) && OF [=n]                                                   │
  │   Location:                                                                                                     │
  │     -> Device Drivers                                                                                           │
  │ (1)   -> PHY Subsystem                                                                                          │
  │         -> MediaTek UFS M-PHY driver (PHY_MTK_UFS [=n])                                                         │
  │ Selects: GENERIC_PHY [=y]                                                                                       │
  │ Selected by [n]:                                                                                                │
  │   - SCSI_UFS_MEDIATEK [=n] && SCSI_UFSHCD [=y] && SCSI_UFSHCD_PLATFORM [=y] && ARCH_MEDIATEK && RESET_CONTROLLE

Symbol: PHY_QCOM_QMP_UFS [=n]                                                                                   │
  │ Type  : tristate                                                                                                │
  │ Defined at drivers/phy/qualcomm/Kconfig:89                                                                      │
  │   Prompt: Qualcomm QMP UFS PHY Driver                                                                           │
  │   Depends on: PHY_QCOM_QMP [=n]                                                                                 │
  │   Location:                                                                                                     │
  │     -> Device Drivers                                                                                           │
  │ (2)   -> PHY Subsystem                                                                                          │
  │         -> Qualcomm QMP PHY Drivers (PHY_QCOM_QMP [=n])                                                         │
  │           -> Qualcomm QMP UFS PHY Driver (PHY_QCOM_QMP_UFS [=n])                                                │
  │ Selects: GENERIC_PHY [=y]

 Symbol: PHY_SAMSUNG_UFS [=n]                                                                                    │
  │ Type  : tristate                                                                                                │
  │ Defined at drivers/phy/samsung/Kconfig:32                                                                       │
  │   Prompt: Exynos SoC series UFS PHY driver                                                                      │
  │   Depends on: OF [=n] && (ARCH_EXYNOS || COMPILE_TEST [=n])                                                     │
  │   Location:                                                                                                     │
  │     -> Device Drivers                                                                                           │
  │ (3)   -> PHY Subsystem                                                                                          │
  │         -> Exynos SoC series UFS PHY driver (PHY_SAMSUNG_UFS [=n])                                              │
  │ Selects: GENERIC_PHY [=y]

This top level was enabled

 Symbol: SCSI_UFSHCD [=y]                                                                                        │
  │ Type  : tristate                                                                                                │
  │ Defined at drivers/ufs/Kconfig:6                                                                                │
  │   Prompt: Universal Flash Storage Controller                                                                    │
  │   Depends on: SCSI [=y] && SCSI_DMA [=y]                                                                        │
  │   Location:                                                                                                     │
  │     -> Device Drivers                                                                                           │
  │ (4)   -> Universal Flash Storage Controller (SCSI_UFSHCD [=y])                                                  │
  │ Selects: PM_DEVFREQ [=y] && DEVFREQ_GOV_SIMPLE_ONDEMAND [=y] && NLS [=y]

This one was not enabled, enabling it in a test

Symbol: SCSI_UFSHCD_PLATFORM [=y]                                                                               │
  │ Type  : tristate                                                                                                │
  │ Defined at drivers/ufs/host/Kconfig:30                                                                          │
  │   Prompt: Platform bus based UFS Controller support                                                             │
  │   Depends on: SCSI_UFSHCD [=y] && HAS_IOMEM [=y]                                                                │
  │   Location:                                                                                                     │
  │     -> Device Drivers                                                                                           │
  │       -> Universal Flash Storage Controller (SCSI_UFSHCD [=y])                                                  │
  │ (6)     -> Platform bus based UFS Controller support (SCSI_UFSHCD_PLATFORM [=y])

Also enabling these other choices

 --- Universal Flash Storage Controller                                                    │ │
  │ │                   [*]   Universal Flash Storage BSG device node                                             │ │
  │ │                   [*]   PCI bus based UFS Controller support                                                │ │
  │ │                   [ ]     DesignWare pci support using a G210 Test Chip                                     │ │
  │ │                   [*]   Platform bus based UFS Controller support                                           │ │
  │ │                   []     Cadence UFS Controller platform driver

Microsoft recently released a “business only” surface Go 4.
This go around they switched from having an NVME controller option to only offering a UFS based option.
See also https://www.microsoft.com/en-us/surface/business/surface-go-4
In the past models you could get either eMMC based storage or NVME based storage.

UFS, being as it has previously not been a common pc form factor storage option, is not supported in the Fog Kernel (I’m on 6.1.22)

I imagine for me to get this thing deployed right now, I’ll need to go build a custom kernel.
Perhaps we can add UFS storage support in the kernel going forward?

A minor bugfix update has been released…and then something more was caught (aliases for functions weren’t being exported) and another version was released right after

Release notes: https://github.com/darksidemilk/FogApi/releases/tag/2310.6.3
PSGallery: https://www.powershellgallery.com/packages/FogApi/2310.6.3

@Lenain Just a little FYI.
I don’t know when I’ll have time to test it out fully before putting it into production, but the idea of adding this to the module was not forgotten and I started a branch for it while I was working on some other bug fixes
https://github.com/darksidemilk/FogApi/commit/03c0342cff759ef272a64bbb91cbb1b5999bab9c

@typotony Sorry I’m late to the party.

I haven’t built a “helper” command yet for removing fog images, but I do have Get-FogImages

https://fogapi.readthedocs.io/en/latest/commands/Get-FogImages/

If you setup the fogapi powershell module, you can use that to get a list of the current images.

You can delete the records of the fog images from the database via

Remove-FogObject

i.e. this would delete ALL your images

$images = get-fogimages
$images | Foreach-object {
  Remove-fogobject -type object -coreObject image -idofobject $_.data.id
}

That little snippet is untested but should be enough to get you started.

I’m pretty sure the api command won’t remove the files from the server though.

However, I believe the path to the files, or at least the relative path is in a child property of the data property when you run Get-FogImages.
So if you set this up to run on the linux based fog server using the powershell api module, you could script something that runs in a cronjob to delete from the database and then the files.

i.e. (again untested as I’m away from my test environment, so the property names are guesses, but it should point you in the right direction)

Get-FogImages | % {
  if ($_.data.name -eq "removeMe") {
    Remove-fogobject -type object -coreObject image -idofobject $_.data.id
    rm -fr $_.data.path
  }
}

This, if run on the fog server where you have configured the fogapi module (see my signature for doc links), would find images named “removeMe” and then delete them from the database and then the files from the server.

@Sebastian-Roth Sadly I have not actually done this.
I believe that it is possible and @tom-elliott may have some insight on accessing the api over https. I believe he has some forum posts elsewhere discussing it. I think there may be some fog server side configuration needed to enable https api. For sure you need to have the root CA certificate that issued your fog server web certificate trusted on the machine issuing the commands. In theory that should be all there is to it, but I haven’t done a ton of testing on it, and while it’s on my to-do list, it’s a bit far down the list. Not to say SSL/TLS isn’t very important even behind a firewall, I just haven’t got there yet.

@glequeau In theory, if you can get all the certificates trusted correctly and can access a fog api url in a browser over https, then it should work for the powershell commands as well. That’s how it typically works with powershell and api commands.

That was long, here’s a short version:

• Go to your https fog site
• view the certificate being used and view the trust chain
• Download the root and any intermediate CA certificates
• Install them on your machine as trusted root ca certs
• Try the api commands again with the https url set

If that doesn’t work, then we just have more development needed on https api commands

You could also try editing your locally installed version of the fogapi module. Specifically the invoke-fogapi command (probably at C:\program files\windowspowershell\modules\fogapi\2303.5.33\fogapi.psm1) then search for that command in the compiled version of the file.

You would add the -SkipCertificateCheck switch to all calls to invoke-restmethod and invoke-webrequest which would bypass that error and still use the ssl connection, but ignore errors about untrusted certificates

@Sebastian-Roth I have a work in progress concept for permanent links. But things have gotten busy and I haven’t had much free time to get it working in full.
Short version is that we’ll be able to create uuids for all pages and that can be used as a permanent link even if it moves elsewhere in the doc.

@Lenain That’s great to hear, glad we could get it figured out.