@sebastian-roth I hope to be building our 20H2 image the start of next year. So I can do some testing along with that for sure on this then.

Due to current global pandemic conditions I find myself with some extra time to work on this. I do also have an infant, so not a ton of time, but more than I usually have to work on these kinds of projects.

I am currently thinking I will focus on the following things in the module but would love input on any features people would like in an API module out of the box.

• Creating a readthedocs or github pages based webpage for help files integrated with powershell’s get-help {function-Name} -online functionality. i.e. https://github.com/darksidemilk/FogApi is a rough draft
• Moving the api to its own repo following best practices for powershell modules. https://github.com/darksidemilk/FogApi.
• Update the build script to utilize the new structure and documentation needs
• Make sure each existing function has documentation, especially including examples
• Make it compatible with Powershell 7 (ideally without losing powershell 5 compatibility)
• Once compatible with powershell 7, add more cross platform compatibility for linux and mac (though I don’t have a way to test mac functionality) on all existing functions
• Add Functions for more common fog tasks (Hoping to get some requests to know where focus is best spent)
• Make it so the return object will work with the small api changes in fog 1.6 by making it just return just the content as the count is automatically included in a powershell object or changing the return object in some way that isn’t a breaking change. i.e. it currently returns something like

count: 100
hosts: {hostJsonContent}

in 1.6 I understand it will change to

count: 100
data: {hostJsonContent}

So I may utilize some method to make it so everything returns an object with count and data or just data or something based on what fog version you have. May also add additional 1.6 options if time permits (I understand there are join functions, and options to return just a specific parameter/member of an object or objects, or at least those were the plans 2 years ago)

• Make it so there’s more pipeable commands. i.e. you put some host in a variable and then just pipe it to actions

Get-FogHost -hostname computername | Add-FogHostMac -macaddress "12:34:56:78:90"

This would probably take quite some time to add

We’ll see how much of this I’m able to do in the undetermined amount of time this pandemic has me at home all day.

@mecsr Have you run test disk on the vm your uploading that image from that gave you that message?

Still unsure why uploading an image would cause a cpu halt

@Wayne-Workman said in Cortana/Windows Search breaks in default profile:

@Quazz That definitely needs integrated into the script.

I believe it already is integrated in the script. The only .dat files it should copy are the ntuser.dat files. But I could be wrong on that one.
I’m still planning on some serious work on this to make it only edit the ntuser.dat hive for each setting stored there instead of copying the whole thing. But I have a lot of other projects that have to take priority at the moment. I will make a note to add in an explicit exclude in the copy for the UsrClass.dat just to be safe.

@sebastian-roth

The smaller to larger worked fine as well.
here are all the d1.partition and d1.minimum.partitions I have (will update this post tomorrow with the recaptured first image) so it’s all in one spot

1909 image

for reference of old windows partition table
This was NOT created with the new init, included this for comparison.

d1.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 134217694
sector-size: 512

/dev/sda1 : start=        2048, size=     1083392, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=75A55945-9352-4F30-A330-67878EC28FA1, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/sda2 : start=     1085440, size=      202752, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=D1336408-C5B3-446F-BE78-725587F7224D, name="EFI system partition", attrs="GUID:63"
/dev/sda3 : start=     1288192, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=C9F8E03C-928A-4233-9837-49CE510E39B0, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda4 : start=     1320960, size=   132894720, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=D7729B0D-A7B4-4E1F-A9F5-F06FAAAE6141, name="Basic data partition"

d1.minimum.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 134217694
sector-size: 512

/dev/sda1 : start=        2048, size=     1083392, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=75A55945-9352-4F30-A330-67878EC28FA1, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/sda2 : start=     1085440, size=      202752, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=D1336408-C5B3-446F-BE78-725587F7224D, name="EFI system partition", attrs="GUID:63"
/dev/sda3 : start=     1288192, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=C9F8E03C-928A-4233-9837-49CE510E39B0, name="Microsoft reserved partition", attrs="GUID:63"
/dev/sda4 : start=     1320960, size=    45462454, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=D7729B0D-A7B4-4E1F-A9F5-F06FAAAE6141, name="Basic data partition"

20H2 smaller image

partitions of image from 60 GB VM

d1.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 125829086
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2C465DF5-10FD-4883-B323-423E3D10FB2E, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7AFE394E-47A0-43FE-A06E-7AFD7286E200, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=   124567040, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=4A0FF43A-6E31-4AC9-8446-073E07C63AA8, name="Basic data partition"
/dev/nvme0n1p4 : start=   124806656, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=AEF97419-7B7D-4EC5-8C19-FF8C0217A965, attrs="RequiredPartition GUID:63"

d1.minimum.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 125829086
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2C465DF5-10FD-4883-B323-423E3D10FB2E, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7AFE394E-47A0-43FE-A06E-7AFD7286E200, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=    21331078, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=4A0FF43A-6E31-4AC9-8446-073E07C63AA8, name="Basic data partition"
/dev/nvme0n1p4 : start=    21571584, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=AEF97419-7B7D-4EC5-8C19-FF8C0217A965, attrs="RequiredPartition GUID:63"

20H2 Larger image

Partitions of image from 128 GB VM

d1.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2C465DF5-10FD-4883-B323-423E3D10FB2E, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7AFE394E-47A0-43FE-A06E-7AFD7286E200, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=   267173376, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=4A0FF43A-6E31-4AC9-8446-073E07C63AA8, name="Basic data partition"
/dev/nvme0n1p4 : start=   267412992, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=AEF97419-7B7D-4EC5-8C19-FF8C0217A965, attrs="RequiredPartition GUID:63"

d1.minimum.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2C465DF5-10FD-4883-B323-423E3D10FB2E, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p2 : start=      206848, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7AFE394E-47A0-43FE-A06E-7AFD7286E200, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      239616, size=    21351120, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=4A0FF43A-6E31-4AC9-8446-073E07C63AA8, name="Basic data partition"
/dev/nvme0n1p4 : start=    21592064, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=AEF97419-7B7D-4EC5-8C19-FF8C0217A965, attrs="RequiredPartition GUID:63"

This module appears to have gained some popularity…

Maybe I need to find some time to do more updating.

@plegrand A couple questions
What are the specs (cpu(s) model and speed, Ram speed and size, virtual/physical) of your FOG server?
What are the specs of the workstations (specifically the slowest one)?
What is the compression rating on the image you’re pushing?

Something to understand about multicast (and someone should correct me if I’m wrong) is that is does all the decompression on the server and then sends the packets out following multicast protocol. The idea of multicast is to do more on the server side and less on the client and get a constant speed. It also is meant to prevent using a file or part of a file that is in use to make a fail safe against corrupting files and such. The downside is that if you’re using an image with a compression level of 9 and your server isn’t all that powerful, multi-cast is going to slow down a lot compared to unicast where decompression is done on the client.

My point is if your server has less power or less optimization for decompressing then your clients, multicast will just be slower. Considering your speed difference of multicast and unicast, my guess would be this is the case.

Also, a side note in favor of just using unicast, I have used fog to deploy to a lab of 40 computers (10 at a time, which can be changed with the Max clients setting in the storage configuration, just fyi) with the same image on unicast and had no problems with corruption and the speed was glorious.

Another limiting factor is client and server port speeds. If your server is on a 10/100Mb port, you should upgrade that because it will make your life painful. But if your server is on a Gigabit switch then 10 clients on 100Mb switches will all run at around 1.7 GB/min (estimated mathematically and based on experience) If everything is on gigabit switches/ports then you might see a decrease in client speed as you add more simultaneous clients , but will still be around 2 or 3 times faster for 10 clients. Just to give you a little reference off the top of my head.

It would probably be a good idea to start recording average speeds for common configurations and post them in the forum somewhere so people have a reference of how fast to configuration ought to be performing.

@Arrowhead-IT Of course since it is in github now, anyone is welcome to help with commits and contributions

@sebastian-roth

Windows 8.1

Image captured and deployed with new init
Did deploy test from larger 128 GB source VM and it had no problem deploying to 60 GB VM

d1.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      614400, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=6D92B4A9-E0B4-4DED-8C4C-E271AA36B06F, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=      616448, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2410B705-A7CB-4CE5-A393-8B3919C909D1, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      821248, size=      262144, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=58F556FE-7D3B-44EA-A846-C0848DB743CB, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1083392, size=   267350016, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=364F4B4F-4D49-4955-A5EB-9CDE151EEBF7, name="Basic data partition"

d1.minimum.partitions

label: gpt
label-id: 68156B04-B4FE-40EF-96CC-747C33F75E54
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 268435422
sector-size: 512

/dev/nvme0n1p1 : start=        2048, size=      614400, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=6D92B4A9-E0B4-4DED-8C4C-E271AA36B06F, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=      616448, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=2410B705-A7CB-4CE5-A393-8B3919C909D1, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      821248, size=      262144, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=58F556FE-7D3B-44EA-A846-C0848DB743CB, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1083392, size=    17806622, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=364F4B4F-4D49-4955-A5EB-9CDE151EEBF7, name="Basic data partition"

@dvbnl It’s the “.” in “Dell Inc.”
I was just implementing this and found an issue with “Vmware Inc.” specifically the “.” not being seen as part of the string.

I chose to remove any trailing dots in the manufacturer name.

add this second manu definition, and it should help move you forward.

manu=`dmidecode -s system-manufacturer`;
manu="${manu%.*}";

I would also add below it something like

if [[ "${manu}" == "Dell Inc" ]]; then manu="Dell"; fi

If you are structuring your folders with the name “Dell” rather then Dell Inc

Updated to ver 6525 before updating the image again.
This time around it worked flawlessly as it should.

I am trying using the mapped letters from when I mount the shares.

I also just discovered how to run a command prompt as the system user via psexec (http://stackoverflow.com/questions/77528/how-do-you-run-cmd-exe-under-the-local-system-account)

So I’m using that method to test whether or not it should work via the fog service.

@r-pawlowski I would be happy to help in anyway I can. Personally, I deploy a bootmanager (grub2win) and the ipxe.efi file to each machines efi partition. Then I can just change the boot managers default to boot straight to fog. It takes a bit of time to get setup, but I found it to be the most reliable as I then didn’t have to worry as much about the bios settings. Theoretically you can use bcdedit commands to change the boot order to boot to network, but that also differs on different hardware.

I digress though, the easy bit for queueing an image to deploy

• Follow the setup instructions for the fog API (involves installing the module from powershell gallery and then inputting your API keys and fog server address)
• Then make sure the host you’re imaging is assigned the correct image, this can be changed in the API if you need that, but for this quick example, we’ll pretend it’s set correct.
• For an example, we’ll say the hostname of a computer in fog is ‘test-computer’, so this will find the host in fog, then queue an image to deploy on it right now

$hostID = (Get-FogHost -hostname 'Test-Computer').id
#create a quick json string, this can also be done in a powershell object and converted, but this is just a quick example.
#tasktypeID of 1 is a deploy task, if no runtime/schedule time is specified, it defaults to instant deploy
# shutdown = 0 is not scheduling with shutdown
# I believe other2 = 0 means not a debug
# other4 = 1 means to enable wol
# isactive =1 means the task is active
$jsonData = "{`"taskTypeID`": 1,`"shutdown`":`"0`",`"other2`":`"0`",`"other4`":`"1`",`"isActive`":`"1`" }";
# make the API call to create the new task for your host
New-FogObject -type objecttasktype -coreTaskObject host -jsonData $jsonData -IDofObject $hostId;

Once you have this all setup in a powershell module or script internally, you can get things automated pretty smoothly. It’s meant to be modular so it can be applied to any infrastructure or workflow and built upon.

If you need more help let me know and I’ll see what I can do.

@chris-whiteley Sorry for the insanely delayed reply.
So you’re looking to find a host by the serial number? And then get the last time it was imaged?
I was coming here to post information on my recent update, but this sounded useful so I went ahead and implemented getting a foghost by the serialnumber in the inventory field and even a get-lastimagetime function that will default to prompting you to scan a serial number barcode
These will be published shortly.

-JJ

Needed to not use root as msqluser for storage node.
Set it back to fogstorage with password from the fog webgui fog config →fog settings → storage node settings (something like that)
Put that in the storage node servers /opt/fog/.fogsettings and redid the install again. All is well again

@mecsr Use powershell instead of bat scripts, you’ll have more power. Pun intended, but it’s also just true. Especially when it comes to licensing tools. Sometimes the tools just don’t have command line options. Powershell has more user friendly tools for editing registry keys and environemnet variables which licenses sometimes require.

Also use the windows system image manager included in the windows ADK for creating an unattend.xml file

Snapins are an excellent method.

@george1421 said in Copy Cloud Storage of /images to Remote Servers Globally:

@JJ-Fullmer do you know if there is an API for exporting image definitions from fog?

https://forums.fogproject.org/topic/12026/powershell-api-module

The other option might be to write some mysql code to export the image def table on the root fog server then to import it on the remote fog server end. This could be added to your cron job as part of the image movements.

As long as the receving end fog server doesn’t create its own images the export and import with mysql should be pretty clean. If the receiving end fog server also created images then there is the problem of duplicate image IDs in the database.

Sorry I just saw this

100% you can get image definitions from the api and you could then store them in json files. It would be a lot simpler, easier and safer than direct database calls in my opinion and it handles the whole column sql stuff.

Here’s a quick sample of what you might do

On the source server with the original image install and configure the fogApi powershell module.

Open up a powershell console

#if you're new to powershell you may need this
Set-executionpolicy RemoteSigned;
#install and import the module
Install-Module FogApi
Import-Module FogApi;
# go obtain your fog server and user api tokens from the fog web gui then put them in this command
Set-FogServerSettings -fogApiToken 'your-server-token' -fogUserToken 'your-user-token' fog-server 'your-fog-server-hostname';

You’ll need to do that above setup for each location to connect to the api

Now you can get the images and store them as json files, could do 1 big one and could use other file formats, this example will create a json file for each image you have and store it in \server\share\fogImages which you should of course to change to some share you have access to.
This is a function you could paste into powershell and change that path using the parameter -exportPath "yourPath\here"

function Export-FogImageDefinitions {
    [cmdletBinding()]
    param (
        $exportPath = "\\sever\share\fogImages"
    )
    $images = Get-FogImages;
    $images | Foreach-object {
        $name = $_.name;
        $_ | ConvertTo-Json | Out-File -encoding oem "$exportPath\$name.json";
    }
}

So you can copy and paste that (or save it into a .ps1 you dotsource, I may also just add it as a function in the fogAPi module as it would be handy for other users especially when migrating servers.)

You’d run it like this (after having setup your fog api connection Export-FogImageDefinitions -exportPth "some\path\here"
Now you have them all in json format on some share so you need an import command

Now this one I can’t easily test at the moment and may need a little bit of TLC, I based the fields to pass in on the ones defined in the matching class https://github.com/FOGProject/fogproject/blob/master/packages/web/lib/fog/image.class.php

This is the function you’d run on the other fog servers after configuring the powershell module for the destination fog server on that destination network.

EDIT: I adjusted this one after some testing. The api will ignore the old id automatically so it doesn’t matter if you pass that and you can pass all the other information that was exported as well, so no need to filter out anything with Select-Object, in fact you can just pass the raw json files

Function Import-FogImageDefinitions {
    [cmdletBinding()]
    param(
         $importPath = "/images/imageDefinitions"
    )
    $curImages = Get-FogImages;
    #get the json files you exported into the given path
    $imageJsons = Get-ChildItem $importPath\*.json;
    $imageJsons | Foreach-Object {
        # get the content of the json, convert it to an object, and create the image definition
        #create the image on the new server if it doesn't already exist
        if ($curImages.name -contains $_.baseName) { 
             #this will assume the exported definition is the most up to date definition and update the image definition.
             Update-FogObject -type object -coreObject image -jsonData (Get-Content $_.Fullname -raw); 
        } else {
             New-FogObject -type object -coreObject image -jsondata (Get-Content $_.Fullname -raw)
        }
    }
} 

So you copy paste this function into powershell and run import-fogImageDefinitions -importPath \\path\to\where\you\exported\images

That should do the trick.

Also, I believe that all of this will work in the linux version of powershell as well if you don’t have windows in your environment or if you wanted to have all of this running on the fog servers themselves. I also didn’t include what hosts have it assigned as that would probably be different on different servers. That’s also possible to export and import if the hosts are all the same.

You could also take this example and expand on it and get things automated via scheduled tasks on a windows system or cron on the fog servers so it essentially auto-replicates these definitions as you add new ones. If you want more information or more guidance on using the api for this let me know

@JJ-Fullmer
A new version has been published!

Release notes here:

https://github.com/darksidemilk/FogApi/releases/tag/2303.5.26

@Jbob You’d think I would have tried that since I suggested it to this guy, but I have not tried that because I am dumb.

See also https://forums.fogproject.org/topic/12915/windows-boot-manager-boot-option-disappears-after-image