USB ethernet adapter mac(s) for imaging multiple hosts. (Universal Imaging Nics) (Wired Nic for All Wireless Devices)

JJ Fullmer

As should have probably been expected since the fog client doesn’t check the uuid, it tried to register a new host when the usb adapter was ignored on client. So not going to ignore on client, just going to try and force a remove of the mac via an api call.

JJ Fullmer

@Tom-Elliott I don’t suppose you’d be able to help me a little bit with the api calls?
I was trying to start with just a search and it wasn’t working.

After reading through the api documentation here https://news.fogproject.org/simplified-api-documentation/ I figure my process would be something like.

GET the Search results for the hostname in fog hosts and get the host id (or otherwise get the current/requesting hosts id)
GET the value of the macs array of the host with the id.
Search the macs array for the usb ethernet macs (this one isn’t an api call, I can do this one)
Remove any usb macs from the array. (another non api, just creating a new value to put)
PUT/UPDATE the value of the primac as the new top of the mac array
PUT/UPDATE the value of the mac array with the new value without any usb-ethernet mac addresses.

I think that would do the trick. Then I just put that in a happy little powershell function/cmdlet within my secondLogonCommands.ps1 script (runs after fog client has renamed and rebooted a just deployed machine) and no longer worry about forgetting to remove the mac of the usb ethernet adapter and can image devices with a usb-ethernet setup in a more streamlined manner.

Please and thank you =).

I would be using powershell to run the api calls with invoke-restmethod and or invoke-webrequest (has an alias of curl). I’ve got a invoke-restmethod command working to queue a capture task, so I figure this is plausible, but I’m a little lost on how to get the search call working (testing all this in insomnia editor) and how to query just one value from the results of a hosts information. If that functionality isn’t built in, I can easily just parse the result as a powershell string.

JJ Fullmer

I found some other examples here https://forums.fogproject.org/topic/10036/api well I found them after I figured a bit more out myself. I’m currently writing a couple quick script cmdlets I’ll post here to invoke the fogapi in powershell and to remove a given list of usbmacs from a host.

JJ Fullmer

Discovered that updating mac address association doesn’t work on the host/hostid path with the api. Need to use the /macaddressassocitaion path and first set the usb adapter to not primary, then set a new primary, then delete the macaddressassociation record. That removes it from the host.

JJ Fullmer

Another fogapi powershell module, I didn’t see the one @Tom-Elliott had posted elsewhere in the forums until I was 95% done with mine…but hey I did it so sharing is still caring.

function Invoke-FogApi {
    <#
        .SYNOPSIS
           a cmdlet function for making fogAPI calls via powershell
        
        .DESCRIPTION
            takes a few parameters with a default that will get all hosts
            Makes a call to the api of a fog server and returns the results of the call
            The returned value is an object that can then be easily filtered, processed, and otherwise manipulated in poweshell.
            i.e. you could take the return value of the default all hosts and run 
                $(invoke-fogapi).hosts | where name -match "$(hostname)"
            to get the host information for the current computer
        
        .PARAMETER fogApiToken
            a string of your fogApiToken gotten from the fog web ui. Can be set in the function as a default or passed to the function
        
        .PARAMETER fogUserToken
           a string of your fog user token gotten from the fog web ui in the user section. Can be set in the function as a default or passed to the function
        
        .PARAMETER fogServer
            The hostname or ip address of your fogserver, defaults to the default fog-server
        
        .PARAMETER uriPath
            Put in the path of the apicall that would follow http://fog-server/fog/
            i.e. 'host/1234' would access the host with an id of 1234
            
        .PARAMETER Method
          Defaults to 'Get' can also be 
        
        .PARAMETER jsonData
            The jsondata string for including data in the body of a request
        
        .EXAMPLE
            #if you had the api tokens set as default values and wanted to get all hosts and info you could run this, assuming your fogserver is accessible on http://fog-server
            Invoke-FogApi;

        .Example
            #if your fogserver was named rawr and you wanted to put rename host 123 to meow
            Invoke-FogApi -fogServer "rawr" -uriPath "host/123" -Method "Put" -jsonData "{ `"name`": meow }";

        .Link
            https://news.fogproject.org/simplified-api-documentation/
        
        .NOTES
            The online version of this help takes you to the fog project api help page
            
    #>

    [CmdletBinding()]
    param (
      #took out my api tokens, you can default apitoken strings here or pass them to the function
        [string]$fogApiToken = 'defaultValueCanBeHardcoded', 
        [string]$fogUserToken = 'defaultValueCanbeHardcoded',
        [string]$fogServer = "fog-server",
        [string]$uriPath = "host", #default to get all hosts
        [string]$Method = "Get",
        [string]$jsonData #default to empty
    )
    
    begin {
        # Create headers
        Write-Verbose "Building Headers...";
        $headers = @{};
        $headers.Add('fog-api-token', $fogApiToken);
        $headers.Add('fog-user-token', $fogUserToken);

        # Set the baseUri
        Write-Verbose "Building api call URI...";
        $baseUri = "http://$fogServer/fog";
        $uri = "$baseUri/$uriPath";
    }
    
    process {

        Write-Verbose "$Method`ing $jsonData to/from $uri";
        if ($Method -eq "Get") { #don't include body with get
            $result = Invoke-RestMethod -Uri $uri -Method $Method -Headers $headers -ContentType "application/json";            
        }
        else {
            $result = Invoke-RestMethod `-Uri $uri -Method $Method -Headers $headers -Body $jsonData -ContentType "application/json";
        }
    }
    
    end {
        Write-Verbose "finished api call";
        return $result;
    }
}

JJ Fullmer

My solution in the functions I call in powershell

function Remove-UsbMac {
    <#
        .SYNOPSIS
            A cmdlet that uses invoke-fogapi to remove a given list of usb mac address from a host
        
        .DESCRIPTION
            When a wireless device is imaged with a usb ethernet adapter, it should be removed when it's done
        
        .PARAMETER fogServer
            passed to calls of invoke-fogapi within this function see help invoke-fogapi -parameter fogserver
        
        .PARAMETER usbMacs
            a string of mac addresses like this @("01:23:45:67:89:10", "00:00:00:00:00:00")
        
        .PARAMETER fogApiToken
            the apitoken for invoke-fogapi calls
        
        .PARAMETER fogUserToken
            the user api token for invoke-fogapi calls
        
        .PARAMETER hostname
            the hostname to remove the usb macs from, defaults to current hostname
        
        .EXAMPLE
            Remove-UsbMacs -fogServer "foggy" -usbMacs @("01:23:45:67:89:10", "00:00:00:00:00:00")
            
        .Link
            https://forums.fogproject.org/topic/10837/usb-ethernet-adapter-mac-s-for-imaging-multiple-hosts-universal-imaging-nics-wired-nic-for-all-wireless-devices/14
        
        .NOTES
            online version of help goes to fog forum post where the idea was conceived
    #>
    [CmdletBinding()]
    param (
        [string]$fogServer = "fog-server",
        [string[]]$usbMacs = @("80:3f:5d:0a:ee:16","80:3f:5d:10:37:10"), #default usb mac list, can be overridden
        [string]$fogApiToken = 'NjEzNDY1MzczOTM0NjI2MzM2MzIzODYxNjI2NDMyMzkzMjY2NjQ2MTM0MzYzMzM0NjIzODM4NjIzMTM0NjM2NjM4MzMzNjM0NjMzOTMzNjY2NjY2NjQ2NjM3MzQzOTY2MzMzMTYxMzQzNDYzMzkzNjM0MzkzNTY2MzgzNDY2NjY=',
        [string]$fogUserToken = 'YzVkYjE2MWU4NDk5ZTczMzc0MjAwZTNlNmIxYjcwZTBjYzlhZjFiODBiM2YxNDE4ZDc5NWEyNjViZDEyNzYwNWNkOGFmZDY5MjIyYjU4MDc5ZTlmZjc0YjIzMDRkMjY1OGNlN2Y1NThjMjEyOGUxZmE5MzcwODA1ZDUwZWE2YzI=',
        [string]$hostname = "$(hostname)",
        $macId #initialize
    )
    
    begin {
        Write-Verbose "remove usb ethernet adapter from host $hostname on fog server $fogServer ....";
        # get the host id by getting all hosts and searching the hosts array of the returned json for the item that has a name matching the current hostname and get the host id of that item
        $hostId = ( (Invoke-FogApi -fogServer $fogServer -fogApiToken $fogApiToken -fogUserToken $fogUserToken).hosts | Where-Object name -match "$hostname" ).id;
        # With the host id get mac associations that match that host id.
        $macs = (Invoke-FogApi -fogServer $fogServer -fogApiToken $fogApiToken -fogUserToken $fogUserToken -uriPath "macaddressassociation").macaddressassociations |
            Where-Object hostID -match "$hostId"; 

        # Copy the return fixedsize json array collection to a new powershell list variable for add and remove functions
        $macList = New-Object System.Collections.Generic.List[System.Object];
        $macs | ForEach-Object { $macList.add("$($_.mac)"); }
        $result = "no usb adapters found"; #replace string if found
    }
    
    process {
        # Check if any usbmacs are contained in the host's macs
        $usbMacs | ForEach-Object { #loop through list of usbMacs
            if ( $macList.contains($_) ) { # check if the usbMac is contained in the mac list of the host
                # Remove from the list so a new primary can be picked if needed
                $macList.Remove($_);
                
                Write-Verbose "$_ is a $usbMac connected to $hostname, checking if it is the primary...";
                $macItem = ($macs | Where-Object mac -eq $_ );
                
                if ( $macItem.primary -eq 1 ) {
                    Write-Verbose "It is primary, let's fix that and set $($macList[0]) to primary";
                    $macItem.primary = 0; 
                    Invoke-FogApi -fogApiToken $fogApiToken -fogUserToken $fogUserToken `
                        -fogServer $fogServer -jsonData ($macItem | ConvertTo-Json) -Method "Put" `
                        -uriPath "macaddressassociation/$($macItem.id)/edit" -Verbose;
                    
                    Write-Verbose "Primary attribute removed, setting new primary...";
                    $newPrimary = ($macs | Where-Object mac -eq $macList[0] );
                    $newPrimary.primary = 1;
                    Invoke-FogApi -fogApiToken $fogApiToken -fogUserToken $fogUserToken `
                    -fogServer $fogServer -jsonData ($newPrimary | ConvertTo-Json) -Method "Put" `
                    -uriPath "macaddressassociation/$($newPrimary.id)/edit" -Verbose;
                }

                Write-Verbose "Remove the usb ethernet mac association";
                $result = Invoke-FogApi -fogApiToken $fogApiToken -fogUserToken $fogUserToken `
                    -fogServer $fogServer -uriPath "macaddressassociation/$($macItem.id)/delete" `
                    -Method "Delete" -Verbose;
                
                Write-Verbose "Usb macs $usbMacs have been removed from $hostname on the $fogServer";
            }
        }
    }
    
    end {
        return $result;
    }
}

function Invoke-FogApi {
    <#
        .SYNOPSIS
           a cmdlet function for making fogAPI calls via powershell
        
        .DESCRIPTION
            takes a few parameters with a default that will get all hosts
            Makes a call to the api of a fog server and returns the results of the call
            The returned value is an object that can then be easily filtered, processed, and otherwise manipulated in poweshell.
            i.e. you could take the return value of the default all hosts and run 
                $(invoke-fogapi).hosts | where name -match "$(hostname)"
            to get the host information for the current computer
        
        .PARAMETER fogApiToken
            a string of your fogApiToken gotten from the fog web ui. Can be set in the function as a default or passed to the function
        
        .PARAMETER fogUserToken
           a string of your fog user token gotten from the fog web ui in the user section. Can be set in the function as a default or passed to the function
        
        .PARAMETER fogServer
            The hostname or ip address of your fogserver, defaults to the default fog-server
        
        .PARAMETER uriPath
            Put in the path of the apicall that would follow http://fog-server/fog/
            i.e. 'host/1234' would access the host with an id of 1234
            
        .PARAMETER Method
          Defaults to 'Get' can also be 
        
        .PARAMETER jsonData
            The jsondata string for including data in the body of a request
        
        .EXAMPLE
            #if you had the api tokens set as default values and wanted to get all hosts and info you could run this, assuming your fogserver is accessible on http://fog-server
            Invoke-FogApi;

        .Example
            #if your fogserver was named rawr and you wanted to put rename host 123 to meow
            Invoke-FogApi -fogServer "rawr" -uriPath "host/123" -Method "Put" -jsonData "{ `"name`": meow }";

        .Link
            https://news.fogproject.org/simplified-api-documentation/
        
        .NOTES
            The online version of this help takes you to the fog project api help page
            
    #>

    [CmdletBinding()]
    param (
        [string]$fogApiToken = 'NjEzNDY1MzczOTM0NjI2MzM2MzIzODYxNjI2NDMyMzkzMjY2NjQ2MTM0MzYzMzM0NjIzODM4NjIzMTM0NjM2NjM4MzMzNjM0NjMzOTMzNjY2NjY2NjQ2NjM3MzQzOTY2MzMzMTYxMzQzNDYzMzkzNjM0MzkzNTY2MzgzNDY2NjY=',
        [string]$fogUserToken = 'YzVkYjE2MWU4NDk5ZTczMzc0MjAwZTNlNmIxYjcwZTBjYzlhZjFiODBiM2YxNDE4ZDc5NWEyNjViZDEyNzYwNWNkOGFmZDY5MjIyYjU4MDc5ZTlmZjc0YjIzMDRkMjY1OGNlN2Y1NThjMjEyOGUxZmE5MzcwODA1ZDUwZWE2YzI=',
        [string]$fogServer = "fog-server",
        [string]$uriPath = "host", #default to get all hosts
        [string]$Method = "Get",
        [string]$jsonData #default to empty
    )
    
    begin {
        # Create headers
        Write-Verbose "Building Headers...";
        $headers = @{};
        $headers.Add('fog-api-token', $fogApiToken);
        $headers.Add('fog-user-token', $fogUserToken);

        # Set the baseUri
        Write-Verbose "Building api call URI...";
        $baseUri = "http://$fogServer/fog";
        $uri = "$baseUri/$uriPath";
    }
    
    process {

        Write-Verbose "$Method`ing $jsonData to/from $uri";
        if ($Method -eq "Get") {
            $result = Invoke-RestMethod -Uri $uri -Method $Method -Headers $headers -ContentType "application/json";            
        }
        else {
            $result = Invoke-RestMethod -Uri $uri -Method $Method -Headers $headers -Body $jsonData -ContentType "application/json";
        }
    }
    
    end {
        Write-Verbose "finished api call";
        return $result;
    }
}

I am putting these in a “secondLogonCommands.ps1” script that I have running after the fog service has renamed and rebooted a newly imaged machine to join it to the domain.

This makes it so any macs not caught by the fog pxe inventory are caught by the service. Then any usb mac addresses are removed from the host and the host will be recognized by uuid from then on. I tested the uuid as well, and it worked as expected. As in, after using these functions to remove the mac, after the host was inventoried and or imaged. I was able to boot with a usb ethernet adapter (that isn’t registered to anything) to the fog menu and the host was recognized by its uuid properly. Hooray!

Playing with the api gave me another idea to implement.
I can use the group membership I assign at initial registration in the fog menu to assign the proper AD OU. So I can make it so I never have to go the web ui to deploy an image. Just boot to fog, do a full inventory, pick my groups and snapins, and then the first logon script will use the api to change the OU to join via group. Then the secondlogon script will remove the usb mac association and the service will happily deploy snapins.

I imagine it’s possible to make this process a little smoother still. But this method works for me at least for now.

Sebastian Roth

@JJ-Fullmer Sorry I didn’t get to look into this for a bit.

So is that a new init you made for this testing purpose, or was that included in the latest commit on the working branch?

The init is just for testing. We added the code and @Tom-Elliott will upload new official inits soon.

I think my current hope is that I will be able to create the host with the uuid from the fog menu, booting with any usb-ethernet-adapter.

Well that is what I’d expect it to work like already. Please let us know if this is not possible at the moment (still using the patched init.xz so far).

Then probably have an api call in a after image script to remove the usb-ethernet mac address(es) from the host.

Yeah right, this is an issue when reusing the same USB NIC adapter again and again. I will need to think this through…

However, that may have the issue of the fog client trying to re-add it if it isn’t removed in time, though I suppose that wouldn’t be a problem since that mac would have to be approved before being saved.

As of now the fog-client (or fogservice as you call it sometimes) is not ready for UUID yet. This is something on my list though and I have started to implement. But just too many things going on.

Great to see that you are working your way through the API. Probably a nice way of fixing this. Possibly we will come up with a general solution to this soon anyway. Let’s see.

JJ Fullmer

@sebastian-roth

@sebastian-roth said in [USB ethernet adapter mac(s) for imaging multiple hosts. \

Yeah right, this is an issue when reusing the same USB NIC adapter again and again. I will need to think this through…

See below…
The api calls are working for what I want. It does seem a little convoluted to go through this much trouble, but it’s worth it to me. I have quite a few non-ethernet devices, and it’s a growing number.

Trying to break this down simpler. I am still using the patched init.

Register host in fog boot menu with usb nic (gets usb nic mac but also uuid)
host is imaged
sysprep does its thing
firstlogoncommands/setupcomplete script runs (for mine, I install the fog service at the end after queueing a secondLogonScript)
fog service adds wifi macs, domain joins, and reboots
SecondLogon script runs, contains the Remove-UsbMac function below. This removes the mac from the host right before shutting down the computer at end of imaging.
Remove usb nic so it doesn’t get added again by the fog service.

There is indeed still a potential problem with the fogservice adding it willy nilly if you forget to unplug it. But hopefully that is properly handled by removing right before a shutdown. This is typically the only time I use the usb nic, so it should work out for most scenarios.

A more dynamic and or integrated solution would be great, but I get that having the service use uuid and eventually phasing out the MAC addresses as a registration entity is that solution and that it will take some time. If there’s anything I can do to help with that, I would love to contribute.

JJ Fullmer

Also, if there’s an interest, I would gladly put in the effort to combine Tom’s ps fog api cmdlet with my own, add some more verbosity and help options, like the possible uri paths being tab completed for example. And encapsulate the whole thing into a installable module. So anyone could download it from the git shared scripts repo and import it and start playing with the api in a windows environment.
Well technically, powershell is in linux now too. So theoretically this same thing could be used on a linux machine with powershell 6.
I am just referring to module-izing the Invoke-FogApi function to give people an easy and installable starting point to play with the api. Just wondering if there’s an interest in that, or maybe someone beat me to it and I just haven’t seen it anywhere and wasted my time writing that like a ninny.

Sebastian Roth

@jj-fullmer:

I have quite a few non-ethernet devices, and it’s a growing number.

Yes, we have more and more users with devices not having an onboard NIC and therefore we want to handle this properly.

A more dynamic and or integrated solution would be great, but I get that having the service use uuid and eventually phasing out the MAC addresses as a registration entity is that solution and that it will take some time. If there’s anything I can do to help with that, I would love to contribute.

Absolutely, this is what we are aiming for. I will get back to you on this I am sure!

PS: About the API cmdlet. You should talk to @Wayne-Workman who’s working on a native Linux API cmd thing.

Wayne Workman

@jj-fullmer I’m writing a fog-cli using Python3, which can run on any OS. You’re of course welcome to keep going down the powershell path, but for the Python3 based project your talents would certainly be appreciated and your welcome to help.

JJ Fullmer

@wayne-workman I’m certainly more fluent in powershell, but I know some python.
Where might I find a git repo or other place to see what you got.
Maybe I’ll still make my powershell module but do it in something of a port style and match any functions you create to give users options.

Just playing with my little invoke-api cmdlet function I posted here I’ve been able to easily create so many more functions and functionalities. I imagine python has similar rest methods built in that create objects and such.

JJ Fullmer

@jj-fullmer In case anyone is wanting to play with this further. Here’s another function to get the current host or get a host by a given uuid, mac, or hostname.

function Get-FogHost {
    [CmdletBinding()]
    param (
        [string]$uuid,
        [string]$hostName,
        [string]$macAddr
    )
    
    begin {
        [bool]$found = $false;
        Write-Verbose 'Checking for passed variables'
        if (!$uuid -and !$hostName -and !$macAddr) {
            Write-Verbose 'no params given, getting current computer variables';
            $uuid = (Get-WmiObject Win32_ComputerSystemProduct).UUID;
            $macAddr = ((Get-NetAdapter | Select-Object MacAddress)[0].MacAddress).Replace('-',':');
            $hostName = $(hostname);
        }
        Write-Verbose 'getting all hosts to search...';
        $hosts = (Invoke-FogApi).hosts;
        Write-Verbose "search terms: uuid is $uuid, macAddr is $macAddr, hostname is $hostName";
    }
    
    process {
        Write-Verbose 'finding host in hosts';
        $hostObj = $hosts | Where-Object {
            ($uuid -ne "" -AND $_.inventory.sysuuid -eq $uuid) -OR `
            ($hostName -ne "" -AND $_.name -match $hostName) -OR `
            ($macAddr -ne "" -AND $_.macs -contains $macAddr);
            if  ($uuid -ne "" -AND $_.inventory.sysuuid -eq $uuid) {
                 Write-Verbose "$($_.inventory.sysuuid) matches the uuid $uuid`! host found"; 
                 $found = $true;
            }
            if  ($hostName -ne "" -AND $_.name -match $hostName) {
                Write-Verbose "$($_.name) matches the hostname $hostName`! host found"; 
                $found = $true;
            }
            if ($macAddr -ne "" -AND $_.macs -contains $macAddr) {
                Write-Verbose "$($_.macs) matches the macaddress $macAddr`! host found";
                $found = $true;
            }
        }

    }
    
    end {
        if ($found){
            return $hostObj;
        }
        return $found; #return false if host not found
    }
}

Wayne Workman

@jj-fullmer said in USB ethernet adapter mac(s) for imaging multiple hosts. (Universal Imaging Nics) (Wired Nic for All Wireless Devices):

current host or get a host by a given uuid, mac, or hostname.

You should fork the fog-community-scripts repository and add this stuff to it. Just label it the powershell cli or something, give it some snazzy name.

For the Python project, I’m calling it fog-cli.

USB ethernet adapter mac(s) for imaging multiple hosts. (Universal Imaging Nics) (Wired Nic for All Wireless Devices)

221

11.7k

17.1k

154.7k