USB ethernet adapter mac(s) for imaging multiple hosts. (Universal Imaging Nics) (Wired Nic for All Wireless Devices)

JJ Fullmer

This post is deleted!

JJ Fullmer

@tom-elliott So I can achieve my goals by adding the usb adapter mac to all hosts and check the ignore on client box?
And the mac won’t make the fog boot menu recognize the wrong host?
Also, how might I add a mac address ignored on client to all hosts. I was thinking utilizing an everyone group. But groups don’t have mac settings.

Sebastian Roth

@JJ-Fullmer As of now - that is current RC version - you should be able to use a single USB NIC with any number of clients as we already prefer UUID as a key to recognize clients. You can see this when you change one client’s MAC either on the machine (when it’s simple as in a VM) or in the web UI. The client should still be recognized when booted to the FOG menu. Give it a go.

That said we still have not testen this feature for all occations and I am not sure if we have it all right to meet your situation. So please try and let us know.

JJ Fullmer

@sebastian-roth I am currently in the process of updating my imaging process to properly use the new(est) and ever changing windows sysprep workflow (which once I have working I intend to document thoroughly and share said documentation on the fog forum, may not show up for another month though)
I am currently capturing an image so I might as well throw some more testing in here for the usbnic.

So I will go make sure that the mac of the usb adapter isn’t currently assigned to any device and give this a try.

Wait, question, how would I add a new wireless device to fog by uuid? Would I still just add it by mac address? If I add a new host from the fog boot menu, but the only way I can get to said boot menu is via a usb ethernet adapter, won’t I still have to deal with this mac address issue, or does it ignore that now in some way? Just trying to wrap my head around how this new registration works.

JJ Fullmer

Test1 - Adding wifi host with usb nic from fog boot menu no pre-registration

Test computer is a Acer Switch 5 tablet
UUID was listed in the bios/efi settings
Fog booted and registered
When checking the client on the web gui found the usb mac was there and no UUID was pulled in the inventory tab.

@Sebastian-Roth @Tom-Elliott So I am using the working branch and am up to date on that one. Do I need to switch to the dev-branch to get the UUID functionality in the latest RC?

x23piracy

@jj-fullmer working is work in progress (alpha), dev is RC’s (beta). So i don’t think so in working branch you will receive latest changes and/or code improvements.

Sebastian Roth

@JJ-Fullmer As x32piracy said you are right on the working branch for the RC version. No need to change to the (actually older) dev-branch!

Somehow I had the impression that registration did work but turned out that it doesn’t. Possibly we’ve never tested this. Sorry for that. Please find a patched init.xz (32 bit init_32.xz is gonna come when Tom added my change officially) here. Move your current /var/www/fog/service/ipxe/init.xz out of the way and put the new one there. After registering a client you should see the UUID filled in the inventory.

JJ Fullmer

@sebastian-roth @x23piracy Thank you, that’s what I thought.

@Sebastian-Roth So is that a new init you made for this testing purpose, or was that included in the latest commit on the working branch?

So I will test it again and see what happens.

I did try a different method and came up with an idea of a workaround, but it sounds like a lot of work.

I took a suggestion Tom gave me and made a dummy host for a usb ethernet adapter.
That host has the usb ethernet mac address and it is ignored on the client.
When I imaged a wifi device using that, it got the uuid during imaging.

Then after imaging the fog service made it a pending host and of course didn’t keep the inventory findings, nor did it keep the attached snapins that I wanted deployed.

So one option would be to have the usb-ethernet host so I can boot to the fog menu and deploy images from there on anydevice. Beforehand having configured desired snapins/active directory settings.
Then in a script then runs after imaging I would make some api calls to get the settings that were set on the usb-ethernet host and copy them to the new pending host the fog service creates. Then approve the host, queue the snapin-tasks, and queue a inventory task. However, two issues would be getting the proper new computer name since I typically utilize Fog to do that by setting it in the initial registration, and queuing the inventory task would prove difficult as it would go back to the problem of not being able to boot to fog via wifi and booting it with the usb host would just put the inventory info in the temp host entry. I suppose I could make some sort of custom tool that copies the database entry of the inventory info from one host to another, but that’s not the best way to do things, especially if we have any schema changes and such.

I think my current hope is that I will be able to create the host with the uuid from the fog menu, booting with any usb-ethernet-adapter. Then probably have an api call in a after image script to remove the usb-ethernet mac address(es) from the host. However, that may have the issue of the fog client trying to re-add it if it isn’t removed in time, though I suppose that wouldn’t be a problem since that mac would have to be approved before being saved.

If I am still overcomplicating this and missing something, let me know.

Thanks,
-JJ

JJ Fullmer

So testing results.

Registering from the fog boot menu did indeed add the UUID using the new init.xz. However it did not add the wifi mac addresses, but that should theoretically not matter with the uuid system. Since the FOS command line registration doesn’t have an option for setting the AD OU to join, I still have to go to the gui anyway once the imaging starts to set that. So I went to do that and set the usb ethernet mac to be ignored by client as well. Now to formulate the command to remove the mac address from the host. May run into an issue with not being able to remove it while it’s the only mac address. May try replacing it with a dummy one for funzies.
Once I have the host deployed with the usb ethernet mac removed and just the wifi macs registered, I’ll test to see if the fog boot menu recognizes by its uuid instead of the unregistered usb ethernet mac.

p.s.
Also, I accidentally tested a method of letting the client add the host as pending and then approving it in the fog boot menu. This created a host in fog with all the mac addresses and then it queued the inventory which added the uuid. This would maybe work for some instances, but would require manually renaming the computer at some point in there in fog or on the host. But it did work. so that’s good/

JJ Fullmer

As should have probably been expected since the fog client doesn’t check the uuid, it tried to register a new host when the usb adapter was ignored on client. So not going to ignore on client, just going to try and force a remove of the mac via an api call.

JJ Fullmer

@Tom-Elliott I don’t suppose you’d be able to help me a little bit with the api calls?
I was trying to start with just a search and it wasn’t working.

After reading through the api documentation here https://news.fogproject.org/simplified-api-documentation/ I figure my process would be something like.

GET the Search results for the hostname in fog hosts and get the host id (or otherwise get the current/requesting hosts id)
GET the value of the macs array of the host with the id.
Search the macs array for the usb ethernet macs (this one isn’t an api call, I can do this one)
Remove any usb macs from the array. (another non api, just creating a new value to put)
PUT/UPDATE the value of the primac as the new top of the mac array
PUT/UPDATE the value of the mac array with the new value without any usb-ethernet mac addresses.

I think that would do the trick. Then I just put that in a happy little powershell function/cmdlet within my secondLogonCommands.ps1 script (runs after fog client has renamed and rebooted a just deployed machine) and no longer worry about forgetting to remove the mac of the usb ethernet adapter and can image devices with a usb-ethernet setup in a more streamlined manner.

Please and thank you =).

I would be using powershell to run the api calls with invoke-restmethod and or invoke-webrequest (has an alias of curl). I’ve got a invoke-restmethod command working to queue a capture task, so I figure this is plausible, but I’m a little lost on how to get the search call working (testing all this in insomnia editor) and how to query just one value from the results of a hosts information. If that functionality isn’t built in, I can easily just parse the result as a powershell string.

JJ Fullmer

I found some other examples here https://forums.fogproject.org/topic/10036/api well I found them after I figured a bit more out myself. I’m currently writing a couple quick script cmdlets I’ll post here to invoke the fogapi in powershell and to remove a given list of usbmacs from a host.

JJ Fullmer

Discovered that updating mac address association doesn’t work on the host/hostid path with the api. Need to use the /macaddressassocitaion path and first set the usb adapter to not primary, then set a new primary, then delete the macaddressassociation record. That removes it from the host.

JJ Fullmer

Another fogapi powershell module, I didn’t see the one @Tom-Elliott had posted elsewhere in the forums until I was 95% done with mine…but hey I did it so sharing is still caring.

function Invoke-FogApi {
    <#
        .SYNOPSIS
           a cmdlet function for making fogAPI calls via powershell
        
        .DESCRIPTION
            takes a few parameters with a default that will get all hosts
            Makes a call to the api of a fog server and returns the results of the call
            The returned value is an object that can then be easily filtered, processed, and otherwise manipulated in poweshell.
            i.e. you could take the return value of the default all hosts and run 
                $(invoke-fogapi).hosts | where name -match "$(hostname)"
            to get the host information for the current computer
        
        .PARAMETER fogApiToken
            a string of your fogApiToken gotten from the fog web ui. Can be set in the function as a default or passed to the function
        
        .PARAMETER fogUserToken
           a string of your fog user token gotten from the fog web ui in the user section. Can be set in the function as a default or passed to the function
        
        .PARAMETER fogServer
            The hostname or ip address of your fogserver, defaults to the default fog-server
        
        .PARAMETER uriPath
            Put in the path of the apicall that would follow http://fog-server/fog/
            i.e. 'host/1234' would access the host with an id of 1234
            
        .PARAMETER Method
          Defaults to 'Get' can also be 
        
        .PARAMETER jsonData
            The jsondata string for including data in the body of a request
        
        .EXAMPLE
            #if you had the api tokens set as default values and wanted to get all hosts and info you could run this, assuming your fogserver is accessible on http://fog-server
            Invoke-FogApi;

        .Example
            #if your fogserver was named rawr and you wanted to put rename host 123 to meow
            Invoke-FogApi -fogServer "rawr" -uriPath "host/123" -Method "Put" -jsonData "{ `"name`": meow }";

        .Link
            https://news.fogproject.org/simplified-api-documentation/
        
        .NOTES
            The online version of this help takes you to the fog project api help page
            
    #>

    [CmdletBinding()]
    param (
      #took out my api tokens, you can default apitoken strings here or pass them to the function
        [string]$fogApiToken = 'defaultValueCanBeHardcoded', 
        [string]$fogUserToken = 'defaultValueCanbeHardcoded',
        [string]$fogServer = "fog-server",
        [string]$uriPath = "host", #default to get all hosts
        [string]$Method = "Get",
        [string]$jsonData #default to empty
    )
    
    begin {
        # Create headers
        Write-Verbose "Building Headers...";
        $headers = @{};
        $headers.Add('fog-api-token', $fogApiToken);
        $headers.Add('fog-user-token', $fogUserToken);

        # Set the baseUri
        Write-Verbose "Building api call URI...";
        $baseUri = "http://$fogServer/fog";
        $uri = "$baseUri/$uriPath";
    }
    
    process {

        Write-Verbose "$Method`ing $jsonData to/from $uri";
        if ($Method -eq "Get") { #don't include body with get
            $result = Invoke-RestMethod -Uri $uri -Method $Method -Headers $headers -ContentType "application/json";            
        }
        else {
            $result = Invoke-RestMethod `-Uri $uri -Method $Method -Headers $headers -Body $jsonData -ContentType "application/json";
        }
    }
    
    end {
        Write-Verbose "finished api call";
        return $result;
    }
}

JJ Fullmer

My solution in the functions I call in powershell

function Remove-UsbMac {
    <#
        .SYNOPSIS
            A cmdlet that uses invoke-fogapi to remove a given list of usb mac address from a host
        
        .DESCRIPTION
            When a wireless device is imaged with a usb ethernet adapter, it should be removed when it's done
        
        .PARAMETER fogServer
            passed to calls of invoke-fogapi within this function see help invoke-fogapi -parameter fogserver
        
        .PARAMETER usbMacs
            a string of mac addresses like this @("01:23:45:67:89:10", "00:00:00:00:00:00")
        
        .PARAMETER fogApiToken
            the apitoken for invoke-fogapi calls
        
        .PARAMETER fogUserToken
            the user api token for invoke-fogapi calls
        
        .PARAMETER hostname
            the hostname to remove the usb macs from, defaults to current hostname
        
        .EXAMPLE
            Remove-UsbMacs -fogServer "foggy" -usbMacs @("01:23:45:67:89:10", "00:00:00:00:00:00")
            
        .Link
            https://forums.fogproject.org/topic/10837/usb-ethernet-adapter-mac-s-for-imaging-multiple-hosts-universal-imaging-nics-wired-nic-for-all-wireless-devices/14
        
        .NOTES
            online version of help goes to fog forum post where the idea was conceived
    #>
    [CmdletBinding()]
    param (
        [string]$fogServer = "fog-server",
        [string[]]$usbMacs = @("80:3f:5d:0a:ee:16","80:3f:5d:10:37:10"), #default usb mac list, can be overridden
        [string]$fogApiToken = 'NjEzNDY1MzczOTM0NjI2MzM2MzIzODYxNjI2NDMyMzkzMjY2NjQ2MTM0MzYzMzM0NjIzODM4NjIzMTM0NjM2NjM4MzMzNjM0NjMzOTMzNjY2NjY2NjQ2NjM3MzQzOTY2MzMzMTYxMzQzNDYzMzkzNjM0MzkzNTY2MzgzNDY2NjY=',
        [string]$fogUserToken = 'YzVkYjE2MWU4NDk5ZTczMzc0MjAwZTNlNmIxYjcwZTBjYzlhZjFiODBiM2YxNDE4ZDc5NWEyNjViZDEyNzYwNWNkOGFmZDY5MjIyYjU4MDc5ZTlmZjc0YjIzMDRkMjY1OGNlN2Y1NThjMjEyOGUxZmE5MzcwODA1ZDUwZWE2YzI=',
        [string]$hostname = "$(hostname)",
        $macId #initialize
    )
    
    begin {
        Write-Verbose "remove usb ethernet adapter from host $hostname on fog server $fogServer ....";
        # get the host id by getting all hosts and searching the hosts array of the returned json for the item that has a name matching the current hostname and get the host id of that item
        $hostId = ( (Invoke-FogApi -fogServer $fogServer -fogApiToken $fogApiToken -fogUserToken $fogUserToken).hosts | Where-Object name -match "$hostname" ).id;
        # With the host id get mac associations that match that host id.
        $macs = (Invoke-FogApi -fogServer $fogServer -fogApiToken $fogApiToken -fogUserToken $fogUserToken -uriPath "macaddressassociation").macaddressassociations |
            Where-Object hostID -match "$hostId"; 

        # Copy the return fixedsize json array collection to a new powershell list variable for add and remove functions
        $macList = New-Object System.Collections.Generic.List[System.Object];
        $macs | ForEach-Object { $macList.add("$($_.mac)"); }
        $result = "no usb adapters found"; #replace string if found
    }
    
    process {
        # Check if any usbmacs are contained in the host's macs
        $usbMacs | ForEach-Object { #loop through list of usbMacs
            if ( $macList.contains($_) ) { # check if the usbMac is contained in the mac list of the host
                # Remove from the list so a new primary can be picked if needed
                $macList.Remove($_);
                
                Write-Verbose "$_ is a $usbMac connected to $hostname, checking if it is the primary...";
                $macItem = ($macs | Where-Object mac -eq $_ );
                
                if ( $macItem.primary -eq 1 ) {
                    Write-Verbose "It is primary, let's fix that and set $($macList[0]) to primary";
                    $macItem.primary = 0; 
                    Invoke-FogApi -fogApiToken $fogApiToken -fogUserToken $fogUserToken `
                        -fogServer $fogServer -jsonData ($macItem | ConvertTo-Json) -Method "Put" `
                        -uriPath "macaddressassociation/$($macItem.id)/edit" -Verbose;
                    
                    Write-Verbose "Primary attribute removed, setting new primary...";
                    $newPrimary = ($macs | Where-Object mac -eq $macList[0] );
                    $newPrimary.primary = 1;
                    Invoke-FogApi -fogApiToken $fogApiToken -fogUserToken $fogUserToken `
                    -fogServer $fogServer -jsonData ($newPrimary | ConvertTo-Json) -Method "Put" `
                    -uriPath "macaddressassociation/$($newPrimary.id)/edit" -Verbose;
                }

                Write-Verbose "Remove the usb ethernet mac association";
                $result = Invoke-FogApi -fogApiToken $fogApiToken -fogUserToken $fogUserToken `
                    -fogServer $fogServer -uriPath "macaddressassociation/$($macItem.id)/delete" `
                    -Method "Delete" -Verbose;
                
                Write-Verbose "Usb macs $usbMacs have been removed from $hostname on the $fogServer";
            }
        }
    }
    
    end {
        return $result;
    }
}

function Invoke-FogApi {
    <#
        .SYNOPSIS
           a cmdlet function for making fogAPI calls via powershell
        
        .DESCRIPTION
            takes a few parameters with a default that will get all hosts
            Makes a call to the api of a fog server and returns the results of the call
            The returned value is an object that can then be easily filtered, processed, and otherwise manipulated in poweshell.
            i.e. you could take the return value of the default all hosts and run 
                $(invoke-fogapi).hosts | where name -match "$(hostname)"
            to get the host information for the current computer
        
        .PARAMETER fogApiToken
            a string of your fogApiToken gotten from the fog web ui. Can be set in the function as a default or passed to the function
        
        .PARAMETER fogUserToken
           a string of your fog user token gotten from the fog web ui in the user section. Can be set in the function as a default or passed to the function
        
        .PARAMETER fogServer
            The hostname or ip address of your fogserver, defaults to the default fog-server
        
        .PARAMETER uriPath
            Put in the path of the apicall that would follow http://fog-server/fog/
            i.e. 'host/1234' would access the host with an id of 1234
            
        .PARAMETER Method
          Defaults to 'Get' can also be 
        
        .PARAMETER jsonData
            The jsondata string for including data in the body of a request
        
        .EXAMPLE
            #if you had the api tokens set as default values and wanted to get all hosts and info you could run this, assuming your fogserver is accessible on http://fog-server
            Invoke-FogApi;

        .Example
            #if your fogserver was named rawr and you wanted to put rename host 123 to meow
            Invoke-FogApi -fogServer "rawr" -uriPath "host/123" -Method "Put" -jsonData "{ `"name`": meow }";

        .Link
            https://news.fogproject.org/simplified-api-documentation/
        
        .NOTES
            The online version of this help takes you to the fog project api help page
            
    #>

    [CmdletBinding()]
    param (
        [string]$fogApiToken = 'NjEzNDY1MzczOTM0NjI2MzM2MzIzODYxNjI2NDMyMzkzMjY2NjQ2MTM0MzYzMzM0NjIzODM4NjIzMTM0NjM2NjM4MzMzNjM0NjMzOTMzNjY2NjY2NjQ2NjM3MzQzOTY2MzMzMTYxMzQzNDYzMzkzNjM0MzkzNTY2MzgzNDY2NjY=',
        [string]$fogUserToken = 'YzVkYjE2MWU4NDk5ZTczMzc0MjAwZTNlNmIxYjcwZTBjYzlhZjFiODBiM2YxNDE4ZDc5NWEyNjViZDEyNzYwNWNkOGFmZDY5MjIyYjU4MDc5ZTlmZjc0YjIzMDRkMjY1OGNlN2Y1NThjMjEyOGUxZmE5MzcwODA1ZDUwZWE2YzI=',
        [string]$fogServer = "fog-server",
        [string]$uriPath = "host", #default to get all hosts
        [string]$Method = "Get",
        [string]$jsonData #default to empty
    )
    
    begin {
        # Create headers
        Write-Verbose "Building Headers...";
        $headers = @{};
        $headers.Add('fog-api-token', $fogApiToken);
        $headers.Add('fog-user-token', $fogUserToken);

        # Set the baseUri
        Write-Verbose "Building api call URI...";
        $baseUri = "http://$fogServer/fog";
        $uri = "$baseUri/$uriPath";
    }
    
    process {

        Write-Verbose "$Method`ing $jsonData to/from $uri";
        if ($Method -eq "Get") {
            $result = Invoke-RestMethod -Uri $uri -Method $Method -Headers $headers -ContentType "application/json";            
        }
        else {
            $result = Invoke-RestMethod -Uri $uri -Method $Method -Headers $headers -Body $jsonData -ContentType "application/json";
        }
    }
    
    end {
        Write-Verbose "finished api call";
        return $result;
    }
}

I am putting these in a “secondLogonCommands.ps1” script that I have running after the fog service has renamed and rebooted a newly imaged machine to join it to the domain.

This makes it so any macs not caught by the fog pxe inventory are caught by the service. Then any usb mac addresses are removed from the host and the host will be recognized by uuid from then on. I tested the uuid as well, and it worked as expected. As in, after using these functions to remove the mac, after the host was inventoried and or imaged. I was able to boot with a usb ethernet adapter (that isn’t registered to anything) to the fog menu and the host was recognized by its uuid properly. Hooray!

Playing with the api gave me another idea to implement.
I can use the group membership I assign at initial registration in the fog menu to assign the proper AD OU. So I can make it so I never have to go the web ui to deploy an image. Just boot to fog, do a full inventory, pick my groups and snapins, and then the first logon script will use the api to change the OU to join via group. Then the secondlogon script will remove the usb mac association and the service will happily deploy snapins.

I imagine it’s possible to make this process a little smoother still. But this method works for me at least for now.

Sebastian Roth

@JJ-Fullmer Sorry I didn’t get to look into this for a bit.

So is that a new init you made for this testing purpose, or was that included in the latest commit on the working branch?

The init is just for testing. We added the code and @Tom-Elliott will upload new official inits soon.

I think my current hope is that I will be able to create the host with the uuid from the fog menu, booting with any usb-ethernet-adapter.

Well that is what I’d expect it to work like already. Please let us know if this is not possible at the moment (still using the patched init.xz so far).

Then probably have an api call in a after image script to remove the usb-ethernet mac address(es) from the host.

Yeah right, this is an issue when reusing the same USB NIC adapter again and again. I will need to think this through…

However, that may have the issue of the fog client trying to re-add it if it isn’t removed in time, though I suppose that wouldn’t be a problem since that mac would have to be approved before being saved.

As of now the fog-client (or fogservice as you call it sometimes) is not ready for UUID yet. This is something on my list though and I have started to implement. But just too many things going on.

Great to see that you are working your way through the API. Probably a nice way of fixing this. Possibly we will come up with a general solution to this soon anyway. Let’s see.

JJ Fullmer

@sebastian-roth

@sebastian-roth said in [USB ethernet adapter mac(s) for imaging multiple hosts. \

Yeah right, this is an issue when reusing the same USB NIC adapter again and again. I will need to think this through…

See below…
The api calls are working for what I want. It does seem a little convoluted to go through this much trouble, but it’s worth it to me. I have quite a few non-ethernet devices, and it’s a growing number.

Trying to break this down simpler. I am still using the patched init.

Register host in fog boot menu with usb nic (gets usb nic mac but also uuid)
host is imaged
sysprep does its thing
firstlogoncommands/setupcomplete script runs (for mine, I install the fog service at the end after queueing a secondLogonScript)
fog service adds wifi macs, domain joins, and reboots
SecondLogon script runs, contains the Remove-UsbMac function below. This removes the mac from the host right before shutting down the computer at end of imaging.
Remove usb nic so it doesn’t get added again by the fog service.

There is indeed still a potential problem with the fogservice adding it willy nilly if you forget to unplug it. But hopefully that is properly handled by removing right before a shutdown. This is typically the only time I use the usb nic, so it should work out for most scenarios.

A more dynamic and or integrated solution would be great, but I get that having the service use uuid and eventually phasing out the MAC addresses as a registration entity is that solution and that it will take some time. If there’s anything I can do to help with that, I would love to contribute.

JJ Fullmer

Also, if there’s an interest, I would gladly put in the effort to combine Tom’s ps fog api cmdlet with my own, add some more verbosity and help options, like the possible uri paths being tab completed for example. And encapsulate the whole thing into a installable module. So anyone could download it from the git shared scripts repo and import it and start playing with the api in a windows environment.
Well technically, powershell is in linux now too. So theoretically this same thing could be used on a linux machine with powershell 6.
I am just referring to module-izing the Invoke-FogApi function to give people an easy and installable starting point to play with the api. Just wondering if there’s an interest in that, or maybe someone beat me to it and I just haven’t seen it anywhere and wasted my time writing that like a ninny.

Sebastian Roth

@jj-fullmer:

I have quite a few non-ethernet devices, and it’s a growing number.

Yes, we have more and more users with devices not having an onboard NIC and therefore we want to handle this properly.

A more dynamic and or integrated solution would be great, but I get that having the service use uuid and eventually phasing out the MAC addresses as a registration entity is that solution and that it will take some time. If there’s anything I can do to help with that, I would love to contribute.

Absolutely, this is what we are aiming for. I will get back to you on this I am sure!

PS: About the API cmdlet. You should talk to @Wayne-Workman who’s working on a native Linux API cmd thing.

Wayne Workman

@jj-fullmer I’m writing a fog-cli using Python3, which can run on any OS. You’re of course welcome to keep going down the powershell path, but for the Python3 based project your talents would certainly be appreciated and your welcome to help.

USB ethernet adapter mac(s) for imaging multiple hosts. (Universal Imaging Nics) (Wired Nic for All Wireless Devices)

144

11.7k

17.2k

154.7k