Best posts made by JJ Fullmer

Hi Friends,

We just got these new Z640 workstations and they come with these NVME 256GB SSDs plugged into a pci-e slot.
They show up in the bios, and when I boot to fog and check compatibility and partition information it says it’s compatible and all the partition info pops up no problem. One possible issue is that it assigns it to something like /dev/nvme instead of the standard /dev/sda

But then when I try to image or even just do a hardware inventory, FOG gives me a “HDD not found on system” error and then reboots after 1 minute.

Is this something that just isn’t supported? Do I need to enable something in a custom kernel? What am I missing here?

Thanks,
-JJ

P.S.
Fog info
Version svn 5676
bzImage Version: 4.3.0
bzImage32 Version: 4.3.0

@JJ-Fullmer Well it wasn’t actually phpipam that was slowing it down.
I had some ssl logs running for a website running on that server. It was tracking access logs for every access of the ssl cert and site. Which was also adding up to over 400 GB of logs. I changed the apache configuration of the site to log much less. That solved the problem. So again, nothing wrong with zstd. Just wanted to have the right answer in case someone else happens to see a constant 5-8 Mbps transmit and receive on a storage node.

First let’s try to gather some more information. Leave the snapin as is for now and only schedule snapin tasks for clients and let those run.

1. Does it randomly fail on the same host? So if you schedule the same task on one host ten times in a row, does it run through all the time if it is fine on the first run?

I don’t typically try it more than once maybe twice, if it fails I just run the command manually. But I will give that a try. See if I can’t recreate it on my computer.

2. When it magically fails on a host can you please grab that host and take a look at C:\Program Files (x86)\FOG\tmp. I have not checked the code yet but I would hope the fog-client leaves a copy of that file there on failure. Now check if that file is zero size or truncated or anything like that.

It does not leave a copy, it does put a copy there and if you’re watching it you can copy it real quick or you can stop the fogservice before the hash check if you’re lucky. I think I did that once when I first discovered the issue but don’t recall the result, I remember it being confusing, I guess I’ll have to try this again too.

3. On snapin failure as well pay attention to the apache and PHP-FPM log files to see if there is an HTTP/PHP error on snapin download (see my signature).

Good thought will give those a post.

4. Please post the full fog-client log here in case of a failure so we can have a look. Sometimes there is something we see that leads us to the real problem.

Standby

If that doesn’t help we can go ahead an try monitor the database for changes of the hash. FOG has a deamon running that updates the snapin hashes from time to time. Unlikely but maybe this is causing an issue. You can take a look in that log file (webUI -> logs -> snapin log) and even disable that in another step just to see if it interferes.

I was hoping there is a way to disable it on the server, maybe something in the database or something, because I didn’t want to go through the trouble of recompiling the fog client. The other trick is that I really only use snapins during provisioning right after I image a computer. But I guess if I want it fixed I oughta suck it up.

The only other thing that may be a factor is I also add the snapins to the host and initiate the task from the api during provisioning. So It’s possible that it being queued from api causes this too, we’ll see if that’s the case though since I’ll be testing by deploying through the gui.

@george1421
Yes that’s how I got the hardware inventory to run without error.
But that doesn’t work for imaging. It acts likes it’s going to work, filling partitions and what not, then it just says database updated and reboots like it was finished instead of launching into partclone.

@mousepl Are you thinking you would want to be able to do this on the fly, or would you want to deploy the snapin on all hosts that have it assigned to it?
By on the fly I mean, select specific hosts from the snapin then deploy, that would be a bit more difficult to implement, but maybe adding a button for deploying on all hosts with a snapin is possible.

If you’re familar with powershell or willing to learn. You could create a function/script that does this with the api module.

i.e.

# install the module
Install-Module FogApi;
# Follow the setup instructions found in the links in my signature. i.e. get the fogapi token and user api token from your web gui
Set-FogServerSettings -fogServer 'serverName' -fogapitoken 'insertServerAPItokenHere' -fogusertoken 'insertFogUserAPItokenHere'

Assuming those steps are done for the user running the function

function Start-SnapinDeploy {
[CmdletBinding()]
param(
    #define the snapin name you want to deploy
    $snapinName
)
#get all the snapins from the server
$allSnapins = Get-FogSnapins;
#get all the host/snapin associations from the server
$snapAssocs = Get-FogObject -type object -coreObject snapinassociation;
#get the snapin you want to deploye in a single object
$snapinToDeploy = $allSnapins | Where-Object name -match $snapinName;
#find all the hosts that have that snapin associated
$hostsWithSnapin = $snapAssocs.snapinassociations | Where-Object snapinID -match "$($snapinToDeploy.ID)"
#loop through the hosts that have that snapin assigned and start a task for that snapin for each of them
$hostsWithSnapin.hostID | ForEach-Object {
    $hostID = $_;
    $json = (@{
        "taskTypeID" = 13
        "deploySnapins"=$snapinToDeploy.ID
    } | ConvertTo-Json);
    New-FogObject -type objecttasktype -coreTaskObject host -jsonData $json -IDofObject $hostID;
    }
}
# call the function to deploy a snapin named office365
Start-SnapinDeploy -snapinName 'office365'

I may be off on the json creation as I haven’t tested a single snapin deploy, so the required body data might be different. I just kinda altered the way I use the start-fogsnapins function https://github.com/FOGProject/fog-community-scripts/blob/master/PowershellModules/FogApi/FogApi/Public/Start-FogSnapins.ps1 for this example. I typically only use fog snapins for the initial deployment at imaging.

But as you can see from the example, while things are linked together, the linkage is really meant to go from host to snapin. Snapins are really meant to be deployed at imaging time, which is why they are all queued when you a image a host. The idea being to include software provisioning with your OS images. The feature request you described would be useful I agree, but one might argue that that makes more sense for a software deployment tool, where fog is an imaging tool.

It can be used as a software deployment tool, but that’s just not how it’s designed.

That’s all really just a disclaimer to say it may be a bit for this to show up as a new feature. It’s possible and it would be useful. I’m all for anything linked in a database having linked buttons in each spot in a gui. But creating that button may just be easier said than done, but may eventually be done.

You could in theory create a gui using the powershell module and built in powershell command show-command and some dynamic paramters that grab all the available snapins, it would take a bit of work but you could make a gui prompt with drop downs/multi-selector for fog host(s), snapins, then a deploy button. Point being, just because it isn’t in the web gui, doesn’t mean the capability doesn’t already exist.

I hope that helps

Just wanted to chime in with another report on a speed change between 1.5.7 and 1.5.8

1.5.7 ~22 GiB/min
1.5.8 ~11 GiB/min

This is on nvme drives, and we have a gigabit port aggregation on the main deploying node (in case you’re wondering how we got it going so fast).

However on 1.5.7 there was always a slow but steady drop in speed. It would start at 20-25 GiB/min and slowly drop GiB/min every couple seconds. But I never cared much since the ~20 GB image was done deploying in 2-3 minutes each time. In 1.5.8 it isn’t doing the speed drop and the overall time taken is about the same. It was just cycling between just below and just above 11 GiB/min (i.e. 10.58 - 11.03 or something along those lines) Looking at some of my recent imaging times just before and now after the upgrade to 1.5.8 they’re all at about 2 minutes 30 seconds. The only real variation appears to be the hardware being imaged, which is to be expected.

Point being, perhaps there isn’t actually a speed change but rather a more accurate overall average speed for the whole process instead of attempting a realtime speed? Or maybe just a generally more steady speed? Or just a better way of calculating the displayed imaging speed?

@Chris-Whiteley Maybe take a look in the web gui at the report viewer -> Imaging log and see if there’s actually a difference in time for your images deploying before and after the upgrade? I’m finding mine are all still within 0-30 seconds of the same time.

I am also trying the linux live cd idea to see if it gets anything different and maybe if there’s a way to change it to something more standard

According to the official page from microsoft https://www.microsoft.com/en-us/windows/windows-11-specifications it just says “secure boot capable” I guess we’ll just have to wait till it’s released to insiders to get some real world information.

@george1421 While I agree and understand how it all works. I have found that we did get an increase in speed when we setup the aggregated adapter on the storage node. Even with just one client going. But perhaps that’s really just agreeing with your statements. As like on a highway if it was 1 lane, you often slowdown cause of the other slow drivers and perhaps I just opened up a metaphorical passing lane for fog images to go the full speed limit at. You do also have to consider all the switches it goes through and yada yada. And of course it’s all more complicated. Point being, I didn’t get a 2x boost when we aggregated the server link, but we did get a boost, so I wouldn’t deter anyone with the equipment capable of it from giving it a try.

Wasn’t able to boot to a live cd just yet but I tried some other stuff.

I tried changing the primary hard disk to /dev/nvme0
because I figure that may be the new /dev/sda
When I tried to image the computer I got a new error

“Erasing Current MBR/GPT Tables… The specified path is a character device!
Warning! GPT Main header not overwritten! Error is 22
Warning! MBR not overwritten! Error is 29
Corrupted partition table was erased. Everything should be fine now”

It does still get through the hardware inventory, but in either case it doesn’t pull and hard disk information.
I’m going to try /dev/nvme next to see if that works.

I also tried changing the bios sata disk mode from RAID to AHCI, but I don’t know if that effects anything on an nvme drive since it’s plugged into pci-express.

Thanks for all the help thus far.

-JJ

@jj-fullmer I haven’t done a full thorough fog windows 11 test. But it seems that some of the cpu and bios security “requirements” aren’t hard requirements. As long as your cpu supports TPM 1.2 you can do a clean install of windows 11, you just can’t in place upgrade (without a registry change).

I am also posting this on a computer with windows 11 on it, with an i7-6700. I didn’t use fog, and secure boot got enabled by the windows 11 installer (it might have already been enabled, I didn’t double check sadly). However I just disabled secure boot and could still boot.

So the concerns about a secure boot requirement may be unfounded. This is my home computer and I don’t have a fog server at home, but I’ll come back here once I get a chance to test creating and deploying a windows 11 image to see if there are any issues with secure boot. If anyone wants to test this out @testers before I get some time, you can download a windows 11 iso here https://www.microsoft.com/en-us/software-download/windows11?ranMID=24542&ranEAID=0JlRymcP1YU&ranSiteID=0JlRymcP1YU-aILwA1rXpThxrraz01AUgg&epi=0JlRymcP1YU-aILwA1rXpThxrraz01AUgg&irgwc=1&irclickid=_2cqgd3xf9kkf6xflm1yfj9km9e2xoz2ov3bwz2yp00

Also, here’s my ranting on unattend and sysprep that I can’t seem to stop myself from interjecting.

There are also multiple places you can put the unattend.xml file to get it to be used. I put it in 2 places C:\unattend.xml and C:\windows\system32\sysprep\unattend.xml I could have swore you had yours in a panther folder when I first read this, but it looks like I read that wrong or you edited while I was writing. I reference the one in the root of C in my call to sysprep and have it in the sysprep folder as a fail safe because it is a place that sysprep looks for the file. Is the unattend file doing what you expect? You have some winpe commands and partition creation commands in there. Are you using winPE in your process or is that just from the guide you found?

There are many ways that you can use sysprep and fog. The way I do it is

Install windows 10 onto vm
Hit Ctrl+shift+f3 at the oobe screen to enter audit mode
Run my base customization script (copies some custom files, copies some network drivers, installs some basic default programs)
Run my cleanup script (defrag, cleanmgr, sfc, dism, chkdsk, provision the metro/uwp apps to my liking…I’m thorough…)
Copy the unattend file and run sysprep with sysprep.exe /audit /reboot /unattend:C:\Unattend.xml

My unattend file is then set to run the Audit System pass, the Audit User pass, then the generalize pass and then shuts down. At that point I capture the image. I use the reseal property in unattend.xml to control which phase comes next.

Then when I deploy the image, it is a generalized image with no drivers but with lots of customization built in. Right after the image deploys the unattend file continues in specialize phase where, among many other things, the network drivers I copied earlier that apply to the hardware get added to ensure network connectivity. Then the oobe phase starts where all the oobe wizard stuff gets skipped because it is all already answered and then I use the firstlogoncommands section to start up a custom provisioning powershell module/script that sets everything up across a few reboots. I use the built-in admin user during those firstlogon setup pieces, I find that much easier and more reliable then setting up a separate user, the builtin admin usually has a better time with initial setup and having admin rights to do everything you want to, you can always disable the admin account once all is done.

That may all sound kinda complicated, but once you get it set up and scripted, it’s pretty simple.

My point in sharing all this, is you mentioned your new, and I want to let you know that there’s not just one way to handle all this. It’s good that you’re wanting to use sysprep at least. I was once a misguided soul and tried to find ways around it.

I see that you used a website to generate the answerfile. It looks like a decent starting point. I would reccomend making the whole thing yourself using the windows system image manager. It’s not quite as easy, but you’ll be able to see how much more you can do. https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/wsim/windows-system-image-manager-technical-reference. Once you download the adk you can extract your windows 10 iso somewhere and load it up into the system image manager and start messing with it.

I could go on and on, but I don’t want to scare you away. Hopefully I haven’t already. It is completely possible to use somone elses or a web generated unattend file and image with fog without breaking any windows licensing or custom ids. I just felt, once I read through the documentation and gave it a try, that it was worth it to make it myself the official windows way. And I found that they have a lot of documentation on the topic. Here’s some more to look at, docs.microsoft.com is your friend in this adventure.

https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unattend/
https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/sysprep--system-preparation--overview

@Andrew-Single This post I made many moons ago at my old job might be helpful to you. It may require some adaptation, but I still use it on a daily basis to update my fog to the latest version…

https://forums.fogproject.org/topic/4578/automated-latest-fog-svn-update-script-1-0-and-later-and-other-helpful-productivity-things

Also this is the wiki on how to install the svn version manually
https://wiki.fogproject.org/wiki/index.php/SVN

hope that helps

I looked through the other post and gave debug mode a try,
I ran lsblk and got the same disk info except it didn’t have any mount information.
But is said nvme0n1 and partition nvme0n1p1

So if the root problem you want to solve is fog joining the domain I think I see an issue, but could be missing something. So in your script that starts sysprep you have it copy a setupComplete.cmd. I also see that you have it disable the fog service. Does setupcomplete re-enable and start the fogservice back up? If it is starting backup are you getting any error messages in C:\fog.log on the imaged machine?

Sadly the resizable download doesn’t work on nvme drives. mps and mpa do work though.

Did a debug deploy
I found that fog saw the partition name as /dev/nvme0n11 instead of what was listed in gdisk -l /dev/nvme0n1p1
tried to change the name to no avail.

@george1421 I see that in the code but this is a method I use to get past dhcp on the daily. I think it’s going out to dhcp just to make sure the machine has an ip address and is on the network and then connects to the tftp server. But since it already has the efi boot file it’s not using dhcp to try to go find it, which is the part I believe he needs to skip. But if my thought doesn’t work then the default.ipxe method you outlined is worth a shot.

I would add to your method that in the case that file isn’t seen (like if it’s a super picky uefi bios) you could create a efi partition instead of fat32 with diskpart But I would bet your method there would work.

@Tom-Elliott How might I test this hunch?

@george1421 That’s what I figured, it seems an odd problem so I’m going to give all the info I can to help find the problem. And plus, these nvme drives think that they are going to become the norm and phase out sata, sas, and scsi, so I’ll do whatever I can to help get FOG working with them.