Posts made by JJ Fullmer

@danieln I don’t know of a way to default assign a snapin to every new host that is built-in to fog.
However, if a snapin is assigned to a host the deployment of said snapins is automatic when you deploy an image.

Here’s a few ways you could make it close to or completely automatic on all new/existing hosts.

• You can assign a snapin when you register a host via pxe, so you just standardize there
• You can make an everyone/all group and assign the snapin in that group periodically.
• You can assign hosts to a snapin from the snapin screen I believe (maybe that’s only in 1.6) point though is you can assign to all from there.
• You can use the FogApi powershell module to write something that adds the snapin to all hosts that don’t have it, and make that a scheduled task somewhere to have that run every day or hour or what have you

If you wanted to bake it into the image you would just have it as part of your setupcomplete.cmd or unattend.xml firstlogoncommands.

@Jim-Graczyk I agree with you and I actually ran into similar issues in my migration and I’m still working out what is the best approach for migrating without losing client connections.
I took notes on my internal migration that I hope to utilize in creating a new doc for migrating a server without losing the client connection, but I also hope to make the PKI/Certs of fog more flexible in general, it’s just a matter of finding the time to work on it.
If you want to help in creating such a doc in the new docs site (see also https://github.com/FOGProject/fog-docs/tree/master) that would be very welcome.

I’d love to build a script for migrating, or include it as an option in the installer, but scaling that to support all supported linux distros and standard supported configurations would take some serious work that I don’t realistically have time to do.

By the by, if the Powershell commands worked, that is something that can be scaled in most windows environments and could be deployed as a group policy script or you can use Powershell methods such as Invoke-Command to run it remotely on all machines.

@Jim-Graczyk said in What Prevents a FOG Client from Connecting to A FOG Server?:

I DID run into 1.5.10.xxx when downloading FOG, somehow. I used the “master” branch in git for my installs. I follow what I believe are FOG wiki pages that contain specific instructions.

Recently, as in back in like June of this year, we started doing more frequent releases and created a ‘stable’ branch. There were some security issues reported in 1.5.10 but the changes made didn’t fully justify a 1.5.11 release in our versioning schema. So we created a flow for more frequent releases, it’s described here https://github.com/FOGProject/fogproject/tree/stable?tab=readme-ov-file#versioning-and-branches
We’ve built out a CI/CD automated release workflow to do monthly releases so that new minor features, bug fixes, and security fixes are released to users more frequently.
The ‘stable’ branch is now the default branch which should be installed from. The wiki is being slowly migrated over to a new docs site and the latest install doc is found here: https://docs.fogproject.org/en/latest/install-fog-server

Also yes, changing the ip address typically does involve a recommendation for updating the certs because the ip address is including in the public cert’s san. If you can re-use the same IP and have used that alias in the public cert, then you should be able to migrate without needing to re-install clients.

@Jim-Graczyk
Re-reading through your post I realized you mentioned a fresh install of the client isn’t working.
There may be a different issue. When I migrated from centos 7 recently I ended up rerunning the fog server installer with --recreate-ca --recreate-keys and reinstalled the fogservice on all my clients to have them use updated certs and an updated fog server name. I didn’t have the time to troubleshoot it and I also don’t have 8 locations so this wasn’t that big of a thing to deploy via other software deployment tools we have.

In theory you should still be able to migrate the private and public keys (note that the private keys are hidden with a prepended . in the filename when you migrate them from the old fog server.

@Jim-Graczyk Firstly, are you running 1.5.10 on the new server or 1.5.10.1615 the latest stable release version? Granted it would be best to have updated the old server first and then migrate so you’re migrating between the same versions.

If you maintained the same server name (or at least used a dns alias to point the old server name to the new server name) and you migrated the /opt/fog/snapins/ssl directory and all other fog stuff (like the database and the /opt/fog/.fogsettings file) before running the installer for the first time (you may be able to throw them in there afterwords in theory)
then you would have the same CA cert and private cert and subject name for the public cert. So if all that PKI stuff remains the same then all your hosts would already trust the fog server ca (it gets added to trusted root certs when installing the fog client) and the public cert should be updated on the client or may even remain the same. Sounds like you’re using a dns alias so as long as that is updated in the migrated locations it can be made to work.

If you ran the install of the new fog, and then migrated stuff, then you might have some conflicting settings and you’ll have generated a new Fog CA with a new private key for the web server certificate that the fog client uses to ensure you’re communicating with the right fog server.

Granted, making a new CA and private key when migrating servers is a good idea and with a new CA it’s easiest to re-install the client to fix the issue so it gets the new CA and cert.

I believe you could do something like this in an admin powershell session to force the fog service to use a new CA.

#stop the service
stop-service fogservice; 
# delete the certs from the fog service program files path
remove-item "C:\Program Files (x86)\fog\ca.cert.der","C:\Program Files (x86)\fog\fog.ca.cer","C:\Program Files (x86)\fog\tmp\public.cer"; 
#remove the old Fog CA cert from the trusted root store
Get-ChildItem Cert:\LocalMachine\Root\ | Where-Object Subject -match 'CN=FOG Server CA' | Remove-item -force -ea 0;
#Download the new ca cert, replace "fog-server" with your fog server's name
iwr "https://fog-server/fog/management/other/ca.cert.der" -OutFile "C:\Program Files (x86)\fog\ca.cert.der"
#trust the new CA
import-certificate -FilePath "C:\Program Files (x86)\fog\ca.cert.der" -CertStoreLocation Cert:\LocalMachine\Root\
#reset the host encryption in the gui (or if you use my FogApi powershell module you can use the Reset-HostEncryption command)
#after resetting the host encryption start the service back up
Start-Service FogService;

I just ran all that on a working client and it connected. I even stopped after removing the ca cert from the store and files to confirm that broke the client. Then imported it again and all was well. It’s possible that resetting the host encryption isn’t needed, but I imagine it’s needed as you have a new private key, etc.

Granted, reinstalling the service should do all of the above, so it may be a null point.

You could also restore the old CA cert and private key from where you’re migrating to and you may be able to make it work without touching the clients besides restarting the service and resetting host encryption.

@Tom-Elliott It sounded like he tried snponly and others in an earlier post.
The initializing devices happens after the efi file is downloaded and booted to, secure boot would stop it from booting because it’s the efi file that isn’t signed.
I’ve had some luck with the delay based ipxe files on troubleshome devices like this, though there also could just be some driver missing from the ipxe build that isn’t initializing for this device. I feel like it doesn’t boot to pxe at all if the ethernet adapter isn’t supported.
@anube The other thing I’ve had luck with is booting directly to the ipxe.efi file. But on a new device that requires either the device to have a built-in uefi shell or you

• setup 2 usbs
  • 1 a refind usb boot disk
  • 2 another usb where you have the efi boot files from /tftpboot (I usually just grab the ones I know work but you could just grab all of /tftpboot via winscp/filezilla/or some other ftp client off the fog server).
    • I would also go here https://www.realtek.com/Download/List?cate_id=584 and download the UEFI UNDI driver. Extract that and grab the x64 RtkUndiDxe.efi file and throw that on the usb too.
• Then you boot the refind usb,
• choose the uefi shell option
• find the usb with the tftpboot folder and efi driver.
  • To find it in the uefi shell you switch to each disk like fs0: then enter, then run ls to see if it’s the right disk, then fs1: fs2: etc.
• Once you find it you can run load RtkUndiDxe.efi hit enter so the latest uefi driver for that adapter is loaded.
• Then find the efi boot file you want to use/try a bunch of them. i.e. /tftpboot/ipxe.efi simply entering the path will boot to the efi file and it will attempt to then boot to the fog server.

If that does the trick you at least have a workaround and a simpler way to find which efi boot file works for that device. It may be possible to load the updated efi driver into the firmware, but that varies by device and I wouldn’t suggest it unless there’s an official way of doing it provided by MSI. It may also work with an older build of ipxe or a future one or there may be something that can be added to the ipxe build. You can also find ways to make this methodology more automated.

@anube Have you tried the 10sec-delay versions of the ipxe files?

@adam1972 said in Linux host name change after imaging?:

There’s nothing in AD

That checkbox at the bottom of AD is where you enforce hostname changes I believe. Which I just noticed is also what @Tom-Elliott said 20 minutes ago.

I would also look at the snapin definition, this screenshot is from 1.6, but these boxes still existed on snapin definitions in 1.5 (minus the No action, it’s just checkboxes for shutdown or reboot I believe)

If those are checked it could be part of why it’s doing so many reboots.

The fog client runs things in windows without being logged in, there’s actually 2 services running for it, a system service and a user service, I imagine linux works the same way.

However, if it’s waiting to perform the hostname change, there could be something else that’s causing a reboot at the linux OS level when the hostname is attempted to be changed without a reboot? If that is waiting to happen I believe the client waits for that to finish before attempting snapins. It causing the random restarts is still perplexing.

@Eazis The storagenode can be a master of a different storage group if I recall correctly, but the default storage group should have the master (where the web server is) as the master. There’s a lot of ways to configure it, but for simplicity in testing this out I would suggest 1 storage group with the master fog server as the master node.

What version of FOG and kernel is on the master and the nodes. They should all be the same, latest stable, latest dev-branch, or latest working-1.6-beta would be best as we may have already fixed the issue if it’s not a configuration issue.

@anube Is the bios/uefi firmware up to date on the computer?
Has this device worked before? Or similar MSI devices?
Do you have a pxe capable usb adapter you could try?

@adam1972 I was about to ask for the log.
So it looks like you don’t have the enforce option enabled for renaming the hose and the client thinks that someone is logged in. I’d try enabling enforce, I think it’s actually under the Active Directory settings a little checkbox there, might be under service or client settings on the host.
I’m using 1.6 where it’s moved to under service settings on a host.

@shatchett0 you would cancel the capture task and then boot the host to pxe and you should see a compatibility check in the fog boot menu.
The error also looks like an issue with ext4 thinking it’s a dirty filesystem being captured. If you schedule the capture task with debug checked you can try running the file system check tools interactively. This looks like a problem with this specific host and the image being captured more than a fog server issue.

@adam1972 the double reboot is reminiscent of joining windows computers to the domain and then rebooting a second time to rename in the domain, though fog client has long since done it in one go. Are you setting the ad settings for the Linux hosts? Do the snapins have the reboot box checked in their definition?

@adam1972 if you want to do this in fos right after deploying you could probably use a postdownload script to set the contents of /etc/hostname
You would just need to mount the disk that was just imaged and use sed to edit the file with the $hostname variable. Well there’s a few other steps but I’m on my phone.
I don’t know for sure if that’s still all there is to it as I feel like hostnamectl has gotten more robust in newer Linux releases across all distros, but it’s a good start.

@anube Have you tried ipxe.efi as the boot file?
What version of FOG are you running?

@Fog_Newb said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

That makes sense to me especially if that command is only able to rename.

Just for clarity. Rename in this context also handles moving the file. By “renaming” the file from its old path of /images/dev/$mac to /images/$imagename it’s really moving the file. But since those paths are on different filesystems, it doesn’t work.
The php ftp_put attempts to use an upload of the file from the server to the other path, but that is when it’s actually making the empty file because it gets confused by the paths being on different disks.

@Tom-Elliott @Fog_Newb

I tried this symlink method on my test server and didn’t have success sadly. Seems nfs doesn’t follow symlinks.
My theory on the put function was also wrong.

How big of an issue is this disk size inflation for you? Typically you’re only capturing 1 image at a time, so you need your /images disk to have
(enough space for all your images) + (the size of your biggest image) + (maybe 10% of that for flexibility)
So if your current images take up say 500 GB, and the biggest of them was 100GB (which would be a very big image)
Then provisioning 660 GB should be sufficient if you didn’t want to just double the provisioned space.
If you want to have space to capture them all at the same time, then 1.1 TB would do the trick.
If your images are just really really big, then consider using snapins or other software deployment methods to keep your images to just the base os and a limited number of default apps so that they stay on the smaller side. For examples, my Windows 11 images take up just under 20 GB on the server and around 30 GB on my hosts and then I add the rest per use case within my environment.

You can also play with the different compression levels of the image. I’ve found a zstd compression level of 11 to be a good balance of speed and size.
You could also throw caution to the wind and not care about the speed of deploy and capture and just put the compression to max if you’re more concerned about taking less server disk space.

The only thing I can think of to make this work natively would require database schema changes so you can define separate paths for /images and /images/dev and reworking the install script to make sure the /etc/exports is created correctly. Sadly I don’t think that amount of effort would benefit any other use case and this is an uncommon use case that’s not in our realm of support right now, especially since it’s mostly due to limitations in linux.

The is one other possibility, and that would be a postinit script, but those run before imaging starts not after like postdownload scripts. But there’s other security issues there with how you’d get the ro /images share to be mounted as rw when your task only has permissions to open /images/dev which does have rw. So you’d have to open an ssh or sftp session and send the command to move the folder from /images/dev to /images that way. But securely creating that connection at that point is a different beast. It’s odd that linux/php are having this limitation but that’s where we’re at.

@Fog_Newb I think I found the problem.

Normally the ftp_rename command successfully handles moving the captured image from /images/dev to /images but for some reason that command ( a built in php/ftp thing) doesn’t like it when the directories are mounted on separate disks. Not sure why, doesn’t make a ton of sense, but that’s the issue.

In Fog, when this fails, we attempt an ftp_put command aka an upload.
This should work, but there’s a path mismatch at the moment.
During capture the “local” file where it is captured is at /images/{mac} instead of /images/dev/{mac} and the put command is getting the latter because that’s what we already have for the rename command that happens completely remotely.
So I’m testing some stuff to fix this in the dev-branch.

Basically I think you may have found a bug in a failsafe method for uploading a captured image.

@Fog_Newb I am able to recreate the problem in the latest stable version (1.5.10.1615). Working it out. it looks like the image does stay in dev on my end.
I’ll debug this when I can this week.

@Fog_Newb

I guess ro is the way /images is supposed to be as far as /etc/exports

https://github.com/FOGProject/fogproject/security/advisories/GHSA-3xjr-xf9v-hwjh

I’m glad you found the security advisory and already changed that back to RO, I panicked a little when I read that.

So in 1.6 does the image stay in the dev folder?

Yes

Then you could at least manually move the image from dev to images to workaround the issue while we work this out.
This feels like a permissions issue, but you already did that step it sounds like.

What is the size of the file when it creates a file instead of a folder?

0 bytes

And what does ls -l /images show, I’m curious if it’s a file or a symlink. Either way it’s not what should be happening

What is your php version? php - v

8.3.6

Might be worth upgrading to the latest php version (8.3.11), there was a bug in 8.3.10 (Which I realize you’re not using) that caused an issue with the part of capture where we delete the file that was fixed in 8.3.11, but it was likely fine in 8.3.6.

I’m going to test some of this on my dev fog server and see what happens.