@sebastian-roth
@Taspharel
@george1421

All of the info provided lent itself to the ultimate solution…

For anyone who wants to create images on one fog server and have them used on a completely separate, unique and different FOG Server, here’s the process I used.

We’ll create an image that has the FOGService on it associated with the first FOG Server (FOGServer1) by running Sysprep with shutdown. We’ll install the second FOG Server (FOGServer2) and acquire the certs from it by installing the FOGService on a PC and associating it with FOGServer2. We’ll edit the uploaded disk by attaching it as a “D:” drive to a PC and create or add to the SetupComplete.cmd file to load the FOGServer2 certs after Sysprep completes.

Here are the steps:

• Build the image PC with OS, applications, etc. - whatever you want already installed. I subscribe to the notion that a windows image should contain the MS OS CD installed, FOGService Installed and then the rest of the content should come from Snapins. As little as possible should be done by hand because can seldom repeat the process if we have to. For this process, however, it doesn’t matter.

• Install the FOG Client Service and associate it with the first FOG Server (FOGServer1). As stated above, I use Snapins to deploy content the makes up my images.

• Run Sysprep with shutdown, then capture the image to the FOG server (FOGServer1).

• Install the FOG Client Service on a PC associated with the 2nd FOG server (FOGServer2), accepting all defaults.

• In the FOG folder (C:\Program Files\FOG or c:\Program Files (x86)\FOG) copy files ca.cert.der and fog.ca.cer to external storage

• Deploy the image from FOGServer1 to a disk (a virtual is vastly perferred) with Shutdown. Do not let the machine attached to the disk boot.

• Mount the disk to another computer as an additional disk (as in D Drive).

• Using this other computer, edit the contents of the sysprep’d drive (D Drive)

• Copy ca.cert.der and fog.ca.cer to the “D:\windows\setup\scripts” folder

• Create SetupComplete.CMD file in “D:\windows\setup\scripts”, or add to the existing file

• Add these lines for a 32 bit OS:
  copy /y %windir%\Setup\Scripts\ca.cert.der “%programfiles%\FOG\ca.cert.der”
  copy /y %windir%\Setup\Scripts\fog.ca.cer “%programfiles%\FOG\fog.ca.cer”
  certutil -delstore Root “FOG Server CA”
  certutil -delstore Root “FOG Project”
  certutil -addstore Root “%programfiles%\FOG\ca.cert.der”
  certutil -addstore Root “%programfiles%\FOG\fog.ca.cer”

• Add these lines for a 64 bit OS:
  copy /y %windir%\Setup\Scripts\ca.cert.der “%programfiles(x86)%\FOG\ca.cert.der”
  copy /y %windir%\Setup\Scripts\fog.ca.cer “%programfiles(x86)%\FOG\fog.ca.cer”
  certutil -delstore Root “FOG Server CA”
  certutil -delstore Root “FOG Project”
  certutil -addstore Root “%programfiles(x86)%\FOG\ca.cert.der”
  certutil -addstore Root “%programfiles(x86)%\FOG\fog.ca.cer”

• Dismount the additional disk, connect it to a machine associated with FOGServer2 as the system drive (C:), and capture the image to FOGServer2.

• When Deploying the image to additional machines from FOGServer2, the machine will associate with the FOG Server2 but will not join the domain or run Snapins until you Reset Encryption for the new host (button found at the top of the “General” tab for the host).

• One caveat - I use a DNS alias for all FOG Servers (creatively, I chose fogserver) so I don’t have to worry about FOG server name differences. If your FOGServer1 and FOGServer2 are in the same DNS zones, this won’t work, so if you have different FOG server names, you can save the setting,json file from the \program files\FOG folder from a PC associated FOGServer2 or just edit the one with the new FOG server name in it. When you’ve mount the imaged disk as an additional drive, copy the altered settings.json file to the \windows\setup\scripts folder (along with the certs) and add an additional ‘copy’ command to SetupComplete.cmd to get the file into the FOG folder (just like the certs). I haven’t tested this, so hopefully some who knows better will comment.

While the above process is a pain to execute for each image you create, each time you need to associate it with a new FOG Server, I find it far more cost effective than uninstalling the FOG Service and reinstalling it after imaging each PC.

• Please note that I had to place some very odd quotes around some paths because drive letters become emojis d: …

Jim Graczyk

@robtitian16

I believe I’ve seen this when I’ve forgotten to disable Secure Boot in the Hyper-V VM definition.

Jim

@tom-elliott

Here are the screenshots for each step of the fog iPXE process:

@Tom-Elliott said in Storage Group in Dashboard Showing 1 Slot in QUEUE But No Tasks Exist:

mysql fixing

I ran the maintenance script found at the bottom of this link:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_MySQL

Several lines deleted rows. The problem is resolved.

Thanks,

Jim

@themcv

OK - so yeah, maybe it was a DNS issue on the client side after all. One of the snapins I’m working on dorked the DNS search list.

I determined this by examining the spanins deployed to the host - 100% alignment with snapin that dorks the DNS searchlist.

Go Figure…

Thanks and sorry for the trouble…

Jim

@tom-elliott

Tom - I tested this on several Gen2 VMs and they all extended to the fill disk size.

Thanks - SOLVED.

Jim

Server

• FOG Version: 1.5.0
• OS: CEntOS7

Client

• Service Version: 0.11.12
• OS: Win7

Description

I’m working on a Snapin for a VPN client install (SoftEther VPN - great freeware, recommend it highly - AND I have a working installation process).

The result is the VPN Client package testing has resulted in the FOG client informing FOG about the MAC address on the VPN vNICs the software creates.

While looking for a way to Dis-Approve Pending MAC addresses, I click on the + button (Add MAC Address) on the General form of the a Host page. Doing so caused the page to list many new places to add MAC addresses:

This screen shot is at 33% and there are 28 similar pages before one get’s to the bottom of the webpage.

Upon hitting the + to add MACs, the webpage becomes unresponsive. I cannot get a new page to come up - at least not too easily - and the server and workstation show substantial increases in load. From task manager:

I’m forced to close the browser and re-log in to be able to continue.

This only appear to occur on hosts with several pending MAC addresses and it may only occur what a browser is zoomed in.

Jim

@avaryan

Thanks. I haven’t had the need for the Access Control Plugin. I supposed it’s time. Thanks. I still believe it should be an option. The Dashboard is a good page, but it’s not the most efficient thing when you want to get on with FOG tasks.

Jim

@birvin

I would also offer the suggestion that you separate the /images folder onto a separate virtual disk. You Could do this without rebuilding. I’ve used USB disks on ‘toaster’ that are mounted as /images as well. In either case, this allows use to expand the /images folder w/o messing with the rest of the install.

Wayne is an expert, so maybe my approach is more work. I’ve backed into this problem over and again, as image space has grown (mostly from uploads, in my case), but we were able to mount the old volume as /oldimages, mount the new empty volume as /images, copy image, then unmount the /oldimages volume. If I recall, it was only a matter of mess with fstab in my case.

Just a thought.

Jim

@Jim-Graczyk

The wiki on migration covers migrating a single FOG server using the only workable process, so I’ll leave that alone.

I think my issue with the entire “Server Migration” process is that it requires you take FOG down at the onset of the process - before you attempt to install FOG on the new server. This obviously stops all the PC IT maintenance processes FOG provides IT support, but also FOG’s benefits to the end users of the hosts, until the new server/system is back up and working.

I’ve used FOG on a fairly large scale - some instances have 10+ remote sites, each with its own FOG Storage Node. To say that shutting it down before starting migration is inconvenient to the business is an understatement. Storage node migration didn’t require that the storage node had use the same IP as the old storage node - so that helped.

I hope that my initial post to you comes up when other FOG users search “FOG Migration”, so they don’t waste the time I did trying to build a completely new FOG system, including storage nodes, to cut over to, only to find that you can image PCs and create new hosts, but the exiting host would not talk to the new server system - so all the existing host configuration is lost (a large chunk of work). Ironically, there are numerous FOG wikis that address most issues that I ran into - creating new certs, the ever-present Reset Encryption on the FOG Client, etc., but none mentioned that the CA of the old system HAD to be used in place of the new.

I’m glad there was some way forward without having to reconfigure everything manually, but like many things in FOG wikis, everything in them is true, but context and limitations are not defined at the beginning. Even the migration process link didn’t spell out requirements, nor include remote sites and storage nodes. It needs to spell out that harvesting the CA and certs and using the old server’s IP address are as important as the database, snapins, and images - and this is the ONLY way to migrate.

So - a suggestion to the great minds working on FOG:

Create a way to change the FOG Client CA in one step - delete old and replace new - from within the FOG Client. This could be something issued from the old FOG server, for security reasons. CA deletion should also be built into the FOG Client Installation MSI/EXE. I use FOG to build portable Windows images that are company and FOG server independent. The FOG client is installed at the initial boot and replaced in the image as FOG evolves.

If that capability could be added, migration would be possible such that the new system w database, snapins, images, host configs, could be build out with multi-storage nodes and multi-locations, side-by-side with the old FOG system, gated only by DHCP boot settings at each site (IP broadcast space).

Jim Graczyk

@Jim-Graczyk

The wiki on migration covers migrating a single FOG server using the only workable process, so I’ll leave that alone.

I think my issue with the entire “Server Migration” process is that it requires you take FOG down at the onset of the process - before you attempt to install FOG on the new server. This obviously stops all the PC IT maintenance processes FOG provides IT support, but also FOG’s benefits to the end users of the hosts, until the new server/system is back up and working.

I’ve used FOG on a fairly large scale - some instances have 10+ remote sites, each with its own FOG Storage Node. To say that shutting it down before starting migration is inconvenient to the business is an understatement. Storage node migration didn’t require that the storage node had use the same IP as the old storage node - so that helped.

I hope that my initial post to you comes up when other FOG users search “FOG Migration”, so they don’t waste the time I did trying to build a completely new FOG system, including storage nodes, to cut over to, only to find that you can image PCs and create new hosts, but the exiting host would not talk to the new server system - so all the existing host configuration is lost (a large chunk of work). Ironically, there are numerous FOG wikis that address most issues that I ran into - creating new certs, the ever-present Reset Encryption on the FOG Client, etc., but none mentioned that the CA of the old system HAD to be used in place of the new.

I’m glad there was some way forward without having to reconfigure everything manually, but like many things in FOG wikis, everything in them is true, but context and limitations are not defined at the beginning. Even the migration process link didn’t spell out requirements, nor include remote sites and storage nodes. It needs to spell out that harvesting the CA and certs and using the old server’s IP address are as important as the database, snapins, and images - and this is the ONLY way to migrate.

So - a suggestion to the great minds working on FOG:

Create a way to change the FOG Client CA in one step - delete old and replace new - from within the FOG Client. This could be something issued from the old FOG server, for security reasons. CA deletion should also be built into the FOG Client Installation MSI/EXE. I use FOG to build portable Windows images that are company and FOG server independent. The FOG client is installed at the initial boot and replaced in the image as FOG evolves.

If that capability could be added, migration would be possible such that the new system w database, snapins, images, host configs, could be build out with multi-storage nodes and multi-locations, side-by-side with the old FOG system, gated only by DHCP boot settings at each site (IP broadcast space).

Jim Graczyk

@jaoyer

Please be advised that the FOG server migration, as described in the link, does not work well for existing Hosts. While the process migrates the hosts’ info that’s stored in the database, it doesn’t take into account that the new FOG server will have a new CA named exactly the same as the old server’s CA. When you follow this migration guideline, your existing hosts will not connect to the new server at a new IP address, that has a new CA unless you remove the old FOG CA from every client. This is possible thru GPOs etc, (as described by @jj=fullmer) but it’s tricky and can’t be done with FOG (since there is no way to run a snaping from the new server until the CA is already removed from the client). If your FOG system is used on more than on AD domain, then the problem becomes compounded.

I opted to harvest the certs and CA from my old server and put them on the new server. This means I had to use the old server’s IP address on the new server. This to make the old cert & CA usable by the old clients). The Reset Encryption button in the WebUI doesn’t reset the CA.

I’m working on a post to document what I did. I have several sites, each with a FOG Storage Node - something else that is omitted in the migration guide link.

I can’t say that my process will work for anyone else. I had v1.5.10 on all servers before and after I migrated. My migration took place on a network that didn’t change. My goal was to retire CEntOS 7 and move to ALMALINUX, which I did. I migrated all my images and snapins and the configuration of every host. With this approach, the Reest Encryption button was necessary, but since the CA was the same on the servers and the hosts, everything worked as it did before.

I’ll get the completed instructions posted as soon as I can.

Jim Graczyk

I believe I have a viable migration process to get FOG services moved from set of servers to another set. This process takes into account Locations, Storage Groups, existing Hosts, etc. The system I’m migrating has 7 remote sites so that’s 8 locations and 8 servers - one FOG server and 7 FOG Storage Nodes, each at the end of a Static VPN WAN. The initial FOG server is an alias ‘fogserver’ in DNS and DHCP is served out by MS Domain Controllers at each site. Each Storage node serves out all that is needed for each remote location (PXE, initrd, snapins, and images). Each storage node is the master of its own storage group. We do replication manually, so there may be some snapin and image replication issues in this process that I can’t account for.

The process requires being able to harvest the digital identity of the initial FOG server for use on the replacement FOG server, so the replacement server needs the same name and IP address. This is necessary to allow existing hosts to be easily ‘acquired’ by the new system since the Hosts. I pursued a side-by-side migration that worked fine PXE booting->host registration->imaging, but failed for existing Hosts and, for unknown reasons, even new Hosts failed all software distribution steps. The idea of placing FOG required scripts in group policies wasn’t appealing for my purposes since we need the FOGService to work without the machine joining a domain.

I’m currently testing and will have a process “doc” soon. I’m interested in posting this process for FOG Project’s use if the powers that be find it useful.

I’ll post it here when it’s completed.

It seems to me that the need to change Linux distros is an age-old requirement as various Linux OSs have been dropped or usurped. Migration of FOG from CEntOS to any other Debian Linux should be well documented.

More soon.

Jim Graczyk

@JJ-Fullmer

Perhaps I misunderstand how all this is supposed to work, but there seems to be a fundamental issue - In the instructions for migrating seem to indicate that one could create a new FOG Server and migrate that new server into operating - side by side with the old server. Any path to doing this includes new certs on a new server that will have a new IP address and a new server name. One can migrate the database to retain Host, Snapins, Image, etc, but nothing is said about migrating existing FOG clients (hosts) to recognize the new FOG server as each’s FOG server. Various articles address this issue superficially by telling us to Reset Encryption for the Host using the FOG WebUI. I don’t think the Reset Encryption button is applicable when the FOG Server has all new Certs, CA, name & IP address.

In one of the posts for this topic, @JJ-Fullmer describes Powershell commands that would make a FOG clients be able to accept a new FOG server, but one is faced with how one would run those PS commands if each host is no longer able to connect to the new server to run snapins.

Part of this is just defining expectations and tell users what migration process is supported.

As of now, I’ve had a week of downtime on all that FOG does because I attempted an approach that at first, appear to provide a migration path that would have the least downtime - install a new FOG server and move to it, so the old FOG server can remain operational until we know the new one works. In this approach, DHCP would be the only thing that would need to be changes to switch to the new server, first for testing and later, permanently.

It seems this approach is a dead end - as there is no way to get the FOG Clients to connect to the new server automatically. Changing the certs to match the old server exactly can only be done using the same name and IP address. So perhaps this is they only approach that can work with existing Hosts in the system.

So, I’ve shut down the old FOG Server after pulling all existing Certs from it. I got the new server renamed to the old server’s name and IP’d to match the old server (following the Change IP process - which according to the wiki I found, includes reissuing certs - which make clients unable to connect until you mover the old certs back manually - which I did. It seems now that existing Hosts are able to connect to the server. And it may be that all but the migrated FOG Storage Nodes are connected and visible in FOG WebUI.

I feel that this approach is a workaround. In time, the certs from the original CEntOS7 install will one day expire. A migration process that includes locations, existing clients and new certificates should be defined.

Given that many Linuxes come and go, well documented upgrade paths for key software systems like FOG have become more important lately.

I’m still testing because there are still locations to migrate from CEntOS7 to AlmaLinux. I will post when I’m back up and verified.

@JJ-Fullmer

I’m running 1.5.10 on all servers (at least that’s what appears on the fog server and the fog storage nodes in the WebUI). Also, the old server was running 1.5.10 before it was shut down and when I dumped the database for migration to the new server.

I DID run into 1.5.10.xxx when downloading FOG, somehow. I used the “master” branch in git for my installs. I follow what I believe are FOG wiki pages that contain specific instructions.

If there is an issue related to upgrading from 1.5.10 to and current version of 1.5.10.xxxx, please let me know.

As I had once again f’d up and put my post under the General topic, it went unnoticed. So in the last few days, I’ve opted move the new FOG server to the Old FOG server name and IP address, and copied the /opt/fog/snapins/ssl folders and content from the old server. This is all on the servers as they were described in the initial post - not a fresh install.

I also completed the steps in the Change IP Address wiki, which specifically rebuilds the certs.

I’m now testing if I can get new and existing hosts connected.

For the record:
I was also chasing what I thought was a more fully logging version of the 0.13 client - some file named debugger.exe - that appears to be some sort of commandline realtime interface to the client that can be used to debug problems. As I have not seen any documentation regarding the innards of the FOGService and its structure, I punted on that tool.

Thanks for the help. I’ll post my results.

Jim

Please delete the version of this post I started under General

Hi and thanks for any help anyone can provide.

I’m in the middle of a FOG server 1.5.10 migration to get off of CEntOS 7 and onto AlmaLinux 9.4 running FOG 1.5.10.

I tested this process in our lab and have overcome several issues. I followed this:
https://wiki.fogproject.org/wiki/index.php?title=Migrate_FOG

The difference is that we have several locations that also have to be migrated. I uninstalled and reinstalled the location Plugin which seemed to be required. I was able to get a 2 site deployment working correctly w images and snapins, pxe boot from the right location FOG Storage Node, etc.

The worst think I have to do was delete a Host and register with the FOG server to get it to work post imaging, aside from Reseting Host Encryption.

SO. I followed the same process (or so I think) in a production instance and have a different result. Since the lab is typically maintained to test deployments, the one thing I lacked when testing was machines that were in FOG and working pre-migration to test post migration. However, resetting a host’s encryption seemed to get all but one host to connect to FOG. Testing here was justs seeing that the fog.,log file showed proper authentication. Most testing was full deployment - image and then snapins - and these tests worked.

In the production instance, PXE booting and imagining work as expected at the 2 locations that have been migrated, but in this case, no PC is able to connect to the new FOG server - existing or reimaged.

Note: we have 8 locations, but I only migrated the FOG server and one location so far. Each location has only one Storage node - the FOG server DefaultMember at the corp HQ and a storage node at the remote site. Each storage node is in its own storage group and its own location.

We build site / domain independent images. To do this we have a DNS Alias “FOGSERVER” that points to the main FOG server A record. We use this alias in a \windows\setup\scripts\setupcomplete.cmd when running the smartinstaller.exe v0.12. at first boot.

In cases of existing hosts, I’ve have reset encryption over and again.

Oddly, in both fresh deployments of images and existing machines, all upgrade to the new FOG Service v0.13 yet the UI shows none of the host “touch” time changes.

Also in both cases (reimages and existing hosts), fog.logs show the standard No Token file / Authentication Error / Invalid security token error over and over. In 10 mins or more, the FOG client is upgraded to v0.13, but resetting the Host’s Encryption doesn’t result in the client ever getting connected. The Host shows in the UI that the OS is Windows, but all tests show “No Data” (implying the Host isn’t actually connected - which the fog.log files all show they aren’t). I’m not certain that the FOG client updates happen only after resetting encryption, but perhaps that is so.

There is a pattern in the clients - they do get upgraded, they all try to authenticate over and again, until you Reset Host Encryption. Then, after 5 attempts to connect, the FOGService on every test PC seem to stop but the service is still “running”. Restarting the FOG Service produces the same result after 5 more authentication attempts.

Here’s a typical end of fog.log:

9/16/2024 1:59:11 PM Client-Info Version: 0.13.0
9/16/2024 1:59:11 PM Client-Info OS: Windows
9/16/2024 1:59:11 PM Middleware::Authentication Waiting for authentication timeout to pass
9/16/2024 1:59:28 PM Controller Stop
9/16/2024 1:59:28 PM Service Stop requested
9/16/2024 1:59:28 PM Bus Emmiting message on channel: Status
9/16/2024 1:59:28 PM Middleware::Authentication ERROR: Could not authenticate
9/16/2024 1:59:28 PM Middleware::Authentication ERROR: Thread was being aborted.

We also see some message about not starting Looper.

--------------------------------Authentication--------------------------------
9/16/2024 3:35:01 PM Client-Info Version: 0.13.0
9/16/2024 3:35:01 PM Client-Info OS: Windows
9/16/2024 3:35:01 PM Middleware::Authentication Waiting for authentication timeout to pass
9/16/2024 3:37:01 PM Middleware::Communication Download: http://roa1fogl02.rq.priv/fog/management/other/ssl/srvpublic.crt
9/16/2024 3:37:01 PM Middleware::Authentication Cert OK
9/16/2024 3:37:01 PM Middleware::Authentication No token found at C:\Program Files (x86)\FOG\token.dat, this is expected if the client has not authenticated before
9/16/2024 3:37:01 PM Middleware::Authentication ERROR: Could not get security token
9/16/2024 3:37:01 PM Middleware::Authentication ERROR: Could not find file ‘C:\Program Files (x86)\FOG\token.dat’.
9/16/2024 3:37:01 PM Middleware::Communication POST URL: http://roa1fogl02.rq.priv/fog/management/index.php?sub=requestClientInfo&authorize&newService
9/16/2024 3:37:01 PM Middleware::Response Invalid security token

9/16/2024 3:37:01 PM Client-Info ERROR: Failed to authenticate, will not run Module Looper.

At this point. all clients stop logging and seem to hang - the timestamp of the fog.log file and the last entry in the log match.

In an attempt to guess what’s going on with server-to-client encryption, since the token errors appear just after the client checks the certificate, we thought maybe the FOG Client connection utilized the ssl cert in some way. We noticed that the cert only had the full server FQDN and its IP in the cert (Subject and AltSubject) (not the FOGSERVER alias we were specifying in the smartinstaller command line install run during the first reboot after imaging, so we altered the image to use the IP address and the latest version at deployment. We see no difference.

I also tried to manually install the FOG Client using both the FQDN and IP address in smartinstaller run manually - and again - still the same result.

I’m thinking is that the issue is on the server side and beyond our understanding.

Let me know if any additional info is needed and thanks for any help you can provide.

Jim Graczyk

Hi and thanks for any help anyone can provide.

I’m in the middle of a FOG server 1.5.10 migration to get off of CEntOS 7 and onto AlmaLinux 9.4 running FOG 1.5.10.

I tested this process in our lab and have overcome several issues. I followed this:
https://wiki.fogproject.org/wiki/index.php?title=Migrate_FOG

The difference is that we have several locations that also have to be migrated. I uninstalled and reinstalled the location Plugin which seemed to be required. I was able to get a 2 site deployment working correctly w images and snapins, pxe boot from the right location FOG Storage Node, etc.

The worst think I have to do was delete a Host and register with the FOG server to get it to work post imaging, aside from Reseting Host Encryption.

SO. I followed the same process (or so I think) in a production instance and have a different result. Since the lab is typically maintained to test deployments, the one thing I lacked when testing was machines that were in FOG and working pre-migration to test post migration. However, resetting a host’s encryption seemed to get all but one host to connect to FOG. Testing here was justs seeing that the fog.,log file showed proper authentication. Most testing was full deployment - image and then snapins - and these tests worked.

In the production instance, PXE booting and imagining work as expected at the 2 locations that have been migrated, but in this case, no PC is able to connect to the new FOG server - existing or reimaged.

Note: we have 8 locations, but I only migrated the FOG server and one location so far. Each location has only one Storage node - the FOG server DefaultMember at the corp HQ and a storage node at the remote site. Each storage node is in its own storage group and its own location.

We build site / domain independent images. To do this we have a DNS Alias “FOGSERVER” that points to the main FOG server A record. We use this alias in a \windows\setup\scripts\setupcomplete.cmd when running the smartinstaller.exe v0.12. at first boot.

In cases of existing hosts, I’ve have reset encryption over and again.

Oddly, in both fresh deployments of images and existing machines, all upgrade to the new FOG Service v0.13 yet the UI shows none of the host “touch” time changes.

Also in both cases (reimages and existing hosts), fog.logs show the standard No Token file / Authentication Error / Invalid security token error over and over. In 10 mins or more, the FOG client is upgraded to v0.13, but resetting the Host’s Encryption doesn’t result in the client ever getting connected. The Host shows in the UI that the OS is Windows, but all tests show “No Data” (implying the Host isn’t actually connected - which the fog.log files all show they aren’t). I’m not certain that the FOG client updates happen only after resetting encryption, but perhaps that is so.

There is a pattern in the clients - they do get upgraded, they all try to authenticate over and again, until you Reset Host Encryption. Then, after 5 attempts to connect, the FOGService on every test PC seem to stop but the service is still “running”. Restarting the FOG Service produces the same result after 5 more authentication attempts.

Here’s a typical end of fog.log:

---

9/16/2024 1:59:11 PM Client-Info Version: 0.13.0
9/16/2024 1:59:11 PM Client-Info OS: Windows
9/16/2024 1:59:11 PM Middleware::Authentication Waiting for authentication timeout to pass
9/16/2024 1:59:28 PM Controller Stop
9/16/2024 1:59:28 PM Service Stop requested
9/16/2024 1:59:28 PM Bus Emmiting message on channel: Status
9/16/2024 1:59:28 PM Middleware::Authentication ERROR: Could not authenticate
9/16/2024 1:59:28 PM Middleware::Authentication ERROR: Thread was being aborted.

We also see some message about not starting Looper.

--------------------------------Authentication--------------------------------

9/16/2024 3:35:01 PM Client-Info Version: 0.13.0
9/16/2024 3:35:01 PM Client-Info OS: Windows
9/16/2024 3:35:01 PM Middleware::Authentication Waiting for authentication timeout to pass
9/16/2024 3:37:01 PM Middleware::Communication Download: http://roa1fogl02.rq.priv/fog/management/other/ssl/srvpublic.crt
9/16/2024 3:37:01 PM Middleware::Authentication Cert OK
9/16/2024 3:37:01 PM Middleware::Authentication No token found at C:\Program Files (x86)\FOG\token.dat, this is expected if the client has not authenticated before
9/16/2024 3:37:01 PM Middleware::Authentication ERROR: Could not get security token
9/16/2024 3:37:01 PM Middleware::Authentication ERROR: Could not find file ‘C:\Program Files (x86)\FOG\token.dat’.
9/16/2024 3:37:01 PM Middleware::Communication POST URL: http://roa1fogl02.rq.priv/fog/management/index.php?sub=requestClientInfo&authorize&newService
9/16/2024 3:37:01 PM Middleware::Response Invalid security token

9/16/2024 3:37:01 PM Client-Info ERROR: Failed to authenticate, will not run Module Looper.

At this point. all clients stop logging and seem to hang - the timestamp of the fog.log file and the last entry in the log match.

In an attempt to guess what’s going on with server-to-client encryption, since the token errors appear just after the client checks the certificate, we thought maybe the FOG Client connection utilized the ssl cert in some way. We noticed that the cert only had the full server FQDN and its IP in the cert (Subject and AltSubject) (not the FOGSERVER alias we were specifying in the smartinstaller command line install run during the first reboot after imaging, so we altered the image to use the IP address and the latest version at deployment. We see no difference.

I also tried to manually install the FOG Client using both the FQDN and IP address in smartinstaller run manually - and again - still the same result.

I’m thinking is that the issue is on the server side and beyond our understanding.

Let me know if any additional info is needed and thanks for any help you can provide.

Jim Graczyk

@wayne-workman

Update:

Replication wasn’t responsible for any of the traffic we’re seeing. We disable replication in the FOG Configuration area and still saw the same 2 GB per day per FOG Storage Node. Our bandwidth monitor showed the traffic to be both TCP and HTTP. We then stopped the services FOGImageReplicator, FOGSnapinReplicator, and FOGSnapinHash services. Still no change. We saw a constant 160-190kb/s traffic from the Main FOG server to the Storage Node. So, I captured packets - only 1000, but that took only 5 seconds. I saw what looked like the storage node replication log transmitted from the storage node to the main FOG Server. Most of the traffic was from the main FOG server to the storage node, but I wouldn’t tell much from the packet.

In the end, I was able to stop almost all traffic by shutting down all FOG services, including the FOGScheduler.

So the questions now are:

Is there any documentation that explains what each service does and what we’re breaking by turning each off?

Are the negative effects mitigated by running any of the service daily?

Any assistance is appreciated.

Jim Graczyk

@wayne-workman

Wow!!! I never would have expected that. I would assume an MD5 hash on each side would be the best way to go.

My design philosophy is always to go as thin as you can manage. It’s based on the notion that even though GB Ethernet is everywhere and my internet connection is 300 x 30, CPUs outperform disk, disks outperform networks, and WAN connection will remain a problem for the rest of my life.

At present, we’re working over VERY thin pipes, but for my future reference, is there any way to manage the frequency of image and snapin replication tasks? If I have a lot of images, I’d likely set replication for once a day, at most, especially with many locations.

Thanks for all your help.

Preliminary info indicates we may have made FOG traffic a footnote compare to all inter-site traffic, instead of being the top user by a factor of 10.

Thanks - more later…
Jim Graczyk

@george1421

Thanks for this fine detailed info. Our possible fallback position might be a full fog server at the remote site, but we have only a dozen or less PC per site. I don’t look forward to trying to keep 5 FOG servers config’d identically.

Also, we are aware that each FOG server has it’s own cert, so moving PCs from server to server would be a challenge. Half of our PCs are laptops with users moving from location to location. With one FOG server, we can easily move a PC to a different location (where ever the user is that day), and re-image. Our approach to Snapins is automatically site aware (DFS and SAMBA), and our FOG Snapin is just a generic 15K stub. All our snapins are accessed over SMB. We rely on FOG to manage and execute Snapins, but not so much store them.

Our bandwidth monitoring is showing very little traffic from the FOG client to the Main FOG server. Personally, I can’t understand how or why the Main FOG server would transmit 1 GB in 12 hours to a server that is in sync. I wondered about the Snapin Hash Global Enable, but our 40 snapins amount to less than 1 MB of data, so I don’t see it there.

We’ve used the FOG Linux Service Enable section of FOG Configuration to uncheck these parameters:
IMAGEREPLICATORGLOBALENABLED
SNAPINREPLICATORGLOBALENABLED
SNAPINHASHGLOBALENABLED

We’ll report back our results after a day.

thanks,

Jim Graczyk