Feature request: a "Troubleshooting" menu in the web UI to find hosts that fail to authenticate

[lperoma]

Hello everyone,

First of all, thank you for FOG. We run it in a school (about 320 hosts, FOG 1.5.x on Debian 12) and it is a core part of how we deploy and maintain our PCs.

I would like to propose a feature: a “Troubleshooting” (or “Host health”) menu in the web UI that lets an administrator find, in a few clicks, which hosts are failing to authenticate and what to do about it. Today this requires digging through server logs and the database, which is complicated and not realistic for most FOG admins. We had to write a script to do it, and I am sharing what it does as a starting point for the design.

The problem

FOG does not tell you when a host fails to authenticate. Nothing appears in the web UI or in any error log, because the server answers these requests with HTTP 200 even when it rejects the login. We only found out when somebody stood in front of the machine. Two cases kept hitting us:

1. Rejected login in the iPXE menu. The user picks Deploy Image, types the credentials, the screen flickers and the menu comes back with no message. Server side, the only trace is the menu (boot.php) being requested again a few seconds later, and the kernel (bzImage / init.xz) never being downloaded. Our working theory (not fully confirmed yet) is the keyboard layout: iPXE uses a US layout, so passwords with symbols typed on another layout would arrive differently.
2. Windows FOG client that cannot authorize. The host retries forever. The database shows no security token, or one that has not been renewed for days (hostSecToken / hostSecTime), and the client’s fog.log shows token or certificate errors. The usual fix is Reset Encryption Data, but nobody knows which hosts need it.

What the menu should show

Each item is something we currently detect by hand:

• Hosts whose FOG client cannot authorize: missing or stale security token, or an abnormal number of authorization requests (a retry loop). Suggested action: Reset Encryption Data, then check fog.log on the client.
• Hosts that fail the PXE login: repeated rejected logins without the kernel ever being downloaded. Suggested action: check the keyboard layout and the credentials.
• Hosts with no contact for N days (configurable): so that machines that no longer exist can be removed and machines in use whose client silently stopped can be fixed.
• For each one: host name, IP, since when, and the suggested action next to it.
• Optionally, an e-mail digest through the existing notification mechanism.

How our script detects it (for reference)

A read-only shell/awk script that runs weekly from cron on the FOG server and e-mails a report. It never writes to FOG. It uses:

• Apache access log (other_vhosts_access.log plus the rotated .gz files):
  • Rejected PXE logins: two POSTs to boot.php from the same IP within 120 s, with no kernel download in between. A host is listed after 3 rejections in the period.
  • Client not authorizing: we take the most frequent response size of /management/index.php?sub=requestClientInfo and of the authorize POSTs as “normal”, and flag IPs with several abnormal responses, or with too many authorizations in a single day (a normal host does about 47 a day; we flag above 100, which means a retry loop).
  • Mapping IP to host: the mac= parameter of requestClientInfo links an IP to the host MACs in the database.
• FOG database (hosts, hostMAC) : the host token and its last renewal time. A host is flagged when its token is missing or older than its last failed request. It also gives the hosts with no sign of life for more than 15 days.

Example of the summary it produces (host names removed):

OVERALL STATUS: NEEDS ATTENTION

SUMMARY
  Hosts whose FOG client cannot authorize....... 4
  Hosts that could not log in on PXE............ 4
  Hosts with no contact (over 15 days).......... 64
  Hosts active this week........................ 189

1) HOSTS THAT CANNOT CONNECT TO FOG (Windows client)
   The client cannot authorize against the server.
   -> In the FOG web UI, open the host and click "Reset Encryption Data";
      then check the FOG log on the client.
   - HOST-A, since 03 Nov
   ...

In a few weeks it found hosts with broken tokens that nobody had reported, a laptop that repeatedly failed the PXE login, and several machines in daily use whose FOG client had silently stopped working.

Why it belongs inside FOG

Parsing Apache logs is fragile and only works because we know the response sizes by heart. FOG already has the information: the hosts table, the token times and the requests it serves. The simplest native design would be to record an event (or a counter on the host) whenever a PXE login or a client authorization is rejected, with the reason, and build the menu on top of that.

You are very welcome to take our script and keep it, use it, change it or build on it however you like, with no conditions. We do not have a public code repository, so if any developer wants it, reply here or send me a private message and I will share it. I can also explain the heuristics and thresholds in more detail, share anonymised log samples, or test a patch or plugin on our server if that helps.

Thank you again for your work.

[Tom Elliott]

@lperoma Thank you for the detailed write-up. The detection notes made this quick to scope.

FOG 1.6 covers this as follows:

FOG Client cannot authorize. This was the real gap, and a pull request now adds it: https://github.com/FOGProject/fogproject/pull/2089. The server records each host whose client fails the handshake: since when, how many times, the source address and the reason. A new list report, Reports > Client Authentication Failures, shows them with the fix next to each one. A host leaves the list when its client authenticates again. Two reasons are recorded:

• The security token does not match. Reset Encryption Data fixes this.
• The server cannot decrypt the client’s key. This means the certificate the server publishes for clients does not pair with its private key, so every host fails at once. A reset does not help. Running the installer with --recreate-keys issues a new pair.

One case stays invisible to the server. A client that trusts a different FOG CA refuses the server’s certificate and never sends a login, so there is nothing to record. Its fog.log shows “Certificate is not from FOG CA”.

Rejected iPXE menu logins. 1.6 already records each rejected login in the audit trail, with the username and the address. Look under Reports > Audit Report.

Hosts with no contact. 1.6 stores the last check-in and the last ping for each host. Fleet Report shows the last check-in, so you can sort on it.

All of this is 1.6 only. 1.5.x does not get these changes, so your script remains the answer there until you move to 1.6.