A reimaged host stays stuck on an old directory join plan ("marker absent; automatic retry refused")
-
Hi Tom and everyone,
First, thanks for the quick work on the shared adapter and enrollment issues. I tested #2116/#2118 today and enrollment behaves now. While doing that I hit something else with the directory join that I think is worth a thread of its own.
Setup: FOG 1.6.0.158 stable moved today to the patches channel (schema 465, so it includes #2111, #2112, #2113, #2116 and #2118). Two Proxmox VMs (test-uefi and test-bios) deployed from the same image, whose source machine already has agent 0.1.28. Both hosts are set to join AD and both had been deployed once before today.
After the second Deploy, test-bios joined the domain without any trouble. test-uefi never did. It kept the Windows name “instalacion”, stayed in the workgroup, and installed its software normally (Chrome, VLC, 7-Zip and Avast, before the activation snapin, which is the order we wanted). The agent log shows this every minute, and nothing else about the directory:
2026-11-20T19:24:29+01:00 this state was made on another machine (a cloned or reimaged disk); generated a new key and enrolling as this machine 2026-11-20T19:24:29+01:00 enrolled as host 666, certificate valid until 2027-11-20 18:24:29 2026-11-20T19:26:01+01:00 facts: sending inventory=true software=6 directory=true printers=4 network=1 2026-11-20T19:26:04+01:00 directory rejoin: directory rejoin marker absent; automatic retry refused 2026-11-20T19:28:55+01:00 directory rejoin: directory rejoin marker absent; automatic retry refusedSo the very first directory poll after the reimage was already refused. On the server,
hostDirectory.hdRejoinStatefor that host had this (trimmed):"phase":"awaiting_reboot", "attempt_id":"add87a0f...", "trigger_task_id":16923, "last_deploy_task_id":16923, "retry_authorized":false, "placement_hold":trueThe Deploy that had just reimaged the machine was task 16927, newer than 16923, but it did not replace that plan. The computer account was fine in AD (it existed and belonged to the account FOG uses to join), so it was not an account problem.
Reading the code, this looks like the intended protection:
PROTECTED_PHASESis started, awaiting_reboot, unknown and failed, and only a plan inreadyis dropped when a newer Deploy arrives (dropSupersededPlan, reasondirectory_task_superseded). That makes sense for the old disk, where the join may really have happened. But after a Deploy the old disk is gone. The plan waits for a local marker and a reboot that can no longer exist, the new install has no marker, so the agent refuses, and the server keeps waiting for the agent. Neither side ever gives up.What happened to this host before is simple: it had a join plan pending that never actually ran, I reinstalled it, and it got stuck on that old plan. I can’t show you the log of the earlier install, because the reimage overwrote it, so the odd part is that the stored state says awaiting_reboot for something that, as far as I know, was never executed. test-bios did not get stuck, and its previous plan had ended in “verified”.
The way out is the “Clear directory operation review” checkbox, and here is where I got a second surprise.
clearReview()writes a new state withlast_deploy_task_idset to the latest Deploy and dropsworkgroupandobservation. As far as I can tell, with the Deploy counted as already used and no workgroup flag, nothing gets planned afterwards. On a real PC earlier today, ticking the box left the host with no plan at all. I added"workgroup":trueby hand and FOG did plan a join, but that agent never ran it and I joined the machine manually.For test-uefi I wrote the cleared state by hand instead:
reason: operator_cleared,workgroup: true, andlast_deploy_task_idset to the previous Deploy (16923), so that 16927 was still pending. On the next poll FOG created a new plan (ready, trigger 16927), the agent ran it (awaiting_reboot, no error), and after a reboot it went toverifiedwith the host joined. So the join itself works, and only the stuck state was the problem.What I would suggest, if you agree:
- A new Deploy for a host should clear everything that was pending for the previous installation: queued tasks, snapin tasks and directory join plans, not only plans in
ready. This includes plans in started, awaiting_reboot, unknown and failed, because they refer to a disk that has been replaced. Whatever the Deploy itself creates (for example the snapins attached to it) would of course stay. I only checked the join plans; I have not tested what happens to old queued tasks and snapins when a Deploy is created, so please take that part as a suggestion. - The review checkbox should keep
workgroupandobservation, and should not mark the latest Deploy as used, so the next poll can plan the join again. - When the agent says “marker absent; automatic retry refused”, the host’s Active Directory panel could say that this needs the checkbox. Right now the only hint is in the agent log on the machine.
Open question on my side: I don’t know why that first plan was in awaiting_reboot if it never ran. If there is something I should capture the next time (agent log before reimaging, a particular field of the state), tell me and I will.
Thanks!
- A new Deploy for a host should clear everything that was pending for the previous installation: queued tasks, snapin tasks and directory join plans, not only plans in
-
@lperoma Thanks for the detailed report. Your reading of the code is right. I opened https://github.com/FOGProject/fogproject/pull/2122 to fix both problems.
On your open question: the first attempt did run. The server stores awaiting_reboot only when the agent reports that Windows accepted the join. Your state also has last_deploy_task_id equal to trigger_task_id (16923), and that is set only when the agent claims the plan. So the join ran after Deploy 16923, and Windows waited for a restart. The disk was reimaged by 16927 before that restart, so the agent never reported “verified”. That also explains why the computer account in AD looked healthy.
What changes in #2122:
- A newer Deploy now supersedes an old attempt in started, awaiting_reboot, unknown or failed. Three things must be true: the agent has no marker for the old attempt, a newer Deploy has completed, and the machine booted after that Deploy ended. FOG then drops the old attempt, writes an audit entry, and plans a new join from the new Deploy. If the agent still has the marker, the old attempt stays.
- The “Clear directory operation review” checkbox no longer marks the newest Deploy as used. It uses up only the cleared attempt’s own Deploy. It also keeps the workgroup flag and the observation. If a newer Deploy exists, the next check-in can plan the join again.
- With the first change, a reimaged host no longer needs the checkbox, so I did not add a hint to the Active Directory panel.
On queued tasks and snapins from the previous install: this PR changes only the directory join plans.
The fix reaches the patches channel after the PR merges. Your hand-written state on test-uefi does no harm.
-
Thanks Tom, I tested #2122 and it does what you described.
I applied the two
srcfiles of the PR by hand on top of thepatcheschannel (schema 465, agent 0.1.28) and reproduced the same situation with test-bios. First Deploy (16928): the agent asked for the directory join and the plan went toawaiting_reboot(attempt12f448e3…,trigger_task_id16928). Before it rebooted I powered the VM off and queued a second Deploy (16929), which completed normally. The agent enrolled by itself after that (rebind, approved via deploy).For a while the old plan was still in
hdRejoinState, stillawaiting_rebootwith trigger 16928, so right after queuing the Deploy nothing had changed. Once the machine booted the new Windows and the agent checked in, FOG replaced it on its own. A minute later there was a new attempt (3e64f53…, trigger 16929,awaiting_reboot). I didn’t touch “Clear directory operation review” or edit anything in the database. After one reboot the host wasverified,hdJoined=1,hdKind=ad,secure_channel: healthy.So the old attempt is dropped after the new Deploy and the first boot, not at the moment the Deploy is created, which matches what you wrote. It only looks odd if you check the host in between and still see the old plan. No problem on my side, just mentioning it in case anyone else notices the same thing.
One idea, only if you think it’s worth it: a checkbox in the Deploy task form, something like “discard every pending task of this host”, unchecked by default. The automatic rules are safer for the normal case, but sometimes the admin knows everything left over from the previous install is dead (queued tasks, snapins, a pending directory join) and wants to force a clean start right when the Deploy is queued, without waiting for the first boot and without the “Clear directory operation review” box. In our case, when a machine misbehaves it often keeps snapin tasks queued, and today you have to open the host, work out which tasks it has and delete them one by one. In a big fleet that is slow and easy to get wrong, whereas one box on the Deploy would clean the slate in a single step.
I’ll drop my manual patch when #2122 reaches
patches.