@Tom-Elliott All is working now

Hey Tom, updated to 1.6.0.97 this morning and everything checks out:

Exports: migrated to /etc/exports.d/fog.exports, and my kubuntu.exports came through untouched.
SSH keys: fog@fogger key landed in authorized_keys, works on my custom port. Pulled the Match password exception, so the box is key-only again.
Dashboard: 1.5 history is back, and the Nov 3 runs are on Nov 3 now.
Capture: new capture shows local time, no asterisk, and moved out of /images/dev over the key with no errors in the log.
Ports: FOG_SSH_PORT now shows up under FOG Settings → General Settings with my 2416, and the per-node fields are blank, so it falls back to that.

Seriously impressive turnaround. Thanks for taking the reports seriously and running with them!

@Tom-Elliott Hey Tom, updated to 1.6.0.97 this morning and everything checks out:

Exports: migrated to /etc/exports.d/fog.exports, and my kubuntu.exports came through untouched.
SSH keys: fog@fogger key landed in authorized_keys, works on my custom port. Pulled the Match password exception, so the box is key-only again.
Dashboard: 1.5 history is back, and the Nov 3 runs are on Nov 3 now.
Capture: new capture shows local time, no asterisk, and moved out of /images/dev over the key with no errors in the log.
Ports: FOG_SSH_PORT now shows up under FOG Settings → General Settings with my port, and the per-node fields are blank, so it falls back to that.

Seriously impressive turnaround. Thanks for taking the reports seriously and running with them!

@Tom-Elliott Awesome, thanks Tom! Didn’t expect all that so fast. Glad the exports.d thing worked out, and stoked about key auth. Can’t wait to ditch the password workaround.

I’ll update when it hits Patches and let you know how it goes.

@Tom-Elliott Thanks Tom! Glad the SSH_PORT report turned up the other six settings too.

For the exports, I went a different route that keeps FOG managing its own lines: NFS also reads /etc/exports.d/*.exports, so I put the Kubuntu export in /etc/exports.d/kubuntu.exports. The installer rebuilds /etc/exports but never touches that folder. I’ll confirm it survives the next update.

Follow-up: I think the cause of 1b (post-upgrade runs missing from the chart).

taskLog.createTime is a TIMESTAMP, so with the session at +00:00 it comes back as UTC (my 9:27 PM EST run reads as 2026-11-04 02:27). But WindowedStats::dailySeries() calls toDisplayStored($value) with the default $isDatetime = true. Inside the 26-hour band, that treats the UTC value as pre-boundary America/New_York time, puts it on Nov 4, past the end of the series, and drops it.

The same applies to every chart that uses dailySeries() on a TIMESTAMP column: imaging (taskLog.createTime), storage (images.imageDateTime), inventory (inventory.iCreateDate) and fleet (hosts.hostCreateDate). Audit and snapin use DATETIME columns, so they’re fine.

Suggested fix: add an $isDatetime parameter to dailySeries(), pass it through to toDisplayStored(), and have the four TIMESTAMP callers pass false:

// WindowedStats::dailySeries(): new last parameter
        $countK = 'c',
        $isDatetime = true
    ) {
...
            $day = self::toDisplayStored(
                (string)$row[$dateK],
                $isDatetime
            )->format('Y-m-d');

// ImagingStats::runsPerDay(): taskLog.createTime is a TIMESTAMP
        return self::dailySeries($rows, $start, $end, 'd', 'c', false);

// StorageStats, InventoryStats, FleetStats: same, add 'd', 'c', false

Here is a full patch against current dev-branch (5 files, +30/−6) if that’s helpful. I haven’t applied it to my server (yet), so it’s untested beyond reading the code. I thought I’d run it by the forum first for feedback. See attached.

Thanks

fog-dashboard-timestamp-fix.patch.txt

Confirmation: after midnight the chart now shows 3 runs on Nov 4, all of which actually ran on the evening of Nov 3 (8:52, 9:27 and 10:07 PM EST). So the runs weren’t lost; they were being placed a day late, off the end of the series until the window rolled over. That matches the UTC TIMESTAMP being read as pre-boundary local time in dailySeries().

TLDR;
FOG read last night run times in UTC, then mistakes them for New York time, which pushes them a day later.

Two related issues after upgrading from the final 1.5 dev build to 1.6.0.69, both apparently tied to the UTC storage change.

1. The dashboard “Imaging Over the last” chart shows 0 for every day

taskLog has completed runs on several days, but the chart is flat at zero. The get30day response confirms it: 0 for every date, including today.

a) Pre-upgrade runs are excluded. The fold in src/Audit/ImagingStats.php (_foldSql()) requires:

AND `logImageName` <> ''

Every taskLog row written before the upgrade has a blank logImageName (and logTaskTypeName), so all 1.5-era history is dropped:

day         logImageName  COUNT(*)
2026-10-07                4
2026-10-10                4
2026-10-12                6
2026-10-27                2
2026-10-28                2
2026-10-30                4
2026-11-03                1
2026-11-03  BigBox        5

Could the upgrade backfill logImageName from tasks/images, or could the fold fall back to the task’s image when it’s blank?

b) Post-upgrade runs are also missing. Running the fold’s SQL by hand with the same window returns today’s three runs, all named BigBox:

taskID  started
11      2026-11-03 20:52:14
12      2026-11-03 21:27:37
13      2026-11-03 22:07:13

But get30day returns [“2026-11-03”, 0]. These runs fall within 26 hours of the UTC storage boundary, and their taskLog.createTime values are in server-local time (America/New_York) rather than UTC, which may be related. In the same tasks rows, taskCreateTime is local while taskCheckIn and taskStateChangedTime are UTC.

2. New captures show UTC times with a misleading tooltip

Under Images → List All Images, captures made after the upgrade show the Captured time in UTC with an asterisk (2026-11-04 03:22:56 * for a capture that finished at 10:22 PM Eastern). The tooltip says “recorded before this server moved to UTC; written in America/New_York”. On 1.5, the same server showed local times.

From src/Base/StorageEpoch.php, I understand this is the BAND_HOURS = 26 window: values near the boundary are treated as pre-boundary and shown unadjusted. That makes sense, but for those in-band values the tooltip says they were written in America/New_York when they’re actually UTC. Wording like “recorded near this server’s move to UTC; time zone uncertain” would be more accurate.

Server time zone settings, for reference: timedatectl, FOG_TZ_INFO, PHP and MariaDB are all America/New_York. (MariaDB’s time zone tables aren’t loaded, but I see FOG deliberately avoids CONVERT_TZ().)

Version: 1.6.0.69 (Patches channel), Ubuntu 26.04 Server VM

The upgrade itself went smoothly, but two things broke afterward.

1. The Kubuntu menu entry lost its NFS export. My Kubuntu entry boots the live ISO over NFS: the ISO is extracted to /srv/kubuntu, the kernel and initrd are served over HTTP, and the kernel line uses boot=casper netboot=nfs nfsroot=[fog server IP] :/srv/kubuntu. That needs its own line in /etc/exports:

/srv/kubuntu *(ro,sync,no_subtree_check,insecure,fsid=2)

The 1.6 installer rewrote /etc/exports (it saved my original as /etc/exports.<timestamp>) and kept only the /images and /images/dev exports, so the Kubuntu entry broke. Re-adding the line and running exportfs -ra fixed it. The menu entry itself carried over fine.

It would be nice if the installer preserved non-FOG lines in /etc/exports instead of replacing the file. Anyone else serving extra NFS shares from their FOG box will hit this on every update.

2. Captures stuck in /images/dev. After the upgrade, captures finished but never moved out of /images/dev/<MAC>. The Apache error log showed:

ssh2_connect(): Unable to connect to [fog server IP] on port 22

1.6 now does the post-capture move over SSH, and my sshd runs on a non-standard port with key-only auth. Two fixes were needed:

FOGSSH.php reads a FOG_SSH_PORT setting, but it didn’t exist in my database, so I (Claude) inserted it:

sql

INSERT INTO globalSettings (settingKey, settingDesc, settingValue, settingCategory)
VALUES ('FOG_SSH_PORT', 'SSH port FOG uses to reach storage nodes', 'custom port of ssh server', 'General Settings');

FOG only authenticates with a password (auth_password), so I allowed password logins for the service account from the server itself only:

Match User fogproject Address [fog server IP],127.0.0.1
    PasswordAuthentication yes

Suggestions: expose FOG_SSH_PORT in FOG Settings by default, have the installer detect sshd’s port, and mention the SSH requirement in the upgrade notes, since FTP alone isn’t enough anymore.

Question: is SSH key authentication on the roadmap for FOG’s storage node connections? Right now FOGSSH.php only uses auth_password, so I had to carve out a password-login exception for fogproject on a box that’s otherwise key-only. Supporting ssh2_auth_pubkey_file, with a key pair the installer generates for the service account, would let servers keep password authentication disabled entirely. Is that possible today through a setting I’m missing, or would it need a code change?

Thanks

@maxcarpone 1.5.10.2402 is out. It works. I was able to capture from both my PC and laptop successfully. But it has a weird issue with null tasks.

https://forums.fogproject.org/topic/18230/1-5-10-2402-additional-null-tasks-created-but-this-time-manages-to-start-a-transfer

1.5.10.2402 - Additional null tasks created but this time manages to start a capture.

@maxcarpone I had the same problem with 2390 and reverted to an older back up. That version was broke AF

@Tom-Elliott Thanks Tom. 1.5.10.2344 seems to be capturing fine. No additional null tasks were created and the PC had no problem checking in and starting the capture.

I’d swear, not only with this version but the earlier ones that were malfunctioning during check in and creating additional tasks, I had to create the task twice. Like I went through the process of setting up a capture but nothing showed up under tasks then had to do it again.

I can’t seem to duplicate it though. Setting up a 2nd capture for the laptop, I only had to do it once (and no additional null tasks were created).

PC finished the capture no problem and laptop checked in and is capturing now. Looks like 2344 did it!

Thanks again.

@Tom-Elliott Hi Tom, Unfortunately, updating to the latest 1.5.10.2340 gave me the same problem. This time, I tried capturing an image from the PC. I managed to take a screenshot of the strange other Tasks added.

The PC is refusing to check in. LMK if you need any logs and which logs. IIRC, do a debug task. If you can point me to any documentation on where the logs are and how to get the logs you want, that would be great.

I captured from both of my PC’s on 8/19 dev version 2294. Tonight, early am, I updated to 2328, the update appeared to go fine. I set a task to capture from my laptop, (not sure if this is important) before I rebooted the laptop I checked the Tasks list in the web UI from another PC and saw a bunch of other tasks labelled ‘fog’ with parentheses along with the capture task for the laptop. I should have taken a pick. I removed the other tasks leaving only the one for the laptop then rebooted the laptop. Laptop would not check in. Over and over it tried.

I deleted the task for the laptop, rebooted the FOG server and set up a capture for the laptop again. Rebooted the laptop, still the laptop was unable to check in and kept trying over and over.

went back to 2294 all good.

I suspect something is borked in this version. Feel like a POS for not gathering logs but it is late.

@Tom-Elliott Tom, looks like everything has been resolved after going from 2144 to 1.5.10.2218! No errors and I was able to capture no problem. Thanks

@Tom-Elliott Hello Tom, I had posted a pic of the RAM0 error. Since I reverted to a backup of 2144. I no longer get any errors so I can’t take a new pic but here is the original pic with 2191 installed - (see below). I will try 1.5.10.2213 as soon as I can. Perhaps tomorrow evening?

The laptop is booting UEFI. It runs windows 11 with secure boot disabled and TPM requirements disabled during install.

Here is a text translation of the error in case you are unable to view attached photo.

RDSEED32 is broken. Disabling the corresponding CPUID bit.
ACPI BIOS Error (bug): Failure creating named object [_SB.PCI0.GPP5.WLAN._PRR],
AE_ALREADY_EXISTS (20250807/dswload2-326)
ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20250807/psobject-220
)
pcieport 0000:01:00.0: Unable to change power state from D3cold to D0, device in
accessible
pcieport 0000:02:00.0: Unable to change power state from D3cold to D0, device in
accessible
Kernel panic - not syncing: VFS: Unable to mount root fs on “/dev/ram0” or unknown-block(1,0)
Kernel Offset: disabled
—[ end Kernel panic - not syncing: VFS: Unable to mount root fs on “/dev/ram0” or unknown-block(1,
0) ]—

“2144 good, 2191 bad”

Yes, that is the kind of in-depth analysis I am known for.

@Tom-Elliott I’m sorry. I restored a previous version of the FOG server and no longer have any error logs from when all this happened. I can go to 1.6… IIRC there is a change that needs to be made to SSHD conf or something. The last time I went to 1.6 FOG was unable to see the default storage node.

Everything working fine until this latest FOG update. Using a Strix Point laptop with a USB Ethernet adaptor. Also trying to install different kernels through the web UI no longer works. HTTP ERROR 500

I tried installing master branch. Failed, couldn’t even boot to memtest. I restored the FOG VM from a back up made a few days ago. 1.5.10.2144 - everything is working as normal now. Afraid to update

Just thinking, sometime between the 1.5.10.2144 backup, there were tons of dev updates, and one time when I did a git pull on FOG, it deleted a bunch of things. But since the gui worked, I assumed it was okay