1.6 upgrade: /etc/exports overwritten, captures stuck in /images/dev (SSH on non-standard port)

[Fog_Newb]

The upgrade itself went smoothly, but two things broke afterward.

1. The Kubuntu menu entry lost its NFS export. My Kubuntu entry boots the live ISO over NFS: the ISO is extracted to /srv/kubuntu, the kernel and initrd are served over HTTP, and the kernel line uses boot=casper netboot=nfs nfsroot=[fog server IP] :/srv/kubuntu. That needs its own line in /etc/exports:

/srv/kubuntu *(ro,sync,no_subtree_check,insecure,fsid=2)

The 1.6 installer rewrote /etc/exports (it saved my original as /etc/exports.<timestamp>) and kept only the /images and /images/dev exports, so the Kubuntu entry broke. Re-adding the line and running exportfs -ra fixed it. The menu entry itself carried over fine.

It would be nice if the installer preserved non-FOG lines in /etc/exports instead of replacing the file. Anyone else serving extra NFS shares from their FOG box will hit this on every update.

2. Captures stuck in /images/dev. After the upgrade, captures finished but never moved out of /images/dev/<MAC>. The Apache error log showed:

ssh2_connect(): Unable to connect to [fog server IP] on port 22

1.6 now does the post-capture move over SSH, and my sshd runs on a non-standard port with key-only auth. Two fixes were needed:

FOGSSH.php reads a FOG_SSH_PORT setting, but it didn’t exist in my database, so I (Claude) inserted it:

sql

INSERT INTO globalSettings (settingKey, settingDesc, settingValue, settingCategory)
VALUES ('FOG_SSH_PORT', 'SSH port FOG uses to reach storage nodes', 'custom port of ssh server', 'General Settings');

FOG only authenticates with a password (auth_password), so I allowed password logins for the service account from the server itself only:

Match User fogproject Address [fog server IP],127.0.0.1
    PasswordAuthentication yes

Suggestions: expose FOG_SSH_PORT in FOG Settings by default, have the installer detect sshd’s port, and mention the SSH requirement in the upgrade notes, since FTP alone isn’t enough anymore.

Question: is SSH key authentication on the roadmap for FOG’s storage node connections? Right now FOGSSH.php only uses auth_password, so I had to carve out a password-login exception for fogproject on a box that’s otherwise key-only. Supporting ssh2_auth_pubkey_file, with a key pair the installer generates for the service account, would let servers keep password authentication disabled entirely. Is that possible today through a setting I’m missing, or would it need a code change?

Thanks

[Tom Elliott]

@Fog_Newb Thanks for the detailed write-up. Taking them in order:

1. /etc/exports. This is long-standing installer behavior: it rebuilds the file and keeps the old one as /etc/exports.<timestamp>. To keep your own exports, run the installer once with -E (--no-exportbuild). That writes STORAGE_rebuild_nfs_exports='no' to /opt/fog/.fogsettings, and later updates leave the file alone. If you do that, you maintain the /images and /images/dev lines yourself.

2. FOG_SSH_PORT missing. That is a bug, and your diagnosis is right. A 1.5 database arrives at a schema step count past the 1.6 step that creates FOG_SSH_PORT, so an upgraded server never gets the row. Six other settings had the same problem. The fix is https://github.com/FOGProject/fogproject/pull/2057: the updater now adds any of those rows that are missing. Your hand-inserted row is fine and stays as it is. Once the fix is in, the setting shows under FOG Settings > General Settings. Note that it is one port for every node FOG connects to.

You are also right that 1.6 moves the finished capture over SSH, where 1.5 used FTP. So the storage node user needs working SSH from the FOG server.

3. Key authentication. Not today. FOGSSH only does password auth, and no setting changes that. Your Match User fogproject Address ... block is the right way to keep the rest of the box key-only in the meantime. Key-based auth would need a code change.

[Fog_Newb]

@Tom-Elliott Thanks Tom! Glad the SSH_PORT report turned up the other six settings too.

For the exports, I went a different route that keeps FOG managing its own lines: NFS also reads /etc/exports.d/*.exports, so I put the Kubuntu export in /etc/exports.d/kubuntu.exports. The installer rebuilds /etc/exports but never touches that folder. I’ll confirm it survives the next update.

[Tom Elliott]

@Fog_Newb A follow-up: two parts of my earlier reply are already out of date, and your exports.d idea is now how FOG will do it.

1. /etc/exports

Your /etc/exports.d approach is the right one, and the installer will now work the same way. The change is https://github.com/FOGProject/fogproject/pull/2061. It is in review, so it is not in a release yet.

How it works:

• FOG writes its two exports (/images and /images/dev) to its own file, /etc/exports.d/fog.exports. It rewrites only that file on each install.
• The installer no longer rewrites /etc/exports.
• On the first install with this change, it removes only the exact /images and /images/dev entries from /etc/exports, so they are not exported twice. Every other line stays as it is. It saves the old file as /etc/exports.fog-migrate.<timestamp> and prints that name.
• It never touches any other file in /etc/exports.d, so your kubuntu.exports stays as it is. Your fsid=2 does not clash, because FOG uses fsid=0 and fsid=1.
• -E (--no-exportbuild) still means “leave NFS alone”.
• An uninstall removes fog.exports and leaves your files alone.

You do not need to change anything. Your current setup keeps working before and after this change.

2. SSH key authentication

I said this needed a code change. That change is now in: https://github.com/FOGProject/fogproject/pull/2060 (issue #2058).

How it works:

• The installer generates a key pair at /etc/fog/ssh/fog_ssh_key. It adds the public key to the storage node user’s ~/.ssh/authorized_keys on the same server.
• A storage node install fetches the same public key from its master and authorizes it. The request is signed with the storage node password, and the master answers only nodes it has registered.
• FOG tries the key first and falls back to the password, so a server without the key keeps working as before.
• Saving a storage node in the web UI tests both the password and the key, and tells you which one fails.

It is on dev-branch now, and it reaches stable with the next stable update. After you update and run the installer:

• Check that the storage node user’s ~/.ssh/authorized_keys holds a fog@<hostname> key.
• Remove your Match User fogproject Address ... block and reload sshd.
• Run one capture to confirm the image moves out of /images/dev.

3. SSH port

I said FOG_SSH_PORT is one port for every node. That will change: each storage node gets its own SSH Port and FTP Port fields in Storage Node Management. That work is in progress, and I will post the link when it is up.

• A node with no port set uses the global setting, FOG_SSH_PORT or FOG_FTP_PORT. If the global setting is not set either, the defaults are 22 and 21.
• The FTP field sets only the control port (21 by default). FOG uses passive mode, so it needs no separate data-port setting.
• On a single server like yours, the global setting you already have is enough.

[Fog_Newb]

@Tom-Elliott Awesome, thanks Tom! Didn’t expect all that so fast. Glad the exports.d thing worked out, and stoked about key auth. Can’t wait to ditch the password workaround.

I’ll update when it hits Patches and let you know how it goes.

[Tom Elliott]

@Fog_Newb The per-node ports change has landed: https://github.com/FOGProject/fogproject/pull/2062

Each storage node now has an SSH Port field and an FTP Port field in Storage Node Management. If you leave a field blank, the node uses FOG_SSH_PORT or FOG_FTP_PORT. If that setting is not set either, it uses 22 or 21. Every SSH and FTP connection to that node uses its port. That includes the capture move, replication, snapins, the file deleter, the log viewer and the dashboard check.

It is on dev-branch now, and it reaches stable with the next stable update, along with the exports and key changes. On your single server, the global setting you already have is enough.