FOG install overwrites /etc/exports with no warning

[mechalas]

We run nightly backups that keep several weeks of history and restoration took only a few minutes. Only three clients got wedged to the point where they had to be rebooted.

But we were lucky because we only have three dozen or so clients in our lab. In my previous role, I was part of a team that supported roughly 20,000 clients. Imagine the havoc that gets wreaked when an exports file with a few hundred filesystems in it goes away. Batch jobs start hanging, user login sessions hang, and so on. In most cases, it’s just a stale NFS filehandle message, but not all applications are tolerant of an outage of minutes to hours (depending on how long it takes for the issue to get noticed).

I appreciate that it’s a small development team, but this stuff has real impact. Overwriting a file that is not owned by the software and is a critical, shared resource rubs me the wrong way, especially when it takes down infrastructure. And yes, I saw the help, but the use of inconsistent language makes it easy to make faulty assumptions about what the installer is actually doing:

	-E    --no-exportbuild		Skip building nfs file
	-P    --no-pxedefault		Do not overwrite pxe default file

One option speaks of building, another says overwrite. The latter sounds dangerous, the former does not. But, more to the point, sometimes I make mistakes because my life is not 100% error-free, even when I am paying close attention. Software can help stop me from doing something I shouldn’t.

Here is a simple solution for the installer that would mitigate this problem: grep the file for the lines you want to add, and if you don’t see them, use echo with an append:

if grep '^my line regex' /etc/exports > /dev/null 2>&1
then
   :
else
  echo "my line" >> /etc/exports
fi

Worst-case scenario is that someone modified the line you were going to add, and you end up adding a duplicate.

[Sebastian Roth]

@mechalas I am absolutely with you here that simply overwriting the file can cause trouble if people rely on the FOG server as general NFS storage (not recommended by the way!) and therefore customize /etc/exports.

We have a similar situation for different configuration files as well (DHCPd, VSFTPd). For those we do a single shot backups (copy before overwrite). So running the installer a second time (for whatever reason) will erase your initial config as well. Not as bad as we have it for NFS but still. I remember that I had thought about how to properly solve this a couple of times in the last years but never got to a proper solution. Too bad we never did.

Thanks for your suggestion but as you already mentioned this will ruin configs that were modified by people manually. Let’s take the time and try to come up with a proper solution for all situations now that you brought up this again!

Checksums would work for /etc/exports but not for dhcpd.conf as that has variable information like IP addresses. Another option would be to use “block comments” like:

### FOG config start - don't touch ###
...
### FOG config end - don't touch ###

But I have seen many situations where this fails terribly and I don’t thing we can make this absolutely fail save.

Hmmm, any other suggestions?

[Quazz]

To be fair, it’s been a longstanding recommendation that the FOG server should be running nothing but the FOG server for a reason.

[Wayne Workman]

@mechalas Please see the NFS section of the .fogsettings documentation, and set it to not overwrite your exports file:

https://wiki.fogproject.org/wiki/index.php?title=.fogsettings#NFS

@Sebastian-Roth there appears to be an /etc/exports.d functionality I’d suggest looking into. Maybe fogs config can be put into its own separate file:
https://serverfault.com/questions/910827/how-do-i-use-etc-export-d
Because this would be an improvement and not a break-fix, I’d suggest backlogging this until more pressing things are sorted.

[mechalas]

@Quazz

To be fair, it’s been a longstanding recommendation that the FOG server should be running nothing but the FOG server for a reason.

If I followed that advice for every service I stood up that suggested this, I’d have a pile of VM’s, Docker containers, or a slew of physical systems (or a combination of all three) that need to be managed. Each of those solutions brings their own administrative overhead and issues. It’s trading one problem for another.

In an ideal world, this is great. In the real world where resources are limited, I have to make decisions about what can go where, and what’s easiest to manage.

@Wayne-Workman

Please see the NFS section of the .fogsettings documentation, and set it to not overwrite your exports file:

I read that document. It says “rebuild” and “rebuilt”. When I have my engine in my car rebuilt, I expect to get back my old engine with new parts in it. I do not expect to get back just the new parts.

The language here matters.

[mechalas]

I like the /etc/exports.d approach the best.

My age is showing: I didn’t even know this functionality had been added to exportfs or I’d have been using it.

[Sebastian Roth]

@mechalas @Wayne-Workman I have done some first tests on Debian and CentOS just to make sure this actually works as expected. Seems like it does so far.

Meanwhile I was wondering about your ideas on how to move over from what we have now to /etc/exports.d schema. There are a couple different situations, some are easy and some are hard to come by.

• Fresh install: Just generate /etc/exports.d/fog.exports and we are good to go.
• Upgrade install with a clean /etc/exports file: either we use checksum or regex to make sure it is not modified and move the file to /etc/exports.d/fog.exports (and maybe create an empty /etc/exports)
• Upgrade install with a modified /etc/exports file: Not really sure what to do in this case. We could try to grep the existing entries, pipe those into a new /etc/exports.d/fog.exports and sed the main file to remove/comment the entries there. But I guess there are so many different files out there that chance is high we break things for some people again - last time though.

Any comments on this?

[george1421]

@Sebastian-Roth said in FOG install overwrites /etc/exports with no warning:

1. Fresh install: Just generate /etc/exports.d/fog.exports and we are good to go.
2. Upgrade install with a clean /etc/exports file: either we use checksum or regex to make sure it is not modified and move the file to /etc/exports.d/fog.exports (and maybe create an empty /etc/exports)
3. Upgrade install with a modified /etc/exports file: Not really sure what to do in this case. We could try to grep the existing entries, pipe those into a new /etc/exports.d/fog.exports and sed the main file to remove/comment the entries there. But I guess there are so many different files out there that chance is high we break things for some people again - last time though.

1. Test to see if /etc/exports.d directory exists, if not fall back to current way of managing exports.
2. (3. can be combined) test to see if /etc/exports.d exists. if so use sed or awk to remove the fog lines from /etc/exports and then move to the exports.d design.
3. Same process as 2.

Questions: what happens if we have 2 exports.d file that contains the same fsid? Will that break something? Do we move FOG’s nfs fsid to something unique like 1001? Does the fsid need to be unique or sequential?

[Sebastian Roth]

@george1421 said in FOG install overwrites /etc/exports with no warning:

1. Test to see if /etc/exports.d directory exists, if not fall back to current way of managing exports.

No good I suppose. Testing on Debian I noticed that /etc/exports.d did not exist on a clean fresh OS install with nfs-server package but it all worked perfectly well after I created it manually…

[Wayne Workman]

@george1421 said in FOG install overwrites /etc/exports with no warning:

Questions: what happens if we have 2 exports.d file that contains the same fsid? Will that break something?

In what scenarios do you see having two?

[Wayne Workman]

@Sebastian-Roth said in FOG install overwrites /etc/exports with no warning:

Upgrade install with a clean /etc/exports file: either we use checksum or regex to make sure it is not modified and move the file to /etc/exports.d/fog.exports (and maybe create an empty /etc/exports)

Just check if it exists.

if [[ ! -f "/etc/exports.d/fog.exports" ]]; then 
    echo "do stuff"
fi

I’d suggest asking the user to confirm the actions to be taken, with a default answer of “y”.
Maybe something like “it’s detected that fog configuration exists in /etc/exports. Would you like the installer to move this to the new location for you?”

Just kicking around ideas…

[george1421]

@Wayne-Workman said in FOG install overwrites /etc/exports with no warning:

In what scenarios do you see having two?

If you were to install fog side by side with another product that supports nfs. What is the likelihood of two installs creating an entry with fsid of 0 or 1? We see that issue today when someone wants to create a second nfs share on the fog server and clones what is currently there but updates the paths only.