Feature request for FOG 1.6.x - Configure image capture to use NFSv4 instead of NFSv3

george1421

@quinniedid Wow this has been 6 months already…
This thread really wasn’t intended to be a how-to document but more of an engineering working document to see if it is really possible to run nfsv4 with FOG. The intent was to minimize the number of open ports needed to make firewall rule crafting a bit easier. NFSv4 brings some interesting but confusing additions to the fog design. What I learned was that both the FOG server and FOS Linux needed some tweaks to get things working like with NFSv3.

george1421

Changes needed to FOS for NFSv4 support

Understand these instructions are for the FOG Devs and not the general FOG admin. You need to know the insides of FOS Linux development to understand some of my notations. I did not test with the option of port=2049. The hope is that is the default so it shouldn’t be needed. By defining a specific port firewall rules can be crafted much easier than with NFSv3 and earlier.

In buildroot the nfs-utils packages in the nfs-utils.mak file nfsv4 support must be enabled.

NFS_UTILS_CONF_OPTS = \
        --enable-nfsv4 \
        --enable-nfsv41 \
...
HOST_NFS_UTILS_CONF_OPTS = \
        --enable-nfsv4 \
        --enable-nfsv41 \
...

nfsvers=4 must be added to the mount command in the following files in the overlay fs directory
./rootfs_overlay/bin/fog line:14
./rootfs_overlay/bin/fog.mount line:17,20
./rootfs_overlay/bin/fog.av line:15
./rootfs_overlay/bin/fog.photorec
Done. Now rebuild initrd filesystem in buildroot

I’ve compiled a FOG 1.5.9 NFSv4 version of the initrd here: https://drive.google.com/file/d/1EHLhmM9-kXpFO7kfk3H1ydEZF3q8lID1/view?usp=sharing

george1421

Changes needed on FOG server to support NFSv4

Build the nfsv4 virtual fs mount points

mkdir -p /opt/fog/data/capture
mkdir -p /opt/fog/data/images

(optional) Provide the same ./postinitscript files for both capture and deploy.

mkdir -p /images/dev/postinitscripts
mkdir -p /images/postinitscripts

Edit the /etc/fstab to bind mount the virtual NFSv4 file system to the physical fog directories

/images/dev    /opt/fog/data/capture            none    bind 0 0
/images        /opt/fog/data/images             none    bind 0 0

If you included optional step #2 append this to the end of the /etc/fstab

/images/dev/postinitscripts   /images/postinitscripts    none    bind 0 0

Connect the virtual fs to the physical fs

mount -a

Now you should be able to run these commands to see if the mount works. Looking at /opt/fog/data/capture should give the same list as /images/dev and looking at /opt/fog/data/images should give the same list as /images. If that is valid then move on to the next step.

Now we need to get the gid and uid of the fog service account fogproject.
5.1 Run the following command to get fogproject’s uid: grep fogproject /etc/passwd | cut -d ":" -f3 note this value. It will most likely be 1000 or 1001 but could be anything, it depends on the host OS.
5.2 Run the following command to get fogproject’s gid: grep fogproject /etc/passwd | cut -d ":" -f4 note this value. It will most likely be 1000 or 1001. You will need the uid and gid values in the next step.
Edit the /etc/exports file. In this case we are converting FOG to only operate in NFSv4 mode. So we will remove all of the FOG NFSv3 export lines and replace them with the NFSv4 export lines. Insert the following into the /etc/exports file.

/opt/fog/data/ *(fsid=0,no_subtree_check,insecure)

/opt/fog/data/images *(ro,sync,nohide,no_subtree_check,all_squash,insecure,anonuid=1001,anongid=1001)

/opt/fog/data/capture *(rw,sync,nohide,no_subtree_check,all_squash,insecure,insecure_locks,no_wdelay,anonuid=1001,anongid=1001)

Be sure to update the anonuid= and anongid= values above to match the values of the uid and gid you collected in the previous step.

The last step is FOG server linux OS dependant. We will need to enable NFSv4 and disable all other NFSvX support. For Debian variants you need to do this:
7.1 Edit /etc/default/nfs-common and make these adujustments

NEED_STATD="no"
NEED_IDMAPD="yes"

7.2 Edit /etc/default/nfs-kernel-server Note that RPCNFSDOPTS is typically not included by default. Please add that option if it is not present.

RPCNFSDOPTS="-N 2 -N 3"
RPCMOUNTDOPTS="--manage-gids -N 2 -N 3"

ref: https://wiki.debian.org/NFSServerSetup

7.3 For RHEL compatible linux OS, edit /etc/nfs.conf and make thest changes:

[nfsd]
tcp=y
vers2=n
vers3=n
vers4=y
vers4.0=y
vers4.1=y
vers4.2=y

Edit the following file on the FOG server. We need to have it tell FOS Linux to use /capture as the target directory instead of /images/dev Use your favorite linux editor and modify this file /var/www/html/fog/lib/fog/bootmenu.class.php starting at 1497
Changing:

                    $storage = escapeshellcmd(
                        sprintf(
                            '%s:/%s/%s',
                            $ip,
                            trim($StorageNode->get('path'), '/'),
                            (
                                $TaskType->isCapture() ?
                                'dev/' :
                                ''
                            )
                        )
                    );

to this

                    $storage = escapeshellcmd(
                         sprintf(
                             '%s:/%s',
                             $ip,
                             (
                                 $TaskType->isCapture() ?
                                 'capture/' :
                                 trim($StorageNode->get('path'), '/')
                             )
                         )
                     );

Reboot the FOG server. This will check your edits as well as restart the NFS server in v4 mode.

quinniedid

@george1421 I am having a hard time understanding what is going on here. I have followed the above steps. I see that it checks the mounted file system and checks it but once we start getting to preparing backup location, I get failed…

Any ideas?

alt text

Sebastian Roth

@quinniedid Looks like a permission issue to me. At this stage FOS just tries to create a directory and errors out if it’s not able to create the sub directory on the NFS share - see code reference.

george1421

@quinniedid There is two places where things could go not as planned.

In the equivelent to windows file share permissions we have the NFS shares, mapping to hopefully the fogproject user here

/opt/fog/data/ *(fsid=0,no_subtree_check,insecure)

/opt/fog/data/images *(ro,sync,nohide,no_subtree_check,all_squash,insecure,anonuid=1001,anongid=1001)

/opt/fog/data/capture *(rw,sync,nohide,no_subtree_check,all_squash,insecure,insecure_locks,no_wdelay,anonuid=1001,anongid=1001)

hopefully the anonuid and gid above point to the uid and gid of the fogproject user. Make sure you have the proper uid and gid for the fogproject user. What we are doing with the above commands is to make all users end up saving files as the fogproject user in the fogproject group. With NFSv3 FOS Linux would connect as root and write files as root (a bit dangerous from a security standpoint). Now all file writes and reads should be as a non-privileged user fogproject.

The next thing we need to check is to ensure the fogproject user has read/write access to /images and everything below it. Lets see the output of ls -la /images and ls -la /images/dev

Finally what host OS is the fog server running under?

quinniedid

@george1421 Here are my settings and OS information. I am using your initrd that you uploaded, don’t know if I mentioned that or not.

alt text

From the client:

alt text

NFS debug enabled I see access denied… It seems like everything is configured correctly:

alt text

george1421

@quinniedid Well this is a bit troubling. From the FOG server side it looks like fogproject has access.

So lets try this. Setup a debug capture (tick the debug check box before scheduling the capture) PXE boot the target computer, that will drop you to the fos linux command prompt on the target computer. Start imaging by keying in fog. You will need to press enter at each break point. Eventually you will get to the error, press ctrl C to exit out. At this point /images should be mapped to /images/dev (acutally the nfsv4 mount point /capture See if you can touch a file in /images from the client computer.

If you can’t. I ran into an anomally when setting my debian system up. The instructions I posted below I created on a centos system, then I duplicated on a debian system. I found that the order of the records in the export file changed the behavior of the readonly vs rw shares. I though it was a fluke, but maybe there is something more to this. If you can’t create a file in /capture (actually /images/dev to the FOG OS) then swap the order of the definition in the exports file to make it look like this:

/opt/fog/data/ *(fsid=0,no_subtree_check,insecure)

/opt/fog/data/capture *(rw,sync,nohide,no_subtree_check,all_squash,insecure,insecure_locks,no_wdelay,anonuid=1001,anongid=1001)

/opt/fog/data/images *(ro,sync,nohide,no_subtree_check,all_squash,insecure,anonuid=1001,anongid=1001)

Putting the rw directory first. Then run the command exportfs -a Lets see if that has an impact on image capture. It shouldn’t matter, but we are still learning here. So…

quinniedid

@george1421 I have been testing quite a bit of the day with all sorts of different configurations on the server. Come to find out it appears that in the initrd that you uploaded, for some reason the NFS mount is being mounted as “Read Only File System”.

If I issue on the FOS client mount -t nfs -o vers=4 10.0.0.5:/capture /images I am able to then write to the /images directory no problem… The only issue is that I still can’t continue to run the capture as it marks /images directory as read-only again.

I don’t know what to look for in FOS for the reason why it is mounting the NFS as a read-only file system?

george1421

@quinniedid Did you update the fog server boot.php file?

The bit that you can map to /capture and you can write the share. That is the right direction.

To help debugging on the target computer, when its in debug mode you can do this. Give root a password with passwd make it something simple like hello this password will be reset on the next reboot. Get the IP address of the target computer with ip a s. From there you can remote into the target computer with Putty logging in with root and hello. Once you are there you can do all of the commands, even imaging from the remote session.

Now that you have a remote session run this command and show me the output cat /proc/cmdline Make sure that /capture is listed in the settings and not /images. Use Putty to copy and paste the output of the command in this thread.

quinniedid

@george1421 Thank you for that SSH tip!!! Saves me from having to go into the office almost everyday to test this stuff.

I don’t have any network access into my test environment but at least I can do it from a Windows or Linux client that’s already on the network instead of the console.

This is the command result:
alt text

Now what I am unsure of is if I did make any modification to the boot.php. I will try and find that file and make the necessary modifications.

UPDATE: I was able to change the boot.php file which is what you had mentioned at the very beginning. I was thinking that was part of the initrd file but clearly it is not. It is capturing right now. After it is done, I will be testing a deploy as well.

george1421

@quinniedid Ok I think you did not update the FOG server file, its referencing /images/dev in the kernel parameters, it should be /capture

WELL for some silly reason I did not include these instructions in the steps. It was listed previously but not in the single post.

Edit /var/www/html/fog/lib/fog/bootmenu.class.php on the fog server starting at line 1497

Replacing

                    $storage = escapeshellcmd(
                        sprintf(
                            '%s:/%s/%s',
                            $ip,
                            trim($StorageNode->get('path'), '/'),
                            (
                                $TaskType->isCapture() ?
                                'dev/' :
                                ''
                            )
                        )
                    );

to this

                    $storage = escapeshellcmd(
                         sprintf(
                             '%s:/%s',
                             $ip,
                             (
                                 $TaskType->isCapture() ?
                                 'capture/' :
                                 trim($StorageNode->get('path'), '/')
                             )
                         )
                     );

quinniedid

@george1421 I was able to capture and deploy!!! Thank you for helping me through this process. I am being forced to use this starting in September now.

How much modification would it take to even just allow NFSv4 to work in FOG as it stands today?

george1421

@quinniedid said in Feature request for FOG 1.6.x - Configure image capture to use NFSv4 instead of NFSv3:

How much modification would it take to even just allow NFSv4 to work in FOG as it stands today?

I guess I don’t understand? You have NFSv4 running to day.

From the perspective of fog supporting nfsv4 out of the box. The request was for FOG 1.6.1 to move over to nfsv4. As you see the tweaks are not that much to do (assuming if someone includes all of the instructions).NFSv4 does bring in some additional levels of security that are not available in v3. I guess if we have fog admins that are willing to test nfsv4 to ensure there are no hidden gotchas it may be an easier sell. But ultimately its up to the developers to decide to include it or not.

Sebastian Roth

@george1421 @quinniedid Regarding this I might point to the discussion we had about replacing NFSv3 - possibly even replacing NFS altogether: https://forums.fogproject.org/topic/14772/feature-request-for-fog-1-6-x-replace-nfsv3

I won’t find the time to lead this discussion on weather FOG should move to NFSv4 or change to an entire different protocol. Though I think an in depth discussion is worthwhile before heading down one or the other road.

quinniedid

@george1421 I was able to get it working, yes. I was able to do both a capture and a deploy with NFSv4 being setup.

I now have the ability to only open two ports in the firewall for all of my FOG clients. Ports 2049/tcp and 80/tcp. I have a DHCP relay server that sits on each network, and it delivers the both ipxe.efi and default.ipxe via TFTP and everything else is done with HTTP. I haven’t figured out a way to boot not using TFTP but it seems that only works if HTTP network boot is an option for the device. This way I do not have to expose a TFTP service on the FOG server or having to use a helper to get it where it needs to be.

@Sebastian-Roth Quickly looking at that discussion it seems that maybe SSH might be the future but it comes with some performance loss and some other struggles.

I would be more than happy to just have NFSv4 be the default standard because at least that is improvement until a more in depth development and assessment can be done to do something different.

Thank you all!!!

george1421

@quinniedid said in Feature request for FOG 1.6.x - Configure image capture to use NFSv4 instead of NFSv3:

I haven’t figured out a way to boot not using TFTP but it seems that only works if HTTP network boot is an option for the device.

The issue is the pxe rom on the target computers. They generally only speak tftp. Once iPXE gets loaded it speaks multiple languages (tftp, http, https, nfs, AoE). The problem is getting iPXE loaded in the first place over the network.

While this isn’t a sustainable solution, you can usb boot iPXE then go 100% http{s}/nfs

quinniedid

@george1421 said in Feature request for FOG 1.6.x - Configure image capture to use NFSv4 instead of NFSv3:

Changes needed to FOS for NFSv4 support

Understand these instructions are for the FOG Devs and not the general FOG admin. You need to know the insides of FOS Linux development to understand some of my notations. I did not test with the option of port=2049. The hope is that is the default so it shouldn’t be needed. By defining a specific port firewall rules can be crafted much easier than with NFSv3 and earlier.

In buildroot the nfs-utils packages in the nfs-utils.mak file nfsv4 support must be enabled.
NFS_UTILS_CONF_OPTS = \
        --enable-nfsv4 \
        --enable-nfsv41 \
...
HOST_NFS_UTILS_CONF_OPTS = \
        --enable-nfsv4 \
        --enable-nfsv41 \
...
nfsvers=4 must be added to the mount command in the following files in the overlay fs directory
./rootfs_overlay/bin/fog line:14
./rootfs_overlay/bin/fog.mount line:17,20
./rootfs_overlay/bin/fog.av line:15
./rootfs_overlay/bin/fog.photorec

Done. Now rebuild initrd filesystem in buildroot

I’ve compiled a FOG 1.5.9 NFSv4 version of the initrd here: https://drive.google.com/file/d/1EHLhmM9-kXpFO7kfk3H1ydEZF3q8lID1/view?usp=sharing

@george1421 It has been quite some time but I have been tasked with having to completely rebuild FOG on a new OS. I was wondering if you could spare some time to generate a new initrd-NFSv4 from the “dev-branch” for me?

Or if there are instructions somewhere I can follow to do this myself, I would be very much willing to do that as well.

Sebastian Roth

@quinniedid said in Feature request for FOG 1.6.x - Configure image capture to use NFSv4 instead of NFSv3:

I have been tasked with having to completely rebuild FOG on a new OS.

Well that’s interesting. Why a new OS? Based on what software stack? Or did I just read this the wrong way and you simply meant adding NFSv4 support?

quinniedid

@Sebastian-Roth

I just meant that I am having to rebuild the FOG server on an updated OS. I figured that an up to date version of initrd with NFSv4 support enabled would be a good idea. We have been utilizing NFSv4 for about a year and a half now. This was a requirement from our Cyber security team.

I am was running Ubuntu 18.04 before and now running Ubuntu 22.04. The new server is currently in production with the old initrd_NFSv4 george1421 made for me, working. I just wanted to make sure there wasn’t new code in the initrd that could be an issue later down the road, that would interfere with new features and what not. Which is why I guess I was asking if we could get the newest version but built to enable NFSv4 support.

Feature request for FOG 1.6.x - Configure image capture to use NFSv4 instead of NFSv3

250

11.6k

17.1k

154.6k