Feature request for FOG 1.6.x - Configure image capture to use NFSv4 instead of NFSv3

  • Moderator

    System security is becoming more of a concern and many compliance certifications will hit on the openness of the FOG imaging share. By switching over to NFSv4 we can use the additional security that is part of nfsv4 as well as reduce the number of open ports on the server to 1 for NFS. Reducing the number of port will allow FOG to integrate firewall rules in the the FOG design much easier than before.

    On the client side this is what is needed to connect to an NFSv4 server on FOS Linux
    mount -t nfs4 -o proto=tcp,port=2049,nolock,proto =tcp,rsize=32768,wsize=32768,intr,noatime "192.168.10.1:/" /images

    We need to be mindful because mounts work a little differently on nfsv4 where the share point becomes the root of the share (akin to how MS Windows shares work) install of the full path. So sharing /images on the fog server the client would mount /images but the path would be in the root of /images path. Its just something FOG will need to take into account

  • Moderator

    Changes needed on FOG server to support NFSv4

    1. Build the nfsv4 virtual fs mount points
    mkdir -p /opt/fog/data/capture
    mkdir -p /opt/fog/data/images
    
    1. (optional) Provide the same ./postinitscript files for both capture and deploy.
    mkdir -p /images/dev/postinitscripts
    mkdir -p /images/postinitscripts
    
    1. Edit the /etc/fstab to bind mount the virtual NFSv4 file system to the physical fog directories
    /images/dev    /opt/fog/data/capture            none    bind 0 0
    /images        /opt/fog/data/images             none    bind 0 0
    

    If you included optional step #2 append this to the end of the /etc/fstab

    /images/postinitscripts  /images/dev/postinitscripts   none    bind 0 0
    
    1. Connect the virtual fs to the physical fs
    mount -a
    

    Now you should be able to run these commands to see if the mount works. Looking at /opt/fog/data/capture should give the same list as /images/dev and looking at /opt/fog/data/images should give the same list as /images. If that is valid then move on to the next step.

    1. Now we need to get the gid and uid of the fog service account fogproject.
      5.1 Run the following command to get fogproject’s uid: grep fogproject /etc/passwd | cut -d ":" -f3 note this value. It will most likely be 1000 or 1001 but could be anything, it depends on the host OS.
      5.2 Run the following command to get fogproject's gid: grep fogproject /etc/passwd | cut -d ":" -f4 note this value. It will most likely be 1000 or 1001. You will need the uid and gid values in the next step.

    2. Edit the /etc/exports file. In this case we are converting FOG to only operate in NFSv4 mode. So we will remove all of the FOG NFSv3 export lines and replace them with the NFSv4 export lines. Insert the following into the /etc/exports file.

    /opt/fog/data/ *(fsid=0,no_subtree_check,insecure)
    
    /opt/fog/data/images *(ro,sync,nohide,no_subtree_check,all_squash,insecure,anonuid=1001,anongid=1001)
    
    /opt/fog/data/capture *(rw,sync,nohide,no_subtree_check,all_squash,insecure,insecure_locks,no_wdelay,anonuid=1001,anongid=1001)
    

    Be sure to update the anonuid= and anongid= values above to match the values of the uid and gid you collected in the previous step.

    1. The last step is FOG server linux OS dependant. We will need to enable NFSv4 and disable all other NFSvX support. For Debian variants you need to do this:
      7.1 Edit /etc/default/nfs-common and make these adujustments
    NEED_STATD="no"
    NEED_IDMAPD="yes"
    

    7.2 Edit /etc/default/nfs-kernel-server Note that RPCNFSDOPTS is typically not included by default. Please add that option if it is not present.

    RPCNFSDOPTS="-N 2 -N 3"
    RPCMOUNTDOPTS="--manage-gids -N 2 -N 3"
    

    7.3 For RHEL compatible linux OS, edit /etc/nfs.conf and make thest changes:

    [nfsd]
    tcp=y
    vers2=n
    vers3=n
    vers4=y
    vers4.0=y
    vers4.1=y
    vers4.2=y
    

    ref: https://wiki.debian.org/NFSServerSetup

    1. Reboot the FOG server. This will check your edits as well as restart the NFS server in v4 mode.
  • Moderator

    Changes needed to FOS for NFSv4 support

    Understand these instructions are for the FOG Devs and not the general FOG admin. You need to know the insides of FOS Linux development to understand some of my notations. I did not test with the option of port=2049. The hope is that is the default so it shouldn’t be needed. By defining a specific port firewall rules can be crafted much easier than with NFSv3 and earlier.

    1. In buildroot the nfs-utils packages in the nfs-utils.mak file nfsv4 support must be enabled.
    NFS_UTILS_CONF_OPTS = \
            --enable-nfsv4 \
            --enable-nfsv41 \
    ...
    HOST_NFS_UTILS_CONF_OPTS = \
            --enable-nfsv4 \
            --enable-nfsv41 \
    ...
    
    1. nfsvers=4 must be added to the mount command in the following files in the overlay fs directory
      ./rootfs_overlay/bin/fog line:14
      ./rootfs_overlay/bin/fog.mount line:17,20
      ./rootfs_overlay/bin/fog.av line:15
      ./rootfs_overlay/bin/fog.photorec

    2. Done. Now rebuild initrd filesystem in buildroot

    I’ve compiled a FOG 1.5.9 NFSv4 version of the initrd here: https://drive.google.com/file/d/1EHLhmM9-kXpFO7kfk3H1ydEZF3q8lID1/view?usp=sharing

  • Moderator

    @quinniedid Wow this has been 6 months already…
    This thread really wasn’t intended to be a how-to document but more of an engineering working document to see if it is really possible to run nfsv4 with FOG. The intent was to minimize the number of open ports needed to make firewall rule crafting a bit easier. NFSv4 brings some interesting but confusing additions to the fog design. What I learned was that both the FOG server and FOS Linux needed some tweaks to get things working like with NFSv3.


  • @george1421 We are now being required to get NFSv4 utilized in our environment; have about 3 months before they shut off our server…

    I have tried to follow some of the steps that you have taken to get this working. What I am confused about is where to make modifications to the FOS linux OS?

    Would you be able to provide a little more details into the steps you took to get this working?

    Thank you!

  • Moderator

    As I continue down this rabbit hole I find that in buildroot the nfs-utils in nfs-utils.mak nfsv4 is specifically disabled.

    NFS_UTILS_CONF_OPTS = \
            --disable-nfsv4 \
            --disable-nfsv41 \
            --disable-gss \
            --disable-uuid \
            --enable-tirpc \
            --enable-ipv6 \
            --without-tcp-wrappers \
            --with-statedir=/run/nfs \
            --with-rpcgen=$(HOST_DIR)/bin/rpcgen
    
    HOST_NFS_UTILS_CONF_OPTS = \
            --disable-nfsv4 \
            --disable-nfsv41 \
            --disable-gss \
            --disable-uuid \
            --disable-ipv6 \
            --without-tcp-wrappers \
            --with-statedir=/run/nfs \
            --disable-caps \
            --disable-tirpc \
            --without-systemd \
    

    Need to be changed to enable nfsv4 to this

    NFS_UTILS_CONF_OPTS = \
            --enable-nfsv4 \
            --enable-nfsv41 \
            --disable-gss \
            --disable-uuid \
            --enable-tirpc \
            --enable-ipv6 \
            --without-tcp-wrappers \
            --with-statedir=/run/nfs \
            --with-rpcgen=$(HOST_DIR)/bin/rpcgen
    
    HOST_NFS_UTILS_CONF_OPTS = \
            --enable-nfsv4 \
            --enaable-nfsv41 \
            --disable-gss \
            --disable-uuid \
            --disable-ipv6 \
            --without-tcp-wrappers \
            --with-statedir=/run/nfs \
            --disable-caps \
            --disable-tirpc \
            --without-systemd \
    
    

    Once enabled the package and initrd need to be recompiled.

    In the initrd in /bin/fog.mount in lines 17 and 19
    ref: mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime “$storage” /images >/tmp/mount-output 2>&1

    The mount command needs to be updated to create an nfsv4 mount.
    from this

        up)
            mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime "$storage" /images >/tmp/mount-output 2>&1
            ;;
        down)
            mount -o nolock,proto=tcp,rsize=32768,intr,noatime "$storage" /images >/tmp/mount-output 2>&1
            ;;
    

    to this

        up)
            mount -o nolock,nfsvers=4,proto=tcp,rsize=32768,wsize=32768,intr,noatime "$storage" /images >/tmp/mount-output 2>&1
            ;;
        down)
            mount -o nolock,nfsvers=4,proto=tcp,rsize=32768,intr,noatime "$storage" /images >/tmp/mount-output 2>&1
            ;;
    

    So far the FOG web code needs to be updated because the nfsv4 shares are presented differently than the >nfsv4.

    nfsv3 share structure

    /images/
    /images/dev/
    

    nfsv4 shares structure

    /
    /images/
    /capture/
    

    ./lib/fog/bootmenu.class.php starting at 1497 https://github.com/FOGProject/fogproject/blob/171d63724131c396029992730660497d48410842/packages/web/lib/fog/bootmenu.class.php#L1497

    Replacing

                        $storage = escapeshellcmd(
                            sprintf(
                                '%s:/%s/%s',
                                $ip,
                                trim($StorageNode->get('path'), '/'),
                                (
                                    $TaskType->isCapture() ?
                                    'dev/' :
                                    ''
                                )
                            )
                        );
    

    to this

                        $storage = escapeshellcmd(
                             sprintf(
                                 '%s:/%s',
                                 $ip,
                                 (
                                     $TaskType->isCapture() ?
                                     'capture/' :
                                     trim($StorageNode->get('path'), '/')
                                 )
                             )
                         ); 
    

    Its still not clear if there is a benefit to the cost of modifying the code to support NFS v4 over just keeping everything the same.

  • Moderator

    @george1421 I’ve worked with nfsv4 for about a week now. Many times I’ve stopped in irritation that the system doesn’t work like the examples but I’ve finally have a solid configuration for NFSv4. While nfsv4 is nfs, its very different in its setup than versions of NFS less than v4. NFSv4 creates a logical file system that doesn’t necessarily align with the physical file system on the server. In some ways it makes things more complex, in other ways it allows for some pretty cool redirection.

    I’ve spent about 5 evenings trying to figure out how to map our current directory structure over to the nfsv4 file system. The problem was what ever the root was shared as ro or wr the child directories shares followed what the root share permissions were and ignored the specific share level permissions that the directories were exported as. I won’t go into details on the tests, but lets say I just ignored what I thought I knew and followed the examples and it worked.
    ref: https://cwiki.apache.org/confluence/display/DIRxINTEROP/NFSv4+Server+Export+Setup

    Just to repeat it again, nfsv4 creates a logical file system with the root of the share, what you define as the root directory in your exportfs. You only assign the fsid=0 what you are calling the root of your logical file system. For my example I’ve created the logical file system structure under /opt/fog/data directory. In this example I’ve created bind mounts to the original FOG directories to maintain backwards compatibility.

    Build the nfsv4 mount points

    mkdir -p /opt/fog/data/capture
    mkdir -p /opt/fog/data/images
    

    Freebie: so we present the same postinitscript files for both capture and imaging.

    mkdir -p /images/dev/postinitscripts
    mkdir -p /images/postinitscripts
    

    Create the bind loop back directories to map the physical file locations to the nfsv4 logical file system. Note this mapping can bring in directories from random locations but present them as a unified logical filesystem in NFS. This is the part that had me stuck for several days, where I thought I new better and this was unnecessary. But without this bind mount the child directories took on the share level permission of the logical nfs root.

    -> /etc/fstab
    /images/dev    /opt/fog/data/capture            none    bind 0 0
    /images        /opt/fog/data/images             none    bind 0 0
    /images/postinitscripts  /images/dev/postinitscripts   none    bind 0 0
    

    Just a placeholder for when creating the fogproject user. I would recommend that the fog installer actually set a unique user and group IP when the fog project user is created. Then we can use only the squash root to assign the files created by the root user to the unique group and user ID. Its really only adding security through obscurity so I did not add it into the install instructions here.

    We’ll add the fogproject user

    adduser fogproject
    

    Then we will grab the uid and gid of the fogproject user to use in the script that adds the nfs exports

    $uid = `grep fogproject /etc/passwd | cut -d ":" -f3`
    $gid = `grep fogproject /etc/passwd | cut -d ":" -f4`
    

    I’m only showing the product of the exports file not how $uid and $gid is set in the file
    -> /etc/exports

    /opt/fog/data/ *(fsid=0,no_subtree_check,crossmnt,all_squash,insecure,anonuid=1000,anongid=1000)
    /opt/fog/data/capture *(rw,sync,no_subtree_check,no_wdelay,no_subtree_check,insecure_locks,all_squash,insecure,anonuid=1000,anongid=1000)
    /opt/fog/data/images *(ro,sync,no_subtree_check,all_squash,insecure,anonuid=1000,anongid=1000)
    

    I understand the above is a big changed from what we have before. All users that connect to the nfs share will be squash all users to the fogproject user. The locks are only needed on a writeable share

    To configure the nfs server to only honor nfsv4 requests the following file needs to be edited
    -> /etc/nfs.conf

    [nfsd]
    tcp=y
    vers2=n
    vers3=n
    vers4=y
    vers4.0=y
    vers4.1=y
    vers4.2=y
    

    Once the services are restarted the NFSv4 service will only accept nfs connects on tcp port 2049 where we can craft a custom firewall rule to protect without opening a large number of ports just for NFS.

    There will be some changes needed in the FOS linux OS to accommodate this new NFSv4 structure. Most notably is to update the mount command to send the connect request

    To capture an image

    mount -t nfs -o nfsvers=4,rw 192.168.112.14:/capture /mnt/images
    
    

    To deploy an image

    mount -t nfs -o nfsvers=4,rw 192.168.112.14:/images /mnt/images
    

    As a side note you can connect to the nfs root directory with this

    mount -t nfs -o nfsvers=4,rw 192.168.112.14:/capture /mnt/images
    

    And then move between the logical shares. The nfsv4 root share is exported as ro. When you change into the capture share with cd the share permissions automatically change to rw. If you cd to /images the share permissions switch to ro. If you cd up one directory the change to ro.

    This is what a you see when you execute these commands.

    # mount -t nfs -o nfsvers=4,rw 192.168.112.14:/ /mnt/images
    # cd images/
    # ls -la
    total 0
    drwxr-xr-x. 4 root  root 35 Dec  4 17:44 .
    drwxr-xr-x. 3 root  root 20 Nov 30 18:49 ..
    drwxrwxrwx. 3 fogproject fogproject 85 Dec  4 17:52 capture
    drwxrwxrwx. 5 fogproject fogproject 82 Nov 29 12:05 images
    

    Traversing these directory will change the share level permissions.

    I’m still working on the ARM project so now I’ll update the FOS Linux client with this new file structure to see how well (if) it works.

  • Senior Developer

    There is a nice article on how to setup NFSv4 with a single port open: https://peteris.rocks/blog/nfs4-single-port/

378
Online

8.4k
Users

15.2k
Topics

142.5k
Posts