Could not mount images folder (bin/fog.download) ... failed: Operation not supported

Kiweegie · Feb 5, 2020, 5:57 PM

Hi folks been a while but we’re back using FOG again I’ve setup a multiple server/location setup all running on Ubuntu 18.04 and FOG 1.55. with a management node and storage node in our main site both set as Master, with the Storage node being Master for all bar one of 11 remote storage nodes.

Using location, WOL broadcast and LDAP plugins.

We had been deploying and capturing images fine after initial setup this week but today started getting this error.

Nothing has changed far as I can see and I’m not in the office today to test the machines myself but will look to do a debug boot tomorrow to see if i can see any errors,

I’ve found similar reports in here with error “permission denied” but not for Operation not supported.

My server setup (same across storage and normal nodes) is:

Ubuntu 18.04.3 virtual machines
2 drives, (dev/sda) x 30GB for OS, (dev/sdb) x 60 GB for images
/dev/sdb mounted to /mnt/images with fstab entry
/images directory has .mntcheck in required locations
ownership of the directory was set to fogproject user during setup
images appear to have owner set to root but changing this to fogproject user does not resolve issue.

This is happening when deploying both pre and post sysprepped images. We have another error on the post-syprepped ones generating an error “Windows could not complete the installation” but this is down to the fogclient being enabled on the base image when sysprep taken I think. I can sort that separately.

Any suggestions on above would be great.

And while I’m here can I just say now great the UI is looking these days! i had already seen the improvements the team had put in place when i was last using FOG about 3 odd years ago but its looking really slick now.

I’ve fully updated my documentation on the installation of above for our service desk teams and in case thats of use to anyone here I’m happy to share.

regards Tom

george1421 · Feb 5, 2020, 6:13 PM

What does the output of this command look like when run on your fog server?

sudo showmount -e 127.0.0.1

Kiweegie · Feb 6, 2020, 9:14 AM

@george1421 said in Could not mount images folder (bin/fog.download) ... failed: Operation not supported:

sudo showmount -e 127.0.0.1

Hi George,

Output of command as follows

clnt_create: RPC: Program not registered

I’ve checked NFS server service:

systemctl status nfs-server

and was not started showing following output:

● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2020-02-04 21:58:25 UTC; 21h ago
  Process: 905 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
  Process: 903 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
  Process: 891 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=1/FAILURE)

Feb 04 21:58:25 geo01vfog01 systemd[1]: Starting NFS server and services...
Feb 04 21:58:25 geo01vfog01 exportfs[891]: exportfs: Failed to stat /mnt/images/dev: No such file or directory
Feb 04 21:58:25 geo01vfog01 systemd[1]: nfs-server.service: Control process exited, code=exited status=1
Feb 04 21:58:25 geo01vfog01 systemd[1]: nfs-server.service: Failed with result 'exit-code'.
Feb 04 21:58:25 geo01vfog01 systemd[1]: Stopped NFS server and services.

I was able to start it again via

sudo systemctl start nfs-server

This was the same on both the Master (normal) and Master (storage) nodes.

I’ve enabled debugging to see if I can figure out why NFS server stopped which I’m guessing is the cause of the issue I’m seeing?

sudo vi /etc/default/nfs-kernel-server

Added following and restarted NFS service

RPCMOUNTDOPTS="--manage-gids --debug all"

Also seeing loads of these errors in syslog not sure if related?

Feb  5 19:05:06 fogserver systemd-resolved[842]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-20
18-0001, retrying transaction with reduced feature level UDP.
Feb  5 19:05:19 geo01vfog01 in.tftpd[21004]: tftp: client does not accept options
Feb  5 19:05:49 geo01vfog01 in.tftpd[21104]: tftp: client does not accept options
Feb  5 19:05:52 geo01vfog01 in.tftpd[21114]: tftp: client does not accept options
Feb  5 19:06:45 geo01vfog01 in.tftpd[21284]: tftp: client does not accept options
Feb  5 19:06:56 geo01vfog01 in.tftpd[21319]: tftp: client does not accept options

I’ll be able to confirm tomorrow when I’m back in the office that this is resolved or not.

I’m seeing a separate issue with the locations plugin doing something odd but will log separate ticket for that.

Thanks again George

regards Tom

george1421 · Feb 5, 2020, 8:11 PM

@Kiweegie The expected output from the show mount is a list of nfs shares. The rpc program not registered is a bit unexpected.

What is the output of cat /etc/exports

Sebastian Roth · Feb 5, 2020, 9:47 PM

@Kiweegie said in Could not mount images folder (bin/fog.download) ... failed: Operation not supported:

… Failed to stat /mnt/images/dev: No such file or directory

Please run ls -alR /mnt/images/ and post output here.

Kiweegie · Feb 6, 2020, 11:21 AM

Hi @george1421

output as follows on all nodes.

/mnt//images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
/mnt/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)

cheers Tom

Kiweegie · Feb 6, 2020, 11:37 AM

@Sebastian-Roth said in Could not mount images folder (bin/fog.download) ... failed: Operation not supported:

ls -alR /mnt/images/

Here you go Sebastien.

Output from geovfog01 (Georgia normal master server)

admin@geo01vfog01:~$ sudo ls -alR /mnt/images/
[sudo] password for admin:
/mnt/images/:
total 52
drwxr-xr-x 10 fogproject root  4096 Feb  4 18:52 .
drwxr-xr-x  3 root       root  4096 Jan 30 11:25 ..
drwxrwxrwx  3 fogproject root  4096 Feb  4 18:52 dev
drwx------  2 fogproject root 16384 Jan 30 11:25 lost+found
-rwxrwxrwx  1 fogproject root     0 Jan 30 14:58 .mntcheck
drwxrwxrwx  2 fogproject root  4096 Jan 30 13:30 postdownloadscripts
drwxrwxrwx  2 fogproject root  4096 Feb  4 18:18 Toysrus_Post_Sysprep
drwxrwxrwx  2 fogproject root  4096 Feb  4 18:03 Toysrus_Pre_Sysprep
drwxrwxrwx  2 fogproject root  4096 Feb  5 15:40 Mattel_Post_Sysprep
drwxrwxrwx  2 fogproject root  4096 Feb  4 18:39 Mattel_Presysprep
drwxrwxrwx  2 fogproject root  4096 Feb  4 15:14 Windows10_presysprep

/mnt/images/dev:
total 12
drwxrwxrwx  3 fogproject root 4096 Feb  4 18:52 .
drwxr-xr-x 10 fogproject root 4096 Feb  4 18:52 ..
-rwxrwxrwx  1 fogproject root    0 Jan 30 13:30 .mntcheck
drwxrwxrwx  2 fogproject root 4096 Jan 30 13:30 postinitscripts

/mnt/images/dev/postinitscripts:
total 12
drwxrwxrwx 2 fogproject root 4096 Jan 30 13:30 .
drwxrwxrwx 3 fogproject root 4096 Feb  4 18:52 ..
-rwxrwxrwx 1 fogproject root  249 Jan 30 13:30 fog.postinit

/mnt/images/lost+found:
total 20
drwx------  2 fogproject root 16384 Jan 30 11:25 .
drwxr-xr-x 10 fogproject root  4096 Feb  4 18:52 ..

/mnt/images/postdownloadscripts:
total 12
drwxrwxrwx  2 fogproject root 4096 Jan 30 13:30 .
drwxr-xr-x 10 fogproject root 4096 Feb  4 18:52 ..
-rwxrwxrwx  1 fogproject root  235 Jan 30 13:30 fog.postdownload

/mnt/images/Toysrus_Post_Sysprep:
total 7629888
drwxrwxrwx  2 fogproject root       4096 Feb  4 18:18 .
drwxr-xr-x 10 fogproject root       4096 Feb  4 18:52 ..
-rwxrwxrwx  1 fogproject root          5 Feb  4 18:14 d1.fixed_size_partitions
-rwxrwxrwx  1 fogproject root    1048576 Feb  4 18:14 d1.mbr
-rwxrwxrwx  1 fogproject root        190 Feb  4 18:14 d1.minimum.partitions
-rwxrwxrwx  1 fogproject root         15 Feb  4 18:14 d1.original.fstypes
-rwxrwxrwx  1 fogproject root          0 Feb  4 18:14 d1.original.swapuuids
-rwxrwxrwx  1 fogproject root    9265465 Feb  4 18:14 d1p1.img
-rwxrwxrwx  1 fogproject root 7802656523 Feb  4 18:18 d1p2.img
-rwxrwxrwx  1 fogproject root        190 Feb  4 18:14 d1.partitions

/mnt/images/Toysrus_Pre_Sysprep:
total 8060100
drwxrwxrwx  2 fogproject root       4096 Feb  4 18:03 .
drwxr-xr-x 10 fogproject root       4096 Feb  4 18:52 ..
-rwxrwxrwx  1 fogproject root          5 Feb  4 17:59 d1.fixed_size_partitions
-rwxrwxrwx  1 fogproject root    1048576 Feb  4 17:59 d1.mbr
-rwxrwxrwx  1 fogproject root        190 Feb  4 17:59 d1.minimum.partitions
-rwxrwxrwx  1 fogproject root         15 Feb  4 17:59 d1.original.fstypes
-rwxrwxrwx  1 fogproject root          0 Feb  4 17:59 d1.original.swapuuids
-rwxrwxrwx  1 fogproject root  453635778 Feb  4 17:59 d1p1.img
-rwxrwxrwx  1 fogproject root 7798828240 Feb  4 18:03 d1p2.img
-rwxrwxrwx  1 fogproject root        190 Feb  4 17:59 d1.partitions

/mnt/images/Mattel_Post_Sysprep:
total 7215252
drwxrwxrwx  2 fogproject root       4096 Feb  5 15:40 .
drwxr-xr-x 10 fogproject root       4096 Feb  4 18:52 ..
-rwxrwxrwx  1 fogproject root          5 Feb  4 18:46 d1.fixed_size_partitions
-rwxr-xr-x  1 fogproject root    1048576 Feb  5 15:40 d1.mbr
-rwxr-xr-x  1 fogproject root        190 Feb  5 15:40 d1.minimum.partitions
-rwxrwxrwx  1 fogproject root         15 Feb  4 18:46 d1.original.fstypes
-rwxrwxrwx  1 fogproject root          0 Feb  4 18:46 d1.original.swapuuids
-rwxr-xr-x  1 fogproject root   10387064 Feb  5 15:40 d1p1.img
-rwxr-xr-x  1 fogproject root 7376949941 Feb  5 15:42 d1p2.img
-rwxr-xr-x  1 fogproject root        190 Feb  5 15:40 d1.partitions

/mnt/images/Mattel_Presysprep:
total 7087908
drwxrwxrwx  2 fogproject root       4096 Feb  4 18:39 .
drwxr-xr-x 10 fogproject root       4096 Feb  4 18:52 ..
-rwxrwxrwx  1 fogproject root          5 Feb  4 18:35 d1.fixed_size_partitions
-rwxrwxrwx  1 fogproject root    1048576 Feb  4 18:35 d1.mbr
-rwxrwxrwx  1 fogproject root        190 Feb  4 18:35 d1.minimum.partitions
-rwxrwxrwx  1 fogproject root         15 Feb  4 18:35 d1.original.fstypes
-rwxrwxrwx  1 fogproject root          0 Feb  4 18:35 d1.original.swapuuids
-rwxrwxrwx  1 fogproject root  454330540 Feb  4 18:35 d1p1.img
-rwxrwxrwx  1 fogproject root 6802601965 Feb  4 18:39 d1p2.img
-rwxrwxrwx  1 fogproject root        190 Feb  4 18:35 d1.partitions

/mnt/images/Windows10_presysprep:
total 8864012
drwxrwxrwx  2 fogproject root       4096 Feb  4 15:14 .
drwxr-xr-x 10 fogproject root       4096 Feb  4 18:52 ..
-rwxrwxrwx  1 fogproject root         13 Feb  4 15:09 d1.fixed_size_partitions
-rwxrwxrwx  1 fogproject root    1048576 Feb  4 15:09 d1.mbr
-rwxrwxrwx  1 fogproject root        861 Feb  4 15:09 d1.minimum.partitions
-rwxrwxrwx  1 fogproject root         15 Feb  4 15:09 d1.original.fstypes
-rwxrwxrwx  1 fogproject root          0 Feb  4 15:09 d1.original.swapuuids
-rwxrwxrwx  1 fogproject root  444522502 Feb  4 15:09 d1p1.img
-rwxrwxrwx  1 fogproject root   11385185 Feb  4 15:10 d1p2.img
-rwxrwxrwx  1 fogproject root     174678 Feb  4 15:10 d1p3.img
-rwxrwxrwx  1 fogproject root 8619581073 Feb  4 15:14 d1p4.img
-rwxrwxrwx  1 fogproject root        861 Feb  4 15:09 d1.partitions
admin@geo01vfog01:~$

Output from geovfog02 (Georgia storage master server)

admin@geo01vfog02:~$ sudo !!
sudo ls -alR /mnt/images/
[sudo] password for admin:
/mnt/images/:
total 36
drwxr-xr-x 6 fogproject root  4096 Feb  4 16:13 .
drwxr-xr-x 3 root       root  4096 Jan 30 18:20 ..
drwxrwxrwx 3 fogproject root  4096 Feb  4 16:13 dev
drwx------ 2 fogproject root 16384 Jan 30 18:20 lost+found
-rwxrwxrwx 1 fogproject root     0 Jan 30 19:19 .mntcheck
drwxrwxrwx 2 fogproject root  4096 Jan 30 19:19 postdownloadscripts
drwxrwxrwx 2 fogproject root  4096 Feb  4 16:13 Mattel_Post_Sysprep

/mnt/images/dev:
total 12
drwxrwxrwx 3 fogproject root 4096 Feb  4 16:13 .
drwxr-xr-x 6 fogproject root 4096 Feb  4 16:13 ..
-rwxrwxrwx 1 fogproject root    0 Jan 30 19:19 .mntcheck
drwxrwxrwx 2 fogproject root 4096 Jan 30 19:19 postinitscripts

/mnt/images/dev/postinitscripts:
total 12
drwxrwxrwx 2 fogproject root 4096 Jan 30 19:19 .
drwxrwxrwx 3 fogproject root 4096 Feb  4 16:13 ..
-rwxrwxrwx 1 fogproject root  249 Jan 30 19:19 fog.postinit

/mnt/images/lost+found:
total 20
drwx------ 2 fogproject root 16384 Jan 30 18:20 .
drwxr-xr-x 6 fogproject root  4096 Feb  4 16:13 ..

/mnt/images/postdownloadscripts:
total 12
drwxrwxrwx 2 fogproject root 4096 Jan 30 19:19 .
drwxr-xr-x 6 fogproject root 4096 Feb  4 16:13 ..
-rwxrwxrwx 1 fogproject root  235 Jan 30 19:19 fog.postdownload

/mnt/images/Mattel_Post_Sysprep:
total 7215252
drwxrwxrwx 2 fogproject root       4096 Feb  4 16:13 .
drwxr-xr-x 6 fogproject root       4096 Feb  4 16:13 ..
-rwxrwxrwx 1 fogproject root          5 Feb  4 16:09 d1.fixed_size_partitions
-rwxrwxrwx 1 fogproject root    1048576 Feb  4 16:09 d1.mbr
-rwxrwxrwx 1 fogproject root        190 Feb  4 16:09 d1.minimum.partitions
-rwxrwxrwx 1 fogproject root         15 Feb  4 16:09 d1.original.fstypes
-rwxrwxrwx 1 fogproject root          0 Feb  4 16:09 d1.original.swapuuids
-rwxrwxrwx 1 fogproject root   10387064 Feb  4 16:09 d1p1.img
-rwxrwxrwx 1 fogproject root 7376949941 Feb  4 16:13 d1p2.img
-rwxrwxrwx 1 fogproject root        190 Feb  4 16:09 d1.partitions
admin@gla01vfog02:~$ sudo ls -alR /mnt/images/
/mnt/images/:
total 36
drwxr-xr-x 6 fogproject root  4096 Feb  4 16:13 .
drwxr-xr-x 3 root       root  4096 Jan 30 18:20 ..
drwxrwxrwx 3 fogproject root  4096 Feb  4 16:13 dev
drwx------ 2 fogproject root 16384 Jan 30 18:20 lost+found
-rwxrwxrwx 1 fogproject root     0 Jan 30 19:19 .mntcheck
drwxrwxrwx 2 fogproject root  4096 Jan 30 19:19 postdownloadscripts
drwxrwxrwx 2 fogproject root  4096 Feb  4 16:13 Mattel_Post_Sysprep

/mnt/images/dev:
total 12
drwxrwxrwx 3 fogproject root 4096 Feb  4 16:13 .
drwxr-xr-x 6 fogproject root 4096 Feb  4 16:13 ..
-rwxrwxrwx 1 fogproject root    0 Jan 30 19:19 .mntcheck
drwxrwxrwx 2 fogproject root 4096 Jan 30 19:19 postinitscripts

/mnt/images/dev/postinitscripts:
total 12
drwxrwxrwx 2 fogproject root 4096 Jan 30 19:19 .
drwxrwxrwx 3 fogproject root 4096 Feb  4 16:13 ..
-rwxrwxrwx 1 fogproject root  249 Jan 30 19:19 fog.postinit

/mnt/images/lost+found:
total 20
drwx------ 2 fogproject root 16384 Jan 30 18:20 .
drwxr-xr-x 6 fogproject root  4096 Feb  4 16:13 ..

/mnt/images/postdownloadscripts:
total 12
drwxrwxrwx 2 fogproject root 4096 Jan 30 19:19 .
drwxr-xr-x 6 fogproject root 4096 Feb  4 16:13 ..
-rwxrwxrwx 1 fogproject root  235 Jan 30 19:19 fog.postdownload

/mnt/images/Mattel_Post_Sysprep:
total 7215252
drwxrwxrwx 2 fogproject root       4096 Feb  4 16:13 .
drwxr-xr-x 6 fogproject root       4096 Feb  4 16:13 ..
-rwxrwxrwx 1 fogproject root          5 Feb  4 16:09 d1.fixed_size_partitions
-rwxrwxrwx 1 fogproject root    1048576 Feb  4 16:09 d1.mbr
-rwxrwxrwx 1 fogproject root        190 Feb  4 16:09 d1.minimum.partitions
-rwxrwxrwx 1 fogproject root         15 Feb  4 16:09 d1.original.fstypes
-rwxrwxrwx 1 fogproject root          0 Feb  4 16:09 d1.original.swapuuids
-rwxrwxrwx 1 fogproject root   10387064 Feb  4 16:09 d1p1.img
-rwxrwxrwx 1 fogproject root 7376949941 Feb  4 16:13 d1p2.img
-rwxrwxrwx 1 fogproject root        190 Feb  4 16:09 d1.partitions
admin@geo01vfog02:~$

I’ve added the server setup to another ticket ref location plugin but might be of use to see here also.

Sebastian Roth · Feb 6, 2020, 3:17 PM

@Kiweegie said in Could not mount images folder (bin/fog.download) ... failed: Operation not supported:

/mnt//images *(ro, …

While I don’t think this can cause the issue I’d still change that to /mnt/images.

Beside that I see different IP addresses: 10.166.141.199 in the picture of your initial post while that IP is not in the list you posted just now…

george1421 · Feb 6, 2020, 6:36 AM

@Kiweegie ok your exports file is correct (hopefully the double slash in the images share was a type-o. That double slash // would keep the nfs server from starting). Lets make sure we have this right. You are NOT using or storing any images in /images directory path only the /mnt/images and /mnt/images is mounted on a local hard drive on the fog server? If you are using only /mnt/images you will need to update the storage node configuration for this server and change the ftp and nfs paths.

If that was/is the case I might go about it a bit differently just to keep everything clean.

Edit: Well it would help if I would read the entire thread before responding to avoid duplicating what Sebastian posted.

Kiweegie · Feb 6, 2020, 3:17 PM

@Sebastian-Roth good spot on the // it was there in exports on geovfog02 server. I’ve now corrected that I’ll blame my poor eyesight on that one…

I’ve double checked all other nodes to be on the safe side and it was only this one.

Ignore the ip from original image was screenshot from an earlier build, albeit same error.

edit: I’ve corrected the original image to avoid confusion and also provide better image - notepad++ with Deep Black theme appplied.

regards Tom

Kiweegie · Feb 6, 2020, 4:06 PM

@george1421 @Sebastian-Roth

Hi both not seeing this error any more so can be marked as resolved. I was not seeing the issue before editing the // in the exports file so can’t say that was the fix. Not sure what has changed if anything here other than permissions on the images themselves within the /mnt/images directory.

Ownership was set on the folder to fogproject:root but the images are getting set as root:root. I amended this to fogproject:root again so not sure if that is needed? If so can add sticky bit to get around this perhaps.

EDIT: Ignore above comment ref permissions - some are set as fogproject:root and some as root:root - both work so the permissions is a red herring.

Thoughts welcome

regards Tom

Sebastian Roth · Feb 6, 2020, 5:09 PM

@Kiweegie said in Could not mount images folder (bin/fog.download) ... failed: Operation not supported:

Thoughts welcome

Do you have firewall restrictions in your MPLS or the VPN tunnel?

Could not mount images folder (bin/fog.download) ... failed: Operation not supported

184

12.0k

17.3k

155.2k