@Sebastian-Roth Thanks Sebastian, I’ll try checking source ports and see if there is any consistency. I find it very strange that accessing the FogServer through the mount command fails to make a connection on port 51853 but telnetting from the same Client on that port works. The difference is the source port though. There is no record though of the firewall (Shorewall) on the Host to the FogVM blocking any incoming packets on port 51853. All very strange. Something else I will try is taking the FogVM off the VLAN to make testing easier.

@Sebastian-Roth Many thanks Sebastian for your great assistance. This problem is now SOLVED!!
For some reason D-Link consider blocking low source port numbers in their switches is a good way to stop DOS attacks. The switch [DGS-3620] can block tcp syn src port <1024 in the DOS Attack settings.
Network based “security” is becoming more prevalent, so using low src ports may be forced to be changed in FOG?

@Sebastian-Roth After taking the FogServer and the client off a VLAN, to enable testing via a hub to monitor packet activity, we have discovered that the problem is within our switch. The switch is ‘swallowing’ packets that originate on a low source port. The FogServer tells the FogClient to call the mountd on a small range of high ports (e.g.:51853) and the FogClient uses a low port in the 800/900 range. This call from the FogClient to the FogServer is blocked by the switch.
Why does the FogClient use a low source port? Looking at MacOS and Ubuntu18.04, this appears to be standard practice but both MacOS and Ubuntu also put out calls to NFS (2049) which is successful and enables mounting of the server drives on the client. The FogClient doesn’t do this, relying upon the call that is blocked by the switch.

@Sebastian-Roth Thanks Sebastian, I’ll try checking source ports and see if there is any consistency. I find it very strange that accessing the FogServer through the mount command fails to make a connection on port 51853 but telnetting from the same Client on that port works. The difference is the source port though. There is no record though of the firewall (Shorewall) on the Host to the FogVM blocking any incoming packets on port 51853. All very strange. Something else I will try is taking the FogVM off the VLAN to make testing easier.

Thanks Sebastian,

rcpinfo -p localhost
 program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  50037  mountd
    100005    1   tcp  42203  mountd
    100005    2   udp  52143  mountd
    100005    2   tcp  55967  mountd
    100005    3   udp  56692  mountd
    100005    3   tcp  51853  mountd
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049
    100003    3   udp   2049  nfs
    100227    3   udp   2049
    100021    1   udp  42980  nlockmgr
    100021    3   udp  42980  nlockmgr
    100021    4   udp  42980  nlockmgr
    100021    1   tcp  40509  nlockmgr
    100021    3   tcp  40509  nlockmgr
    100021    4   tcp  40509  nlockmgr

iptables -L -n -v
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

What I have discovered in the dump from our client, there is a GETPORT CALL to program MOUNT (100005) which is responded to by the server with GETPORT REPLY Port:51853 which is a port monitored by mountd, and not NFS. This explains why our client just stops, but not sure what mountd and the client are then meant to do. Obviously nothing is happening at the moment but is this because mountd doesn’t pass back to portmap to get the NFS port or the client should actually ask for a NFS port and not a mountd port?

Given that there are thousands of clients out there without this problem, the answer resides within our FOG server but where? Is there a config file in the OS (Ubuntu18.04) to setup portmapper? Or doies the problem lie with mountd?

Thanks
David…

@Sebastian-Roth Thank you for the offer. The problem is at a school where I volunteer on a Friday, so will be there tomorrow to have a look. Not sure to where I can upload a tcpdump file at this moment but will let you know. Thanks again.

Some more information:
Doing some tcpdumps and watching traffic flow, this is what APPEARS to be happening.
The Client puts out call on port 111 to the Server (rpcbind) which responds. There are 7 packets exchanged between the Client and Server which appears to be a ‘conversation’ with the portmapper (rpcbind) telling the client where to locate the NFS server (port 2049). There are then some ARP packets between the Client and Server establishing their mac addresses. Then nothing. No traffic.
It is as if the Server has told the client (running the FOG debug) to call back using NFS (port 2049) to complete the NFS mount, but this never occurs and thus the Client “times out” the mount command.

selinux is not part of a standard Ubuntu install and not involved here. There is no firewall running either. All computers are on the same network too, so there “should be” no difference networkwise between the Mac and the Linux computers. Not sure why the standard Fog debug, and I assume the upload and download commands, use sunrpc and not NFS.
Thanks

We have just upgraded our Fog server to a Ubuntu 18.04 VM running the latest version of Fog but are having problems mounting the images from the Fog server. Error message is always a connection timeout and thus a failure to mount.
Testing using a “Debug” job instead of a upload/download and issuing a mount command from the debug prompt shows that the Fog Debug mont command uses port 111 (sunrpc) and fails. BUT using the same mount command in a MacOS Terminal shows it using port 2019 (nfs) and correctly mounting the Fog server /images directory.
So why does Fog use sunrpc and fail but other computers use NFS and work?
I’ve gone through the “Troubleshoot NFS” many times but still can’t work this out. All help appreciated.

@quazz Apologies for not forwarding the netstat results but I would have to manually copy them from one machine (CLI) to another. All looked OK with rpc and nfs running.
I have been following the Fog NFS Troubleshoot guide and not having any luck with mounting a PC client after having it download the Fog debug program from the Fog server. I then tried to do a mount of the Fog /images and /images/dev directories using another Linux box. SUCCESS, at least to mount and copy a file from /images to /images/dev on the Fog server. I then tried the same exercise using a Mac and successfully mounted it to the Fog Server.
Along this process (had to install nfs-common on the Linux client) it was drawn to my attention that /sbin/mount.nfs should exist on the client. This made me check the Fog Debug CLI on the PC client and such a file does not exist suggesting to me that I cannot use the Fog Debug CLI to mount a NFS directory on the Fog server. Anyway, testing mount: successfully with Linux and MacOS says to me that the Fog Server is working properly to some extent. BUT…
Setting up the Fog server to upload an image from the PC-client still does not work resulting in failing to mount the /images/dev folder. Here is (in part) what is displayed on the PC-client.
Send method … NFS
Checking in … Done
Mounting File System …

mount: mounting failed {FogServer IP}:/images/dev on /images failed: Connection timed out.
Done
* Preparing to send image file to server.
* Checking Mounted File System
**************************
* An error has been detected!!
*************************
Fatal Error: Failed to mount NFS Volume

Computer then reboots after 1 minute.

This appears to be a FOG problem with getting PC clients to successfully mount the NFS Volumes (/images and /images/dev) but at this stage of the process FOG is ‘controlling’ everything and I’m at a loss how to proceed from here. I can successfully mount volumes using Linux and MacOS but Fog client won’t. Frustrating.

All suggestions on debugging this issue greatly appreciated.

@quazz Selinux is not installed (default Ubuntu16.04 in KVM VM) and although there is a firewall running on the Host, packets are not routed through the Host. The FOG VM is on the same subnet as the PC client and packets are routed through a NIC in Bridge mode straight to the FOG VM which isn’t running a firewall either.
I can see a number of packets being exchanged (tcpdump) between the VM and the PC client using Port 111 (sunrpc / nfs) so I assume that NFS is working to some extent but the client always times out trying to mount the remote drive on the Fog server. Thanks.