Posts made by DJ...

@Sebastian-Roth Many thanks Sebastian for your great assistance. This problem is now SOLVED!!
For some reason D-Link consider blocking low source port numbers in their switches is a good way to stop DOS attacks. The switch [DGS-3620] can block tcp syn src port <1024 in the DOS Attack settings.
Network based “security” is becoming more prevalent, so using low src ports may be forced to be changed in FOG?

@Sebastian-Roth After taking the FogServer and the client off a VLAN, to enable testing via a hub to monitor packet activity, we have discovered that the problem is within our switch. The switch is ‘swallowing’ packets that originate on a low source port. The FogServer tells the FogClient to call the mountd on a small range of high ports (e.g.:51853) and the FogClient uses a low port in the 800/900 range. This call from the FogClient to the FogServer is blocked by the switch.
Why does the FogClient use a low source port? Looking at MacOS and Ubuntu18.04, this appears to be standard practice but both MacOS and Ubuntu also put out calls to NFS (2049) which is successful and enables mounting of the server drives on the client. The FogClient doesn’t do this, relying upon the call that is blocked by the switch.

@Sebastian-Roth Thanks Sebastian, I’ll try checking source ports and see if there is any consistency. I find it very strange that accessing the FogServer through the mount command fails to make a connection on port 51853 but telnetting from the same Client on that port works. The difference is the source port though. There is no record though of the firewall (Shorewall) on the Host to the FogVM blocking any incoming packets on port 51853. All very strange. Something else I will try is taking the FogVM off the VLAN to make testing easier.

Thanks Sebastian,

rcpinfo -p localhost
 program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  50037  mountd
    100005    1   tcp  42203  mountd
    100005    2   udp  52143  mountd
    100005    2   tcp  55967  mountd
    100005    3   udp  56692  mountd
    100005    3   tcp  51853  mountd
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049
    100003    3   udp   2049  nfs
    100227    3   udp   2049
    100021    1   udp  42980  nlockmgr
    100021    3   udp  42980  nlockmgr
    100021    4   udp  42980  nlockmgr
    100021    1   tcp  40509  nlockmgr
    100021    3   tcp  40509  nlockmgr
    100021    4   tcp  40509  nlockmgr

iptables -L -n -v
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

What I have discovered in the dump from our client, there is a GETPORT CALL to program MOUNT (100005) which is responded to by the server with GETPORT REPLY Port:51853 which is a port monitored by mountd, and not NFS. This explains why our client just stops, but not sure what mountd and the client are then meant to do. Obviously nothing is happening at the moment but is this because mountd doesn’t pass back to portmap to get the NFS port or the client should actually ask for a NFS port and not a mountd port?

Given that there are thousands of clients out there without this problem, the answer resides within our FOG server but where? Is there a config file in the OS (Ubuntu18.04) to setup portmapper? Or doies the problem lie with mountd?

Thanks
David…

@Sebastian-Roth Thank you for the offer. The problem is at a school where I volunteer on a Friday, so will be there tomorrow to have a look. Not sure to where I can upload a tcpdump file at this moment but will let you know. Thanks again.

Some more information:
Doing some tcpdumps and watching traffic flow, this is what APPEARS to be happening.
The Client puts out call on port 111 to the Server (rpcbind) which responds. There are 7 packets exchanged between the Client and Server which appears to be a ‘conversation’ with the portmapper (rpcbind) telling the client where to locate the NFS server (port 2049). There are then some ARP packets between the Client and Server establishing their mac addresses. Then nothing. No traffic.
It is as if the Server has told the client (running the FOG debug) to call back using NFS (port 2049) to complete the NFS mount, but this never occurs and thus the Client “times out” the mount command.

selinux is not part of a standard Ubuntu install and not involved here. There is no firewall running either. All computers are on the same network too, so there “should be” no difference networkwise between the Mac and the Linux computers. Not sure why the standard Fog debug, and I assume the upload and download commands, use sunrpc and not NFS.
Thanks

We have just upgraded our Fog server to a Ubuntu 18.04 VM running the latest version of Fog but are having problems mounting the images from the Fog server. Error message is always a connection timeout and thus a failure to mount.
Testing using a “Debug” job instead of a upload/download and issuing a mount command from the debug prompt shows that the Fog Debug mont command uses port 111 (sunrpc) and fails. BUT using the same mount command in a MacOS Terminal shows it using port 2019 (nfs) and correctly mounting the Fog server /images directory.
So why does Fog use sunrpc and fail but other computers use NFS and work?
I’ve gone through the “Troubleshoot NFS” many times but still can’t work this out. All help appreciated.

@quazz Apologies for not forwarding the netstat results but I would have to manually copy them from one machine (CLI) to another. All looked OK with rpc and nfs running.
I have been following the Fog NFS Troubleshoot guide and not having any luck with mounting a PC client after having it download the Fog debug program from the Fog server. I then tried to do a mount of the Fog /images and /images/dev directories using another Linux box. SUCCESS, at least to mount and copy a file from /images to /images/dev on the Fog server. I then tried the same exercise using a Mac and successfully mounted it to the Fog Server.
Along this process (had to install nfs-common on the Linux client) it was drawn to my attention that /sbin/mount.nfs should exist on the client. This made me check the Fog Debug CLI on the PC client and such a file does not exist suggesting to me that I cannot use the Fog Debug CLI to mount a NFS directory on the Fog server. Anyway, testing mount: successfully with Linux and MacOS says to me that the Fog Server is working properly to some extent. BUT…
Setting up the Fog server to upload an image from the PC-client still does not work resulting in failing to mount the /images/dev folder. Here is (in part) what is displayed on the PC-client.
Send method … NFS
Checking in … Done
Mounting File System …

mount: mounting failed {FogServer IP}:/images/dev on /images failed: Connection timed out.
Done
* Preparing to send image file to server.
* Checking Mounted File System
**************************
* An error has been detected!!
*************************
Fatal Error: Failed to mount NFS Volume

Computer then reboots after 1 minute.

This appears to be a FOG problem with getting PC clients to successfully mount the NFS Volumes (/images and /images/dev) but at this stage of the process FOG is ‘controlling’ everything and I’m at a loss how to proceed from here. I can successfully mount volumes using Linux and MacOS but Fog client won’t. Frustrating.

All suggestions on debugging this issue greatly appreciated.

@quazz Selinux is not installed (default Ubuntu16.04 in KVM VM) and although there is a firewall running on the Host, packets are not routed through the Host. The FOG VM is on the same subnet as the PC client and packets are routed through a NIC in Bridge mode straight to the FOG VM which isn’t running a firewall either.
I can see a number of packets being exchanged (tcpdump) between the VM and the PC client using Port 111 (sunrpc / nfs) so I assume that NFS is working to some extent but the client always times out trying to mount the remote drive on the Fog server. Thanks.

@george1421 Thanks for your help, much appreciated. And apologies for the ‘slowness’ of this conversation as I am volunteering at a school and only get to look at FOG problems on a Friday.
The command showmount -e localhost returns
Export list for localhost:
/images/dev *
/images *

The Fog server is a VM running Ubuntu and selinux is not installed. FW is open, so no access problems that I am aware of.

There is a problem mounting the FOG server /images directory on the PC client. We have followed the instructions in the Fog Wiki regarding Troubleshoot NFS but so far to no solution.
We have created a debug task and have the client up and running with a basic linux kernel and are able to ping the Fog server. The client and server are on the same subnet. The nfs server and rpcbind have both been restarted and the /etc/exports file updated to reflect the recommendation in this troubleshoot guide. The .mntcheck file exists (in both /images and /images/dev on the Fog server).
A tcpdump packet capture sees traffic both ways between the server and the PC client (although no idea what it says) when a mount command is given on the client using the debug console. “mount 192.168.30.2:/images /images” (with the recommended switches) but nothing happens on the client until a time out.
Running out of ideas as to how to find the reason for this error. All help appreciated.

We now have the client PC booting and downloading the undionly.kpxe file but had to use the above mentioned “class” option. Further investigating the “Connection timed out” issue has raised another set of problems/questions which I will followup in a new thread.

@tom-elliott Thanks Tom but we’ve tried that many times. Giving an empty filename just upsets the test on dhcpd.conf and using a filename with a space still results in the additional strange characters. Anyway, we have solved (or rather avoided) this issue by commenting out the bootfile-name line and adding…

  class "Legacy" {
    match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
    filename "undionly.kkpxe";
    }

which seems to work. Now having another problem with an error message on the PCclient saying “Mounting /image/dev on /images failed: Connection timed out”, when trying to upload or download an image.
Still… one step forward. Thanks.

@quazz Thanks for the suggestion but unfortunately quotes are not the problem. They possibly appeared different in this post because of the way the post was created but we have used vi directly on the server console and checked the dhcpd.conf file with dhcpd -t -cf dhcpd.conf and all checks out OK. We’ve also tried single quotes but checking the dhcpd.conf file then fails.
Getting to wits end with this problem, so all help greatly appreciated.

Thanks for the suggestion but unfortunately quotes are not the problem. They possibly appeared different in this post because of the way the post was created but we have used vi directly on the server console and checked the dhcpd.conf file with dhcpd -t -cf dhcpd.conf and all checks out OK. We’ve also tried single quotes but checking the dhcpd.conf file then fails.
Getting to wits end with this problem, so all help greatly appreciated.

We are using FOG with Ubuntu 16.04 in a Guest VM and providing DHCP from the host server. When a client PC attempts to get the bootfile name from the DHCP server in order to tftp that bootfile an error is returned saying that the bootfile cannot be found.
DHCP is set up with
option tftp-server-name “192.168.30.2”;
next-server 192.168.30.2;
option bootfile-name “undionly.kpxe”;

which I believe to be correct. The problem arises in that the filename being returned is corrupted and packet sniffing indicates that the PC Client is trying to get “undionly.kpxeM-^? 7;{” - there being a string of strange characters at the end of the filename. Off course the PC Client tftp protocol then issues the error message that this file cannot be found.
We have tried changing the name of the bootfile-name to just a space character which results in the return of " M-? ^7" or other such string of semi-random characters.
Doing a manual tftp get from the PC Client after it has booted into Windows works fine, as does using a Linux Client, so the TFTP server works fine when asked to get the correct filename.
How do we get rid of these odd characters attaching themselves to the bootfile-name returned to the PC Client so it can then TFTP the desired undionly.kpxe file? Alternatively can anyone please suggest an appropriate debugging process.

with thanks
DJ…

We have been trying for weeks to get a lab of Win7 PCs to join a domain but keep getting this error message in fog.log.
[SIZE=6][B][URL=‘http://fogproject.org/forum/search/8005637/?searchform=1&q=Failed%3A+Incomplete+server+response%3B+got%3A+7%3B+wanted%3A+6&o=date&c[date]=1396357200’][I]Failed: Incomplete server response; got: 7; wanted: 6[/I][/URL][/B][/SIZE]

Using a browser to go to [url]http://fogServer//fog/service/hostname.php?mac=ac220b4e8f93[/url] we get back …
#!ok=LabPC-22 #AD=1 #ADDom=mdcs-student.local #ADOU=MDCS/Lab #ADUser=mdcs-student.local\administrator #ADPass={encryted password} #Key=vqmcc-amq38-stfdq-py468-rtqr6

This has 7 items in it but hostnamechanger.cs source indicates that the above error message is generated when more than 6 items result, so I assume this is the problem area.

Is this the probelm, if so, how do we resolve it?

[SIZE=6][B][B][/B][/B][/SIZE]

[I][/I]