NFS problems after upgrade to trunk

[Wayne Workman]

@John-Sartoris Are you using the location plugin?

[John Sartoris]

@Wayne-Workman said:

@John-Sartoris Are you using the location plugin?

Yes, I am using the location plugin. Is there a known issue?

I completely understand. When it doesn’t make sense double check things the wouldn’t make sense…

IP addresses and Paths are correct. Re-saved, I also double checked and re-saved the node choice in the location plugin.

Still no luck. Is there anyway I can get more detail on the client machine to see exactly what is erroring? It just says “An error has been detected!”, it doesn’t specify.

[Wayne Workman]

@John-Sartoris Do a debug download. on the task confirmation page, the debug option is a checkbox. This boots the target host into a shell. On the target host, there is a variable dump initially on the screen, it can be quite valuable to see what it says.

Additionally, I would recommend removing the location plugin entirely (via plugin management) and then reinstalling it and re-setting it up for your various locations.

[John Sartoris]

@Wayne-Workman

From the client in debug…

mount: mounting 10.2.yyy.xxx:/images on images failed: Connection refused

sounds to me like it’s still a permissions or acl type issue.

[Wayne Workman]

@John-Sartoris refused is different than denied… Have you checked for IP conflicts?

[george1421]

I didn’t read all of the posts here, but could you do a showmount -e 127.0.0.1 This will show us what you have NFS shared on your FOG server. You could then do the same command but use your FOG servers external IP address instead of the loopback interface. It could be a firewall issue.

[John Sartoris]

@george1421 said:

I didn’t read all of the posts here, but could you do a showmount -e 127.0.0.1 This will show us what you have NFS shared on your FOG server.

That’s shows that the exports are proper and they do actually work from the storage node cross site, but a client on the local site can’t connect.

[John Sartoris]

@Wayne-Workman said:

@John-Sartoris refused is different than denied… Have you checked for IP conflicts?

I haven’t specifically checked, but this server is configured in the same way as the rest in our environment. DHCP with a reservation. This is the only server that should get this address. Access ports are not configured in this vlan. I have not had any issues connecting to the server expect via NFS. Ping and SSH work from debug client.

[Wayne Workman]

@John-Sartoris I’m thinking it is probably something network related.

[John Sartoris]

@Wayne-Workman

I understand and agree with the assessment, but nothing has changed on the LAN in weeks other that the fog server updates. Firewalls are disabled and allowed for good measure. I even just tried added iptables rules without success.

[Wayne Workman]

@John-Sartoris can you quickly throw together a CentOS 7 VM and install fog trunk to test?

[george1421]

@Wayne-Workman I’m in a meeting right now (yeah its a bit boring) so I can’t test. But from a debug (boot) session, is the showmount command installed in FOS? It would be interesting to know from the client perspective if the FOG server is showing its mount information.

[edit] The other thing would be to try to do a manual nfs mount from the FOS client to the FOG server. If it maps then there is a parameter setup incorrectly (somewhere) in the FOG GUI. The mount command would be something like mount <fog_server_ip>:/images /img (or what ever the local directory is called on the FOG client) [/edit]

[John Sartoris]

@george1421

I’m trying to manually mount from the FOG Client now, and I’m receiving a connection refused.

@Wayne-Workman
Configuring another VM would be possible, but quite a heavy bit of work for what sure seems to be a firewall config or NFS ACL issue that happen during OS or Fog upgrade.

[Wayne Workman]

@John-Sartoris said:

Configuring another VM would be possible, but quite a heavy bit of work for what sure seems to be a firewall config or NFS ACL issue that happen during OS or Fog upgrade.

That’s just the thing though. you’ve disabled UFW, and NFS has no protections. Ubuntu does not come pre-loaded with Security Enhanced Linux, either, like other distributions do https://wiki.ubuntu.com/SELinux

[John Sartoris]

@Wayne-Workman

Do NFS connections get logged on the server? I’ve tried several suggestions I’ve found but I don’t see any messages for successful or failed connections. I’m sure I’ve seen connection attempts in a log before when testing VMWare ESXi at home, just can’t recall where.

Looks like the export option “no_acl” deals with the file system ACLs not anything network related. And the export is to * so in theory anyone should be able to connect.

[george1421]

Just out of curiosity, do the /etc/export files match between the master fog server and the storage node? Can you post the export files here?

Other random thoughts:

Are you running NFSv4 on either of your server? NFSv4 has additional security requirements.

Does the FOG user ID have the same group and user IDs on all fog servers? (this may not be mandatory)

[John Sartoris]

@george1421

As posted earlier, config matches on both servers.

/etc/exports

/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)

nfs-kernel-server 1.2.5-3ubuntu3.2 does appear to be NFSv4, however this is the version running on both servers. It may have been original with the working storage node, while the master may have been upgraded to 3 to 4.

http://packages.ubuntu.com/precise-updates/nfs-kernel-server

also NFS does appear to be reaching the server from the client.

# netstat -t -u -c | grep 10.2.ccc.bbb
tcp        0      0 fog-01:ssh  10.2.ccc.bbb:41650       ESTABLISHED
tcp        0      0 fog-01:nfs  10.2.ccc.bbb:692         ESTABLISHED

[george1421]

Thank you for posting the export file. Understand from my perspective this is a system that was setup by hand (no offense intended). So we must look in every corner.

OK so this is a NFSv4 setup. I know the access controls are greater for NFSv4 over v3. Let me do a little google–fu and see what I can find.

[edit] Just for clarity I’m from the rhel world, so ubuntu is just enough different to be maddening at times. There appears to be two files that should be compared between the working server and the master fog server. /etc/default/nfs-common and /etc/default/nfs-kernel-server these hold the settings for the nfs server. Also ensure that the rpc.idmapd process is running [/edit]

[John Sartoris]

@george1421

I’ve tried to disable NFSv4 as per http://andy.delcambre.com/2007/06/25/disabling-nfsv4-on-ubuntu.html and the comments in “/etc/default/nfs-kernel-server” however the problem still exists.

@Wayne-Workman

Just wanted to say I really appreciate all the help you have both been.

I’m out for the day. I’ll pick this up again in the morning.

[Wayne Workman]

@John-Sartoris I exhausted what I know about it.

If I were in your shoes, I’d rebuild the server. This thread has been open for 8+ hours when a server rebuild would fix it in under 4. It’s just my opinion, but again if it were me, I’d cut my losses and rebuild.