Mounting File System Failed - No route to host

[Bobbyfrank]

I’ve looked at https://forums.fogproject.org/topic/6898/fog-problem (also forgot to change my timezone), but was not able to fix the root problem.

This issue occurs on multiple computer models. I recently got done customizing a boot loader and am booting FOG through GRUB. We can successfully register hosts, but when testing image creating/deployment this popped up.

As a part of the testing process I deleted the default storage node and storage group and created new ones.

The IP address is correct, but for a few other fields we have the FQDN instead of the IP. It doesn’t look like that made a difference here.

Any ideas/tips?

[Culture20]

Figured it out!
Short answer: disabled IPv6 (and allowed NFSv3)
Long answer: On the server, set all the ports for nfsv3 to manual static ports, added the new iptables entries; Nada. Just for giggles, set a static address on the laptop with FOS USB, unplugged the server from the network and plugged both machines directly together, NIC-cable-NIC. Nada. Disabled the firewall on the server. Nada. Pings still worked, ssh from server to FOS USB worked, so the cable was working. Completely disabled IPv6 on a whim, and mounts from the debug console worked. Restarted iptables, and everything still worked. plugged the server back into the wall, walked the laptop to another location with a different subnet and several physical routers in between, and it still worked. Thanks for letting us know about the nfs version!

[Bobbyfrank]

[george1421]

This appears to be an interesting issue. I might suspect that either the target computer either doesn’t have an IP network or something else is going on here.

Is 149.166.139.13 the IP address of your fog server?
If so then on your fog server key in the following command:
showmount -e 127.0.0.1

I see that you are using the https protocol? The ip address [149.166.139.13] points to in-info-fog.soic.iupui.edu?

Is the target (pxe booting host) on the same subnet (vlan) as the fog server?

[george1421]

I just saw something, you are booting this target computer with a FOS USB drive [boottype=usb]?

[Bobbyfrank]

@george1421 here are the results of showmount -e 127.0.0.1

HTTPS is being used. 149.166.139.13 does point to in-info-fog.soic.iupui.edu.

Our current boot setup is using a FOS USB drive (TFTP issues, we’re working with our overarching network team on that). The target is not currently on the same subnet as the fog server. When testing targets on the 139 subnet, the boot process failed on the step where it verifies internet by running ‘curl -Ikfso /dev/null https://in-info-fog.soic.iupui.edu/fog/index.php --connect-timeout 5’ in init.d/S40Network, so I’m expecting our networking friends have something funky going on with the routing. I will be able to do more tests and update in the next few hours.

[george1421]

@bobbyfrank OK are you using the FOS usb boot device or something you created?

If you are using the FOS usb boot, pick option 6 to get a debug console on the target computer. That should drop you to a linux command prompt on the target computer. From there see if you can ping the FOG server by IP address and dns name. Make sure both work correctly. I can tell you that typically FOG doesn’t use the dns names, but instead uses the IP address for everything. that way FOS doesn’t have to deal with misconfigured dhcp dns settings.

[Bobbyfrank]

@george1421 I am using the FOS USB Boot Device.

I am able to ping the FOG server on a different subnet using both IP and FQDN.

Currently, my status is that I am able to register hosts, but not able to deploy/capture images. Here’s a pic of a GOOD error when registering a host with passed arguments (Registering works 100%).

Here’s a pic of the BAD error when attempting to deploy/capture an image. I no longer see the message about no route to host. It is now a NULL message.

The same arguments are being passed. What part is causing the first menu option to fail when the second menu option works?

Here’s a pic of my grub.cfg (/boot/grub/grub.cfg)

[george1421]

@bobbyfrank OK the null message is a good thing, I think. The Null means that (translation provided) “You did not schedule anything for the FOS engine to do on the FOG server before you booted via USB” Please schedule a task on the fog server first then usb boot into FOS.

I really need to get that NULL message fixed so its a bit more meaningful. The FOS USB was only intended to be used for debugging, but now its being used a bit more.

[Bobbyfrank]

@george1421 We’re back to no route on host

Going back a bit - no FQDNs are being used anymore and there is no difference between doing this on the same or different subnet. We tested trying this from a blocked subnet and the curl command came back with a similar message - no route to host.

What protocols/services are used to capture and deploy the image through the FOS USB? I can try testing those individually to see if our network friends are blocking them similar to how TFTP was blocked.

[Sebastian Roth]

@Bobbyfrank No route to host just means that the client is not able to contact the server as it does not seem to be on the same subnet and routing is not setup properly. You need to tell us more about your network layout: network mask, client IP, gateway information sent to the client, DHCP server handing out the network settings.

[george1421]

@bobbyfrank ok I’m seeing a pattern here.

The FOS engine uses tftp, (dhcp if using FOG as dhcp server), (proxydhcp if you have dnsmasq installed), http/https and NFS protocol (this is the point where its currently failing, making the NFS mount to the fog server).

Do you have some kind of screening router/firewall between the target computer and FOG server?

[Culture20]

@george1421, I’ve been working with @Bobbyfrank onsite with this problem the last two work days (2018-02-23/26) and FOG is new to me, but I’m definitely not new to Linux. Here’s an update:

The routing tables look fine.
We can definitely get NFS exports to mount if we use a different OS on the same client machine (Ubuntu 16.04 and 17.10 mount.nfs4 confirmed to work).
The server has ports for NFSv4 open to the subnets we’re needing. iptables has been quadruple checked by two people (and works fine with the other OSes as mentioned).
On the FOS USB, the debug console can ping to the server just fine, and curl works, but it doesn’t have the root certs needed to verify our server’s cert (easy to ignore for the moment).
Finally: NFS traffic isn’t seen via tcpdump on the server when the FOS USB attempts to NFS mount either auto or in the debug console (it definitely is when other OSes are mounted to the exports).
The only thing I can think of is this is an oddity with busybox mount vs regular mount[.nfs4] (FOS USB is using mount via busybox, yes?)

[george1421]

@culture20 first, FOS/FOG uses nfs v3 not v4. Not sure if there are different ports in play or not.

I would drop iptables as a test to ensure you have not missed a port.

If you usb boot into FOS, I would like to see if you can manually mount fog fog server. I’ll provide the command in a moment.

Something else not related to your nfs issue, but when booted into FOS, if you give root a password (any password like ‘hello’) and get the IP address of FOS, you can connect to it using putty allowing you the chance to remote debug FOS. Without setting root’s password the ssh server won’t allow remote connections.

[george1421]

@george1421 Looking in the inits FOS mounts the fog server with this command
mount -o nolock,proto=tcp,rsize=32768,intr,noatime <fog_ip>:/images /images

From a USB booted FOS image, make the /images directory in FOS and then see if you can mount it using the above command.

The mount command can be found in /bin/fog.mount in the FOS environment.

[Culture20]

@george1421: Ah, NFSv3… I rejoiced when NFSv4 came to be. NFSv4 is TCP only and uses one static port. NFSv3 was from pre-firewall days and uses a port manager daemon on a static port and other daemons that open up on random ports to do the actual data transfers. I can see the allure of NFSv3 for the UDP connections if you have a dedicated network.

How modifiable is the FOS USB? I’ve dealt with it for about 45 minutes total so far, but have made custom live CDs before.

[Tom Elliott]

@culture20 While UDP is the “default” for NFS, we (fog/fos) are using TCP proto’s.

Hence the proto=tcp in the mount -o nolock,proto=tcp,rsize=32768,intr,noatime <fog_ip>:/images /images statement

[Culture20]

Figured it out!
Short answer: disabled IPv6 (and allowed NFSv3)
Long answer: On the server, set all the ports for nfsv3 to manual static ports, added the new iptables entries; Nada. Just for giggles, set a static address on the laptop with FOS USB, unplugged the server from the network and plugged both machines directly together, NIC-cable-NIC. Nada. Disabled the firewall on the server. Nada. Pings still worked, ssh from server to FOS USB worked, so the cable was working. Completely disabled IPv6 on a whim, and mounts from the debug console worked. Restarted iptables, and everything still worked. plugged the server back into the wall, walked the laptop to another location with a different subnet and several physical routers in between, and it still worked. Thanks for letting us know about the nfs version!

[george1421]

@culture20 FWIW, The developers put this document together a while ago. https://forums.fogproject.org/topic/6162/firewall-configuration

There is also guidance on selinux: https://forums.fogproject.org/topic/6154/selinux-policy

I have not heard of any issues having ipv6 enabled with FOG.