Unable to deploy - No route to host

Mashley

I am unable to deploy images. I see a similar problem in this forum post but I’m still getting the same error after working through what I could: https://forums.fogproject.org/topic/9677/could-not-mount-images-folder-bin-fog-upload/2
I have FOG installed on an Ubuntu VM.
I have followed the Troubleshooting NFS wiki ( https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_NFS ), however, I am still getting the error:

Mounting File System …Failed
Could not mount images folder (/bin/fog.download)
Args Passed:
Reason: mount: mounting 10.0.4.25:/images on images failed: No route to host

The FOG server IP address is correct.

My /etc/exports file is correct and is as follows:

/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)

Output of df -h command:

Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 389M 1.9M 388M 1% /run
/dev/sda1 19G 9.4G 8.0G 54% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/loop0 128K 128K 0 100% /snap/bare/5
/dev/loop1 62M 62M 0 100% /snap/core20/1494
/dev/loop2 62M 62M 0 100% /snap/core20/1434
/dev/loop3 249M 249M 0 100% /snap/gnome-3-38-2004/99
/dev/loop4 45M 45M 0 100% /snap/snapd/15534
/dev/loop5 55M 55M 0 100% /snap/snap-store/558
/dev/loop6 82M 82M 0 100% /snap/gtk-common-themes/1534
/dev/loop7 66M 66M 0 100% /snap/gtk-common-themes/1519
/dev/loop8 45M 45M 0 100% /snap/snapd/15904
/dev/sda2 3.7G 15M 3.4G 1% /swap
/dev/sda3 9.2G 1.2G 7.5G 14% /home
/dev/sda5 854G 117G 694G 15% /images
tmpfs 389M 12K 389M 1% /run/user/125
tmpfs 389M 24K 389M 1% /run/user/1000

Output of lsblk command:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 4K 1 loop /snap/bare/5
loop1 7:1 0 61.9M 1 loop /snap/core20/1494
loop2 7:2 0 61.9M 1 loop /snap/core20/1434
loop3 7:3 0 248.8M 1 loop /snap/gnome-3-38-2004/99
loop4 7:4 0 44.7M 1 loop /snap/snapd/15534
loop5 7:5 0 54.2M 1 loop /snap/snap-store/558
loop6 7:6 0 81.3M 1 loop /snap/gtk-common-themes/1534
loop7 7:7 0 65.2M 1 loop /snap/gtk-common-themes/1519
loop8 7:8 0 44.7M 1 loop /snap/snapd/15904
sda 8:0 0 900G 0 disk
├─sda1 8:1 0 18.6G 0 part /
├─sda2 8:2 0 3.7G 0 part /swap
├─sda3 8:3 0 9.3G 0 part /home
├─sda4 8:4 0 1K 0 part
└─sda5 8:5 0 868.3G 0 part /images
sr0 11:0 1 1024M 0 rom

Output of “ls -lhartR /images” command:
/images:
total 40K
drwxrwxrwx 2 fogproject root 16K Apr 19 12:55 lost+found
drwxrwxrwx 2 fogproject root 4.0K Apr 19 14:28 postdownloadscripts
drwxrwxrwx 2 root root 4.0K May 17 16:00 HostImage_102
drwxrwxrwx 3 fogproject root 4.0K May 17 16:01 dev
drwxrwxrwx 5 fogserver fogserver 4.0K May 18 12:17 .Trash-1000
drwxrwxrwx 7 fogproject root 4.0K May 26 14:45 .
-rwxrwxrwx 1 fogproject root 0 May 26 14:45 .mntcheck
drwxr-xr-x 24 root root 4.0K May 26 14:48 …

/images/lost+found:
total 20K
drwxrwxrwx 2 fogproject root 16K Apr 19 12:55 .
drwxrwxrwx 7 fogproject root 4.0K May 26 14:45 …

/images/postdownloadscripts:
total 12K
-rwxrwxrwx 1 fogproject root 235 Apr 19 14:28 fog.postdownload
drwxrwxrwx 2 fogproject root 4.0K Apr 19 14:28 .
drwxrwxrwx 7 fogproject root 4.0K May 26 14:45 …

/images/HostImage_102:
total 117G
-rwxrwxrwx 1 root root 16 May 17 15:12 d1.size
-rwxrwxrwx 1 root root 1.0M May 17 15:12 d1.mbr
-rwxrwxrwx 1 root root 1.2K May 17 15:12 d1.partitions
-rwxrwxrwx 1 root root 51M May 17 15:12 d1p1.img
-rwxrwxrwx 1 root root 137K May 17 15:12 d1p2.img
-rwxrwxrwx 1 root root 19G May 17 15:21 d1p3.img
-rwxrwxrwx 1 root root 15 May 17 15:50 d2.size
-rwxrwxrwx 1 root root 1.0M May 17 15:50 d2.mbr
-rwxrwxrwx 1 root root 496 May 17 15:50 d2.partitions
-rwxrwxrwx 1 root root 14M May 17 15:50 d2p1.img
-rwxrwxrwx 1 root root 598M May 17 15:51 d1p5.img
-rwxrwxrwx 1 root root 73G May 17 15:51 d1p4.img
-rwxrwxrwx 1 root root 1.3G May 17 15:52 d1p6.img
drwxrwxrwx 2 root root 4.0K May 17 16:00 .
-rwxrwxrwx 1 root root 24G May 17 16:01 d2p2.img
drwxrwxrwx 7 fogproject root 4.0K May 26 14:45 …

/images/dev:
total 12K
drwxrwxrwx 2 fogproject root 4.0K Apr 19 14:28 postinitscripts
drwxrwxrwx 3 fogproject root 4.0K May 17 16:01 .
drwxrwxrwx 7 fogproject root 4.0K May 26 14:45 …
-rwxrwxrwx 1 fogproject root 0 May 26 14:45 .mntcheck

/images/dev/postinitscripts:
total 12K
-rwxrwxrwx 1 fogproject root 249 Apr 19 14:28 fog.postinit
drwxrwxrwx 2 fogproject root 4.0K Apr 19 14:28 .
drwxrwxrwx 3 fogproject root 4.0K May 17 16:01 …

/images/.Trash-1000:
total 20K
drwxrwxrwx 2 fogserver fogserver 4.0K May 18 12:17 info
drwxrwxrwx 2 fogserver fogserver 4.0K May 18 12:17 files
drwxrwxrwx 5 fogserver fogserver 4.0K May 18 12:17 .
drwxrwxrwx 2 fogserver fogserver 4.0K May 18 12:17 expunged
drwxrwxrwx 7 fogproject root 4.0K May 26 14:45 …

/images/.Trash-1000/info:
total 8.0K
drwxrwxrwx 5 fogserver fogserver 4.0K May 18 12:17 …
drwxrwxrwx 2 fogserver fogserver 4.0K May 18 12:17 .

/images/.Trash-1000/files:
total 8.0K
drwxrwxrwx 5 fogserver fogserver 4.0K May 18 12:17 …
drwxrwxrwx 2 fogserver fogserver 4.0K May 18 12:17 .

/images/.Trash-1000/expunged:
total 8.0K
drwxrwxrwx 5 fogserver fogserver 4.0K May 18 12:17 …
drwxrwxrwx 2 fogserver fogserver 4.0K May 18 12:17 .
root@fogserver-virtual-machine:/home/fogserver#

Output of “service nfs-kernel-server status” command:

● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/generator/nfs-server.service.d
└─order-with-mounts.conf
Active: active (exited) since Thu 2022-05-26 14:52:50 PDT; 27min ago
Process: 1011 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
Process: 1017 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
Main PID: 1017 (code=exited, status=0/SUCCESS)

May 26 14:52:49 fogserver-virtual-machine systemd[1]: Starting NFS server and services…
May 26 14:52:50 fogserver-virtual-machine systemd[1]: Finished NFS server and services.

Changed permissions in /images following the forum post above -

Initial output of ls -la /images:
total 40
drwxrwxrwx 7 fogproject root 4096 May 26 14:45 .
drwxr-xr-x 24 root root 4096 May 26 14:48 …
drwxrwxrwx 3 fogproject root 4096 May 17 16:01 dev
drwxrwxrwx 2 root root 4096 May 17 16:00 HostImage_102
drwxrwxrwx 2 fogproject root 16384 Apr 19 12:55 lost+found
-rwxrwxrwx 1 fogproject root 0 May 26 14:45 .mntcheck
drwxrwxrwx 2 fogproject root 4096 Apr 19 14:28 postdownloadscripts
drwxrwxrwx 5 fogserver fogserver 4096 May 18 12:17 .Trash-1000

Ran chown -R fogserver /images (fogserver is the user I created, more info on my users is at the bottom of this post:

Current output of “ls -la /images” command:
total 40
drwxrwxrwx 7 fogserver root 4096 May 26 14:45 .
drwxr-xr-x 24 root root 4096 May 26 14:48 …
drwxrwxrwx 3 fogserver root 4096 May 17 16:01 dev
drwxrwxrwx 2 fogserver root 4096 May 17 16:00 HostImage_102
drwxrwxrwx 2 fogserver root 16384 Apr 19 12:55 lost+found
-rwxrwxrwx 1 fogserver root 0 May 26 14:45 .mntcheck
drwxrwxrwx 2 fogserver root 4096 Apr 19 14:28 postdownloadscripts
drwxrwxrwx 5 fogserver fogserver 4096 May 18 12:17 .Trash-1000

User info -
Strangely enough, I do not have a fog user -

Output of “cat /etc/passwd | grep fog” command:
fogserver1000:1000:FogServer,:/home/fogserver:/bin/bash
fogproject1001:1001::/home/fogproject:/bin/bash

Output of “ls -la /home” command:

total 40
drwxr-xr-x 7 root root 4096 May 17 14:50 .
drwxr-xr-x 24 root root 4096 May 26 14:48 …
drwxr-xr-x 2 root root 4096 May 17 14:50 fogDBbackups
drwxr-xr-x 14 fogproject fogproject 4096 Apr 27 10:53 fogproject
drwxr-xr-x 17 fogserver fogserver 4096 May 17 14:45 fogserver
drwxr-xr-x 10 root root 4096 May 17 14:50 fog_web_1.5.9.BACKUP
drwx------ 2 root root 16384 Apr 19 12:55 lost+found

Even more strangely, I can register a host from the FOS and when I attempt to do that or join a multicast session, FOS allows me to use the “fog” user to start that. That got me thinking - the FOG user manager GUI shows a user named fog, so I’m assuming that’s the one, but I’m not seeing it as a user in my server, as shown above; I only have users called fogserver and fogproject. Is it possible I deleted the main user and that has messed something up?

GUI showing fog user:

This is my third build and I feel like in all the troubleshooting, I must have done something small but significant to prevent this from working. Any help is welcome and appreciated.

PS - I was able to deploy a single image at some point… I can no longer do that and I’m not sure when that happened.

george1421

@mashley said in Unable to deploy - No route to host:

Reason: mount: mounting 10.0.4.25:/images on images failed: No route to host

This right here concerns me. Why wouldn’t the pxe booting computer have a route to 10.0.4.25 (assuming that is your fog server IP)? I would have previously had a route since iPXE was loaded to get the FOG menu and bzImage was also downloaded.

So to the questions.

Is the FOG server and pxe booting client on the same subnet?
Is there some kind of screening router/firewall that is sitting between the FOG server and target computer, where it might be filtering out the NFS protocol?

Mashley

@george1421 Thanks for your response.

Why wouldn’t the pxe booting computer have a route to 10.0.4.25 (assuming that is your fog server IP)? I would have previously had a route since iPXE was loaded to get the FOG menu and bzImage was also downloaded.

You must be on to something here. I was able to PXE boot and I was also able to unicast to that lab before, and somewhere along the way, I no longer can. When could it have lost that route? I’m sure there’s a solution if I messed up a configuration along the way.

The client and FOG server are on different subnets.
There is a firewall that I don’t (yet) have access to.

george1421

@mashley If you want you could debug this a bit more, or wait until you can review the rules on the firewall.

Mashley

@george1421 Thanks. I’m going to debug a bit more next time I’m in the office.

Unable to deploy - No route to host

141

11.7k

17.2k

154.7k