Cannot deploy image - Connection Time Out

[grobinson2k1]

I’ve been running through the forums trying to solve this situation, but I cannot seem to figure out the issue.

The server IP is correct, this last worked back in June or July when I deployed the images last to this computer. The only changes I had made were to setting the FOG Boot Settings afterwards of REFIND_EFI and SANBOOT following issues where other machines (the same laptop Make/Model) would not correctly boot into Windows. I successfully fixed the issue, but never deployed that fix to a different group of machines, which I am trying to do now, as they no longer boot into Windows 10.

The FOG server is deployed on an Ubuntu 20.04.3 server on an ESXi 7.0+ host. I have FOG 1.5.9

I get the following error:
Init Version: 20200906
Could not mount images folder (/bin/fog.download)
Args Passed:
Reason: mount: mounting 192.168.3.200:/images/ on /images failed: Connection timed out

neo@fog-server:~$ showmount -e 127.0.0.1
Export list for 127.0.0.1:
/images/dev *
/images     *

neo@fog-server:~$ cat /etc/exports
/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)

neo@fog-server:~$ ls -alR /images
/images:
total 24
drwxrwxrwx  6 fogproject root 4096 Jul 26 14:13 .
drwxr-xr-x 23 root       root 4096 Jun 10 10:51 ..
drwxrwxrwx  2 fogproject root 4096 Jun 11 13:03 Classroom_10June21
drwxrwxrwx  2 fogproject root 4096 Jul 26 14:13 CyberDawg
drwxrwxrwx  3 fogproject root 4096 Jul 26 14:13 dev
-rwxrwxrwx  1 fogproject root    0 Jun 10 10:51 .mntcheck
drwxrwxrwx  2 fogproject root 4096 Jun 10 10:51 postdownloadscripts

/images/Classroom_10June21:
total 21735316
drwxrwxrwx 2 fogproject root        4096 Jun 11 13:03 .
drwxrwxrwx 6 fogproject root        4096 Jul 26 14:13 ..
-rwxrwxrwx 1 fogproject root           4 Jun 11 12:57 d1.fixed_size_partitions
-rwxrwxrwx 1 fogproject root     1048576 Jun 11 12:57 d1.mbr
-rwxrwxrwx 1 fogproject root        1034 Jun 11 12:57 d1.minimum.partitions
-rwxrwxrwx 1 fogproject root          60 Jun 11 12:57 d1.original.fstypes
-rwxrwxrwx 1 fogproject root           0 Jun 11 12:57 d1.original.swapuuids
-rwxrwxrwx 1 fogproject root    51680073 Jun 11 12:57 d1p1.img
-rwxrwxrwx 1 fogproject root      139889 Jun 11 12:57 d1p2.img
-rwxrwxrwx 1 fogproject root 20907221295 Jun 11 13:03 d1p3.img
-rwxrwxrwx 1 fogproject root   502906493 Jun 11 13:03 d1p4.img
-rwxrwxrwx 1 fogproject root   793919046 Jun 11 13:04 d1p5.img
-rwxrwxrwx 1 fogproject root        1034 Jun 11 12:57 d1.partitions

/images/CyberDawg:
total 22033668
drwxrwxrwx 2 fogproject root        4096 Jul 26 14:13 .
drwxrwxrwx 6 fogproject root        4096 Jul 26 14:13 ..
-rwxrwxrwx 1 fogproject root           8 Jul 26 14:06 d1.fixed_size_partitions
-rwxrwxrwx 1 fogproject root     1048576 Jul 26 14:06 d1.mbr
-rwxrwxrwx 1 fogproject root        1034 Jul 26 14:06 d1.minimum.partitions
-rwxrwxrwx 1 fogproject root          20 Jul 26 14:06 d1.original.fstypes
-rwxrwxrwx 1 fogproject root           0 Jul 26 14:06 d1.original.swapuuids
-rwxrwxrwx 1 fogproject root    48645785 Jul 26 14:07 d1p1.img
-rwxrwxrwx 1 fogproject root      139889 Jul 26 14:07 d1p2.img
-rwxrwxrwx 1 fogproject root 21215771924 Jul 26 14:13 d1p3.img
-rwxrwxrwx 1 fogproject root   502910420 Jul 26 14:13 d1p4.img
-rwxrwxrwx 1 fogproject root   793918878 Jul 26 14:13 d1p5.img
-rwxrwxrwx 1 fogproject root        1034 Jul 26 14:06 d1.partitions

/images/dev:
total 12
drwxrwxrwx 3 fogproject root 4096 Jul 26 14:13 .
drwxrwxrwx 6 fogproject root 4096 Jul 26 14:13 ..
-rwxrwxrwx 1 fogproject root    0 Jun 10 10:51 .mntcheck
drwxrwxrwx 2 fogproject root 4096 Jun 10 10:51 postinitscripts

/images/dev/postinitscripts:
total 12
drwxrwxrwx 2 fogproject root 4096 Jun 10 10:51 .
drwxrwxrwx 3 fogproject root 4096 Jul 26 14:13 ..
-rwxrwxrwx 1 fogproject root  249 Jun 10 10:51 fog.postinit

/images/postdownloadscripts:
total 12
drwxrwxrwx 2 fogproject root 4096 Jun 10 10:51 .
drwxrwxrwx 6 fogproject root 4096 Jul 26 14:13 ..
-rwxrwxrwx 1 fogproject root  235 Jun 10 10:51 fog.postdownload

When I ssh into the machine on debug I still get the connection timeout notification.

[Wed Oct 13 root@fogclient ~]# mkdir /images
[Wed Oct 13 root@fogclient ~]# ping 192.168.3.200
PING 192.168.3.200 (192.168.3.200): 56 data bytes
64 bytes from 192.168.3.200: seq=0 ttl=63 time=0.717 ms
^C
--- 192.168.3.200 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.717/0.717/0.717 ms
[Wed Oct 13 root@fogclient ~]# mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime "192.168.3.200:/images/dev" /images
mount: mounting 192.168.3.200:/images/dev on /images failed: Connection timed out
[Wed Oct 13 root@fogclient ~]# mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime "192.168.3.200:/images" /images
mount: mounting 192.168.3.200:/images on /images failed: Connection timed out

Any help is greatly appreciated.

[grobinson2k1]

@george1421 I’ve updated to NFSv4 and the firewall has been reenabled and FOG is working.

[george1421]

@grobinson2k1 said in Cannot deploy image - Connection Time Out:

Where do I place your custom compiled init_nfsv4?

it goes into /var/www/html/fog/service/ipxe directory. Then in FOG Configuration->FOG Settings click on the expand all button and search for init.xz replace that with init_nfsv4.xz and save the settings. This will force the nfsv4 disk to be loaded for every computer.

[Sebastian Roth]

@grobinson2k1 said in Cannot deploy image - Connection Time Out:

I dropped it and it worked, …

Great to see you and George figured this out so quickly!

Correct, they are on two different subnets, but the same layer 3 switch. There was no other firewall between them.

Allow me a quick comment on this. Two different subnets can be on the same layer 3 switch but they won’t be able to communication with each other unless there is a router involved! Sure router does not need to be a firewall but most often is a combined thing. So what I am saying is, that there needs to be a router in between those two subnets and this might also play a role - maybe not with the NFS issue you see now but keep that in mind if you want to do milticasting as well!!

[grobinson2k1]

Fortunately, I can snapshot the VM for this and I get the steps all the way until I get to the custom init file.

Where do I place your custom compiled init_nfsv4?

[george1421]

@grobinson2k1 I have a how to that I never wrote (yet). The discussion is in this tread: https://forums.fogproject.org/post/143625

The changes are pretty simple. you need to make one edit to the FOG programming code and then use the nfsv4 init.

And then you need to use my custom init (virtual hard drive for FOS Linux): https://drive.google.com/file/d/1EHLhmM9-kXpFO7kfk3H1ydEZF3q8lID1/view?usp=sharing

Read over the entire thread to make sure you understand what is happening. At the very least if you mess up FOG for some reason you can just rerun the fog installer and it will fixup what was changed.

[grobinson2k1]

@george1421 Sounds good, can we mark this as solved. I hope you can get NFSv4 ported for FOG.

I guess I will either leave the firewall off for now, until I have to figure out how to make it secure later. Or when I need to deploy I will turn the firewall off and back on after I am done.

[george1421]

@grobinson2k1 see the issue with NFSv3 is that its not very firewall friendly. You need to have a large port range open unless you can specifically define a smaller range. I have been working on a NFSv4 port for FOG that works just over the single 2048 port which is much easier to firewall off.

[grobinson2k1]

@george1421 said in Cannot deploy image - Connection Time Out:

So I’m guessing that the target computer and the fog server are on different subnets? Is there any type of screening firewall between the two?

Correct, they are on two different subnets, but the same layer 3 switch. There was no other firewall between them.

Now as to this…

Can you drop the firewall on the FOG server and test to see if you can connect?

I dropped it and it worked, which pisses me off. I have not modified the firewall settings since when it last worked. Now, I guess I need to play with the ports to solve that issue.

[george1421]

@grobinson2k1 OK good so we know a bunch more than we did a few minutes ago.

So I’m guessing that the target computer and the fog server are on different subnets? Is there any type of screening firewall between the two?

21,22,80,111,443,2049,20499/tcp
69,111,2049,6080/udp

For NFSv3 I don’t think this is enough ports open. For NFSv4 its right.

Can you drop the firewall on the FOG server and test to see if you can connect?

Also be aware that I’m building a truth table here and if I don’t focus on something then its OK. There is a logic to how we debug this.

[grobinson2k1]

Hi George. Thanks for the quick response.

@george1421 said in Cannot deploy image - Connection Time Out:

IP address of fog server?

192.168.3.200

IP address of fogclient?

192.168.121.2

Did someone enable the firewall on the FOG server?

neo@fog-server:~$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
123/udp                    ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
21,22,80,111,443,2049,20499/tcp ALLOW       192.168.121.0/24
69,111,2049,6080/udp       ALLOW       192.168.121.0/24
21,22,80,111,443,2049,20499/tcp ALLOW       192.168.122.0/24
69,111,2049,6080/udp       ALLOW       192.168.122.0/24
21,22,80,111,443,2049,20499/tcp ALLOW       192.168.11.0/24
69,111,2049,6080/udp       ALLOW       192.168.11.0/24
3000                       ALLOW       Anywhere
123/udp (v6)               ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)
3000 (v6)                  ALLOW       Anywhere (v6)

Are the needed nfs services running on the FOG server? (I think yes because showmount returned the exported directories. But it doesn’t hurt to confirm).

neo@fog-server:~$ sudo systemctl status nfs-server
● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
    Drop-In: /run/systemd/generator/nfs-server.service.d
             └─order-with-mounts.conf
     Active: active (exited) since Fri 2021-10-08 11:23:40 EDT; 5 days ago
    Process: 1042 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
    Process: 1043 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
   Main PID: 1043 (code=exited, status=0/SUCCESS)

Oct 08 11:23:39 fog-server systemd[1]: Starting NFS server and services...
Oct 08 11:23:40 fog-server systemd[1]: Finished NFS server and services.

[george1421]

@grobinson2k1 said in Cannot deploy image - Connection Time Out:

192.168.3.200

Ok there are a few bits of info missing yet.

1. IP address of fog server?
2. IP address of fogclient?
3. Did someone enable the firewall on the FOG server?
4. Are the needed nfs services running on the FOG server? (I think yes because showmount returned the exported directories. But it doesn’t hurt to confirm).