Imaging works in VM and not on bare metal
-
@tom-elliott
If its boots off of the HTML sources and not the NFS could that still be a firewall issue. I use PFsense and have never had a single issue with it. Been running it for years now. Pretty close to a decade and I have hosted so many kinds of servers over the years, it would be strange to see it causing an issue now.I’m starting to wonder if its an issue with the hardware (bare metal side) I have 2 old Lenovo M72E systems I’m using for this experiment is it possible that the NIC is the cause?
Claw
-
@claw22000 said in Imaging works in VM and not on bare metal:
If its boots off of the HTML sources and not the NFS could that still be a firewall issue.
Sure! HTTP is port 80 (or 443 for HTTPS) and NFS uses completely different ports. So any firewall being setup correctly would not allow both to do through if not told so for a good reason.
While it is possible the NIC in the Lenovo M72E could cause an issue I don’t think it would cause the “mount: mounting x.x.x.x:/images/dev on /images failed: Connection timed out.”. This is because on boot up the FOS inits do a quick network check to see if the FOG server is reachable. If NIC has a general issue that would not work and it would bail out way earlier.
If you need further help with this we need to know what you setup really looks like. Why is a pfSense between the FOG server and the machines you PXE boot? Are they in two different subnets? And if it’s not pfSense maybe there is a local firewall on the VM host?
If you have Linux installed on one of your PCs you can just boot that up and to NFS mount testing: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_NFS (I know this article is pretty dated but you will find useful information in there too)
-
@sebastian-roth said in Imaging works in VM and not on bare metal:
If you need further help with this we need to know what you setup really looks like. Why is a pfSense between the FOG server and the machines you PXE boot? Are they in two different subnets? And if it’s not pfSense maybe there is a local firewall on the VM host?
PFSense is not between the systems (Hosts) and the server Its is only a firewall to the outside world.
Everything is on the Same subnet
I followd the instructions to make sure the firewall is disabled and tried the debug test
came back…
mount: mounting x.x.x.x:/images on /images failed: connection timed outis it possible debian 10? do I need a different OS. I typicall use ubuntu server but several spots online say it doesn’t work and recomend using debian.
In the end I don’t care which OS it is since its inside my network it doesn’t need a firewall. I typically disable them on all my servers since only the ports needed to be outside are open and none of them are SSH or anything like it.
-
@claw22000 said in Imaging works in VM and not on bare metal:
came back…
mount: mounting x.x.x.x:/images on /images failed: connection timed outBut it does work from local VMs, right? Can you do the mount test in a VM as well?
I have used FOG in Debian 10 and others have too. Not to blame in general I would say.
-
@sebastian-roth
What I did to test this further was to install and recreate the fog server on a bare metal machine. Then try it the other way round. Neither the bare metal machine or the VM can image on this. Only way it works is VM to VM. Tonight after work I will turn on the VM and do the test again VM to VM.I appreciate all the suggestions as I have no ideas as to what could be the problem.
Claw
-
@sebastian-roth
Tried VM to VM and was able to use the mount command to connect just can’t connect from bare metal to the server.Also tried reinstalling and using Ubuntu tested exactly the same can connect VM to VM and cannot Bare metal to VM
Claw
-
@Claw22000 So what exactly is between bare metal and the FOG server? Host firewall in the VM server? Layer 7 switch with application layer gateway/firewall?
-
@sebastian-roth
My setup is is thisHypervisor: VMware ESXi, 6.5.0, 14320405
Model: PowerEdge R815
Processor Type: AMD Opteron
Processor 6276
Logical Processors: 64
Ram: 186 Gigs2 NICS are use and 1 DRAC
1 NIC is directly to the Modem and only accessed by PFSense
the other NIC is shared across all the the VMs PFSense feeding the internet.
The DRAC and internal Network run to a SG 200-08 8-Port Gigabit Smart Switch (Spanning Tree is disabled)
This runs to the other side of the lab to a Netgear JGS516 16 Port unmanaged switchAll computers in the house are then wired to this switch.
VM for the FOG server is
4 CPU’s across 2 sockets with at least 1GHZ reservation
8GB Ram with at least 4Gig Reservation
HD is Thick provisioned with 300gigs. I will increase this once I document how this works and it will get a dedicated drive for this
NIC Adapter Type is VMXNET 3Install steps of current server
sudo -i
wget https://github.com/FOGProject/fogproject/archive/1.5.9.tar.gz
tar -xzvf 1.5.9.tar.gz
rm 1.5.9.tar.gz
cd fogproject-1.5.9/bin
./installfog.sh
click button in browser
press enter in termanal
log in
change default password
create new user for self
chmod -R 777 /images
chown fogproject:nogroup /imagesInfo after all this is done.
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destinationChain FORWARD (policy ACCEPT)
target prot opt source destinationChain OUTPUT (policy ACCEPT)
target prot opt source destination/etc/hosts.deny # /etc/hosts.deny: list of hosts that are _not_ allowed to access the system. # See the manual pages hosts_access(5) and hosts_options(5). # # Example: ALL: some.host.name, .some.domain # ALL EXCEPT in.fingerd: other.host.name, .other.domain # # If you're going to protect the portmapper use the name "rpcbind" for the # daemon name. See rpcbind(8) and rpc.mountd(8) for further information. # # The PARANOID wildcard matches any host whose name does not match its # address. # # You may wish to enable this to ensure any programs that don't # validate looked up hostnames still leave understandable logs. In past # versions of Debian this has been the default. # ALL: PARANOID/etc/exports /images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0) /images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)ls -al / drwxr-xr-x 23 root root 4096 Mar 20 22:50 . drwxr-xr-x 23 root root 4096 Mar 20 22:50 .. lrwxrwxrwx 1 root root 7 Jul 31 2020 bin -> usr/bin drwxr-xr-x 3 root root 4096 Mar 20 18:57 boot drwxr-xr-x 2 root root 4096 Mar 20 17:15 cdrom drwxr-xr-x 17 root root 4000 Mar 20 18:51 dev drwxr-xr-x 102 root root 4096 Mar 21 15:55 etc drwxr-xr-x 4 root root 4096 Mar 20 22:41 home drwxrwxrwx 5 fogproject nogroup 4096 Mar 20 23:15 images lrwxrwxrwx 1 root root 7 Jul 31 2020 lib -> usr/lib lrwxrwxrwx 1 root root 9 Jul 31 2020 lib32 -> usr/lib32 lrwxrwxrwx 1 root root 9 Jul 31 2020 lib64 -> usr/lib64 lrwxrwxrwx 1 root root 10 Jul 31 2020 libx32 -> usr/libx32 drwx------ 2 root root 16384 Mar 20 17:14 lost+found drwxr-xr-x 2 root root 4096 Jul 31 2020 media drwxr-xr-x 2 root root 4096 Jul 31 2020 mnt drwxr-xr-x 3 root root 4096 Mar 20 22:41 opt dr-xr-xr-x 291 root root 0 Mar 20 18:51 proc drwx------ 5 root root 4096 Mar 21 07:27 root drwxr-xr-x 36 root root 1140 Mar 21 15:51 run lrwxrwxrwx 1 root root 8 Jul 31 2020 sbin -> usr/sbin drwxr-xr-x 6 root root 4096 Mar 20 18:52 snap drwxr-xr-x 4 root root 4096 Mar 20 22:41 srv -rw------- 1 root root 4294967296 Mar 20 17:18 swap.img dr-xr-xr-x 13 root root 0 Mar 20 18:51 sys drwxr-xr-x 5 fogproject root 4096 Mar 20 22:50 tftpboot drwxr-xr-x 2 root root 4096 Mar 20 22:50 tftpboot.prev drwxrwxrwt 15 root root 4096 Mar 21 15:39 tmp drwxr-xr-x 14 root root 4096 Jul 31 2020 usr drwxr-xr-x 14 root root 4096 Mar 20 22:34 varls -al /images -rwxrwxrwx 1 fogproject root 0 Mar 21 07:20 .mntcheck drwxrwxrwx 3 fogproject root 4096 Mar 20 23:15 dev drwxrwxrwx 2 fogproject root 4096 Mar 20 22:50 postdownloadscripts drwxrwxrwx 2 root root 4096 Mar 20 23:15 win10basicls -al /images/dev drwxrwxrwx 3 fogproject root 4096 Mar 20 23:15 . drwxrwxrwx 5 fogproject nogroup 4096 Mar 20 23:15 .. -rwxrwxrwx 1 fogproject root 0 Mar 20 22:50 .mntcheck drwxrwxrwx 2 fogproject root 4096 Mar 20 22:50 postinitscriptsHope this helps I really appreciate your help. Sorry it takes so long to get back some times I work 50 - 60 hours a week so get wrapped up alot
Claw
-
@Claw22000 Can’t see anything obvious causing the described issue.
When you boot up a machine (bare metal), are you able to ping the FOG server and access it’s web interface in the browser from that machine?
-
@sebastian-roth Yes you can access the HTML side and ping the server. I’m stumped. I would be glad to let you see it your self if you have time. I’m a mid to low level linux user so I might be missing something obvious.
Claw
-
@sebastian-roth Ok some progress today. I tried the following commands and it failed
mkdir /images
mkdir /images/dev
mount -o nolock,proto=tcp,rsize=32768,intr,noatime x.x.x.x:/images /images
mount -o nolock,proto=tcp,rsize=32768,intr,noatime x.x.x.x:/images/dev/ /images/devHowever the following command worked
mount -o nolock,proto=udp,rsize=32768,intr,noatime x.x.x.x:/images /images
mount -o nolock,proto=udp,rsize=32768,intr,noatime x.x.x.x:/images/dev/ /images/devI am able to list the files in the folder and it works correctly.
So now the issue is why TCP doesn’t work on the bare metal and does work on the VM.
Suggestions?
Claw
-
@claw22000 so I presume there is a firewall between the fog server vm and the bare metal machines.
The reason vm to vm works is because they reside on the same side of the switch within the same subnet that the fog vm does. Your firewall likely allows port 80/443 from bare metal to the fog vm network. UDP may be fully allowed on the firewall? Not 100% sure of the network layout but this seems like a firewall issue. The only reason I think udp is working is because maybe an assumption was made that the fog server needed multicast capabilities?
-
I appreciate the help. When you say firewall are you talking about my PFsense Box or are we talking about something that resides in the FogServer?
Claw
-
@claw22000 said in Imaging works in VM and not on bare metal:
I appreciate the help. When you say firewall are you talking about my PFsense Box or are we talking about something that resides in the FogServer?
From what you posted so far (Debian 10 and output iptables command) I would not think this is an issue on the FOG server itself.
While I would not think the SG 200-08 (Cisco, right?) or the Netgear JGS516 do block such traffic it’s still worth to try and rule those out one by one. Please connect one of the bare metal machines directly to the SG 200-08 and see if that makes a difference. If NFS in TCP mode still doesn’t work, then could you take out the Cisco switch of the setup by connecting the Netgear uplink cable to your ESXi directly - just for a quick test I mean.
-
@claw22000 The unfortunate part is we don’t know. Could it be the PFSense box? Yes. Could it be a switch misconfiguration? Possibly.
Based on the information you’ve given us so far, though, it really seems to be a firewall type thing. Does this mean it absolutely is? No. As @sebastian-roth has alluded to, we have to take out and replace variables to more definitively get to the root of the issue.
-
@Tom-Elliott From the description so far I wouldn’t think that pfSense is connected in between.
-
@sebastian-roth Great info guys I ordered an unmanaged switch to replace the managed on to see if that corrects the issue. I don’t use any of its features since Its just My family and I. I just like to nerd out and this was a gift to play with. Haven’t change a thing on it since the day I received it. I will report back as soon as I am able to test. Should be here Tuesday.
Claw
-
@sebastian-roth Great news the unmanaged switch showed up and its solved the issue I was having. I have had that thing for going on a decade and never had an issue with it. Well I learned a big lesson if you don’t need a managed switch and your not using VLANs your better off with an unmanaged switch.
Crazy thing is never had any other issues with it.
Thank you all for helping me trouble shoot this. If any one comes across this a cheap Netgear unmanaged switch from amazon will let you test to see if this was also your issue!
Claw
-
@Claw22000 Great to hear!! So did swapping out the Cisco or the Netgear switch solve the issue? You saying better use an unmanaged is somehow confusing.
-
@sebastian-roth Sorry about the confusion. The Cisco is managed and the new Netgear is unmanaged. taking the Cisco out of the mix fixed the issue.
Claw
