DHCP problems on storage nodes

[Greg Plamondon]

@sebastian-roth
If I set the “Location” to use the main management fogserver it works but I am pulling a lot of data across the wire consuming the entire circuit.

[Wayne Workman]

@greg-plamondon I just got done skimming through this thread, I’m unsure of the problem now? Is it still DHCP related or what?

[george1421]

@greg-plamondon Sorry for the delay on this, I needed some time to research this issue. I think I have a test for us to understand what is going on here.

Hopefully you have a virtualization environment at one of the remote locations. Assuming you do, this is what I want to test.

1. Create a new (tiny) virtual machine. This just needs to be the bare minimum, it needs to be able to pxe boot.
2. Manually register that virtual machine with fog.
3. Schedule a capture / deploy (doesn’t matter) but be sure to enable the debug option.
4. PXE boot that target computer. It should boot to a command prompt. It may complain about the same ip address thing but it should display a bunch of commands and then drop you at a linux command prompt.
5. At the linux command prompt I want you to key in the following command. set | grep web. The hope is the response will be web=http://192.168.10.238/fog/
6. Now try to ping your main fog server ip address at 192.168.10.238

Something should fail above. I looked at the script where its going sideways in FOS. That script is here: https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/etc/init.d/S40network specifically at line 40. FOS will make a curl call back to your master FOG server to test if the link is up. FOS uses this command
curl -Ikfso /dev/null "${web}"/index.php --connect-timeout 5
Expanded the command is
curl -Ikfso /dev/null http://192.168.10.238/fog/index.php --connect-timeout 5

curl flags decoded
-I Doc Info
-k allow ssl connection, ignore cert
-f fail silently
-s silent
-o output file /dev/null

The other thing that could be stopping the works could be the 5 second timeout. But I would hope your fog server can respond within 5 seconds…

Wait didn’t you comment it was taking a long time to call up a web page? If your master fog server is overloaded and doesn’t respond with the index.php page within 5 seconds the remote clients will give up pxe booting. The curl command calls up a web page and not just a simple ping to see if it can reach the fog server.

[Greg Plamondon]

@george1421
I ran the command you requested on the TEST client PC.

I didn’t get any out from the command I rand…
I was able to ping the fogserver.

There are parts of the web interface that take a VERY long time to load and those are the “Storage” and “Fog Configuration” pages. The fogserver dashboard is very responsive.

[george1421]

@greg-plamondon This vm is at one of the remote locations?

[Greg Plamondon]

@wayne-workman

Yes as far as I can tell.

[george1421]

@greg-plamondon As for the output of the curl command we probably want to remove the “fso /dev/null” from the command so it prints out the results on the screen. The script uses the error level generated by curl to know success or fail.

Make the command now curl -Ik http://192.168.10.238/fog/index.php --connect-timeout 5

[Greg Plamondon]

@george1421

yes,

[Greg Plamondon]

@george1421

[george1421]

@greg-plamondon you want to change the option switches as in the command I posted. That error threw me for a second.

[Greg Plamondon]

@george1421

[george1421]

@greg-plamondon OK, that isn’t exactly what I expected. I expected a 200 code, not 302 returned (may show my ignorance). The 302 code means success but redirect which is what is happening in FOG.

I just tried this on my fog server running 1.4.4 and I got the same 302 return code. So yours is normal. That doesn’t explain why the target computer can’t access the fog server during startup.

[george1421]

@greg-plamondon Well I think we need to get better debugging information into the inits to see exactly what is going wrong.

There is a wiki page that explains how to unpack the inits. Copy the init.xz (virtual hard drive) from /var/www/html/fog/service/ipxe to /tmp on your fog server then follow the instructions in the wiki: https://wiki.fogproject.org/wiki/index.php?title=Modifying_the_Init_Image

cp /var/www/html/fog/service/ipxe/init.xz /tmp
cd /tmp
xz -d init.xz
mkdir initmountdir
mount -o loop init initmountdir

Then change into /tmp/initmountdir/etc/init.d

I’m going to work on updating S40Network to add more info so we know what its doing (wrong).

[george1421]

This is the updated S40network file.

#!/bin/bash
#
# Start the network....
#
if [[ -n $has_usb_nic ]]; then
    echo "Please unplug your device and replug it into the usb port"
    echo -n "Please press enter key to connect [Enter]"
    read -p "$*"
    echo "Sleeping for 5 seconds to allow USB to sync back with system"
    sleep 5
fi
# Enable loopback interface
echo -e "auto lo\niface lo inet loopback\n\n" > /etc/network/interfaces
/sbin/ip addr add 127.0.0.1/8 dev lo
/sbin/ip link set lo up

sleep 10

# Generated a sorted list with primary interfaces first
read p_ifaces <<< $(/sbin/ip -0 -o addr show | awk -F'[: ]+' 'tolower($0) ~ /link[/]?ether/ && tolower($0) ~ /'$mac'/ {print $2}' | tr '\n' ' ')
read o_ifaces <<< $(/sbin/ip -0 -o addr show | awk -F'[: ]+' 'tolower($0) ~ /link[/]?ether/ && tolower($0) !~ /'$mac'/ {print $2}' | tr '\n' ' ')
ifaces="$p_ifaces $o_ifaces"
for iface in $ifaces; do
    echo "Starting $iface interface and waiting for the link to come up"
    echo -e "auto $iface\niface $iface inet dhcp\n\n" >> /etc/network/interfaces
    /sbin/ip link set $iface up

    # Wait till the interface is fully up and ready (spanning tree)
    timeout=0
    linkstate=0
    until [[ $linkstate -eq 1 || $timeout -ge 35 ]]; do
        let timeout+=1
        linkstate=$(/bin/cat /sys/class/net/$iface/carrier)
        [[ $linkstate -eq 0 ]] && sleep 1 || break
    done
    [[ $linkstate -eq 0 ]] && echo "No link detected on $iface for $timeout seconds, skipping it." && continue
    for retry in $(seq 3); do
		echo "## Bringing up interface $iface ##"
		/sbin/udhcpc -i $iface --now
        ustat="$?"
		echo "## Calling the fog server ${web}/index.php ##"
        curl -Ikfso /dev/null "${web}"/index.php --connect-timeout 5
        cstat="$?"
        # If the udhcp is okay AND we can curl our web
        # we know we have link so no need to continue on.
        # NOTE: the link to web is kind of important, just
        # exiting on dhcp request is not sufficient.
		
		if [[ $ustat -eq 0 && $cstat -eq 0 ]]; then
			echo "## We have an IP address on $iface and the Master FOG server responded to our query ##"
		fi 
        [[ $ustat -eq 0 && $cstat -eq 0 ]] && exit 0

		if [[ $ustat -eq 1 ]]; then
			echo "## DHCP failed on $iface ##"
		fi
		if [[ $cstat -eq 1 ]]; then
			echo "## The Master FOG server failed responded to our query ##"
		fi
		echo "Either DHCP failed or we were unable to access ${web}/index.php for connection testing."
        sleep 1
    done
    echo "No DHCP response on interface $iface, skipping it."
done

# If we end up here something went wrong as we do exit the script as soon as we get an IP!
if [[ -z $ifaces ]]; then
    echo "No network interfaces found, your kernel is most probably missing the correct driver!"
else
    echo "Failed to get an IP via DHCP! Tried on interfaces(s): $ifaces"
fi
echo "Please check your network setup and try again!"
[[ -z $isdebug ]] && sleep 60 && reboot
echo "Press enter to continue"
read
exit 1

My comments have a double pound on both sides. Once you update the S40network file then you need to repack the inits and then move to your storage node at this test location. Understand this is only a test init so that we can find out what is going on. You will want to keep your untouched init file after the debugging is over.

---

My intuition is still telling me this could be a spanning tree issue, even though you said you checked that. Your debug FOS system has access so that doesn’t specifically have the same conditions as a physical system. I guess you could always try to image a virtual machine at the remote location and see if it works (even before tweaking the inits). In the case of a VM it will not drop the link on the physical switch because the VM is connected to a vswitch.

[Greg Plamondon]

@george1421

ok, I have the S40network modiefied and the init repacked. do I need to run the same command for testing?

[Greg Plamondon]

@george1421

I am getting an error in the S40network file.

[george1421]

@greg-plamondon Whelp, that’s why I’m not a programmer

Here is what needs to be fixed, sorry.

This is the bad line

if [ $ustat -eq 0 && $cstat -eq 0 ]; then

This is what it should have been

if [[ $ustat -eq 0 && $cstat -eq 0 ]]; then

Awe, crud and then the next errors you will find are a few lines down.

		if [ $ustat -eq 1 ]; then
			echo "## DHCP failed on $iface ##"
		fi
		if [ $cstat -eq 1 ]; then
			echo "## The Master FOG server failed responded to our query ##"
		fi

need to have the brackets too

		if [[ $ustat -eq 1 ]]; then
			echo "## DHCP failed on $iface ##"
		fi
		if [[ $cstat -eq 1 ]]; then
			echo "## The Master FOG server failed responded to our query ##"
		fi

[Greg Plamondon]

@george1421

[Greg Plamondon]

@george1421

My question is if the connection is good “## We have an IP address on eth0 and the Master FOG server responded to our query ##” Why does it then disconnect the eth0 interface and attempt to obtain an IP when it already has one that is working?

[Greg Plamondon]

@george1421

now if I issue a: /etc/init.d/S40network restart
I get this: