i219LM NIC, ASUS Q170M-C Motherboard

[george1421]

@Sebastian-Roth Maybe even like if you don’t enumerate the network interfaces on the first pass, wait for 30 seconds (or what ever time) and then try one more time. This would keep booting fast for well behaved network interfaces and then give the “slow” ones a second chance to come up.

Tom: What about the usb kernel flag, would that give enough pause for the network to come up on these boards? I don’t know where else this flag is used or the impact.

[Tom Elliott]

@Sebastian-Roth I think the problem is the network interface is not showing up at all, not due to a timing thing. For whatever reason, the device never thinks it’s NIC is in an “Up” state. Because it never detects it as up, it never adds it to the list.

Maybe we should move the “link up” state and just try to bring the nic up ourselves?

[Sebastian Roth]

@Tom-Elliott We don’t actually check the “up” state but the carrier link state. Possibly the NIC / driver just does not show up as link connected until the interface is brought UP. This is kind of stupid, but well…

[dustindizzle11]

@Sebastian-Roth @Tom-Elliott @george1421

I took your advice and modified the init.xz to make a custom init, and all I did was just add “auto eth0” and “auto iface eth0 inet dhcp” to the interfaces file. It brings up eth0 and it images after adding this. So at this point it works. The fix is probably not the best fix in the world, but hey, it works!

[Tom Elliott]

https://github.com/FOGProject/fogproject/blob/dev-branch/src/buildroot/system/skeleton/etc/init.d/S40network

This is the file that handles network loading and generating the /etc/network/interfaces file.

If you look at line 24 we’re defining the “up” state of the interface. Then we check the carrier which should tell us if a link is detected or not.

I think, in this particular system at least, the link isn’t able to be detected. Do you see any messages while it’s booting? Particularly the message would be: “No link detected on <interface> for # seconds, skipping it.” It tries to do this for 1 second every time. Maybe we need to up the “sleep” to use the “timeout” value instead of only waiting for 1 minute?

I would think the carrier state would be able to update within 36 seconds, but maybe our thoughts are wrong? We aren’t bringing the interface down and back up through each check (not that that should be necessary).

One thing, I should add, is that we do add interfaces to the interfaces file. So it’s more interesting, based on what I can see, that it’s not finding an interface to insert to begin with. If you look at lines 18 and 19 of the file, then at line 23 within the for loop, you can clearly see it’s trying to append the interface to the /etc/network/interfaces file. The fact that it’s not inserting the interface tells me we’re not finding the interface to begin with.

If you can boot the system into “debug” again, can you try the commands:

ip -0 -o addr show | awk -F'[: ]+' '/link[/]?ether/ && /<MACOFHOST>/ {print $2}'
ip -0 -o addr show | awk -F'[: ]+' '/link[/]?ether/ && !/<MACOFHOST>/ {print $2}'

Change “<MACOFHOST>” with the mac address with colons of that host. You may need to try this command twice. Once with ALL CAPS, once with all lower. Maybe it’s the upper/lower case that’s the issue?

@Sebastian-Roth I don’t think timing is the issue here. At the point where S40network is run, the kernel has loaded all the drivers. I’m suspecting, possibly, it may be the awk -F statement? (Maybe ether isn’t shown until the link is up?) That doesn’t make sense why it works otherwise though.

If it’s not the awk, (I doubt it), it is probably the ip addr show. Maybe we should use ip -0 -o link show?

[Sebastian Roth]

@dustindizzle11 Would you please show us the script you used?

With the unmodified init.xz what do you see on screen? I guess you see No network interfaces found, your kernel is most probably missing the correct driver!, right?

[dustindizzle11]

@Tom-Elliott Thanks for the response! So I got some mixed results with the commands you gave me, specifically the “awk” commands. It seems that capitalization does matter. To make sense of the picture below…

First I ran “ip -0 -o link show”, which lists eth0 as a network device.

Next I ran “ip -0 -o addr show | awk -F’[: ]+’ ‘/link[/]?ether/ && !/<MACOFHOST>/ {print $2}’” with the MACOFHOST in all Capitals (notice the mac only has one letter to capitalize in the picture). This gave me “eth0” in return.

After that I ran the same exact command as above, but with the MACOFHOST being lowercase (the “f” is lowercase). In return I got nothing. So lowercase did not output anything for me in this case.

Then I ran the version without the exclamation point, (which I’m assuming means NOT command wise), with the MAC being CAPITALIZED “ip -0 -o addr show | awk -F’[: ]+’ ‘/link[/]?ether/ && /<MACOFHOST>/ {print $2}’”. In return I got nothing.

Finally, I ran the above command again, but lowercase instead. In return I got “eth0” again.

So it seems that capitalizing does the opposite for each command. With the “!” before the MACOFHOST, having the MAC be Capitalized provides “eth0” in return. Without the “!” before MACOFHOST, having the MAC be Lowercase provides “eth0” in return. Hopefully this helps troubleshoot the issue in some way.

[dustindizzle11]

@Sebastian-Roth When imaging it just zips to a black screen. I am not able to see any error messages. All I did to the init.xz was just modifying the /etc/network/interfaces and adding eth0 to the list to make it work.

[Sebastian Roth]

@dustindizzle11 Thanks for doing the tests and posting a picture. As we see eth0 is showing up right when you enter the first command. Is this with your modified init.xz or with the original one?

[dustindizzle11]

@Sebastian-Roth Original

[Sebastian Roth]

@Tom-Elliott This definitely looks like a timing issue to me. As he said th last picture is from starting the original init.xz in debug and then running ip command without any other actions I suppose. As we see in the picture the interface eth0 is shown properly. So I am pretty sure that it is timing. The interface is not “available” on kernel driver load time straight away but somehow delayed…

@dustindizzle11 Thanks again! Can you please try the following stuff: Use the original init.xz and add has_usb_nic=1 to the kernel parameters in the host configuration for one of your test clients having this issue. Then when you boot up the client it asks you to replug the (USB-)NIC. Just ignore the message, wait for 10 seconds and hit ENTER to proceed. Does it solve the issue?!

[Tom Elliott]

@Sebastian-Roth I say it isn’t timing not because it couldn’t be, but rather I think the device is there but for whatever reason the carrier link is unrecognized. So I think to further test this theory we would need to see the carrier state not the simple ip command. If it is timing that’s alright as we just increase wait time and all should work fine. If it’s not then I think we need to figure out why the carrier link is unseen.

[dustindizzle11]

@Sebastian-Roth I added “has_usb_nic=1” to the Host Kernel Arguments section and it did the same thing it would do before, which is just zip to a black screen. This is the behavior I get when I don’t get an address. I am using the 4.6.2 Kernel. I didn’t get any messages about re-plugging the USB NIC in though. Let me know if you need to me try anything else.

[Sebastian Roth]

@dustindizzle11 said:

I didn’t get any messages about re-plugging the USB NIC in though. Let me know if you need to me try anything else.

Well then there is no delay and you will run into the same issue again and again I am sure. Can you please post a screenshot of the host kernel setting in the web GUI…

[dustindizzle11]

@Sebastian-Roth

[Sebastian Roth]

@dustindizzle11 Is your kernel file actually named 4.6.2? What do you see when running ls -al /var/www/{,html}/fog/service/ipxe/?

As well would you mind posting a sceenshot of the host settings where we see the MAC address as well. I just want to make sure that we are not running in circles not seeing the obvious reason…

[dustindizzle11]

@Sebastian-Roth Yes, I named it “4.6.2” in “/var/www/{,html}/fog/service/ipxe/”. I am positive the kernel file is named correctly and that it loads when PXE booting. Below are the screenshots you requested.

[Sebastian Roth]

@dustindizzle11 Looks all good to me too. So even more I am wondering why you don’t see the waiting time when setting the has_usb_nic parameter… Possibly this is an older version of the init.xz file (maybe from FOG 1.2.0 when I look at the size of just about 10 MB??). The most current one is around 17 MB (more drivers). So I am pretty sure this is what’s going on.

As you already know how to modify the init.xz file please do so and copy & paste the full content of the file etc/init.d/S40network extracted from the init.xz file you have.

[Tom Elliott]

I’ve updated the S40network script to use a case insensitive approach.

The beauty is all you need to do to test it is rerun the installer and you will have the latest init.

Please update if you can and let us know if it’s working any better.

[dustindizzle11]

@Tom-Elliott I installed Fog1.2 on another system and copied over the init.xz to my current fog setup. After sending the image, it just zipped through to a black screen again . So it did the same thing that it did before. shoot…

@Sebastian-Roth Below is the S40network contents from my original init.xz…

#!/bin/sh

Start the network…

case “$1” in
start)
echo “Starting network…”
/sbin/ifup -a
;;
stop)
echo -n “Stopping network…”
/sbin/ifdown -a
;;
restart|reload)
“$0” stop
“$0” start
;;
*)
echo “Usage: $0 {start|stop|restart}”
exit 1
esac

exit $?