No network interfaces found (verifyNetworkConnection)

[Wayne Workman]

@Sebastian-Roth @george1421 @Tom-Elliott

I’ve had more issues with this error, and I’ve done more experimentation.

Turns out, changing switches isn’t the thing that fixes it, changing to any other network port is.

Also - when this problem is experienced (verifyNetworkConnection), that problem lingers on the port itself. It has nothing to do with the connected computer.

Ideas?

[MRCUR]

@Wayne-Workman Have you verified that the problem port on the switch isn’t being put into blocking mode by STP? Running a sh span should give you the port status (at least on an HP switch, Cisco is likely the same). This would “linger” depending on your switch STP config.

[Wayne Workman]

Had a talk with my network people. They said BPDU guard is turned off for the switch I’m using.

Also - just noticed, this particular machine I’m testing with right now (an optiplex 7010) has indicator lights on the NIC port.

When the computer’s firmware is just starting the NIC, the lights come on. There are two lights on the left and right side. one is solid orange, the other blinks yellow.

As soon as bzImage is loaded, the lights turn off and do not come back on and the above OP error is displayed.

Could it be something to do with the kernel turning on the interface? @Sebastian-Roth @Tom-Elliott @george1421 @MRCUR

See the below video, I recorded in a way that you can see the NIC lights:

https://www.youtube.com/embed/Hgn1mozsQsE

[george1421]

Are you booting the 7010 in bios mode or efi. I still have that 7010 on my workbench.

I did note when I was playing with the intel nuc a few weeks ago. It would drop the link light ever stage of the booting process. PXE Rom->iPXE->bzImage This why standard stp is not where we need to be because it takes 20-30 seconds for stp to start forwarding after each transition. Not saying this is your case, but I can understand why.

After watching the video, I see that your system has a longer than normal pause for the pxe roms to init. In my case, I pressed F12 to get the boot menu. When I selected the network boot option, my link light went out for 2 seconds then it started the dhcp process right away. The same for the transition from pxe to iPXE the link light is out for about 2-3 seconds then the process continues.

[Wayne Workman]

@george1421 said:

Are you booting the 7010 in bios mode or efi.

Legacy mode.

[george1421]

Sorry I was editing my previous post, when you posted. I’m only seeing between 2-3 seconds of where the link light is out per transition. And In my office I just have a really old Linksys SLM2008 switch.

[Wayne Workman]

@george1421 I have an old 100 meg unmanaged switch I can put between the computer and the building network… I’ll see if that makes a difference… if it does, the kernel is not at fault.

[george1421]

@Wayne-Workman I think the problem (according to your video) is way before the kernel gets loaded. The unmanaged switch will be a good test. It will stink for download speed, but it will be a good test for link recovery.

[Wayne Workman]

@george1421 The most interesting thing happened. I’m going to explain as best I can the exact sequence of events.

I hooked up this little unmanaged 100 Mbps 5 port switch between the buiding network and this Optiplex 7010. The optiplex was connected to port 1, the building connected to port 5 (not that it matters).

I didn’t get the OP error. Things started working!

I thought, OK, I need to get a video of what it looks like when it’s working properly. So I turn off the computer, and get my phone setup and the switch in just the right spot for filming and then I fire the computer back on.

Neither the NIC on the computer nor port 1 on the switch ever lit up… what?

I traded what building port that port 5 was connected to, and then it worked again. I got a video of that:

https://www.youtube.com/embed/SUJPdjvZEPc

Any ideas? Man this is nuts.

[george1421]

As you were getting setup to video, did you restart the switch or plug the wiring going to the building at all?

In the last video, the booting timing, was similar to what I’m seeing on my bench. Its about 2 seconds between stages. Because you have a dumb switch between the computer and the building wiring, the computer going up and down shouldn’t impact the port forwarding state on the building network. Its not clear why transferring between building side port 5 to another port had any impact other than it would have transitioned the link state on the building switch.

Are you guys using any type of nac or nap?

[Sebastian Roth]

@george1421 said:

Are you guys using any type of nac or nap?

Same question came to my mind! Are you somehow able to get a serial connection on that building switch? Would be most interesting to see what it talks about the link going down/up and possible other messages (NAC related maybe).

@Wayne-Workman said:

Turns out, changing switches isn’t the thing that fixes it, changing to any other network port is.

So this issue only occurs on one switch port (building switch)?? As soon as you plug into a different port on the same switch things are fine? Either your network guys should have a close look at the specific port configuration or they better abandon that port altogether (if it’s just that single one!).

[Wayne Workman]

@Sebastian-Roth said:

So this issue only occurs on one switch port (building switch)?? As soon as you plug into a different port on the same switch things are fine? Either your network guys should have a close look at the specific port configuration or they better abandon that port altogether (if it’s just that single one!).

No, not just one port. Once I get the error message in the OP for any port, that port seems to be permanently voo-doo’d.

I don’t know what nac or nap is. If I ask my network people this, will they know what I’m talking about? We are 100% Cisco end-to-end.

[Sebastian Roth]

Hopefully they have an idea what NAC is: https://en.wikipedia.org/wiki/Network_Admission_Control

[george1421]

If you have to register (with some kind of network controller) new out of the box computers before you can use them then you have a NAC/NAP (different name same function) that manages access for your network. Since you don’t know the name you probably don’t have this.

When you say you have cisco throughout. Is that real cisco switches (catalyst series) or cisco small business switches (SG series)? The reason why I ask, is the SG series has a nasty feature that is turned on by default called smart port. That feature will try to determine what device is connected to the port then assign the device to the predefined vlan. This “feature” listens for the connected device to announce what it is. This listening phase is several seconds, much like stp learning phase. I’m not a cisco person, so I don’t know if the catalyst series has this same function. But this smart port has caused me pain several times in that with it on, any vlan I set of a port will get overwritten with the smart port setting.

[Wayne Workman]

@george1421 They are all catalyst switches - varying models but none are over maybe 4 or 5 years old (yay free government money).

[Wayne Workman]

@george1421 said:

As you were getting setup to video, did you restart the switch or plug the wiring going to the building at all?

I don’t think I did, I think I just moved the switch to be on top of the computer. And of course I held the power button on the computer until it turned off.

I’ll repeat this tomorrow and see if it’s repeatable.

[Wayne Workman]

More info on this.

I removed the mini-switch from the equation, and tried to image the computer.

It wouldn’t even get a DHCP lease, but the NIC lights did turn on after a long while… then it failed. PXE-M0F.

Turned it off, turned it back on.

It got DHCP proper, but then got the OP error.

Turned it off, turned it back on.

Same thing, OP error.

I put the switch between the computer and the building network again, imaging works perfectly.

[MRCUR]

@Wayne-Workman I would ask for verification that “spanning-tree bpdufilter enable” is set for the port OR that “spanning-tree portfast (int num) enable” is set.

[george1421]

@Wayne-Workman I agree that something is not configured correctly on the building switch. By putting an unmanaged switch in between it fixes (masks) the issue. Because the building switch is never seeing the port drop as the target computer transitions between phases during booting.

[Tom Elliott]

@george1421 I’ve actually seen this exact issue with our systems as well. Not very often, and the fix is indeed to put a managed switch between. Or, alter the firmware on windows to now put the nic into a “Power-saving” mode.

The latter is not a simple thing and is only good for that instance of the OS, if your image doesn’t have those firmware changes in place, they will not hold.