Fog iPXE Menu no input

[AxeMeAQuestion22]

We are experiencing a kind of weird issue. When booting a machine via iPXE the Fog menu comes up as expected, but it does not allow keyboard input. I have tried both the dev and stable branches. I have also reinstalled.

On a potentially unrelated note we were getting an error for missing autoexec.ipxe. I created the file with the below:

#!ipxe
ifopen net0
dhcp net0

chain ${boot-url}/scripts/menu_EFI.ipxe

Found Here

This resolved the missing file issue. However the problem of not being able to navigate the menu persisted. I updated to the most recent branches and reinstalled again, still to no avail. I updated the iPXE binaries following This link.

This also did not resolve the issue. I feel like I’m missing something simple here. Any help would be appreciated. Thank you!

iPXE version (1.21.1+)
Fog Version 1.5.10.1657
Kernel Version 1.5.10.1657

[george1421]

@AxeMeAQuestion22 What hardware has this issue? Is it one model/manufacturer, all models/manufacturer?

Is the device a laptop or desktop? If laptop does using an external keyboard plugged directly into the laptop work?

What language is this keyboard?

I have to think there is something unique with this hardware that is causing a keyboard issue with ipxe.

[AxeMeAQuestion22]

This post is deleted!

[AxeMeAQuestion22]

@george1421 We did some further testing with this. It seems to be all USB keyboards. Most of our machines are Lenovo, but I also have a few custom-built devices, and they all seem to experience this.

Our laptops, which are also Lenovo-branded, can utilize the menu normally.

[george1421]

@AxeMeAQuestion22 I see a slight contradiction, maybe in the way I read it. You have some lenovos that work and some that don’t.

It almost sounds like a usb controller issue (just a wild guess at the moment).

I just want to say this is an issue with the ipxe binaries since ipxe manages the FOG iPXE menu. This has nothing to do with FOS (yet) that would be bzImage and init.xz which haven’t been sent to the target computer at this point.

When on the ipxe menu does it accept the enter key where the arrow keys are not working?

Is this a US english keyboard? If no what language.

Just to confirm that you recompiled iPXE using the instructions you pointed to? Verify the files in tftp directory have the current date that you recompiled them.

What model/make does not work vs what make/model does work?

[AxeMeAQuestion22]

@george1421 The enter key does not work either; you are trapped in the menu until time expires.

These are US English Keyboards.

Yes, I followed those linked instructions to rebuild the binaries, and it did indeed update the date.

It is all of our desktops experiencing the issue. Most of them are lenovo m715q and lenovo p3 tiny. The custom built desktops also do, they have aorus motherboards in them.

The laptops, which are a variety of Lenovo laptops are working as expected.

[Tom Elliott]

@AxeMeAQuestion22 Do the BIOS options have a “Legacy USB” mechanism to them?

[AxeMeAQuestion22]

@Tom-Elliott The new BIOS flash does not have CSM/Legacy support any longer.

[jfernandz]

I’m experiencing the same problem with keyboard input, in fact the keyboard input doesn’t work either till the OS boots, it doesn’t work in Grub, I feel like if iPXE were having a kind of issue with the firmware.

Also I’ve noticed … I’ve had to replace ${next-server} in the ipxescripts files with the actual FOG Server IP as it seems iPXE isn’t able to retrieve this value from the DHCP service in the FOG Server. Not sure if the firmware issues could be causing this also.

[Tom Elliott]

@jfernandz next-server comes from the dhcp server option 66 (I think that’s what it is in Windows DHCP at least?)

That said, often when next-server is empty it’s usually because there’s multiple dhcp servers trying to tell the same single client their next-server and it doesn’t know which one to use, so it just goes blank. That’s not to say it couldn’t be a firmware issue, but just expressing my past experiences.

As for the Keyboard:

@AxeMeAQuestion22 @jfernandz

The “legacy” setting shouldn’t be a Firmware related/removed thing. It’d be under (generally) the USB options of the BIOS settings. Not “Legacy BIOS modes” or anything. Specifically it’d usually be surrounding the USB specifically.

Now, I don’t have this particular gear, so I could be wrong, just again speaking from prior experiences of many different systems (in general.)

Edit:

Looking for what I see, if you’re using 3rd party keyboards (non standard that came with the system, or basic basic keyboards, it seems lenovo just might drop them beyond initial key entries (f1 for bios setup mode or whatever.)

https://forums.lenovo.com/t5/ThinkCentre-A-E-M-S-Series/M90q-gen-5-usb-keyboard-not-working-in-bios/m-p/5358388

This isn’t quite the same machine, but just kind of describing what I was thinking about with the USB Keyboard/Legacy Support.

[jegan1337]

@jfernandz For what it’s worth, I have a bunch of m920q and brand new m75q Lenovo desktops that are all suddenly experiencing the same issue. Previously any keyboard (including 3rd party and bluetooth) could interact with the Fog iPXE menu with no problem. There hasn’t been any BIOS updates for the m920q models in quite some time and we’ve standardized our BIOS settings. It just seemingly stopped working after updating fog.

[Tom Elliott]

@jegan1337 That’s possible, that maybe something is missing (or was removed from IPXE)

If someone’s able to do a git bisect, figure out when it worked, figure out when it stopped.

Likely you would first do:

1.5.10

now

then use bisect to let it jujmp around and until you see when it narrows down exactly when it broke?

[jfernandz]

@Tom-Elliott said in Fog iPXE Menu no input:

@jfernandz next-server comes from the dhcp server option 66 (I think that’s what it is in Windows DHCP at least?)

That said, often when next-server is empty it’s usually because there’s multiple dhcp servers trying to tell the same single client their next-server and it doesn’t know which one to use, so it just goes blank. That’s not to say it couldn’t be a firmware issue, but just expressing my past experiences.

Hmm … my problem isn’t an empty value for next-server, rather … it comes with an IP, but its the DNS server IP address! …

Regarding the keyboard issue … we are working with Dell Precission 3930 Rack machines, but we also have in the same cluster a couple of Dell Precission 7960 Rack machines where I didn’t tested, I can say a couple of days ago, we upgraded the BIOS of the 3930 which are the ones I can detect this issue, that’s why I was guessing this could be a problem between iPXE and upgraded firmware

[Tom Elliott]

@jfernandz It’s probably a combination of both. At least, best what i can tell about the 3930’s at least?

Either way, similar to the post about bisect, below, might help us out as well.

I mean firmware nearly always plays a part in things, but that’s not to say that something with the iPXE code is not at fault, either.

[jfernandz]

@Tom-Elliott said in Fog iPXE Menu no input:

Either way, similar to the post about bisect, below, might help us out as well.

I’d need more details, some article or document on how to perform this bisect, but I’m interested in contributing, sure

[Tom Elliott]

@jfernandz https://stackoverflow.com/questions/4713088/how-do-i-use-git-bisect

This is just very basic overview and probably unhelpful.

If you’re familiar with git, you can switch to different points in time (via the dev-branch) to figure out when we last knew things were good. So first, I’d suggest trying to find exactly when that would’ve been.

I’d likely start, using dev-branch as the primary “tester case”

git checkout -b dev-branch

If you haven’t already done so of course.

Then I’d put the branch at the same level as when we pushed out 1.5.10:

git checkout 081d1f4

Run your install and see if things worked, if they do:

git checkout dev-branch
git bisect start
git bisect bad
git bisect good 081d1f4

Of course the 081d1f4 is the hash of the commit where things were good: this might or might not be the 1.5.10 release, so your mileage may very and may need some more adjustment.

You can find these hashes on https://github.com/FOGProject/fogproject/commits/dev-branch/?after=f7fd3a7a3f188b4540f089a7f38c4d368b53376c+174 using the different pages at the bottom to find a point in time, and they show the hashes on the right and side of each commit.

It will iterate through things and you just mark after each install/test what worked (as good or bad) by testing:

If it still fails, you’d run git bisect bad, if it succeeds, you would run git bisect good each iteration.

It’s tedious I know. but it will help us find the exact point things broke and hopefully from what (maybe it’s something we did, maybe something within ipxe, but gives us a starting point to test things out.)

Thanks