Fog iPXE Menu no input

jfernandz

I’m experiencing the same problem with keyboard input, in fact the keyboard input doesn’t work either till the OS boots, it doesn’t work in Grub, I feel like if iPXE were having a kind of issue with the firmware.

Also I’ve noticed … I’ve had to replace ${next-server} in the ipxescripts files with the actual FOG Server IP as it seems iPXE isn’t able to retrieve this value from the DHCP service in the FOG Server. Not sure if the firmware issues could be causing this also.

Tom Elliott

@jfernandz next-server comes from the dhcp server option 66 (I think that’s what it is in Windows DHCP at least?)

That said, often when next-server is empty it’s usually because there’s multiple dhcp servers trying to tell the same single client their next-server and it doesn’t know which one to use, so it just goes blank. That’s not to say it couldn’t be a firmware issue, but just expressing my past experiences.

As for the Keyboard:

@AxeMeAQuestion22 @jfernandz

The “legacy” setting shouldn’t be a Firmware related/removed thing. It’d be under (generally) the USB options of the BIOS settings. Not “Legacy BIOS modes” or anything. Specifically it’d usually be surrounding the USB specifically.

Now, I don’t have this particular gear, so I could be wrong, just again speaking from prior experiences of many different systems (in general.)

Edit:

Looking for what I see, if you’re using 3rd party keyboards (non standard that came with the system, or basic basic keyboards, it seems lenovo just might drop them beyond initial key entries (f1 for bios setup mode or whatever.)

https://forums.lenovo.com/t5/ThinkCentre-A-E-M-S-Series/M90q-gen-5-usb-keyboard-not-working-in-bios/m-p/5358388

This isn’t quite the same machine, but just kind of describing what I was thinking about with the USB Keyboard/Legacy Support.

jegan1337

@jfernandz For what it’s worth, I have a bunch of m920q and brand new m75q Lenovo desktops that are all suddenly experiencing the same issue. Previously any keyboard (including 3rd party and bluetooth) could interact with the Fog iPXE menu with no problem. There hasn’t been any BIOS updates for the m920q models in quite some time and we’ve standardized our BIOS settings. It just seemingly stopped working after updating fog.

Tom Elliott

@jegan1337 That’s possible, that maybe something is missing (or was removed from IPXE)

If someone’s able to do a git bisect, figure out when it worked, figure out when it stopped.

Likely you would first do:

1.5.10

now

then use bisect to let it jujmp around and until you see when it narrows down exactly when it broke?

jfernandz

@Tom-Elliott said in Fog iPXE Menu no input:

@jfernandz next-server comes from the dhcp server option 66 (I think that’s what it is in Windows DHCP at least?)

That said, often when next-server is empty it’s usually because there’s multiple dhcp servers trying to tell the same single client their next-server and it doesn’t know which one to use, so it just goes blank. That’s not to say it couldn’t be a firmware issue, but just expressing my past experiences.

Hmm … my problem isn’t an empty value for next-server, rather … it comes with an IP, but its the DNS server IP address! …

Regarding the keyboard issue … we are working with Dell Precission 3930 Rack machines, but we also have in the same cluster a couple of Dell Precission 7960 Rack machines where I didn’t tested, I can say a couple of days ago, we upgraded the BIOS of the 3930 which are the ones I can detect this issue, that’s why I was guessing this could be a problem between iPXE and upgraded firmware

Tom Elliott

@jfernandz It’s probably a combination of both. At least, best what i can tell about the 3930’s at least?

Either way, similar to the post about bisect, below, might help us out as well.

I mean firmware nearly always plays a part in things, but that’s not to say that something with the iPXE code is not at fault, either.

jfernandz

@Tom-Elliott said in Fog iPXE Menu no input:

Either way, similar to the post about bisect, below, might help us out as well.

I’d need more details, some article or document on how to perform this bisect, but I’m interested in contributing, sure

Tom Elliott

@jfernandz https://stackoverflow.com/questions/4713088/how-do-i-use-git-bisect

This is just very basic overview and probably unhelpful.

If you’re familiar with git, you can switch to different points in time (via the dev-branch) to figure out when we last knew things were good. So first, I’d suggest trying to find exactly when that would’ve been.

I’d likely start, using dev-branch as the primary “tester case”

git checkout -b dev-branch

If you haven’t already done so of course.

Then I’d put the branch at the same level as when we pushed out 1.5.10:

git checkout 081d1f4

Run your install and see if things worked, if they do:

git checkout dev-branch
git bisect start
git bisect bad
git bisect good 081d1f4

Of course the 081d1f4 is the hash of the commit where things were good: this might or might not be the 1.5.10 release, so your mileage may very and may need some more adjustment.

You can find these hashes on https://github.com/FOGProject/fogproject/commits/dev-branch/?after=f7fd3a7a3f188b4540f089a7f38c4d368b53376c+174 using the different pages at the bottom to find a point in time, and they show the hashes on the right and side of each commit.

It will iterate through things and you just mark after each install/test what worked (as good or bad) by testing:

If it still fails, you’d run git bisect bad, if it succeeds, you would run git bisect good each iteration.

It’s tedious I know. but it will help us find the exact point things broke and hopefully from what (maybe it’s something we did, maybe something within ipxe, but gives us a starting point to test things out.)

Thanks

jikemin

Has there been a solution to this problem? I have tried many versions of FOG, but I can’t input anything on the keyboard when entering the iPEX menu.

hyperbrute

@jikemin same here.
Exactly same problems:
No keyboard input in ipxe menu.
(and no autoexec.ipxe too but i created the file as @AxeMeAQuestion22 did)
With computers:

Dell OptiPlex 3040 Bios 1.20.1 (last update)
Lenovo Thinkcentre M910T Bios 1.5A (last update)

iPXE version (1.21.1+)
Fog Version 1.5.10.1660
Kernel Version 1.5.10.1660

Thanks

pposca

Any update on this? I am having the same issue with the keyboard when booting from UEFI. It seems, however, work properly in Legacy mode (BIOS).

SotY

I’m new to FOG and this is an issue I also faced from the beggining. We have multiple Lenovo tiny computers: M710q, M910q, M70q gen 2, M70q gen 3 and usb keyboard is not working in iPXE menu for any of them when booting in EFI mode. Everything is fine when booting in bios mode.

I found this discussion on iPXE project github: https://github.com/ipxe/ipxe/discussions/308
After recompiling iPXE binaries with define>USB_HCD_USBIO uncommented keyboard started working for all in EFI mode. I used FOG build script but had to modify it a little bit. Compilation of ncm–ecm–axge.efi fails when USB_HCD_USBIO is enabled, but I won’t use it, so I just disabled compilation of it. All other images compile just fine. I also modified make to use all available cpu cores to make compilation faster. This is modified script: https://pastebin.com/9ViJBATg

SotY

@SotY said in Fog iPXE Menu no input:

I’m new to FOG and this is an issue I also faced from the beggining. We have multiple Lenovo tiny computers: M710q, M910q, M70q gen 2, M70q gen 3 and usb keyboard is not working in iPXE menu for any of them when booting in EFI mode. Everything is fine when booting in bios mode.

I found this discussion on iPXE project github: https://github.com/ipxe/ipxe/discussions/308
After recompiling iPXE binaries with define>USB_HCD_USBIO uncommented keyboard started working for all in EFI mode. I used FOG build script but had to modify it a little bit. Compilation of ncm–ecm–axge.efi fails when USB_HCD_USBIO is enabled, but I won’t use it, so I just disabled compilation of it. All other images compile just fine. I also modified make to use all available cpu cores to make compilation faster. This is modified script: https://pastebin.com/9ViJBATg

After upgrading FOG from release 1.5.10.1660 to release 1.5.10.1667 iPXE images won’t compile with USB_HCD_USBIO enabled. I didn’t dig deep yet, but it seems that changes in console.h, general.h or settings.h that script use causes this.

Tom Elliott

@SotY The files of console.h/general.h/settings.h were updated 2 - 4 months ago. I don’t think anything here is the cause of the failure. I’ve attempted building and am seeing the same error.

This leads me to think something changed with ipxe code particularly surrounding USB_HCD_USBIO and efi built binaries. It seems the efi builder has a native compiler point for it’s own USBIO that’s being constructed separately from the “common” source.

If you just build the binaries without the USBIO configured, does it break things? I know you found a fix and wanted to try it, but it seems mcb30 (main developer of ipxe) made a change that seems to make the usbio drivers act differently in the past. What those changes were, I do not know at this time.

I am more surprised, in my quick review of things, that this even worked before as the components I see erroring haven’t change significantly.

If I change the usbio_driver line to efi_usbio_driver in the drivers/usb/usbio.c file, and re-run the builder, things build successfully. (Including the ncm–ecm-axge.).

If that works, I can put my diff file and copy it with the build script but I’m scared something I’m doing in the driver/usb/usb.c file might be unintentionally making things worse (not better) even though I’m getting successful builds now.

SotY

@Tom-Elliott said in Fog iPXE Menu no input:

@SotY The files of console.h/general.h/settings.h were updated 2 - 4 months ago. I don’t think anything here is the cause of the failure. I’ve attempted building and am seeing the same error.

This leads me to think something changed with ipxe code particularly surrounding USB_HCD_USBIO and efi built binaries. It seems the efi builder has a native compiler point for it’s own USBIO that’s being constructed separately from the “common” source.

If you just build the binaries without the USBIO configured, does it break things? I know you found a fix and wanted to try it, but it seems mcb30 (main developer of ipxe) made a change that seems to make the usbio drivers act differently in the past. What those changes were, I do not know at this time.

I am more surprised, in my quick review of things, that this even worked before as the components I see erroring haven’t change significantly.

If I change the usbio_driver line to efi_usbio_driver in the drivers/usb/usbio.c file, and re-run the builder, things build successfully. (Including the ncm–ecm-axge.).

If that works, I can put my diff file and copy it with the build script but I’m scared something I’m doing in the driver/usb/usb.c file might be unintentionally making things worse (not better) even though I’m getting successful builds now.

I have config files from previous version and all 3 were changed. I just narrowed it down to changes in general.h file. Compiling with USB_HCD_USBIO enabled works fine when using previous general.h and new console.h and settings.h files. I need more time to find out which change broke it. These are the changes:

------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
                                                 -+ #define	NET_PROTO_EAPOL		/* EAP over LAN protocol */
                                                    #define NET_PROTO_LLDP	    /* Link Layer Discovery protocol */
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
                                                 -+  * 802.1x EAP authentication methods
                                                     *
                                                     */
                                                    #define EAP_METHOD_MD5		/* MD5-Challenge port authentication */
                                                    //#define EAP_METHOD_MSCHAPV2	/* MS-CHAPv2 port authentication */

                                                    /*
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
                                                 -+ #define	IMAGE_ZLIB		/* ZLIB image support */
                                                    #define	IMAGE_GZIP		/* GZIP image support */
                                                    //#define	IMAGE_UCODE		/* Microcode update image support */
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
                                                 -+ #define FORM_CMD		/* Form commands */
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
                                                 -+ //#define IMAGE_CRYPT_CMD	/* Image encryption management commands */
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
#define PARAM_CMD		/* Form parameter commands */ <> #define PARAM_CMD		/* Request parameter commands */
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
                                                 -+ #define IMAGE_MEM_CMD		/* Read memory command */
                                                    #define IMAGE_ARCHIVE_CMD	/* Archive image management commands */
                                                    #define SHIM_CMD		/* EFI shim command (or dummy command) */
                                                    #define USB_CMD		/* USB commands */
                                                    //#define FDT_CMD		/* Flattened Device Tree commands */

                                                    /*
                                                     * Certificate sources
                                                     *
                                                     */
                                                    //#undef CERTS_EFI		/* EFI certificate sources */
------------------------------------------------------------------------------------------------------------------------

It’s weird, git shows the last update for general.h on May 11, but I installed FOG on June 10 and that’s when I did git clone and back then the files were different than they are now. These are the files I got on June 10:

console.h: https://pastebin.com/wmy449T0
general.h: https://pastebin.com/GUyqbY3F
settings.h: https://pastebin.com/Ta738Fmn

Fog iPXE Menu no input

158

12.1k

17.3k

155.4k