Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system

mjaskowski

I’m looking at https://wiki.fogproject.org/wiki/index.php/Building_a_Custom_Kernel and it’s not clear to me where I’m to perform step 1:
Build Process

Install [WHERE?] the required packages to build the kernel with:
RHEL/Fedora/CentOS/.rpm other: (need confirmation that this is all the packages required?!?, may also need qt-devel and a few others)
yum groupinstall “Development Tools”
Ubuntu/Debian/.deb other:
sudo apt-get install qt3-dev-tools libqt3-mt-dev

Sebastian Roth

@mjaskowski From the things you wrote weeks ago I though you where way beyond that point already. The development tools are needed to compile the kernel from source code. You install those on a machine you want to use for playing with the kernel build. You can use your FOG server for this but any other Linux machine will do just as well. Doesn’t really matter.

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Great to see you back. When you see rEFInd that means your client chainloaded to boot from disk. Make sure it is registered and a task is scheduled for it.

That’s the thing, I never get to the FOG menu options including the Quick Registration and all. I’ve previously tried to manually register and perform a task to no avail but I’ll try that again.

mjaskowski

@mjaskowski said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Great to see you back. When you see rEFInd that means your client chainloaded to boot from disk. Make sure it is registered and a task is scheduled for it.

That’s the thing, I never get to the FOG menu options including the Quick Registration and all. I’ve previously tried to manually register and perform a task to no avail but I’ll try that again.

So I get the following:
Succeed to download NBP file.
iPXE initialising devices…ok

iPXE 1.0.0+ (9907f) – Open Source Network Boot Firmware – http://ipxe.org
Features: DNS FTP HTTP HTTPS iSCSI NFS TFTP SRP VLAN AoE EFI Menu
Configuring (net0 3c:18:a0:11:dc:62)… ok
Received DHCP answer on interface net0
tftp://xx.xx.x.xx/default.ipxe… ok
http://xx.xx.x.xx/fog/service/ipxe/boot.php… ok
bzImage-earlyprintk… ok
init.xz… ok
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output

to the end of the visible screen.

Sebastian Roth

@mjaskowski Ok, now we are on the right track!! Well done. Can you post the modified part of your eboot.c file here so we know where exactly you added the “Text output” prints.

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Ok, now we are on the right track!! Well done. Can you post the modified part of your eboot.c file here so we know where exactly you added the “Text output” prints.

        struct desc_ptr *gdt = NULL;
        efi_printk(sys_table, "Text output\n");
        efi_loaded_image_t *image;
        efi_printk(sys_table, "Text output\n");
        struct setup_header *hdr = &boot_params->hdr;
        efi_printk(sys_table, "Text output\n");
        efi_status_t status;
        efi_printk(sys_table, "Text output\n");
        struct desc_struct *desc;
        efi_printk(sys_table, "Text output\n");
        void *handle;
        efi_system_table_t *_table;
        unsigned long cmdline_paddr;
 efi_printk(sys_table, "Text output\n");

        efi_early = c;

        _table = (efi_system_table_t *)(unsigned long)efi_early->table;
        handle = (void *)(unsigned long)efi_early->image_handle;
        efi_printk(sys_table, "Text output\n");

        sys_table = _table;

        /* Check if we were booted by the EFI firmware */
        if (sys_table->hdr.signature != EFI_SYSTEM_TABLE_SIGNATURE)
                goto fail;
        efi_printk(sys_table, "Text output\n");

        if (efi_is_64bit())
                setup_boot_services64(efi_early);
        else
                setup_boot_services32(efi_early);
        efi_printk(sys_table, "Text output\n");

        /*
         * make_boot_params() may have been called before efi_main(), in which
         * case this is the second time we parse the cmdline. This is ok,
         * parsing the cmdline multiple times does not have side-effects.
         */
        cmdline_paddr = ((u64)hdr->cmd_line_ptr |
                         ((u64)boot_params->ext_cmd_line_ptr << 32));
        efi_parse_options((char *)cmdline_paddr);
        efi_printk(sys_table, "Text output\n");

        /*
         * If the boot loader gave us a value for secure_boot then we use that,
         * otherwise we ask the BIOS.
         */
        if (boot_params->secure_boot == efi_secureboot_mode_unset)
                boot_params->secure_boot = efi_get_secureboot(sys_table);
        efi_printk(sys_table, "Text output\n");

        /* Ask the firmware to clear memory on unclean shutdown */
        efi_enable_reset_attack_mitigation(sys_table);
        efi_retrieve_tpm2_eventlog(sys_table);
        efi_printk(sys_table, "Text output\n");

        setup_graphics(boot_params);
        efi_printk(sys_table, "Text output\n");

        setup_efi_pci(boot_params);
        efi_printk(sys_table, "Text output\n");

        setup_quirks(boot_params);
        efi_printk(sys_table, "Text output\n");
 
       efi_printk(sys_table, "Text output\n");
                if (status != EFI_SUCCESS) {
                        efi_printk(sys_table, "efi_relocate_kernel() failed!\n");
                        goto fail;
                }
        efi_printk(sys_table, "Text output\n");

Sebastian Roth

@mjaskowski Hmmm, that’s interesting. In the output you posted I counted “Text output” 17 times. Now when I look at the code you posted it also has 17 lines of “Text output”. What that means is that it gets as far as the last “Text output”. That’s really good news because on most systems we debugged this we only got as far as setup_efi_pci…

Ok, please add another efi_printk after the exit_boot call and check - see code here: https://elixir.bootlin.com/linux/v4.19.48/source/arch/x86/boot/compressed/eboot.c#L841

Best if you make it a different text, e.g. “After exit_boot” so we don’t have to count the outputs.

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Hmmm, that’s interesting. In the output you posted I counted “Text output” 17 times. Now when I look at the code you posted it also has 17 lines of “Text output”. What that means is that it gets as far as the last “Text output”. That’s really good news because on most systems we debugged this we only got as far as setup_efi_pci…

Ok, please add another efi_printk after the exit_boot call and check - see code here: https://elixir.bootlin.com/linux/v4.19.48/source/arch/x86/boot/compressed/eboot.c#L841

Best if you make it a different text, e.g. “After exit_boot” so we don’t have to count the outputs.

So not even the ‘Exit boot 1’ prints

    efi_printk(sys_table, "Text output 17\n");

            hdr->pref_address = hdr->code32_start;
            hdr->code32_start = bzimage_addr;
    }

    status = exit_boot(boot_params, handle);
    if (status != EFI_SUCCESS) {
            efi_printk(sys_table, "exit_boot() failed!\n");
            goto fail;
    }
    efi_printk(sys_table, "Exit boot 1\n");

    memset((char *)gdt->address, 0x0, gdt->size);
    desc = (struct desc_struct *)gdt->address;

    /* The first GDT is a dummy. */
    desc++;

    if (IS_ENABLED(CONFIG_X86_64)) {
            /* __KERNEL32_CS */
            desc->limit0    = 0xffff;
            desc->base0     = 0x0000;
            desc->base1     = 0x0000;
            desc->type      = SEG_TYPE_CODE | SEG_TYPE_EXEC_READ;
            desc->s         = DESC_TYPE_CODE_DATA;
            desc->dpl       = 0;
            desc->p         = 1;
            desc->limit1    = 0xf;
            desc->avl       = 0;
            desc->l         = 0;
            desc->d         = SEG_OP_SIZE_32BIT;
            desc->g         = SEG_GRANULARITY_4KB;
            desc->base2     = 0x00;

            desc++;
    } else {
            /* Second entry is unused on 32-bit */
            desc++;
    }
    efi_printk(sys_table, "Exit boot 2\n");

Sebastian Roth

@mjaskowski Ok, seems like it actually hangs on the exit_boot call. Doing a bit of research on this I found that we have another Lenovo device that does the same thing: https://forums.fogproject.org/topic/11060/debugging-lenovo-ideapad-miix-310

Doesn’t help much but let’s keep that in mind.

Ok, now we need to step into the exit_boot function, starting in line 704. Add those print statements here as well to see which call is the last.

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Ok, seems like it actually hangs on the exit_boot call. Doing a bit of research on this I found that we have another Lenovo device that does the same thing: https://forums.fogproject.org/topic/11060/debugging-lenovo-ideapad-miix-310

Doesn’t help much but let’s keep that in mind.

Ok, now we need to step into the exit_boot function, starting in line 704. Add those print statements here as well to see which call is the last.

The last print I get is Exit boot 3

    status = allocate_e820(boot_params, &e820ext, &e820ext_size);
    if (status != EFI_SUCCESS)
            return status;
            efi_printk(sys_table, "Exit boot 3\n");

    /* Might as well exit boot services now */
    status = efi_exit_boot_services(sys_table, handle, &map, &priv,
                                    exit_boot_func);
            efi_printk(sys_table, "Exit boot a\n");
    if (status != EFI_SUCCESS)
            return status;
            efi_printk(sys_table, "Exit boot 4\n");

Sebastian Roth

@mjaskowski Ok, I think we are down to the bones of this. Now we have to take one step back and also enable iPXE debugging to see what happens when the kernel quickly returns control to the UEFI firmware before it takes over for the rest of the boot process.

Please leave all the kernel source code as you have it right now as we might need to get back to that to implement a fix at some point.

Take a look at what is needed to compile iPXE here: https://wiki.fogproject.org/wiki/index.php?title=IPXE#Compile

Beside the prerequisites you only need the following commands:

git clone git://git.ipxe.org/ipxe.git ipxe-efi
cd ipxe-efi/src/config
rm console.h general.h settings.h
wget -O console.h "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/config/console.h"
wget -O general.h "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/config/general.h"
wget -O settings.h "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/config/settings.h"
cd ..
wget -O ipxescript "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/ipxescript"
make bin-x86_64-efi/ipxe.efi EMBED=ipxescript DEBUG=efi_wrap

Then install the new iPXE binary:

sudo -i
mv /tftpboot/ipxe.efi /tftpboot/ipxe.efi.orig
cp path/to/ipxe-efi/src/bin-x86_64-efi/ipxe.efi /tftpboot/ipxe.efi

Boot the machine and you should get some very funky colored output on screen (for example see here). Please take pictures of the output you get on the screen and post here. We need all the details you see!

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Ok, I think we are down to the bones of this. Now we have to take one step back and also enable iPXE debugging to see what happens when the kernel quickly returns control to the UEFI firmware before it takes over for the rest of the boot process.

Please leave all the kernel source code as you have it right now as we might need to get back to that to implement a fix at some point.

Take a look at what is needed to compile iPXE here: https://wiki.fogproject.org/wiki/index.php?title=IPXE#Compile

Beside the prerequisites…

In doing the prerequisites I have the following roadblock:

[root@MCS-FOG ipxe]# git clone git://git.ipxe.org/ipxe.git ipxe-bios
Cloning into ‘ipxe-bios’…
fatal: unable to connect to git.ipxe.org:
git.ipxe.org[0: 212.13.204.60]: errno=Connection timed out
git.ipxe.org[1: 2001:ba8:0:1d4::6950:5845]: errno=Network is unreachable

Sebastian Roth

@mjaskowski Try this: git clone https://git.ipxe.org/ipxe.git ipxe-bios

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

Boot the machine and you should get some very funky colored output on screen (for example see here). Please take pictures of the output you get on the screen and post here. We need all the details you see!

IMG_2019.3.jpg
IMG_2022.3.jpg
IMG_2023.3.jpg

Sebastian Roth

@mjaskowski Reading about “ExitBootServices” and “INVALID_PARAMETER” I found a few resources saying something about memory map being wrong:

https://forum.osdev.org/viewtopic.php?f=1&t=32519
http://mail-index.netbsd.org/port-amd64/2019/09/06/msg003003.html
https://lists.freebsd.org/pipermail/freebsd-hackers/2015-January/046956.html
https://uefi.org/sites/default/files/resources/ACPI_6_3_May16.pdf (p. 117,

Either it’s people developing their own operating system code and running into issues with their coding or it’s people running known operating systems that fail on particular hardware (probably because of UEFI firmware problems).

I am not sure if this is an issue that can be solved in iPXE but I will talk to the developers to see if they can help.

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

I am not sure if this is an issue that can be solved in iPXE but I will talk to the developers to see if they can help.

Argh! Thanks for all your help on this. I’ll see about getting Lenovo developers to cooperate or return their products.

Sebastian Roth

@mjaskowski Are there any firmware updates available for those devices?

EDIT: Just posted in the iPXE forum: http://forum.ipxe.org/showthread.php?tid=20820

Sebastian Roth

@mjaskowski Still not much of an answer. I posted to the iPXE mailing list now too. Should be soon in the archive: http://lists.ipxe.org/pipermail/ipxe-devel/2019-November/thread.html

Did you get any response from Lenovo yet?

Sebastian Roth

@mjaskowski I just remembered this and wanted to hear if you had any reply from Lenovo on this? No answer from iPXE developers mailing list unfortunately.

Have we talked about trying to boot this device using the USB boot method? Maybe it’s a good workaround for you to get around this. Take a look at this forum topic and let us know if you need help with it.

tunatoksoz

@Sebastian-Roth Hi Sebastian, I don’t use Fog project, but i use IPXE, and encountered this very issue myself. This is on a machine with elitegroup bios, and it has 2 nics.

I noticed couple things:

If i open BIOS setup screen, change nothing, but “save changes and exit” i do not see this issue.
If i open BIOS setup screen, change nothing, but “save changes and reset”, i get exit_boot failed error.
If i remove the nic that’s on pci, so i only am left with the one on motherboard, I do not see this issue.

Something with the memory map & nic card & kernel is a problem.

I am using similar script to this:
https://ipxe.org/appnote/debian_live

on debian buster.

Just in case this extra info helps.

Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system

200

12.0k

17.3k

155.2k