Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system

mjaskowski

We just got 28 new Lenovo 300e WinBooks for our Math tutors that we’d like to clone with FOG with the following specs:

BIOS: 6HCN23WW
EC: 6HEC23WW
MTM: 81FY000BUS
CPU: Intel Celeron CPU N3450 @ 1.10GHz
Sys Memory: 4096 MB
eMMC Card: SanDisk iNAND 128GB
OS: Win10 Pro Education
Secure Boot Status: Disabled

No built-in NIC so I’m using a lenovo ThinkPad USB 3.0 Ethernet Adapter Model: PN:RTL8153

We can reach the Fog server registration page and start a Quick Registration but it never completes and neither does the Client System Information (Compatibility), the screen goes black and we get nothing after that. It will just sit that way for hours until I shut it down. Then I tried a manual registration which does get recognized and an image capture appears to start but also stalls and never proceeds.

I’ve even tried Acronis and Clonezilla already which load and then fail to find a hard drive. These units have 128gb eMMCs as primary drives.

Please advise. Thanks.

Sebastian Roth

@tunatoksoz Hmmmm, thanks for joining the discussion here! Seems like you are making progress on this more than we have.

May I ask you if this is on Lenovo 300e as well? As you are saying yours has a build in NIC plus one on PCI I kind of doubt you have the same machine. Can you be more specific on what make and model you see the issue on?

tunatoksoz

@Sebastian-Roth Hi Sebastian, I don’t use Fog project, but i use IPXE, and encountered this very issue myself. This is on a machine with elitegroup bios, and it has 2 nics.

I noticed couple things:

If i open BIOS setup screen, change nothing, but “save changes and exit” i do not see this issue.
If i open BIOS setup screen, change nothing, but “save changes and reset”, i get exit_boot failed error.
If i remove the nic that’s on pci, so i only am left with the one on motherboard, I do not see this issue.

Something with the memory map & nic card & kernel is a problem.

I am using similar script to this:
https://ipxe.org/appnote/debian_live

on debian buster.

Just in case this extra info helps.

Sebastian Roth

@mjaskowski I just remembered this and wanted to hear if you had any reply from Lenovo on this? No answer from iPXE developers mailing list unfortunately.

Have we talked about trying to boot this device using the USB boot method? Maybe it’s a good workaround for you to get around this. Take a look at this forum topic and let us know if you need help with it.

Sebastian Roth

@mjaskowski Still not much of an answer. I posted to the iPXE mailing list now too. Should be soon in the archive: http://lists.ipxe.org/pipermail/ipxe-devel/2019-November/thread.html

Did you get any response from Lenovo yet?

Sebastian Roth

@mjaskowski Are there any firmware updates available for those devices?

EDIT: Just posted in the iPXE forum: http://forum.ipxe.org/showthread.php?tid=20820

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

I am not sure if this is an issue that can be solved in iPXE but I will talk to the developers to see if they can help.

Argh! Thanks for all your help on this. I’ll see about getting Lenovo developers to cooperate or return their products.

Sebastian Roth

@mjaskowski Reading about “ExitBootServices” and “INVALID_PARAMETER” I found a few resources saying something about memory map being wrong:

https://forum.osdev.org/viewtopic.php?f=1&t=32519
http://mail-index.netbsd.org/port-amd64/2019/09/06/msg003003.html
https://lists.freebsd.org/pipermail/freebsd-hackers/2015-January/046956.html
https://uefi.org/sites/default/files/resources/ACPI_6_3_May16.pdf (p. 117,

Either it’s people developing their own operating system code and running into issues with their coding or it’s people running known operating systems that fail on particular hardware (probably because of UEFI firmware problems).

I am not sure if this is an issue that can be solved in iPXE but I will talk to the developers to see if they can help.

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

Boot the machine and you should get some very funky colored output on screen (for example see here). Please take pictures of the output you get on the screen and post here. We need all the details you see!

IMG_2019.3.jpg
IMG_2022.3.jpg
IMG_2023.3.jpg

Sebastian Roth

@mjaskowski Try this: git clone https://git.ipxe.org/ipxe.git ipxe-bios

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Ok, I think we are down to the bones of this. Now we have to take one step back and also enable iPXE debugging to see what happens when the kernel quickly returns control to the UEFI firmware before it takes over for the rest of the boot process.

Please leave all the kernel source code as you have it right now as we might need to get back to that to implement a fix at some point.

Take a look at what is needed to compile iPXE here: https://wiki.fogproject.org/wiki/index.php?title=IPXE#Compile

Beside the prerequisites…

In doing the prerequisites I have the following roadblock:

[root@MCS-FOG ipxe]# git clone git://git.ipxe.org/ipxe.git ipxe-bios
Cloning into ‘ipxe-bios’…
fatal: unable to connect to git.ipxe.org:
git.ipxe.org[0: 212.13.204.60]: errno=Connection timed out
git.ipxe.org[1: 2001:ba8:0:1d4::6950:5845]: errno=Network is unreachable

Sebastian Roth

@mjaskowski Ok, I think we are down to the bones of this. Now we have to take one step back and also enable iPXE debugging to see what happens when the kernel quickly returns control to the UEFI firmware before it takes over for the rest of the boot process.

Please leave all the kernel source code as you have it right now as we might need to get back to that to implement a fix at some point.

Take a look at what is needed to compile iPXE here: https://wiki.fogproject.org/wiki/index.php?title=IPXE#Compile

Beside the prerequisites you only need the following commands:

git clone git://git.ipxe.org/ipxe.git ipxe-efi
cd ipxe-efi/src/config
rm console.h general.h settings.h
wget -O console.h "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/config/console.h"
wget -O general.h "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/config/general.h"
wget -O settings.h "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/config/settings.h"
cd ..
wget -O ipxescript "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/ipxescript"
make bin-x86_64-efi/ipxe.efi EMBED=ipxescript DEBUG=efi_wrap

Then install the new iPXE binary:

sudo -i
mv /tftpboot/ipxe.efi /tftpboot/ipxe.efi.orig
cp path/to/ipxe-efi/src/bin-x86_64-efi/ipxe.efi /tftpboot/ipxe.efi

Boot the machine and you should get some very funky colored output on screen (for example see here). Please take pictures of the output you get on the screen and post here. We need all the details you see!

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Ok, seems like it actually hangs on the exit_boot call. Doing a bit of research on this I found that we have another Lenovo device that does the same thing: https://forums.fogproject.org/topic/11060/debugging-lenovo-ideapad-miix-310

Doesn’t help much but let’s keep that in mind.

Ok, now we need to step into the exit_boot function, starting in line 704. Add those print statements here as well to see which call is the last.

The last print I get is Exit boot 3

    status = allocate_e820(boot_params, &e820ext, &e820ext_size);
    if (status != EFI_SUCCESS)
            return status;
            efi_printk(sys_table, "Exit boot 3\n");

    /* Might as well exit boot services now */
    status = efi_exit_boot_services(sys_table, handle, &map, &priv,
                                    exit_boot_func);
            efi_printk(sys_table, "Exit boot a\n");
    if (status != EFI_SUCCESS)
            return status;
            efi_printk(sys_table, "Exit boot 4\n");

Sebastian Roth

@mjaskowski Ok, seems like it actually hangs on the exit_boot call. Doing a bit of research on this I found that we have another Lenovo device that does the same thing: https://forums.fogproject.org/topic/11060/debugging-lenovo-ideapad-miix-310

Doesn’t help much but let’s keep that in mind.

Ok, now we need to step into the exit_boot function, starting in line 704. Add those print statements here as well to see which call is the last.

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Hmmm, that’s interesting. In the output you posted I counted “Text output” 17 times. Now when I look at the code you posted it also has 17 lines of “Text output”. What that means is that it gets as far as the last “Text output”. That’s really good news because on most systems we debugged this we only got as far as setup_efi_pci…

Ok, please add another efi_printk after the exit_boot call and check - see code here: https://elixir.bootlin.com/linux/v4.19.48/source/arch/x86/boot/compressed/eboot.c#L841

Best if you make it a different text, e.g. “After exit_boot” so we don’t have to count the outputs.

So not even the ‘Exit boot 1’ prints

    efi_printk(sys_table, "Text output 17\n");

            hdr->pref_address = hdr->code32_start;
            hdr->code32_start = bzimage_addr;
    }

    status = exit_boot(boot_params, handle);
    if (status != EFI_SUCCESS) {
            efi_printk(sys_table, "exit_boot() failed!\n");
            goto fail;
    }
    efi_printk(sys_table, "Exit boot 1\n");

    memset((char *)gdt->address, 0x0, gdt->size);
    desc = (struct desc_struct *)gdt->address;

    /* The first GDT is a dummy. */
    desc++;

    if (IS_ENABLED(CONFIG_X86_64)) {
            /* __KERNEL32_CS */
            desc->limit0    = 0xffff;
            desc->base0     = 0x0000;
            desc->base1     = 0x0000;
            desc->type      = SEG_TYPE_CODE | SEG_TYPE_EXEC_READ;
            desc->s         = DESC_TYPE_CODE_DATA;
            desc->dpl       = 0;
            desc->p         = 1;
            desc->limit1    = 0xf;
            desc->avl       = 0;
            desc->l         = 0;
            desc->d         = SEG_OP_SIZE_32BIT;
            desc->g         = SEG_GRANULARITY_4KB;
            desc->base2     = 0x00;

            desc++;
    } else {
            /* Second entry is unused on 32-bit */
            desc++;
    }
    efi_printk(sys_table, "Exit boot 2\n");

Sebastian Roth

@mjaskowski Hmmm, that’s interesting. In the output you posted I counted “Text output” 17 times. Now when I look at the code you posted it also has 17 lines of “Text output”. What that means is that it gets as far as the last “Text output”. That’s really good news because on most systems we debugged this we only got as far as setup_efi_pci…

Ok, please add another efi_printk after the exit_boot call and check - see code here: https://elixir.bootlin.com/linux/v4.19.48/source/arch/x86/boot/compressed/eboot.c#L841

Best if you make it a different text, e.g. “After exit_boot” so we don’t have to count the outputs.

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Ok, now we are on the right track!! Well done. Can you post the modified part of your eboot.c file here so we know where exactly you added the “Text output” prints.

        struct desc_ptr *gdt = NULL;
        efi_printk(sys_table, "Text output\n");
        efi_loaded_image_t *image;
        efi_printk(sys_table, "Text output\n");
        struct setup_header *hdr = &boot_params->hdr;
        efi_printk(sys_table, "Text output\n");
        efi_status_t status;
        efi_printk(sys_table, "Text output\n");
        struct desc_struct *desc;
        efi_printk(sys_table, "Text output\n");
        void *handle;
        efi_system_table_t *_table;
        unsigned long cmdline_paddr;
 efi_printk(sys_table, "Text output\n");

        efi_early = c;

        _table = (efi_system_table_t *)(unsigned long)efi_early->table;
        handle = (void *)(unsigned long)efi_early->image_handle;
        efi_printk(sys_table, "Text output\n");

        sys_table = _table;

        /* Check if we were booted by the EFI firmware */
        if (sys_table->hdr.signature != EFI_SYSTEM_TABLE_SIGNATURE)
                goto fail;
        efi_printk(sys_table, "Text output\n");

        if (efi_is_64bit())
                setup_boot_services64(efi_early);
        else
                setup_boot_services32(efi_early);
        efi_printk(sys_table, "Text output\n");

        /*
         * make_boot_params() may have been called before efi_main(), in which
         * case this is the second time we parse the cmdline. This is ok,
         * parsing the cmdline multiple times does not have side-effects.
         */
        cmdline_paddr = ((u64)hdr->cmd_line_ptr |
                         ((u64)boot_params->ext_cmd_line_ptr << 32));
        efi_parse_options((char *)cmdline_paddr);
        efi_printk(sys_table, "Text output\n");

        /*
         * If the boot loader gave us a value for secure_boot then we use that,
         * otherwise we ask the BIOS.
         */
        if (boot_params->secure_boot == efi_secureboot_mode_unset)
                boot_params->secure_boot = efi_get_secureboot(sys_table);
        efi_printk(sys_table, "Text output\n");

        /* Ask the firmware to clear memory on unclean shutdown */
        efi_enable_reset_attack_mitigation(sys_table);
        efi_retrieve_tpm2_eventlog(sys_table);
        efi_printk(sys_table, "Text output\n");

        setup_graphics(boot_params);
        efi_printk(sys_table, "Text output\n");

        setup_efi_pci(boot_params);
        efi_printk(sys_table, "Text output\n");

        setup_quirks(boot_params);
        efi_printk(sys_table, "Text output\n");
 
       efi_printk(sys_table, "Text output\n");
                if (status != EFI_SUCCESS) {
                        efi_printk(sys_table, "efi_relocate_kernel() failed!\n");
                        goto fail;
                }
        efi_printk(sys_table, "Text output\n");

Sebastian Roth

@mjaskowski Ok, now we are on the right track!! Well done. Can you post the modified part of your eboot.c file here so we know where exactly you added the “Text output” prints.

mjaskowski

@mjaskowski said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Great to see you back. When you see rEFInd that means your client chainloaded to boot from disk. Make sure it is registered and a task is scheduled for it.

That’s the thing, I never get to the FOG menu options including the Quick Registration and all. I’ve previously tried to manually register and perform a task to no avail but I’ll try that again.

So I get the following:
Succeed to download NBP file.
iPXE initialising devices…ok

iPXE 1.0.0+ (9907f) – Open Source Network Boot Firmware – http://ipxe.org
Features: DNS FTP HTTP HTTPS iSCSI NFS TFTP SRP VLAN AoE EFI Menu
Configuring (net0 3c:18:a0:11:dc:62)… ok
Received DHCP answer on interface net0
tftp://xx.xx.x.xx/default.ipxe… ok
http://xx.xx.x.xx/fog/service/ipxe/boot.php… ok
bzImage-earlyprintk… ok
init.xz… ok
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output

to the end of the visible screen.

mjaskowski

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Great to see you back. When you see rEFInd that means your client chainloaded to boot from disk. Make sure it is registered and a task is scheduled for it.

That’s the thing, I never get to the FOG menu options including the Quick Registration and all. I’ve previously tried to manually register and perform a task to no avail but I’ll try that again.

Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system

130

10.6k

16.5k

151.1k