Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system

[Sebastian Roth]

@mjaskowski And when the kernel build fine you just need to copy it over to your FOG server in /var/www/html/fog/service/ipxe/. And I’d still name it bzImage-earlyprintk so you don’t overwrite the original kernel binary.

[mjaskowski]

the last build failed with MK_FW firmware/bnx2x/bnx2x-e1h-7.13.1.0.fw.gen.S
make[1]: *** No rule to make target linux-firmware/bnx2x/bnx2x-e1h-7.13.1.0.fw', needed by firmware/bnx2x/bnx2x-e1h-7.13.1.0.fw.gen.o'. Stop.
make: *** [firmware] Error 2
Since I’m starting it over I’ll edit those files prebuild but what about this error, how do I prevent a repeat?

[george1421]

@mjaskowski said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

make[1]: *** No rule to make target linux-firmware/bnx2x/bnx2x-e1h-7.13.1.0.fw’, needed by firmware/bnx2x/bnx2x-e1h-7.13.1.0.fw.gen.o’. Stop.

I’ve done my share of linux kernel building (watch that statement come back and bit me in the backside). What this is saying is that you don’'t have the linux kernel firmware downloaded and saved into the root of the directory where the .config file is.

git this: https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git

[Sebastian Roth]

@mjaskowski That’s kind of strange as the firmware should be downloaded by the build.sh script for you - see here. Please run ls -al fos/kernelsourcex64/linux-firmware and post output here. If you get no such file or directory run as suggested by George:

cd fos/kernelsourcex64/
git clone git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git

[mjaskowski]

running ls -al fos/kernelsourcex64/linux-firmware resulted in ls: cannot access fos/kernelsourcex64/linux-firmware: No such file or directory there is however a fos/kernelsourcex64/firmware directory

[Sebastian Roth]

@mjaskowski said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

there is however a fos/kernelsourcex64/firmware directory

This is not the same as fos/kernelsourcex64/linux-firmware. You need both!

[Sebastian Roth]

@mjaskowski Did you get to compile the kernel yet?

[mjaskowski]

I got pulled off this project and just back to it now. So I just updated to Kernel to bzImage Version: 4.19.48 and when I pxe boot to it I only get as far as rEFInd - Booting OS in a blue bar at the top of the screen and below prints:

Starting bootmgfw.efi
Using load options ’ ’

and that’s all folks. What next?

[Sebastian Roth]

@mjaskowski Great to see you back. When you see rEFInd that means your client chainloaded to boot from disk. Make sure it is registered and a task is scheduled for it.

[mjaskowski]

I’m looking at https://wiki.fogproject.org/wiki/index.php/Building_a_Custom_Kernel and it’s not clear to me where I’m to perform step 1:
Build Process

• Install [WHERE?] the required packages to build the kernel with:
  RHEL/Fedora/CentOS/.rpm other: (need confirmation that this is all the packages required?!?, may also need qt-devel and a few others)
  yum groupinstall “Development Tools”
• Ubuntu/Debian/.deb other:
  sudo apt-get install qt3-dev-tools libqt3-mt-dev

[Sebastian Roth]

@mjaskowski From the things you wrote weeks ago I though you where way beyond that point already. The development tools are needed to compile the kernel from source code. You install those on a machine you want to use for playing with the kernel build. You can use your FOG server for this but any other Linux machine will do just as well. Doesn’t really matter.

[mjaskowski]

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Great to see you back. When you see rEFInd that means your client chainloaded to boot from disk. Make sure it is registered and a task is scheduled for it.

That’s the thing, I never get to the FOG menu options including the Quick Registration and all. I’ve previously tried to manually register and perform a task to no avail but I’ll try that again.

[mjaskowski]

@mjaskowski said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Great to see you back. When you see rEFInd that means your client chainloaded to boot from disk. Make sure it is registered and a task is scheduled for it.

That’s the thing, I never get to the FOG menu options including the Quick Registration and all. I’ve previously tried to manually register and perform a task to no avail but I’ll try that again.

So I get the following:
Succeed to download NBP file.
iPXE initialising devices…ok

iPXE 1.0.0+ (9907f) – Open Source Network Boot Firmware – http://ipxe.org
Features: DNS FTP HTTP HTTPS iSCSI NFS TFTP SRP VLAN AoE EFI Menu
Configuring (net0 3c:18:a0:11:dc:62)… ok
Received DHCP answer on interface net0
tftp://xx.xx.x.xx/default.ipxe… ok
http://xx.xx.x.xx/fog/service/ipxe/boot.php… ok
bzImage-earlyprintk… ok
init.xz… ok
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output
Text output

to the end of the visible screen.

[Sebastian Roth]

@mjaskowski Ok, now we are on the right track!! Well done. Can you post the modified part of your eboot.c file here so we know where exactly you added the “Text output” prints.

[mjaskowski]

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Ok, now we are on the right track!! Well done. Can you post the modified part of your eboot.c file here so we know where exactly you added the “Text output” prints.

        struct desc_ptr *gdt = NULL;
        efi_printk(sys_table, "Text output\n");
        efi_loaded_image_t *image;
        efi_printk(sys_table, "Text output\n");
        struct setup_header *hdr = &boot_params->hdr;
        efi_printk(sys_table, "Text output\n");
        efi_status_t status;
        efi_printk(sys_table, "Text output\n");
        struct desc_struct *desc;
        efi_printk(sys_table, "Text output\n");
        void *handle;
        efi_system_table_t *_table;
        unsigned long cmdline_paddr;
 efi_printk(sys_table, "Text output\n");

        efi_early = c;

        _table = (efi_system_table_t *)(unsigned long)efi_early->table;
        handle = (void *)(unsigned long)efi_early->image_handle;
        efi_printk(sys_table, "Text output\n");

        sys_table = _table;

        /* Check if we were booted by the EFI firmware */
        if (sys_table->hdr.signature != EFI_SYSTEM_TABLE_SIGNATURE)
                goto fail;
        efi_printk(sys_table, "Text output\n");

        if (efi_is_64bit())
                setup_boot_services64(efi_early);
        else
                setup_boot_services32(efi_early);
        efi_printk(sys_table, "Text output\n");

        /*
         * make_boot_params() may have been called before efi_main(), in which
         * case this is the second time we parse the cmdline. This is ok,
         * parsing the cmdline multiple times does not have side-effects.
         */
        cmdline_paddr = ((u64)hdr->cmd_line_ptr |
                         ((u64)boot_params->ext_cmd_line_ptr << 32));
        efi_parse_options((char *)cmdline_paddr);
        efi_printk(sys_table, "Text output\n");

        /*
         * If the boot loader gave us a value for secure_boot then we use that,
         * otherwise we ask the BIOS.
         */
        if (boot_params->secure_boot == efi_secureboot_mode_unset)
                boot_params->secure_boot = efi_get_secureboot(sys_table);
        efi_printk(sys_table, "Text output\n");

        /* Ask the firmware to clear memory on unclean shutdown */
        efi_enable_reset_attack_mitigation(sys_table);
        efi_retrieve_tpm2_eventlog(sys_table);
        efi_printk(sys_table, "Text output\n");

        setup_graphics(boot_params);
        efi_printk(sys_table, "Text output\n");

        setup_efi_pci(boot_params);
        efi_printk(sys_table, "Text output\n");

        setup_quirks(boot_params);
        efi_printk(sys_table, "Text output\n");
 
       efi_printk(sys_table, "Text output\n");
                if (status != EFI_SUCCESS) {
                        efi_printk(sys_table, "efi_relocate_kernel() failed!\n");
                        goto fail;
                }
        efi_printk(sys_table, "Text output\n");

[Sebastian Roth]

@mjaskowski Hmmm, that’s interesting. In the output you posted I counted “Text output” 17 times. Now when I look at the code you posted it also has 17 lines of “Text output”. What that means is that it gets as far as the last “Text output”. That’s really good news because on most systems we debugged this we only got as far as setup_efi_pci…

Ok, please add another efi_printk after the exit_boot call and check - see code here: https://elixir.bootlin.com/linux/v4.19.48/source/arch/x86/boot/compressed/eboot.c#L841

Best if you make it a different text, e.g. “After exit_boot” so we don’t have to count the outputs.

[mjaskowski]

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Hmmm, that’s interesting. In the output you posted I counted “Text output” 17 times. Now when I look at the code you posted it also has 17 lines of “Text output”. What that means is that it gets as far as the last “Text output”. That’s really good news because on most systems we debugged this we only got as far as setup_efi_pci…

Ok, please add another efi_printk after the exit_boot call and check - see code here: https://elixir.bootlin.com/linux/v4.19.48/source/arch/x86/boot/compressed/eboot.c#L841

Best if you make it a different text, e.g. “After exit_boot” so we don’t have to count the outputs.

So not even the ‘Exit boot 1’ prints

    efi_printk(sys_table, "Text output 17\n");

            hdr->pref_address = hdr->code32_start;
            hdr->code32_start = bzimage_addr;
    }

    status = exit_boot(boot_params, handle);
    if (status != EFI_SUCCESS) {
            efi_printk(sys_table, "exit_boot() failed!\n");
            goto fail;
    }
    efi_printk(sys_table, "Exit boot 1\n");

    memset((char *)gdt->address, 0x0, gdt->size);
    desc = (struct desc_struct *)gdt->address;

    /* The first GDT is a dummy. */
    desc++;

    if (IS_ENABLED(CONFIG_X86_64)) {
            /* __KERNEL32_CS */
            desc->limit0    = 0xffff;
            desc->base0     = 0x0000;
            desc->base1     = 0x0000;
            desc->type      = SEG_TYPE_CODE | SEG_TYPE_EXEC_READ;
            desc->s         = DESC_TYPE_CODE_DATA;
            desc->dpl       = 0;
            desc->p         = 1;
            desc->limit1    = 0xf;
            desc->avl       = 0;
            desc->l         = 0;
            desc->d         = SEG_OP_SIZE_32BIT;
            desc->g         = SEG_GRANULARITY_4KB;
            desc->base2     = 0x00;

            desc++;
    } else {
            /* Second entry is unused on 32-bit */
            desc++;
    }
    efi_printk(sys_table, "Exit boot 2\n");

[Sebastian Roth]

@mjaskowski Ok, seems like it actually hangs on the exit_boot call. Doing a bit of research on this I found that we have another Lenovo device that does the same thing: https://forums.fogproject.org/topic/11060/debugging-lenovo-ideapad-miix-310

Doesn’t help much but let’s keep that in mind.

Ok, now we need to step into the exit_boot function, starting in line 704. Add those print statements here as well to see which call is the last.

[mjaskowski]

@Sebastian-Roth said in Lenovo 300e ipxe boots ok but ipxe.efi can't resolve the system:

@mjaskowski Ok, seems like it actually hangs on the exit_boot call. Doing a bit of research on this I found that we have another Lenovo device that does the same thing: https://forums.fogproject.org/topic/11060/debugging-lenovo-ideapad-miix-310

Doesn’t help much but let’s keep that in mind.

Ok, now we need to step into the exit_boot function, starting in line 704. Add those print statements here as well to see which call is the last.

The last print I get is Exit boot 3

    status = allocate_e820(boot_params, &e820ext, &e820ext_size);
    if (status != EFI_SUCCESS)
            return status;
            efi_printk(sys_table, "Exit boot 3\n");

    /* Might as well exit boot services now */
    status = efi_exit_boot_services(sys_table, handle, &map, &priv,
                                    exit_boot_func);
            efi_printk(sys_table, "Exit boot a\n");
    if (status != EFI_SUCCESS)
            return status;
            efi_printk(sys_table, "Exit boot 4\n");

[Sebastian Roth]

@mjaskowski Ok, I think we are down to the bones of this. Now we have to take one step back and also enable iPXE debugging to see what happens when the kernel quickly returns control to the UEFI firmware before it takes over for the rest of the boot process.

Please leave all the kernel source code as you have it right now as we might need to get back to that to implement a fix at some point.

Take a look at what is needed to compile iPXE here: https://wiki.fogproject.org/wiki/index.php?title=IPXE#Compile

Beside the prerequisites you only need the following commands:

git clone git://git.ipxe.org/ipxe.git ipxe-efi
cd ipxe-efi/src/config
rm console.h general.h settings.h
wget -O console.h "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/config/console.h"
wget -O general.h "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/config/general.h"
wget -O settings.h "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/config/settings.h"
cd ..
wget -O ipxescript "https://github.com/FOGProject/fogproject/raw/master/src/ipxe/src-efi/ipxescript"
make bin-x86_64-efi/ipxe.efi EMBED=ipxescript DEBUG=efi_wrap

Then install the new iPXE binary:

sudo -i
mv /tftpboot/ipxe.efi /tftpboot/ipxe.efi.orig
cp path/to/ipxe-efi/src/bin-x86_64-efi/ipxe.efi /tftpboot/ipxe.efi

Boot the machine and you should get some very funky colored output on screen (for example see here). Please take pictures of the output you get on the screen and post here. We need all the details you see!