Tablet PC hangs on bzImage

Sebastian Roth

@Zerpie Good to hear. Now give it a first try by copying it to the right location and configure this new one as “Host Kernel” in one of your hosts’ settings. This way you can simply test with one or a few clients without causing an issue for other clients.

cp build/linux-4.x.y/arch/x86/boot/bzImage /var/www/html/fog/service/ipxe/bzImage_debug

Only clients having set bzImage_debug as “Host Kernel” will receive this kernel image on boot.
If you got this to work you can go back to your kernel source and modify the kernel config (build/linux-4.x.y/.config). Find CONFIG_EARLY_PRINTK and chnage from # CONFIG_EARLY_PRINTK is not set to CONFIG_EARLY_PRINTK=y (make sure you remove the hash tag at the beginning of the line too). Save the file and run:

make oldconfig
make bzImage
cp arch/x86/boot/bzImage /var/www/html/fog/service/ipxe/bzImage_debug

Build won’t take very long this time!

Now where you set the “Host Kernel” for one of the clients you need to add “Host Kernel Arguments” as well: debug earlyprintk=efi loglevel=7

Keeping my fingers crossed that we see some early kernel messages on screen then. If not there is still more we can do.

Zerpie

@Sebastian-Roth Now it just hangs on

BzImage_debug...

Sebastian Roth

@Zerpie Ok, now we get into the code an add our own debug statements. Edit linux-x.y/arch/x86/boot/compressed/eboot.c and jump to where function efi_main is defined, should look like this or pretty similar depending on the kernel version:

/*
 * On success we return a pointer to a boot_params structure, and NULL
 * on failure.
 */
struct boot_params *efi_main(struct efi_config *c,
                             struct boot_params *boot_params)
{
        struct desc_ptr *gdt = NULL;
        efi_loaded_image_t *image;
...
        sys_table = _table;
        efi_printk(sys_table, "Hello World!\n"); 

        /* Check if we were booted by the EFI firmware */
        if (sys_table->hdr.signature != EFI_SYSTEM_TABLE_SIGNATURE)
...

That efi_printk call is not part of the original code. That’s what I ask you to add, just that one line. Then compile once again, copy the new kernel over and try booting that same client where you have set the debug kernel image and parameters in host’s settings.

Zerpie

@Sebastian-Roth It’s still hanging on BzImage_debug… and it will usually get to a percentage and stop. Just now it stopped at

BzImage_debug... 15%

No “Hello World” message.

Sebastian Roth

@Zerpie Ok, well that’s interesting. So it might not even load the full kernel image but hang on the network connection. So let’s tackle this from a different angle. Install package tcpdump and run:

sudo tcpdump -w hang.pcap host x.x.x.x

Put in the client’s IP address instead of x.x.x.x and let the command just sit there. Now boot up the client and wait till it hangs. Wait another 10 seconds and then stop tcpdump (ctrl+c). Upload the generated hang.pcap file (will be around 30 MB or more) and post a download link here in the forums (or send me a PM).

Zerpie

@Sebastian-Roth When I run the command I get the following

# tcpdump -w hang.pcap host 153.86.19.24
tcpdump: NFLOG link-layer type filtering not implemented

Sebastian Roth

@Zerpie Ok, usually tcpdump is able to select the proper network interface just by itself but does not seem to work here - which Linux OS and version do you use?

# tcpdump -i eth0 -w hang.pcap host 153.86.19.24

Put in the interface FOG is using. If you are not sure run grep interface /opt/fog/.fogsettings to find out.

The command should return a message like listening on ..., link-type EN10MB (Ethernet), capture size ... and won’t return to the shell. Leave it like that till you booted the client to hang.

Zerpie

@Sebastian-Roth I’m using CentOS 7. Specifying the network interface worked, though. Here’s a link to the hang.pcap file shared on my Google Drive.

https://drive.google.com/file/d/1SzQGpk4DsDf3yuB2P7AemvxMXzMdcmYf/view?usp=sharing

Sebastian Roth

@Zerpie That’s very strange. In the PCAP I don’t even see the kernel download at all. The only thing I see is ARP requests and TFTP transfer (download iPXE from FOG server). Seems almost like a filter was used that filters out TCP/HTTP?! Are you sure you let the tcpdump run till you had the bzImage_debug hang on screen? We should see at least some HTTP communication because before the kernel starting iPXE will request boot information via HTTP. We should see this. This is really strange.

Mind trying again? Other than that I’d advice you to try using a different iPXE binary just to see if that makes a difference. Is your FOG providing DHCP to your clients or is it an external DHCP? Currently this client uses i386-efi/ipxe.efi (seen in the PCAP). Change that and tell it to use i386-efi/snponly.efi. See if that makes a difference.

Zerpie

@Sebastian-Roth I let it run a little longer this time. Well after it would hang on BzImage_debug…

Here’s the new link.
https://drive.google.com/file/d/154AX_8LHOGKVLBHetWh9-RxihPoj3qFe/view?usp=sharing

I’ll try changing it to use i386-efi/snponly.efi. Where do I change that?

Sebastian Roth

@Zerpie said in Tablet PC hangs on bzImage:

I’ll try changing it to use i386-efi/snponly.efi. Where do I change that?

That depends on your DHCP server. As you ask about this I suppose you just use the DHCP server that was setup by FOG when running the installer. For this edit /etc/dhcp/dhcpd.conf on your FOG server and find those lines:

...
    class "UEFI-32-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-32-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
        filename "i386-efi/ipxe.efi";
    }
...

Simply change the filename to i386-efi/snponly.efi, save and restart DHCP (service dhcpd restart).

Zerpie

@Sebastian-Roth Thank you for that. I went ahead and changed it. It still hangs on BzImage so I went ahead and captured another hang.pcap.

Here’s the link.
https://drive.google.com/file/d/1fX5xdrz0b_SIPogj7eQFtChOyoDhcf-7/view?usp=sharing

Zerpie

Actually. I just checked the pcap file myself and it looks like it’s still using the ipxe.efi and not snponly.efi. Let me try that again.

Zerpie

Here’s the latest file. I made sure it was grabbing i386-efi/snponly.efi this time.

https://drive.google.com/file/d/1Il1L7msBxgOD2ZIPD3oLr4FifenjwT6E/view?usp=sharing

Sebastian Roth

@Zerpie Something is really going wrong here. Not sure why we see so little information in the PCAP file. There should be much more in it (e.g. at least DHCP). Let’s try doing the same thing just this time simply run tcpdump -i eth0 -w hang3.pcap (this is without a capture filter)

Do you have more than one FOG server?

Other than that, would you be able to capture traffic on the client side as well? Usually you need to have an old hub to connect between client and switch where you hook on another PC and run tcpdump or wireshark to capture. Or you have access to your switch where the client is connected and can setup a monitoring port to capture all the packets on that client port.

Zerpie

@Sebastian-Roth I’m only running one Fog server.

Here’s the hang3.pcap
https://drive.google.com/file/d/1uA54wIeUgPbpduEJZp_RhobKP1Xh4aWq/view?usp=sharing

Sebastian Roth

@Zerpie Ok, this time we see a bit more in the PCAP and I kind of know why my last filter was wrong. Thanks for sticking with me and being patient.

One thing (probably unrelated) that jumped at me is that the kernel args/parameters (debug earlyprintk=efi loglevel=7) are not set. Maybe you just removed them after testing because we saw that the kernel does not even load properly. Just wanted to mention this in case you thing they are still in place. Then maybe something else is wrong here.

So now let’s get to the interesting bits of the PCAP. Near the end we can see the client starting the download of the kernel (GET /fog/service/ipxe/bzImage_debug). And it starts of as normal. Well kind of. Looking at the response from the webserver in detail I have a couple of things bogging me. See the picture where I compare the startup of one of my clients (left) with yours (right) - click on the picture to get a readable version:

PHP version string showing PHP/5.6.37 - should not be possible to run a recent version of FOG with that PHP version. Please run rpm -qa | grep php and post output here.
Content-Length: 1727680 seems like my kernel image is 4-5 times as big as yours is. Why? Please run ls -alk /var/www/html/fog/service/ipxe/bzImage*
The binary data on my side starts off with the magic MZ header [ref] - this is because the kernel is build as EFI executable binary.
The later two things might just be an issue of the bzImage_debug home-brew kernel. Maybe even my fault when I gave you the instructions to build it. Not sure though.

But back to the kernel transfer. It does actually run for a bit including the client sending proper TCP acknowledge packets - as well fairly quickly. Seems all pretty perfect. But then fairly soon (20 ms) the client just stalls out of nowhere, not sending TCP ACKs or any other packets back to the server. The server on the other side keeps asking for confirmation for a couple of seconds and gives up then.

So it looks like iPXE causing the hang - maybe when transferring data over network or just after some amount of time. When I told you to try and use snponly.efi I hoped that this might fix the issue because a different network stack/driver is used. Didn’t help it.

So before we get into building iPXE from source (as easy as building the kernel!) I would like you to try an old iPXE binary that we have used with other tablets when there was an EFI timer issue in iPXE more than two years ago. That issue was fixed in iPXE and therefore I didn’t think about that till now.
ipxe.efi: https://github.com/FOGProject/fogproject/raw/9213bd2a456718b2ce00fa46de4982d35f2703be/packages/tftp/i386-efi/ipxe.efi
snponly.efi https://github.com/FOGProject/fogproject/raw/9213bd2a456718b2ce00fa46de4982d35f2703be/packages/tftp/i386-efi/snponly.efi (only in case you wanna give it a try but I guess if the other one doesn’t do it this won’t either)
Just download the binary and put into your /tftpboot directory on the FOG server (rename original binary I’d suggest).

Zerpie

@Sebastian-Roth said in Tablet PC hangs on bzImage:

PHP version string showing PHP/5.6.37 - should not be possible to run a recent version of FOG with that PHP version. Please run rpm -qa | grep php and post output here.

# rpm -qa | grep php
php-gd-5.6.37-1.el7.remi.x86_64
php-mcrypt-5.6.37-1.el7.remi.x86_64
php-pecl-jsonc-1.3.10-2.el7.remi.5.6.x86_64
php-5.6.37-1.el7.remi.x86_64
php-ldap-5.6.37-1.el7.remi.x86_64
php-pdo-5.6.37-1.el7.remi.x86_64
php-process-5.6.37-1.el7.remi.x86_64
php-common-5.6.37-1.el7.remi.x86_64
php-cli-5.6.37-1.el7.remi.x86_64
php-bcmath-5.6.37-1.el7.remi.x86_64
php-mbstring-5.6.37-1.el7.remi.x86_64
php-mysqlnd-5.6.37-1.el7.remi.x86_64
php-pecl-zip-1.15.3-1.el7.remi.5.6.x86_64
php-fpm-5.6.37-1.el7.remi.x86_6

Content-Length: 1727680 seems like my kernel image is 4-5 times as big as >yours is. Why? Please run ls -alk /var/www/html/fog/service/ipxe/bzImage*

# ls -alk /var/www/html/fog/service/ipxe/bzImage*
-rwxr-xr-x. 1 fog  fog  8118832 Sep  6 16:21 /var/www/html/fog/service/ipxe/bzImage
-rwxr-xr-x. 1 fog  fog  7562352 Sep  6 16:21 /var/www/html/fog/service/ipxe/bzImage32
-rw-r--r--. 1 root root 1727680 Sep 10 13:23 /var/www/html/fog/service/ipxe/bzImage_debug

I re-added the Host kernel Arguments (debug earlyprintk=efi loglevel=7) and tried the old iPXE binary, but it’s still hanging on BzImage.

Sebastian Roth

@Zerpie said in Tablet PC hangs on bzImage:

tried the old iPXE binary, but it’s still hanging on BzImage

Too bad. So I guess we need to get into debugging iPXE. Start by building your own ipxe.efi binary following the instructions here: https://wiki.fogproject.org/wiki/index.php?title=IPXE#Compile

Which version of FOG are you running? Sorry if I asked this before but cannot find it in the thread.

Zerpie

@Sebastian-Roth said in Tablet PC hangs on bzImage:

Which version of FOG are you running?

1.5.4

Tablet PC hangs on bzImage

216

11.7k

17.1k

154.6k