• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Tablet PC hangs on bzImage

    Scheduled Pinned Locked Moved Solved
    FOG Problems
    5
    80
    17.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Z
      Zerpie @Sebastian Roth
      last edited by

      @Sebastian-Roth Sure! I’m not as familiar with Linux as I’d like to be so bear with me if I have a lot of questions or make some mistakes. But I’m up for doing whatever I can to figure this out. Our office has had a lot of trouble finding a solution to easily image these tablets so that we can quickly get them out to the customers. I’m hoping Fog is going to be the solution that works for us.

      1 Reply Last reply Reply Quote 0
      • S
        Sebastian Roth Moderator
        last edited by Sebastian Roth

        @Zerpie said in Tablet PC hangs on bzImage:

        I’m hoping Fog is going to be the solution that works for us.

        First I have to say that there is hope but I can’t promise you a full solution. I remember one time when we were debugging such an issue and it turned out the firmware was buggy. We boiled it down to the bits but the firmware company never officially released a fixed version. So we had to live with a patch in the kernel to work around that firmware bug.

        Anyhow, let’s get started. Make sure you have sufficient space on your FOG server to install developer tools and extract a couple of source code packages - 10 GB should be heaps.

        Install developer tools (commands from our wiki, have not tried those myself lately):

        debian/ubuntu# sudo apt-get install git build-essential zlib1g-dev binutils-dev
        fedora/centos# sudo yum install git gcc gcc-c++ make zlib-devel binutils-devel
        

        See which kernel you are running on your FOG server right now. Not the host OS kernel but the one delivered to the clients. Depending on your OS you can use the file command for that: file /var/www/html/fog/service/ipxe/bzImage*

        Download that same kernel version (put in the version number instead of linux-4.x.y below) and start building:

        mkdir build
        cd build
        wget https://mirrors.edge.kernel.org/pub/linux/kernel/v4.x/linux-4.x.y.tar.xz
        tar xJf linux-4.x.y.tar.xz
        cd linux-4.x.y
        git clone git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
        wget -O .config https://github.com/FOGProject/fos/raw/master/configs/kernelx64.config
        make oldconfig
        make bzImage
        

        Now this may take half an hour, possibly longer depending on your processor and RAM.

        See if you can build that 64 bit kernel properly. If not, post the exact error message here.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        Z 1 Reply Last reply Reply Quote 0
        • Z
          Zerpie @Sebastian Roth
          last edited by

          @Sebastian-Roth said in Tablet PC hangs on bzImage:

          make oldconfig

          I got as far as this command but when I run the command I get the following.

          # make oldconfig
            HOSTCC  scripts/basic/fixdep
            HOSTCC  scripts/kconfig/conf.o
            YACC    scripts/kconfig/zconf.tab.c
          /bin/sh: bison: command not found
          make[1]: *** [scripts/kconfig/zconf.tab.c] Error 127
          make: *** [oldconfig] Error 2
          
          1 Reply Last reply Reply Quote 0
          • S
            Sebastian Roth Moderator
            last edited by

            @Zerpie My fault, missing developer package…

            debian/ubuntu# sudo apt-get install bison
            fedora/centos# sudo yum install bison
            

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            Z 1 Reply Last reply Reply Quote 0
            • Z
              Zerpie @Sebastian Roth
              last edited by

              @Sebastian-Roth Thanks. To be fair, I should have been able to figure that out, myself, but I’m still getting comfortable with Linux.

              Alright, I built the kernel. What is the next step?

              Thanks again for your help.

              1 Reply Last reply Reply Quote 0
              • S
                Sebastian Roth Moderator
                last edited by Sebastian Roth

                @Zerpie Good to hear. Now give it a first try by copying it to the right location and configure this new one as “Host Kernel” in one of your hosts’ settings. This way you can simply test with one or a few clients without causing an issue for other clients.

                cp build/linux-4.x.y/arch/x86/boot/bzImage /var/www/html/fog/service/ipxe/bzImage_debug
                

                Only clients having set bzImage_debug as “Host Kernel” will receive this kernel image on boot.
                If you got this to work you can go back to your kernel source and modify the kernel config (build/linux-4.x.y/.config). Find CONFIG_EARLY_PRINTK and chnage from # CONFIG_EARLY_PRINTK is not set to CONFIG_EARLY_PRINTK=y (make sure you remove the hash tag at the beginning of the line too). Save the file and run:

                make oldconfig
                make bzImage
                cp arch/x86/boot/bzImage /var/www/html/fog/service/ipxe/bzImage_debug
                

                Build won’t take very long this time!

                Now where you set the “Host Kernel” for one of the clients you need to add “Host Kernel Arguments” as well: debug earlyprintk=efi loglevel=7

                Keeping my fingers crossed that we see some early kernel messages on screen then. If not there is still more we can do.

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                Z 1 Reply Last reply Reply Quote 0
                • Z
                  Zerpie @Sebastian Roth
                  last edited by

                  @Sebastian-Roth Now it just hangs on

                  BzImage_debug...
                  
                  1 Reply Last reply Reply Quote 0
                  • S
                    Sebastian Roth Moderator
                    last edited by Sebastian Roth

                    @Zerpie Ok, now we get into the code an add our own debug statements. Edit linux-x.y/arch/x86/boot/compressed/eboot.c and jump to where function efi_main is defined, should look like this or pretty similar depending on the kernel version:

                    /*
                     * On success we return a pointer to a boot_params structure, and NULL
                     * on failure.
                     */
                    struct boot_params *efi_main(struct efi_config *c,
                                                 struct boot_params *boot_params)
                    {
                            struct desc_ptr *gdt = NULL;
                            efi_loaded_image_t *image;
                    ...
                            sys_table = _table;
                            efi_printk(sys_table, "Hello World!\n"); 
                    
                            /* Check if we were booted by the EFI firmware */
                            if (sys_table->hdr.signature != EFI_SYSTEM_TABLE_SIGNATURE)
                    ...
                    

                    That efi_printk call is not part of the original code. That’s what I ask you to add, just that one line. Then compile once again, copy the new kernel over and try booting that same client where you have set the debug kernel image and parameters in host’s settings.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    Z 1 Reply Last reply Reply Quote 0
                    • Z
                      Zerpie @Sebastian Roth
                      last edited by

                      @Sebastian-Roth It’s still hanging on BzImage_debug… and it will usually get to a percentage and stop. Just now it stopped at

                      BzImage_debug... 15%
                      

                      No “Hello World” message.

                      1 Reply Last reply Reply Quote 0
                      • S
                        Sebastian Roth Moderator
                        last edited by

                        @Zerpie Ok, well that’s interesting. So it might not even load the full kernel image but hang on the network connection. So let’s tackle this from a different angle. Install package tcpdump and run:

                        sudo tcpdump -w hang.pcap host x.x.x.x
                        

                        Put in the client’s IP address instead of x.x.x.x and let the command just sit there. Now boot up the client and wait till it hangs. Wait another 10 seconds and then stop tcpdump (ctrl+c). Upload the generated hang.pcap file (will be around 30 MB or more) and post a download link here in the forums (or send me a PM).

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        Z 1 Reply Last reply Reply Quote 0
                        • Z
                          Zerpie @Sebastian Roth
                          last edited by

                          @Sebastian-Roth When I run the command I get the following

                          # tcpdump -w hang.pcap host 153.86.19.24
                          tcpdump: NFLOG link-layer type filtering not implemented
                          
                          1 Reply Last reply Reply Quote 0
                          • S
                            Sebastian Roth Moderator
                            last edited by Sebastian Roth

                            @Zerpie Ok, usually tcpdump is able to select the proper network interface just by itself but does not seem to work here - which Linux OS and version do you use?

                            # tcpdump -i eth0 -w hang.pcap host 153.86.19.24
                            

                            Put in the interface FOG is using. If you are not sure run grep interface /opt/fog/.fogsettings to find out.

                            The command should return a message like listening on ..., link-type EN10MB (Ethernet), capture size ... and won’t return to the shell. Leave it like that till you booted the client to hang.

                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                            Z 1 Reply Last reply Reply Quote 0
                            • Z
                              Zerpie @Sebastian Roth
                              last edited by

                              @Sebastian-Roth I’m using CentOS 7. Specifying the network interface worked, though. Here’s a link to the hang.pcap file shared on my Google Drive.

                              https://drive.google.com/file/d/1SzQGpk4DsDf3yuB2P7AemvxMXzMdcmYf/view?usp=sharing

                              1 Reply Last reply Reply Quote 0
                              • S
                                Sebastian Roth Moderator
                                last edited by

                                @Zerpie That’s very strange. In the PCAP I don’t even see the kernel download at all. The only thing I see is ARP requests and TFTP transfer (download iPXE from FOG server). Seems almost like a filter was used that filters out TCP/HTTP?! Are you sure you let the tcpdump run till you had the bzImage_debug hang on screen? We should see at least some HTTP communication because before the kernel starting iPXE will request boot information via HTTP. We should see this. This is really strange.

                                Mind trying again? Other than that I’d advice you to try using a different iPXE binary just to see if that makes a difference. Is your FOG providing DHCP to your clients or is it an external DHCP? Currently this client uses i386-efi/ipxe.efi (seen in the PCAP). Change that and tell it to use i386-efi/snponly.efi. See if that makes a difference.

                                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                Z 1 Reply Last reply Reply Quote 0
                                • Z
                                  Zerpie @Sebastian Roth
                                  last edited by

                                  @Sebastian-Roth I let it run a little longer this time. Well after it would hang on BzImage_debug…

                                  Here’s the new link.
                                  https://drive.google.com/file/d/154AX_8LHOGKVLBHetWh9-RxihPoj3qFe/view?usp=sharing

                                  I’ll try changing it to use i386-efi/snponly.efi. Where do I change that?

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    Sebastian Roth Moderator
                                    last edited by Sebastian Roth

                                    @Zerpie said in Tablet PC hangs on bzImage:

                                    I’ll try changing it to use i386-efi/snponly.efi. Where do I change that?

                                    That depends on your DHCP server. As you ask about this I suppose you just use the DHCP server that was setup by FOG when running the installer. For this edit /etc/dhcp/dhcpd.conf on your FOG server and find those lines:

                                    ...
                                        class "UEFI-32-2" {
                                            match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
                                            filename "i386-efi/ipxe.efi";
                                        }
                                        class "UEFI-32-1" {
                                            match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
                                            filename "i386-efi/ipxe.efi";
                                        }
                                    ...
                                    

                                    Simply change the filename to i386-efi/snponly.efi, save and restart DHCP (service dhcpd restart).

                                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                    Z 1 Reply Last reply Reply Quote 0
                                    • Z
                                      Zerpie @Sebastian Roth
                                      last edited by Zerpie

                                      @Sebastian-Roth Thank you for that. I went ahead and changed it. It still hangs on BzImage so I went ahead and captured another hang.pcap.

                                      Here’s the link.
                                      https://drive.google.com/file/d/1fX5xdrz0b_SIPogj7eQFtChOyoDhcf-7/view?usp=sharing

                                      Z 1 Reply Last reply Reply Quote 0
                                      • Z
                                        Zerpie @Zerpie
                                        last edited by Zerpie

                                        Actually. I just checked the pcap file myself and it looks like it’s still using the ipxe.efi and not snponly.efi. Let me try that again.

                                        1 Reply Last reply Reply Quote 0
                                        • Z
                                          Zerpie
                                          last edited by

                                          Here’s the latest file. I made sure it was grabbing i386-efi/snponly.efi this time.

                                          https://drive.google.com/file/d/1Il1L7msBxgOD2ZIPD3oLr4FifenjwT6E/view?usp=sharing

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            Sebastian Roth Moderator
                                            last edited by

                                            @Zerpie Something is really going wrong here. Not sure why we see so little information in the PCAP file. There should be much more in it (e.g. at least DHCP). Let’s try doing the same thing just this time simply run tcpdump -i eth0 -w hang3.pcap (this is without a capture filter)

                                            Do you have more than one FOG server?

                                            Other than that, would you be able to capture traffic on the client side as well? Usually you need to have an old hub to connect between client and switch where you hook on another PC and run tcpdump or wireshark to capture. Or you have access to your switch where the client is connected and can setup a monitoring port to capture all the packets on that client port.

                                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                            Z 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 1 / 4
                                            • First post
                                              Last post

                                            173

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project