HP Stream 11 G3 - locks up after init.xz?

[theterminator93]

Hi all,

I’m trying to get FOG to work with some HP Stream 11 G3s for a client. I recently imaged all of them with CloneZilla (no Ethernet port), but wanted to try out FOG since it would reduce deployment time in our environment under our particular circumstances.

I wanted to ensure maximum likelihood that the USB-Ethernet adapter I purchased would properly PXE boot, so I picked up HP’s branded USB 3.0 to Ethernet adapter. It PXE boots perfectly, and I get the FOG menu. When I try to do a full registration and inventory, it shows that it loads the kernel and init.xz, then it locks up. No caps lock LED response, stays that way after 10-15 minutes, and I have to hold the power button for 10 seconds to get it to power off.

I’m running 1.4.0 RC6 (and will update, try again, and report after I get onsite this morning) and I made sure I had the most up-to-date kernel - but no change. Any suggestions or config changes which might resolve this?

[Sebastian Roth]

@theterminator93 Have you tried other iPXE binaries yet? See in the /tftpboot directory on your FOG server. Possibly ipxe7156.efi will do the trick for you.

[theterminator93]

Thanks for the suggestion. Now running the latest git version of 1.4.2 - no changes.

I tried ipxe7156.efi but I got an error saying it was too large to fit in base memory. Realtek7156.efi just locked up after DHCP got an IP, ipxe.pxe loaded but instantly booted to the hard disk (no FOG menu). I ran out of time to try more, but I will do so next week. None of the EFI options will work as I’m booting legacy, I assume.

[Sebastian Roth]

@theterminator93 Oh well, I somehow assumed you were running those UEFI enabled, my fault. Surely disabled secure boot, have you?

Then try other iPXE binaries: undionly.kpxe (default you see the problem with I suppose), undionly.kkpxe (notice the double ‘k’), undionly.pxe and then ipxe.kpxe, ipxe.kkpxe and ipxe.pxe… Please lat us know what happens with each of these?!

I wanted to ensure maximum likelihood that the USB-Ethernet adapter I purchased would properly PXE boot, so I picked up HP’s branded USB 3.0 to Ethernet adapter.

I have heard of vendor USB NICs that were not properly PXE booting. The BIOS of your device has to have a driver to be able PXE boot on the USB NIC. Not all vendors play this nicely. But from what you have told so far it’s looking pretty good! When you see the FOG menu you have made the first couple of hurdles already. So we should be able to make the rest of them as well.

I remember extensively debugging a iPXE to kernel handover issue on a different HP device a good year ago. Read through this if you like. But this is a little different because it was UEFI booting and it turned out that HP had a bugby firmware that got fixed after we told them about it. I really hope this is not the case here too.
To make a long story short you can try those kernels I compiled back then. Try bzImage_ng, add the USB NICs MAC as new host in the FOG web gui by hand and add debug earlyprintk=vga as host kernel parameters too. Schedule an upload job for this host and boot it up. See what you get.

[theterminator93]

Thanks again for the response. Yep, Secure Boot is off as well. Initially I had been using undionly.kpxe. Here are my observations with each of the binaries I found in /tftpboot which weren’t EFI:

default.ipxe ----> Stops after getting an IP from DHCP (will reboot with CTRL+ALT+DEL)
intel.pxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
intel.kpxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
intel.kkpxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
ipxe.pxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
ipxe.krn ----> Stops after getting an IP from DHCP (will reboot with CTRL+ALT+DEL)
ipxe.kpxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
ipxe.kkpxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
pxelinux.0.old —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
realtek.pxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
realtek.kpxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
realtek.kkpxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
undionly.pxe —> Appears to try to boot to network, skips FOG menu and boots to hard disk immediately
undionly.kpxe —> Boots normally. I can load memtest fine from the FOG menu. I was also able to ‘approve this host’ but after it tried to boot the kernel to do an inventory, it locked up at init.xz (had to hard reset).
undionly.kkpxe —> Booted normally. Locked up at init.xz when trying to do hardware inventory from before. Canceled the task, memtest loaded fine.

Fortunately with the new FOG client it registered itself automatically (love that feature!) so I didn’t need to manually enter the MAC to register it. I put bzImage_ng into the /tftpboot directory and changed the kernel under TFTP settings in the web interface, but when attepmting to boot it just keeps adding “…” as it waits, so I fear I may not have set permissions on the file/directory properly (chmod -R 777 /tftpboot).

[george1421]

It would be interesting to know if usb booting directly info FOS gives you the same issue as ipxe -> fos. I have seen a few other threads where the target computer is having an issue transitioning from iPXE to FOS. Booting from a usb flash drive might tell us if there is an issue with the FOS kernel and the hardware or its with the hand-off from iPXE to FOS.

There is a guide here on how to build a usb boot drive to boot into FOS via grub. https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image

[Sebastian Roth]

@theterminator93 said in HP Stream 11 G3 - locks up after init.xz?:

Fortunately with the new FOG client it registered itself automatically (love that feature!) so I didn’t need to manually enter the MAC to register it. I put bzImage_ng into the /tftpboot directory and changed the kernel under TFTP settings in the web interface, but when attepmting to boot it just keeps adding “…” as it waits, so I fear I may not have set permissions on the file/directory properly (chmod -R 777 /tftpboot).

Thanks for all the testing. Unfortunately no luck with a different iPXE binary. Can you please download 01_undionly.kpxe from here (it’s compiled with DEBUG=bzimage,image options so we hopefully see a bit more) and put it into /tftpboot directory on your server. Then either change your DHCP settings to use this file or rename those files. Better try using the original bzImage first so we can see the iPXE debugging information on screen. Please take a picture and post here.

[theterminator93]

Couple updates. I realized that I had incorrectly put bzImage_ng into /tftpboot - so I moved it to /var/www/html/fog/service/ipxe and reset the permissions according to the wiki article. Booted using undionly.kpxe and it still locked up after init.xz.

So then I tried booting with 01_undionly.kpxe. I have attached what I get after it locks up…

I then tried creating and booting to a FOS USB as George suggested. It loaded fine, and since I had already registered and approved the host in the web GUI, I attempted to capture the image - which worked. I made sure I was booting to the USB media in legacy (not EFI) mode as well. I’m guessing that means it’s an issue with iPXE to FOS handoff?

[george1421]

@theterminator93 said in HP Stream 11 G3 - locks up after init.xz?:

I made sure I was booting to the USB media in legacy (not EFI) mode as well

FYI The usb FOS boot supports both efi and bios modes if that matters.

[Sebastian Roth]

@theterminator93 Thanks for testing and posting a picture here. I am not exactly sure yet but booting this 01_undionly.kpxe in a virtual machine I see the this…

Up till now I wasn’t sure if iPXE or the kernel is causing the hang. I am still not exactly sure if we are right on spot here but comparing the debug outputs I think in your case iPXE is playing up. So I compiled a new binary (02_undionly.kpxe, download) with added debug statements. Please use this new binary and take another picture. Unfortunately this will be a step by step thing till we find out what is causing the hang…

[theterminator93]

All righty - here’s what we get with 02_undionly.kpxe:

[Sebastian Roth]

This is how it looks booting this in a virtualbox:

I don’t know the iPXE code and stuff too well. Hopefully the iPXE devs can help us here. I will get in contact and hopefully get some information. Meanwhile you can try 03_undionly.kpxe which I just uploaded. It will end up in a panic after loading the kernel but it would still be interesting to see if it’s doing the same on your HP device.

[theterminator93]

That binary isn’t loading, for some reason it is trying to pull bzImage from a class C private address rather than the address of the server onsite after it gets an IP.

[george1421]

@theterminator93 Wow that is strange. The system that manages where it gets the bImage file is managed by your fog server.

Can you call this url with a web browser, replacing the fog server IP that is correct for your network?

http://<fog_server_ip>/fog/service/ipxe/boot.php?mac=00:00:00:00:00:00

Look through that text file and see if you see that unknown class C address.

If you go into the fog web gui and schedule a deployment/capture task for some test system and then update the mac address above with the actual mac address of the test system does it show you that class C address?

[Sebastian Roth]

@theterminator93 Sorry for that. My fault. I compiled and uploaded this binary in a rush and still had a different iPXE script embedded that I use for testing with my virtualbox environment. Please try 04_undionly.kpxe… (same download link as before)

[Sebastian Roth]

@theterminator93 Could you please boot George’s USB stick again, schedule a debug upload task and run the following command when you get to the command shell: lspci -nn | grep -i ethernet

As well I build a few more binaries to test with after talking to one of the iPXE developers. Please skip number four and go straight to 05_undionly.kpxe (DEBUG=init,device,undi,pci and added statements).

Then there is 06_undionly.kpxe which is a build from the iPXE source without using our FOG modified header files (suggested by the iPXE develover). Only change I made was adding CMD_PARAM because we need this! Don’t worry about the FOG menu being very basic with that one.

And finally it would be interesting to know if iPXE also hangs when a normal exit is done. 07_undionly.kpxe does just that. What happens when you use this?

Please take pictures of all four and post here. Thanks!

[george1421]

@Sebastian-Roth said in HP Stream 11 G3 - locks up after init.xz?:

, schedule a debug upload task

Psssst… in the grub menu, menu item #6 (I think) you can jump right to debug mode no capture or deploy step required. You can’t continue to capture or deploy from there, but you can run the commands you outlined.

[theterminator93]

Strange. I rebooted the server and the 192.168.x.x address was no longer interfering with PXE booting. No signs of it in boot.php based on the suggestions from George at this point either.

In any case, if I run the first command from a FOS USB, it comes back with no results. What’s interesting is it tries three times to get an IP but seems to give up, despite its actually getting an address…?

Here’s what it locked up at with 05_undionly and the added host debug parameters DEBUG=init,device,undi,pci.

06_undionly with the extra debug params was just a black screen when it locked up, with bzImage… ok and init.xz… ok and a blinking cursor.

And 07, with debug params, just skips the FOG menu and boots to the OS. This is what’s on the screen for a split-second before it starts booting Windows.

[Sebastian Roth]

@theterminator93 Thanks again for testing and posting pictures!

Strange. I rebooted the server and the 192.168.x.x address was no longer interfering with PXE booting.

That was my fault. I had the wrong script embedded in one of the older binaries. See my post here further down. Don’t worry about it.

In any case, if I run the first command from a FOS USB, it comes back with no results. What’s interesting is it tries three times to get an IP but seems to give up, despite its actually getting an address…?

FOS tries to contact the FOG webserver after getting an IP via DHCP to make sure it’s fully connected. When building George’s USB FOS you need to change myfogip=x.x.x.x in that script to match your FOG server IP. Otherwise you’ll run into this issue.
About the empty output, again my fault, sorry! Should have asked you to run lsusb instead! Post a picture of the full output please. One first info we already have thanks to the picture. Seems like the driver r8152 is handling eth0… Will be interesting to see the USB IDs.

Here’s what it locked up at with 05_undionly

Alright, it hangs when trying to unload the UNDI root bus. Can’t tell you exactly what that is but it sounds like the UNDI implementation of that USB NIC is faulty. Possibly we can work around this but I need a little more time. Let me see.

06_undionly with the extra debug params was just a black screen when it locked up, with bzImage… ok and init.xz… ok and a blinking cursor.

Fine, so none of our iPXE header configs is causing this issue. 06 was the clean build.

And 07, with debug params, just skips the FOG menu and boots to the OS…

Nice! See this, when iPXE simply exits (shutting itself down to boot from hard disk) there is no hang on “Removed UNDI root bus”!

Hope I can give you some more information or binaries to test soon!

[Sebastian Roth]

@theterminator93 Ok, here is the latest binary with even more debug output. It’s called 08_undionly.kpxe (link). Let’s hope we reach the end soon!