Very slow boot.php and bzImage during pxe boot

[rodluz]

FOG Version: 1.5.9
OS: Ubuntu 20.04 - up to date

Hello, I am having a very strange issue. I have 30 new HP EliteBook 850 G6 laptops with sequential serial numbers. Some of the laptops are working perfectly when trying to register them as hosts. Others, are not booting to the host registration screen because the bzImage part takes around a minute to go up 1%.
When going into pxe boot, the laptop take a while to get to the iPXE menu items.

http://10.27.70.100/fog/service/ipxe/boot.php... ok  #This happens instantly
http://10.27.70.100/fog/service/ipxe/boot.php................ ok  # This takes around 10-15 seconds
http://10.27.70.100/fog/service/ipxe/boot.php......... n%  # This takes about 5-10 seconds

Once I press enter on the Perform Full Host Registration and Inventory menu option, I get:

bzImage................................ n%  # Takes around a minute to go up 1%

At first I thought I may have a batch of laptops that are bad, so I looked through the serial numbers and noticed its all over the place.

I took a laptop that I know doesn’t work to a coworker’s area and used her FOG to see what would happen and there was no issue. This makes me think I screwed up on something with my FOG server.

Only changes I have done to the server recently are post download scripts and add new images/snapins.

Any ideas?

Thanks in advanced!

[Sebastian Roth]

@rodluz said in Very slow boot.php and bzImage during pxe boot:

Only changes I have done to the server recently are post download scripts and add new images/snapins.

Not the things that would cause the described issue I think.

I took a laptop that I know doesn’t work to a coworker’s area and used her FOG to see what would happen and there was no issue. This makes me think I screwed up on something with my FOG server.

That’s interesting. Surely tried a couple of times to make sure it works fine consitently on her FOG server? What FOG version is the other server?

The next thing to look at is the network components. Try connecting the laptop as close to the FOG server as possible. Best if you can take it to the server room and plug it to the very same switch the FOG server is on just for testing the PXE boot.

[rodluz]

@sebastian-roth Hello and thank you for the reply. I tried multiple times on the other FOG server just to make sure with different “bad” laptops and they all worked.

The other FOG server is on 1.5.9.94.

I took a “bad” laptop to the server room and plugged it in to the same switch that my FOG server is connected to and I am still having the same issue.

I originally thought it could be the NIC on those “bad” laptops so ran speed tests using Windows and Linux(Ubuntu) and they are fine. Maybe it could be something with the MAC on the laptops and FOG not accepting it.
I tried using a USB-C to Ethernet adapter to test and it works perfectly with that.

I could just use multiple of those adapters to image the computers but they hold the host data after I register the laptops.

EDIT:
I originally had the kernel version using 4.19.64. Updated it last week to 5.10.34. Just updated it to 5.10.50 today and still having the same issue.

[Sebastian Roth]

@rodluz said in Very slow boot.php and bzImage during pxe boot:

I originally had the kernel version using 4.19.64. Updated it last week to 5.10.34. Just updated it to 5.10.50 today and still having the same issue.

At this stage it’s definitely not the FOS Linux kernel causing the slowness. So swapping that out does not make a difference.

You need to swap out the iPXE binaries (manual download from github) or update to the latest dev-branch version as we see you colleague’s FOG server.

[rodluz]

@sebastian-roth Thanks for the help. I prefer to swap out the iPXE binaries first and see if that helps.

Do I just copy those linked files into /tftpboot then change ownership to fogproject:root and chmod 655 the files, or is there something else I need to do?

[Sebastian Roth]

@rodluz said in Very slow boot.php and bzImage during pxe boot:

Do I just copy those linked files into /tftpboot then change ownership to fogproject:root and chmod 655 the files, or is there something else I need to do?

Yeah, that should do. Though I’d do chmod 644 as they don’t need execution. But would work either way.

[rodluz]

@sebastian-roth Okay so I did that and it’s still acting the same way.

I guess the next step would be to update to the dev-branch?

[Sebastian Roth]

@rodluz No, from the description of the issue I am pretty sure dev-branch won’t help you because you’ve already used the iPXE binaries from dev-branch.

First let’s make sure they are in place. When you PXE boot the machine take a close look at the iPXE header printed on screen. In does print a version number in between braces, like this: iPXE ... (g....) ... - always starts with the letter ‘g’. Note that down and post that here or take a picture of you are not sure what to look at.

Second ask your colleague what kind of DHCP server she’s using and what options for filename (DHCP option 67) is set.

[rodluz]

@sebastian-roth Okay so this is the whole output

>>>Start PXE over IPv4.
  Station IP address is 10.28.68.52

  Server IP address is 10.27.70.100
  NBP filename is ipxe.efi
  NBP filesize is 1060352 Bytes
Downloading NBP file...

  Succeed to download NBP file.
iPXE initializing devices...ok


iPXE 1.21.1+ (gc64d) -- Open source Network Boot Firmware -- http://ipxe.org
Features: DNS FTP HTTP HTTPS iSCSI NFS TFTP SRP VLAN AoE EFI Menu
Configuring (net0 30:24:A9:...)..... ok
Received DHCP answer on interface net0
tftp://10.27.70.100/default.ipxe... ok
http://10.27.70.100/fog/service/ipxe/boot.php... ok
Booting... (Press ESC to access the menu)

The iPXE version is newer, because it used to be 1.20.1+ before the change.

I was the one that set up the DHCP setting for my colleague’s and my VLANs so they are both ipxe.efi.
I checked to make sure that nothing is broken between VLANs and switches and that everything is Gigabit.
I also tried disabling the ADVANCE MENU LOGIN & PXE MENU HIDDEN just to check and still same issue. I went into the FOG Configuration and updated the MAC addresses too but no luck.

[Sebastian Roth]

@rodluz said in Very slow boot.php and bzImage during pxe boot:

(gc64d)

OK that’s definitely the current iPXE version I pushed to dev-branch not long ago.

Thought that you might have different DHCP options set that might explain the issue. But doesn’t look like it.

Hmm, you can try updating to dev-branch but I can’t think of a change we pushed since 1.5.9 that would explain what you describe. Anyhow, give it a go.

[rodluz]

@sebastian-roth I will have to wait until late next month to update. I have to image around 200 computers in the coming weeks so I’ll leave those laptops as the last devices to image in case something breaks with the update.
Thanks for all your help, I will reply to the thread after I update the server.

[Warsonix]

Yo!

DId you find any solution for this!

I am fogging a batch of laptop , everything going fine so far

But I’m stuck on 2 dynabook, I’ve fogged 3 so far no problem , but now I have 2 dynabook side by side , 1 has completed fog deploy image and the other one at each time I boot for PXE it stuck at bzimage going up verrrryyy slow and cant reach 100%

Any idea?

Thanks

[rodluz]

@Warsonix I really needed to get those laptops imaged asap, so I just used a USB-C to Ethernet adapter on those and it worked. I haven’t touched those laptops since then.
Funnily enough, I have to re-image them next week, so I’ll let you know what happens.
I have done a lot of changes to my FOG server since then, so it’s really going to depend on your setup too.