Fog Menu loop

Sebastian Roth

:menuAccess
login

Please take a look through the iPXE menu options in the web UI. Maybe there is something wrong there.

Rebelman

Please take a look through the iPXE menu options in the web UI. Maybe there is something wrong there.

Is there something specific I need to be looking for, I’m under the iPXE Menu Item Settings and not sure what to be looking for.

Sebastian Roth

@Rebelman See if the option for hidden menu is checked for example.

I need to take a look at the code later on to hopefully get an idea where this might be coming from. Will do that on the way home soon.

Sebastian Roth

@Rebelman Taking a look at the code I can confirm that this part of the menu is related to the “hide menu” setting. Though I am still not sure if we see a bug here or if there is something going wrong only in your particular situation (maybe update problem, but really shouldn’t be).

Did you enable the hidden menu intentionally? Is is enabled at all when you look at the settings in the web UI?

Rebelman

@Sebastian-Roth
Here are my Menu hide settings, nothing is checked as far as hiding the menu.

Sebastian Roth

@Rebelman I still can’t replicate this issue unless I enable the hidden menu. Please check in the database to make sure the web UI is not playing tricks on us.

shell> mysql -u root -p
Enter password:
...
mysql> use fog;
...
mysql> SELECT * FROM globalSettings WHERE globalSettings.settingKey LIKE '%HID%';
...
mysql> quit

Please take a picture of the output and post here.

Rebelman

@Sebastian-Roth said in Fog Menu loop:

mysql> SELECT * FROM globalSettings WHERE globalSettings.settingKey LIKE ‘%HID%’;

Here are my lines that return when I type that in.

+-----------+----------------------------+-----------------------------------------------------------------------------+--------------+-------------------+
| settingID | settingKey                 | settingDesc                                                                 | settingValue | settingCategory   |
+-----------+----------------------------+-----------------------------------------------------------------------------+--------------+-------------------+
|        64 | FOG_PXE_MENU_HIDDEN        | This setting defines if you would like the FOG pxe menu hidden or displayed | 0            | FOG Boot Settings |
|       114 | FOG_PXE_HIDDENMENU_TIMEOUT | This setting defines the default value for the pxe hidden menu timeout.     | 30           | FOG Boot Settings |
+-----------+----------------------------+-----------------------------------------------------------------------------+--------------+-------------------+
2 rows in set (0.01 sec)

Sebastian Roth

@Rebelman This is not making sense to me. Seems like hidden menu is definitely not active on your server but still you get the output from boot.php as if it would be enabled.

One thing I noticed in the output you posted is that the settingID of FOG_PXE_MENU_HIDDEN on your server is 64. On mine it is 69. But FOG_PXE_HIDDENMENU_TIMEOUT is 114 same as yours. The settingID does not play a role as far as I know but I wonder if this is a first sign of some database issue/difference that might be playing a role in this.

Let’s step back from the details a little bit. Re-reading your posts I wonder about what you are saying this only being an issue with UEFI machines. You said: “I also just tested on an older machine that still uses Legacy and I have no problems imaging it”. Was this legacy machine as well new to FOG, I mean unregistered?

You might want to run the database maintenance just to make sure: https://wiki.fogproject.org/wiki/index.php/Troubleshoot_MySQL#Database_Maintenance_Commands

Could you possibly take a video of the boot process? Would be great to see the exact difference. Take one video where you boot an unregistered UEFI machine. As well take another video where you do the exact same steps with an unregistered legacy machine. Upload those videos and post links here.

For a clear good readable video capture you can rest your camera or smartphone on a pile of books on your desk. Will be way less shaky than doing it free hand.

Rebelman

@Sebastian-Roth

I did the database maintenance and every line I put in on my server came back “Query OK, 0 rows affected”

The two links direct to the videos you were wanting to see.
FOG Legacy Boot
FOG UEFI Boot

I’m wondering if I should just do a reinstall on my server and start fresh, ( update OS while I’m at it since I’m running an older version) maybe something goofed up during install and that’s what’s causing my problems.

Sebastian Roth

@Rebelman Great you took those videos. Well done!! It’s way easier to follow than reading text and trying to wind my head around what is being said.

So first thing I noticed is that in the text output where it asks you to press a key to get to the menu I see ESC on the UEFI machine and Escape on the legacy machine. I still can’t make sense of this difference and so allow me to ask if those two clients surely booted from the exact same FOG server? Looking at the text output I should be able to see if they both boot from the same IP but the text is too blurry on the UEFI boot video that I just can’t say for sure it’s the exact same IP they boot from. Just want to make sure it is.

Second we see that both systems seem to boot from a hidden menu. But the option is not set in the web UI or the database as we checked. Did you ever intend to hide the FOG menu? Do you remember since when it is like that (need to hit ESC and enter password to get the menu)?

Just to make sure we are not hunting down the wrong track here, can you please grab a copy of the file /var/www/html/fog/lib/fog/bootmenu.class.php, upload it to a file sharing platform and post a link here.

Ok, watching the videos over and over I have a feeling that the difference between legacy and UEFI is that loading the kernel/init (FOS) after selecting Full Registration from the menu simply fails on UEFI and because of that you are thrown back to the menu (which is black and white due to a minor issue we have in the code - should fix that but it’s not causing an issue for you here). So the big question is, why does loading the kernel/init actually fail on this UEFI machine??? It really shouldn’t as this is working for so many other people on UEFI machines. Do you have other UEFI models around? Can you please test if you have the same behavior with all UEFI machines?! Also check if there is a firmware upgrade for this particular machine model you see the issue with.

I have to say again that this would not have been possible without you taking the videos. There are small bits of information I can get out of such a video that we would not be able to get to by me just asking questions and you explaining what is going on.

I’m wondering if I should just do a reinstall on my server and start fresh

Maybe this might help in the hidden menu issue but it most probably won’t change anything for the UEFI booting issue. I am still in hope that we can figure out the hidden menu thing without you re-installing the server.

Sebastian Roth

@Rebelman One last thing I forgot earlier. Please open this URL in your browser and post the full output here: http://10.254.1.20/fog/service/ipxe/boot.php?menuAccess=1

Rebelman

@Sebastian-Roth
Yes, I only have it booting to one FOG server, the picture shows the UEFI boot screen.

For as long as I have had FOG running, I’ve always had to press escape to get into the menu. I’ve never marked it to be hidden, at least not intentionally.

This is the link for the bootmenu.class.php file
Bootmenu

This is my menuAccess info you are also wanting.

#!ipxe
set fog-ip 10.254.1.20
set fog-webroot fog
set boot-url http://${fog-ip}/${fog-webroot}
cpuid --ext 29 && set arch x86_64 || set arch i386
goto get_console
:console_set
colour --rgb 0x00567a 1 ||
colour --rgb 0x00567a 2 ||
colour --rgb 0x00567a 4 ||
cpair --foreground 7 --background 2 2 ||
goto MENU
:alt_console
cpair --background 0 1 ||
cpair --background 1 2 ||
goto MENU
:get_console
console --picture http://10.254.1.20/fog/service/ipxe/bg.png --left 100 --right 80 && goto console_set || goto alt_console
:MENU
menu
colour --rgb 0xff0000 0 ||
cpair --foreground 1 1 ||
cpair --foreground 0 3 ||
cpair --foreground 4 4 ||
item --gap Host is NOT registered!
item --gap -- -------------------------------------
item fog.local Boot from hard disk
item fog.memtest Run Memtest86+
item fog.reginput Perform Full Host Registration and Inventory
item fog.reg Quick Registration and Inventory
item fog.deployimage Deploy Image
item fog.multijoin Join Multicast Session
item fog.sysinfo Client System Information (Compatibility)
choose --default fog.local --timeout 0 target && goto ${target}
:fog.local
sanboot --no-describe --drive 0x80 || goto MENU
:fog.memtest
kernel memdisk initrd=init.xz iso raw
initrd init.xz
boot || goto MENU
:fog.reginput
kernel loglevel=4 initrd=30 root=/dev/ram0 rw ramdisk_size=127000 web=http://10.254.1.20/fog/ consoleblank=0 rootfstype=ext4 storage=10.254.1.20:/images/ storageip=10.254.1.20 loglevel=4 mode=manreg
imgfetch 30
boot || goto MENU
:fog.reg
kernel loglevel=4 initrd=30 root=/dev/ram0 rw ramdisk_size=127000 web=http://10.254.1.20/fog/ consoleblank=0 rootfstype=ext4 storage=10.254.1.20:/images/ storageip=10.254.1.20 loglevel=4 mode=autoreg
imgfetch 30
boot || goto MENU
:fog.deployimage
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param qihost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.multijoin
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param sessionJoin 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.sysinfo
kernel loglevel=4 initrd=30 root=/dev/ram0 rw ramdisk_size=127000 web=http://10.254.1.20/fog/ consoleblank=0 rootfstype=ext4 storage=10.254.1.20:/images/ storageip=10.254.1.20 loglevel=4 mode=sysinfo
imgfetch 30
boot || goto MENU
:bootme
chain -ar http://10.254.1.20/fog/service/ipxe/boot.php##params ||
goto MENU
autoboot

And finally no, I don’t have any other models per say I have different machines I can test, we just bought a batch of computers for a new lab and all are the same model and specs. I can try to upgrade the firmware and see if that makes a difference.

george1421

@Rebelman Lets take a divide an concur approach here.

Lets assume the upgrade did not go as expected. This will be part 1

From the linux console change to /var/www
If there is a fog directory there rename it to fog.old.
Change to /var/www/html
If there is a fog directory there rename it to fog.old
Change to the directory where you downloaded the fog installer. Depending on which instructions it might be /root/fogproject
Change to the bin directory and rerun the fog installer. The fog installer should not ask the questions over again since it should have detected the settings from the previous install. That will rebuild the entire web ui fog directory. Don’t skip the part about updating the database instructions in the linux console. You will need to use a web browser and key in the url listed on the web console.
Reboot the fog server after the reinstall is done.

Now test the fog menu. You should see the fog menu without having to hit escape. There is no negative impact of doing this step.

Sebastian Roth

@Rebelman I am ob the road and can’t post a long answer right now. Looking at the boot.php output you posted I already see that there seems to be an issue with the iPXE code generated. I will have a closer look when I get home tonight.

Tom Elliott

@Rebelman said in Fog Menu loop:

s

What’s the output of mysql statement:

SELECT settingKey,settingValue FROM globalSettings WHERE settingKey IN ('FOG_EFI_BOOT_EXIT_TYPE','FOG_KERNEL_ARGS','FOG_KERNEL_DEBUG','FOG_KERNEL_LOGLEVEL','FOG_KERNEL_RAMDISK_SIZE','FOG_KEYMAP','FOG_KEY_SEQUENCE','FOG_MEMTEST_KERNEL','FOG_PXE_BOOT_IMAGE','FOG_PXE_BOOT_IMAGE_32','FOG_PXE_HIDDENMENU_TIMEOUT','FOG_PXE_MENU_HIDDEN','FOG_PXE_MENU_TIMEOUT','FOG_TFTP_PXE_KERNEL','FOG_TFTP_PXE_KERNEL_32') ORDER BY settingKey;

There should be 15 values returned, and I’m suspecting one of these keys is missing. Please copy and paste the statement into your shell to get the return, then copy and paste the return values into a posting so we can see what’s going on a little easier.

To make things easier to see (and for us to easily view the items more directly) the values in order would be something like:

FOG_EFI_BOOT_EXIT_TYPE
FOG_KERNEL_ARGS
FOG_KERNEL_DEBUG
FOG_KERNEL_LOGLEVEL
FOG_KERNEL_RAMDISK_SIZE
FOG_KEYMAP
FOG_KEY_SEQUENCE
FOG_MEMTEST_KERNEL
FOG_PXE_BOOT_IMAGE
FOG_PXE_BOOT_IMAGE_32
FOG_PXE_HIDDENMENU_TIMEOUT
FOG_PXE_MENU_HIDDEN
FOG_PXE_MENU_TIMEOUT
FOG_TFTP_PXE_KERNEL
FOG_TFTP_PXE_KERNEL_32

I suspect something between 1-12 is actually missing, causing the MENU_TIMEOUT value to to be put in place of the MENU_HIDDEN value (timeout is usually defaulted to 3 - 5 seconds) which is triggering the boot menu to think it is a hidden item.

Tom Elliott

@Rebelman I’m noticing here, that the bzImage is not being generated (specifically it should be trying to get a bzImage32 but it’s blank, leading me more suspicion to my prior posting.)

Rebelman

@Tom-Elliott
So I’m getting an error when typing that in on my FOG database -
ERROR 1146 (42S02): Table ‘fog.global’ doesn’t exist

Tom Elliott

@Rebelman I updated the query, sorry there shouldn’t have been a space between global and Settings. It should’ve appeared as globalSettings

Rebelman

@Tom-Elliott

mysql> SELECT settingKey,settingValue FROM globalSettings WHERE settingKey IN ('FOG_EFI_BOOT_EXIT_TYPE','FOG_KERNEL_ARGS','FOG_KERNEL_DEBUG','FOG_KERNEL_LOGLEVEL','FOG_KERNEL_RAMDISK_SIZE','FOG_KEYMAP','FOG_KEY_SEQUENCE','FOG_MEMTEST_KERNEL','FOG_PXE_BOOT_IMAGE','FOG_PXE_BOOT_IMAGE_32','FOG_PXE_HIDDENMENU_TIMEOUT','FOG_PXE_MENU_HIDDEN','FOG_PXE_MENU_TIMEOUT','FOG_TFTP_PXE_KERNEL','FOG_TFTP_PXE_KERNEL_32') ORDER BY settingKey;
+----------------------------+--------------+
| settingKey                 | settingValue |
+----------------------------+--------------+
| FOG_EFI_BOOT_EXIT_TYPE     | refind_efi   |
| FOG_KERNEL_ARGS            |              |
| FOG_KERNEL_DEBUG           |              |
| FOG_KERNEL_LOGLEVEL        | 4            |
| FOG_KERNEL_RAMDISK_SIZE    | 127000       |
| FOG_KEY_SEQUENCE           |              |
| FOG_MEMTEST_KERNEL         | memtest.bin  |
| FOG_PXE_BOOT_IMAGE         | init.xz      |
| FOG_PXE_BOOT_IMAGE_32      | init_32.xz   |
| FOG_PXE_HIDDENMENU_TIMEOUT | 30           |
| FOG_PXE_MENU_HIDDEN        | 0            |
| FOG_PXE_MENU_TIMEOUT       | 3            |
| FOG_TFTP_PXE_KERNEL        | bzImage      |
| FOG_TFTP_PXE_KERNEL_32     | bzImage32    |
+----------------------------+--------------+
14 rows in set (0.00 sec)```

Tom Elliott

@Rebelman So you’re missing the FOG_KEYMAP value.

Please run:

INSERT INTO fog.globalSettings (settingKey,settingDesc,settingValue,settingCategory) VALUES ('FOG_KEYMAP','This setting defines the keymap used on the client boot image.','','General Settings');

You should have more success in the “hidden” menu issue, as well as no longer have the loop.

Fog Menu loop

151

11.8k

17.2k

154.9k