FOG menu boot loop after image deployment

[AllFoggedOut]

@george1421 said in FOG menu boot loop after image deployment:

@AllFoggedOut When you don’t hit F10 you have the default boot set to the LAN?

While this isn’t an answer to the issue, do you need unattended imaging with these NUCs? If not why not just change the boot order to boot to the local hard drive and then press F10 if you need to reimage them.?

Yes, the default boot option is LAN. Yes, unfortunately, I do need unattended imaging

[george1421]

@AllFoggedOut Did you try an exit mode using rEFInd (yes I know its intended for EFI systems, but the docs also say it supports bios based systems too)? Its a long shot but it might find the boot disk. You need to change this in the host definition and for the bios exit mode.

The sanboot should use the first hard drive reported to the bios, which is why I find it a bit surprising that it doesn’t work. But I can’t say its consistent for M.2 disks.

[AllFoggedOut]

@george1421 said in FOG menu boot loop after image deployment:

@AllFoggedOut Did you try an exit mode using rEFInd (yes I know its intended for EFI systems, but the docs also say it supports bios based systems too)? Its a long shot but it might find the boot disk. You need to change this in the host definition and for the bios exit mode.

I haven’t looked at rEFInd yet - I’ll take a look and see what’s involved.

The sanboot should use the first hard drive reported to the bios, which is why I find it a bit surprising that it doesn’t work. But I can’t say its consistent for M.2 disks.

Yeah, it’s odd to say the least.

[AllFoggedOut]

I’ve set “Host Bios Exit Type” to “REFIND_EFI” under the host definition. I’ve edited “packages/web/service/ipxe/refind.conf” to have “scanfor” set to “hdbios”. All I seem to be getting is a blank screen with a flashing cursor in the top left.

I also set “scanfor” to “manual”, and defined a manual stanza at the bottom of refind.conf. Also bumped timeout to 10. Same behaviour.

Same behaviour with SATA enabled/disabled, pressing/not pressing F10 (as described previously),

Do I need to restart something for changes to take effect?

[Tom Elliott]

@AllFoggedOut This sounds an awful lot two possibilities causing the issue you’re seeing.

1. The image was pushed up and the disk was detected as GPT even though you have it setup as MBR in windows. This, by itself, shouldn’t cause any major issues but the deploy back to disk might. Though it would be ultimately better to fix the partition layout for the capture, you can fix this for deploy by using postdownloadscripts. Essentially, if you can boot the system into a FOS Debug mode and run fixparts <devicename> I suspect you’ll see it asking to fix the partition table. If it is, confirm and save, cancel your created tasking and reboot the system that booted up.
2. (Least likely if image was captured relatively recently) The MBR is not setting the partition boot partition in a “bootable” state.

Just my thoughts.

[Tom Elliott]

In a mild attempt to check if the GPT partition thing I’m suspecting is the issue, I may have found a way to properly fix the “hung disk” issue that once was. This same fix should work for upload/capture etc…

The reason things weren’t getting hung is because we were piping yes into the gdisk command. This meant imaging would continue going and all would be more or less fine. (The data is actually copied or placed back on the disk.) But it could also leave the system in a strange state (for example the system being unable to boot after being deployed to.)

I’ve updated the init’s in hopes to try out a method to verify the partition table first. If the partition table is invalid, try to run fixparts on the disk.

The source has been updated within the working-1.3.2 branch, and the development init’s have been updated to contain this new test.

If the debug->fixparts method works to make the system bootable would you mind seeing if the development init’s are working for you too?
If so, please try downloading the dev init’s on your system and seeing if your systems are working after a deploy. I don’t know IF they will work and I have no means to replicate the problem currently.

wget --no-check-certificate -O /var/www/fog/service/ipxe/init.xz https://fogproject.org/inits/init.xz
wget --no-check-certificate -O /var/www/fog/service/ipxe/init_32.xz https://fogproject.org/inits/init_32.xz

[AllFoggedOut]

@Tom-Elliott said in FOG menu boot loop after image deployment:

1. The image was pushed up and the disk was detected as GPT even though you have it setup as MBR in windows. This, by itself, shouldn’t cause any major issues but the deploy back to disk might. Though it would be ultimately better to fix the partition layout for the capture, you can fix this for deploy by using postdownloadscripts. Essentially, if you can boot the system into a FOS Debug mode and run fixparts <devicename> I suspect you’ll see it asking to fix the partition table. If it is, confirm and save, cancel your created tasking and reboot the system that booted up.

In FOG Debug, I run ‘fixparts /dev/nvme0n1’, and I get ‘MBR command (? for help)’. No sign of an error state or request to fix the partition table.

2. (Least likely if image was captured relatively recently) The MBR is not setting the partition boot partition in a “bootable” state.

If I print out the existing MBR Partition Table via ‘p’ command, I get 2 partitions, the first is set with the boot flag; all looks healthy.

Just my thoughts.

I appreciate your thoughts!

[AllFoggedOut]

@Tom-Elliott said in FOG menu boot loop after image deployment:

If the debug->fixparts method works to make the system bootable would you mind seeing if the development init’s are working for you too?
If so, please try downloading the dev init’s on your system and seeing if your systems are working after a deploy. I don’t know IF they will work and I have no means to replicate the problem currently.

FWIW, these work. I was able to capture and deploy my Win7x64 image w/o any obvious errors. Windows boots when I select the NVMe drive in F10 Boot Menu. Unfortunately, my original problem persists.

I’m going to install/clone/deploy Win10 in EFI mode and see how that goes.

[AllFoggedOut]

Win10x64 unattended capture/deployment works fine using rEFInd (SANBOOT does not work - same issue as with Win7x64 above).

Initially I had rEFInd complaining about my scanfor line containing legacy BIOS options which were incompatible since my BIOS lacked the necessary Compatibility Support Module. Removing hdbios from the list made the error message go away. Had a minor bit of confusion initially because I was editing the wrong refind.conf - correct path is ‘/var/www/html/fog/service/ipxe/refind.conf’.

I might try rEFInd again for my Win7x64 issue (now that I know which file to edit), but I don’t think it’s going to help - the fact that I got a blank screen and flashing cursor vs a rEFInd menu + error message under EFI suggests it’s not happy.

During the course of testing I had to move my image storage directory owing to a lack of space. I moved all files under /images to another LVM volume group. I then updated the path in Storage Management. I then ran into an issue post-image capture (repeated “Database Update failed” messages) which was resolved by changing permissions on the new folder (and sub-folders to fog:fog and mode 775). Not sure if this is correct, but it worked. Apache error_log showed the FTP rename operation failing. I then had a 2nd issue during image deployment where “Checking Mounted File System” failed. Seems the script “src/buildroot/package/fog/scripts/bin/fog.checkmount” still contained the old storage path (possible bug?). I also must have missed the “.mntcheck” file when moving files around - had to recreate it.

Anyway, that aside, Win10x64 clone/deploy is now working seamlessly.

[Tom Elliott]

@AllFoggedOut as for the ‘possible bug’ you are not experiencing a bug for that. If the .mntcheck file is missing it will fail because it has no way else to know whether the system mounted or not. We use the .mntcheck file to determine if this is the case or not.

[george1421]

@AllFoggedOut OK, I’m a bit confused now.

It was my understanding that imaging worked perfectly, but where the issue was when you exited from the FOG iPXE menu to boot the local host OS. The target computer would not boot the local OS, but if you changed the bios boot order to the disk first (instead of PXE) the system would boot properly.

Shouldn’t this should be an iPXE kernel exit mode issue??

[Tom Elliott]

@george1421 it is, the OP is just stating what was tested that appears to be working. During those tests they ran into a full disk and added more space. When adding more space they forgot to setup the permissions and mntcheck files.

[AllFoggedOut]

My “possible bug” comment related to the fact that, after changing “Image Path” and “FTP Path” in Storage Management -> DefaultMember, to a new path, the FOG script file “src/buildroot/package/fog/scripts/bin/fog.checkmount” still contained the old path, which caused image deployment to fail at the “Checking Mounted File System” stage. The reason it failed is because I deleted the old, defunct path from disk.

@george1421, yes, this is still an iPXE exit mode issue.

In summary, Win7x64 Legacy == iPXE Exit Mode loop. Win10x64 EFI == no issues.

[Tom Elliott]

@AllFoggedOut Unfortunately, once the client is loaded into FOS, the only way to change the kernel based variables is to reboot the client. You could make a call to get new data, but that would/could put tremendous load back on the server (imagine 20 hosts booting to perform an imaging task, all calling out to the main server to update their scripts).

It’s simpler just to reboot the machine (or machines as the case may be) to pick up the new data.

[Omar.rodriguez]

This post is deleted!

[AllFoggedOut]

@Omar.rodriguez said in FOG menu boot loop after image deployment:

I understand I might be late to this thread… but we’ve ran into this issue here at the office I work at. Do you happen to have 2 hard drives on your computer?

The NUC contains a single M.2 SSD

When you get the FOG menu on the PC and you press the “Esc” key does it boot into the OS?

No, I get an error message about chainloading failed, and an auto reboot in 10 seconds. This is for Win7x64 using SANBOOT.

[AllFoggedOut]

@Tom-Elliott said in FOG menu boot loop after image deployment:

It’s simpler just to reboot the machine (or machines as the case may be) to pick up the new data.

The script I’m referring to is/was on the server.

[Tom Elliott]

@AllFoggedOut You’re starting to confuse me.

The “bugs” you were referring to were in direct relation to the imaging things. What I’m trying to state is the “bugs” you saw were in regards to the FOS system. This has nothing to do with what you’re seeing in regards to the boot menu loop.

The only reason you saw issues in imaging earlier is because you made a change somewhere. These issues should no longer be present since you’ve made the corrective actions.

[Tom Elliott]

@AllFoggedOut said in FOG menu boot loop after image deployment:

During the course of testing I had to move my image storage directory owing to a lack of space. I moved all files under /images to another LVM volume group. I then updated the path in Storage Management. I then ran into an issue post-image capture (repeated “Database Update failed” messages) which was resolved by changing permissions on the new folder (and sub-folders to fog:fog and mode 775). Not sure if this is correct, but it worked. Apache error_log showed the FTP rename operation failing. I then had a 2nd issue during image deployment where “Checking Mounted File System” failed. Seems the script “src/buildroot/package/fog/scripts/bin/fog.checkmount” still contained the old storage path (possible bug?). I also must have missed the “.mntcheck” file when moving files around - had to recreate it.

This is what I’m referring to.

Your first issue is because permissions, once that was done the next issue is because the .mntcheck file didn’t exist. Now that those issues are corrected (these are both within the init’s btw) the bugs you thought you were seeing will no longer be there.

[AllFoggedOut]

Ok, I’ll try and explain this from a different angle.

I have an issue with iPXE Exit Mode loop that is affecting Win7x64/Legacy BIOS.

In the process of trying to diagnose this issue, I decided to install Windows 10/EFI to see if this also suffered from the same problem.

During the cloning of Windows 10 I realised I was running out of disk space in “/images” on the FOG server.

I aborted the clone, cancelled the imaging task and set about creating a folder for use by FOG to store images.

I moved everything under “/images” to “/stor/fog”. Via the FOG web interface, I went into Storage Management -> DefaultMember, and changed “Image Path” and “FTP Path” to the new folder.

I then recreated the imaging task which failed because I had the wrong permissions on my folder (my fault, not a bug).

When I then attempted to deploy the same image, it failed because the FOG script “fog.checkmount” still contained the OLD storage path. Why did this script still contain the old path? Shouldn’t it have been updated to the new path when I updated DefaultMember?

I also failed to copy “.mntcheck” from /images (my fault, not a bug).

I also deleted “/images” from the server. If I had left this folder in place, I’m guessing the “Checking Mounted File System” would have just continued to work, despite pointing to a defunct/no longer used location.

Now that I’m thinking about it, the NFS exports were also not updated. I had to manually edit /etc/exports, put the new folder in and run exportfs -a.

My assumption was that updating DefaultMember would mean:

a) NFS exports is updated with the new folder
b) fog.checkmount is updated with the new folder

Perhaps the preferred/supported approach is to simply add new storage areas, rather that editing the existing ones.

None of the above has any relation to my iPXE Boot Menu issue with Win7x64/Legacy. It is a coincidental observation that I thought I’d bring to your attention in case it’s a bug with updating Storage Management paths.