USB Boot target device into FOG OS Live (FOSL) for debugging

[Wayne Workman]

Going in the #wiki when complete.

[Sebastian Roth]

@george1421 This is a great idea!! (Wonder why I haven’t come up with this already )

Reading through your posts I thought about building a hybrid ISO file that people could dump onto their USB sticks. Might be a little easier for most users. But it’s not perfect as people still need to use tools like dd/unetbootin/… What do you think? At least it would be a little more straight forward instead of using grub here and syslinux in another case.

Edit: In the grub.cfg the kernel parameter “initrd=init.xz” is missing. Is this still working? Maybe this is only needed with iPXE but I remember we had trouble if we didn’t add this parameter.

[george1421]

@Sebastian-Roth said:

Edit: In the grub.cfg the kernel parameter “initrd=init.xz” is missing. Is this still working? Maybe this is only needed with iPXE but I remember we had trouble if we didn’t add this parameter.

Yes its working no problem. Actually I had the opposite luck when I included it in-line. When I did that I got the kernel panic about unable to mount root fs block (1,0). As seen here. https://forums.fogproject.org/topic/6496/hp-compaq-elite-8300-don-t-boot/14
Once I moved it out of the kernel line and into its own parameter then it booted correctly.

I’m working on a multi boot version that should work for both bios and efi booting. (what I’ve done thus far only works for bios). I think when that is done and working. The easiest thing for users would be for the devs in the fog program to either use dd or some process to capture a functioning usb and make that available for download for testing (or provide the instructions).

I went this route to completely eliminate the whole pxe part and the chaining and such. If the kernel won’t boot off usb flash, its simply not going to work from a production server. It would allow quick changes of the kernel since all you need to do is replace the OS files on the flash drive and you are running the latest FOG Client OS. Right now I’m not seeing a downside for target debugging. <…> Just thinking a bit, if you needed additional debugging tools, the devs could do something like how the linux router project [LRP] does and build tar packages and overlay the tar packages on the root file system to add additional support files. Because really we can change the size of the virtual hard drive to accommodate these additional (layered on) files by just increasing the size in the grub cfg file. This is only a debug kernel not intended for production.

[Edit] thinking a bit maybe and iso image would work just as well. You would still need unetbootin or dd as you noted to copy the iso to flash media if the user wanted to go that way. [/Edit]

[Sebastian Roth]

@george1421 said:

The easiest thing for users would be for the devs in the fog program to either use dd or some process to capture a functioning usb and make that available for download for testing.

That’s exactly the point why I thought about an ISO image. Sure you can take an image (using dd or other tools) from a good prepared USB stick but to me that seams a bit like putting the cart before the horse (please don’t get me wrong, not meaning to be offensive here!)

Generating an ISO that is multiboot on BIOS and EFI (32bit and 64bit!) is possible without much effort. So Tom can add this step to his kernel compile/upload toolchain and we will have current kernel/init.xz and bootable ISO file ready to download for users all the time.
Hugh advantage would be that we only need one “simple” description of how to dump this ISO onto USB sticks. As well a generated ISO is just big enough to take up all the things needed. Whereas an image file (dd or what) is usually created with a fixed size. Sure you can grow/shrink it to suit your needs but it adds some unnecessary inconvenience to it.

Thanks heaps for all the effort you are putting into this and all the tutorials you wrote. I am sure users will find this really helpful even if we end up providing an easy to handle ready to go ISO file!

I’ve played with syslinux (http://www.syslinux.org/wiki/index.php/Isohybrid) for about two hours just to find out that its UEFI support is still kind of young. Does not mean it is impossible to achieve but I found a way easier solution using GRUB2. To generate a FOG debug ISO I downloaded the kernel/init files, wrote a grub.cfg like you have, put it all in a local directory called ISO_root and called grub-mkrescue -o fog_debug.iso ISO_root/. This is on Debian Jessie. Out comes a ready to go ISO file which I can boot in qemu! Testing it on USB Stick now…

Edit: Sorry, I was wrong. This simple command only generates an ISO usable on BIOS machines - at least on my BIOS machine right now. Might take some extra effort to generate BIOS/UEFI ISOs or just the right packages installed. Reading on…

Edit2: Needed to install grub-efi-amd64-bin and grub-efi-ia32-bin to have grub-mkrescure build a multiboot ISO. Here is a first try which also tries to detect 32/64bit Arch (grub cmd cpuid): https://drive.google.com/folderview?id=0B-bOeHjoUmyMM2c4LV90Z1NyNWc&usp=sharing (wow, 60 MB just for a plain BIOS/EFI 32/64bit debug ISO… may I ask you guys to give this a try on various platforms? Just dump it on USB stick using dd/win32 disk imager/unetbootin etc.)

[george1421]

@Sebastian-Roth No offense taken here. In the very first iteration of this document I did say that I didn’t have a use case for doing this. It was more about could I do it vs I need to do this. The idea was to provide a tool were we could boot the fog kernel on new and emerging hardware to be able to collect information necessary to help with the debugging process much like the live boot recommendation. If the fog devs can create a tool with the precise utilities or data collection bits needed to better support fog development then that is a plus too.

I was surprised how small the actual functioning usb is and it actually works. Looking at the next steps the iso image IS the right decision as long as multiboot works as needed. As you noted you can write the iso image to flash, cdrom, or even boot with a virtual machine (not sure of the value with this). With only 60MB being used you have plenty of room to install any missing (i.e. not required for fog capture or deployment) utilities that would make the debugging process easier.

I think a perfect use/test case for your image is for these new UEFI based tablets that seem to be giving the current fog client OS fits. The only problem is if the fog kernel fails to boot via the iso image, how do we fix it?

The one thing that is missing (I think) is once you have your usb flash drive, is a way to refresh the kernels as new kernels are released over time. It could just be a simple update script. When I booted my flash drive (created with my process) the flash drive did not show up using lsblk. At the time it wasn’t important, but if I need to update the kernels I’ll have to do it through a different process.

[Wayne Workman]

I think having a troubleshooting ISO is an amazing idea.

Just be sure to include “troubleshoot” in the name / menus so that people don’t get the wrong idea and think this is the way to go for every time they wanna use fog.

[Sebastian Roth]

@george1421 said:

The one thing that is missing (I think) is once you have your usb flash drive, is a way to refresh the kernels as new kernels are released over time.

I really hope that we can get this streamlined into Tom’s compile tool chain. So if he builds new kernels and inits and uploads those to the website he could also add a current ISO to https://fogproject.org/isos for example. Self updating process within the ISO is not worth the scripting effort I reckon as we will have lots of trouble with NIC issues and people behind proxies. So if we update the kernel people just need to download the most current ISO again.

@Wayne-Workman said:

Just be sure to include “troubleshoot” in the name / menus so that people don’t get the wrong idea and think this is the way to go for every time they wanna use fog.

Sure we will! But this is still a very early stage and I am not sure how well this will work. Would you mind giving it a try? Just download the ISO, dump it onto an empty stick and try booting it on various machines/architectures. I don’t have access to different machines right now as I am not working at university anymore…

[Wayne Workman]

@Sebastian-Roth I will, just bumping the link for now: https://drive.google.com/folderview?id=0B-bOeHjoUmyMM2c4LV90Z1NyNWc&usp=sharing

[george1421]

@Sebastian-Roth I’ll run into the office and grab one of the newer dells to test the efi booting. I have to run into the office anyways since I forgot my travel computer bag on Friday. I won’t be back in the office until Wed, so I may not have a chance to test it if I don’t have a viable system.

[george1421]

Critical review of FOG Debug booting.

Geo image:
This image booted on crusty Lenovo T43 and e6400 in 32 bit bios mode no problem
Booted OK on e6220 in bios mode both 32 and 64 bit
Booted with error on e6220 in EFI mode with both 32 and 64 bit with error “No C<something> No suitable video mode found. Booting in blind mode” The <something> part was due to sloppy hand writing. Since there was no console I can’t tell if the system actually booted or not. I’m suspecting that it was running since it didn’t panic anywhere. There probably needs a kernel parameter passed since the font was really tiny like a 132x56 console resolution, where bios mode had traditional 80x25 resolution.

ISO image:
Used unetbootin to “burn” iso image to 16GB usb drive.

Issue observed, unetbootin creates an entry called default with a 10 second timeout.

1. This default value selected the x64 bit kernel. This fails if the booted device has a 32 bit CPU. I assume it is picking this up from the grub menu. Suggestion make the 32bit kernel default or users will see you have the wrong processor try again message.
2. The default option (on a computer with 64 bit processor causes a kernel panic “unable to mount root filesystem”

Dell e6220
BIOS mode:
Default: Resulted in kernel panic
x64 booted to command prompt
x32 booted to command prompt
EFI mode:
Selecting the usb flash drive from the efi boot menu caused a pause then started booting from hard drive. I never got to the grub menu.

Lenovo T43
BIOS Mode (only)
Default resulted in kernel panic
32 bit booted to command prompt
64 (not supported)

Dell e6400
Bios Mode (only)
Default resulted in kernel panic
32 bit booted to command prompt
64 (not supported)

I looked at the syslinux.cfg that unetbootin created. And I can see that the 64 bit kernel does have the lead over the 32 bit kernel. I suggest that this order be changed in the grub menu to make the 32 bit kernel default. I’m not sure what happened. But as you see unetbootin copied the 64 bit bzimage and renamed it to ubnkern and copied init.xz to ubninit. The copied bzImage has the right size, but the copied init.xz is only 2.8MB in size.

label unetbootindefault
menu label Default
kernel /ubnkern
append initrd=/ubninit loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 pcie_aspm=off consoleblank=0 isdebug=yes

label ubnentry0
menu label FOG 64-bit Debug Kernel
kernel /boot/bzImage
append initrd=/boot/init.xz loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 pcie_aspm=off consoleblank=0 isdebug=yes

[george1421]

Critial review FOG Debug booting

ISO image:
Burned iso image to flash drive using: dd if=multi.iso of=/dev/sdb

BIOS booting on e6220
Invalid partition table! and will sit there until enter is pressed. Then GRUB menu is displayed but only option is x64 kernel. x64 kernel does boot to command prompt

UEFI booting on e6220
Grub menu not displayed. Error message displayed: “Error: disk `’ not found. Entering rescue mode… grub>”

When I reinsert the flash drive into my computer, ubuntu could not mount it. I assume I did something wrong with either the format or the way dd was used.

For reference this is the structure of the flash drive

sdb      8:32   1    15G  0 disk 
├─sdb1   8:33   1   136K  0 part 
├─sdb2   8:34   1   2.8M  0 part 
└─sdb3   8:35   1  56.4M  0 part 

In preparation for rebuilding my direct path, when I ran fdisk it reported that the usb flash drive was in GPT mode and it only reported one partition in fdisk. I did see the procedure to go the gpt route but I elected to go the vfat route which only required one partition with MBR. Just saying not implying anything.

[george1421]

Working back with my image. For EFI booting when I boot with grub2 in efi mode I get the following error message.

Error: no suitable video mode found.
Booting in blind mode

After a little research I found I needed to add this to the grub menu then the fog kernel would boot in EFI mode.
insmod efi_gop
insmod efi_uga
insmod font

So I updated the grub.cfg menu to this:

set timeout=10
set default=0


menuentry "FOG 32-bit Debug Kernel" {
 linux  /boot/bzImage32 loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 pcie_aspm=off consoleblank=0 isdebug=yes
 initrd /boot/init_32.xz
}

menuentry "FOG 64-bit Debug Kernel" {
 insmod efi_gop
 insmod efi_uga
 insmod font

 linux  /boot/bzImage loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 pcie_aspm=off consoleblank=0 isdebug=yes
 initrd /boot/init.xz
}

If I picked the 64 bit kernel it would boot without issue. If I selected the 32 bit kernel it would get the error message.

Now the thing I don’t know, is this error message because of FOG or is it because of something is missing in the kernel itself that grub is inserting? Would this explain why some devices which boot in efi get a black screen?

As a side note, if I leave those settings in place and boot in BIOS mode and select the 64 bit kernel I get warning messages and have to press enter to start booting the kernel, so this is not a viable option for multi booting.

[Sebastian Roth]

Awesome stuff! We are making progress here I reckon. Some of my findings:

• need to add --modules=part_gpt to the grub-mkrescue call to make it work with UEFI machines (don’t ask me why!) - updated the
• automatic arch selection is not a great idea - using two menu entries like George is better I find
• instead of grub modules efi_gop and efi_uga we probably can use all_video which is handled a lot better if we are using a multiboot (bios+uefi) ISO
• don’t use unetbootin to “burn” the ISO onto a USB stick as it is altering the bootloader (fail!) - use dd, rawwrite, win32diskimager or any other simple tool that does a sector by sector copy (you might need to rename .iso to .img)

Uploaded a new ISO image called debug_v2.iso: https://drive.google.com/folderview?id=0B-bOeHjoUmyMM2c4LV90Z1NyNWc&usp=sharing

[george1421]

I started testing the debug_v2.iso

To move the iso image to my usb flash drive I used the dd command.

sudo dd bs=4M if=debug_v2.iso of=/dev/sdb

The copy worked as it should. I ejected the usb drive and reinserted it into my ubuntu laptop. This error was thrown when I inserted it and the laptop tried to mount it.

Error mounting /dev/sdb3 at /media/jondoe/ISOIMAGE1: Command-line `mount -t "hfsplus" -o "uhelper=udisks2,nodev,nosuid" "/dev/sdb3" "/media/jondoe/ISOIMAGE1"' exited with non-zero exit status 32: mount: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail  or so

I’m suspecting this is related to the laptop being in bios mode and using mbr disks. I’m not really concerned about it. Just noting that once you create the usb flash drive its not mountable. I don’t have a efi linux system to confirm if it is a bios vs efi vs gpt issue.

The usb booted correctly in efi mode on an e6220 laptop. It also booted correctly in bios mode. I did note something strange. If I booted in 32bit efi mode, issue a reboot at the command prompt and then rebooted in 64 bit efi mode they kernel would throw an error “Fixing recursion fault but reboot is needed” it waits about 10 seconds then does something similar to a core dump all over the screen. The only seems to be an issue if you switch between 32 bit and 64 bit in efi mode. Rebooting multiple times in 32 bit efi or 64 bit efi works as it should.

Booting in bios mode did thow an error about invalid partition table. I’m not sure if that was an issue with my flash drive from before or not. I’m going to try to fix my flash drive again to make a clean boot then reapply the iso image.

[george1421]

Method #4 Create automated multiboot

I revived / revised my attempt at making an automated fog debug kernel boot image.

This image in contrast to what grub-mkrescue makes, creates a vFAT image that can be mounted by both gpt and mbr systems. The issue I had with the grub-mkrescue flash image is that I could not mount it on my BIOS ubuntu computer or my windows XP computer (not a big loss).

The following script creates a complete boot image that can be moved to a flash drive with dd, win32diskimager or any other disk image tool (not unetbootin).

File: mk.fogdebugboot

#!/bin/bash

if [ -f /tmp/fogkern.img ]; then
    echo Nuking old FOG Debug image
    rm -f /tmp/fogkern.img
fi

echo Make a blank 128MB disk image
dd if=/dev/zero of=/tmp/fogkern.img bs=1M count=128
 
echo Make the partition table, partition and set it bootable.
parted --script /tmp/fogkern.img mklabel msdos mkpart p fat32 1 128 set 1 boot on
 
echo Map the partitions from the image file
kpartx -a /tmp/fogkern.img
 
# sleep 2 seconds, wait for kpartx to create the device nodes
sleep 2
 
echo Make an vfat filesystem on the first partition.
mkfs -t vfat -n GRUB /dev/mapper/loop0p1
 
echo Mount the filesystem via loopback
mount /dev/mapper/loop0p1 /mnt

echo Install GRUB
grub-install --removable --no-nvram --no-uefi-secure-boot --efi-directory=/mnt --boot-directory=/mnt/boot --target=i386-efi
grub-install --removable --no-nvram --no-uefi-secure-boot --efi-directory=/mnt --boot-directory=/mnt/boot --target=x86_64-efi
grub-install --removable --no-floppy --boot-directory=/mnt/boot --target=i386-pc /dev/loop0

echo Download the FOG kernels and inits
wget -P /mnt/boot/ https://fogproject.org/inits/init.xz
wget -P /mnt/boot/ https://fogproject.org/inits/init_32.xz
wget -P /mnt/boot/ https://fogproject.org/kernels/bzImage
wget -P /mnt/boot/ https://fogproject.org/kernels/bzImage32

echo Create the grub configuration file
cat > /mnt/boot/grub/grub.cfg << EOF

set timeout=10
set default=0
insmod all_video

menuentry "FOG 32-bit Debug Kernel" {
 linux  /boot/bzImage32 loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 pcie_aspm=off consoleblank=0 isdebug=yes
 initrd /boot/init_32.xz
}

menuentry "FOG 64-bit Debug Kernel" {
 linux  /boot/bzImage loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 pcie_aspm=off consoleblank=0 isdebug=yes
 initrd /boot/init.xz
}

EOF
 
echo Unmount the loopback
umount /mnt
 
echo Unmap the image
kpartx -d /tmp/fogkern.img
 
# Write the file to flash drive
# sudo dd bs=1M if=/tmp/fogkern.img of=/dev/sdX

[Sebastian Roth]

@george1421 said:

Booting in bios mode did thow an error about invalid partition table. I’m not sure if that was an issue with my flash drive from before or not.

The error is because on the third partition there is stuff to boot this ISO on Mac OS X. Looks like it is trying to mount it as hfsplus but fails. Not sure why that is.

PS: Great script! Maybe you want to download files first and do a rough size calculation before creating the image file. For example:

...
size=$((`du -lsm KERNEL_DIR | cut -f1` + 4))
dd if=/dev/zero of=/tmp/fogkern.img bs=1M count=$size
parted --script /tmp/fogkern.img mklabel msdos mkpart p fat32 1 $size set 1 boot on

[george1421]

@Sebastian-Roth Interesting on the boot stuff for OSX. I think any error would confuse the user beyond the need to use the debug kernel. I’m not saying its a bad thing but if we can avoid the error, the less questions we will have to answer.

It would be interesting to know if the MBR image I created would work on OSX. I think I remember reading somewhere that the disk needs to be GPT.

I can’t remember right now if I tried the grub-mkrescue image on my T43 (32 bit crusty old laptop). I’ll recreate the flash drive based on the your file and see.

As for the dynamic size, I could do that. Good point on predownloading the boot kernel files. I was just being lazy and picked 128MB knowing that was about twice the size I needed of 60MB.

[sudburr]

This post is deleted!