Unable to build FOS

[Sebastian Roth]

@mpmackenna Awesome to see someone is actually using this stuff! Though I am not sure if you actually need that for what you are trying to do. If it’s only for creating a FOS USB boot medium you can follow George’s instructions on this topic. If you need to modify the scripts within the init you wanna read this wiki article.

But to answer you question. You’d run it like this docker run -e BUILDROOT_VERSION='2018.08.2' -e KERNEL_VERSION='4.19.6' ...

I am sorry, should update the readme at some point. We don’t use the docker build much at the moment and I am not exactly sure if it still works. To build you don’t actually need docker. You should be able to build on a plain Linux system (we only tested on Debian-based and RedHat-based so far).

git clone https://github.com/fogproject/fos
cd fos
export BUILDROOT_VERSION='2018.08.2'
export KERNEL_VERSION='4.19.6'
./build.sh -a x64
...
./build.sh -a x86
...

Note that compiling the whole buildroot environment will take a fair amount of time. On our workhorse power machine it takes about an hour for each architecture. On my laptop it takes 6 hours each I’d think. Haven’t done this in a long time.

[mpmackenna]

@Sebastian-Roth I omitted all the docker stuff and started the build on my Ubuntu station. It is off and running. I will check back on it in a number of hours. I am looking forward to testing it out. I will post back once the build is complete and I get the opportunity to try it. Thank you for your help.

[Sebastian Roth]

@mpmackenna All the output is sent to a logfile. You might want to run tail -f fssourcex64/buildrootx64.log in another console so you see what’s happening. I say this because the build script is not perfect and might now stop the Tue Mar 2 02:48:58 CDT 2019 - building... output loop on the console you started building even if it stops with an error. So keeping an eye on the log is always a good idea.

[george1421]

Just for clarity are you wanting to usb boot into a FOS image, or do you want to create the inits directly (the virtual hard drive used for FOS)?

Using buildroot it sounds like you want to create the inits and not the bootable usb stick?

[mpmackenna]

@Sebastian-Roth Build appears to be running still but CPU has dropped off. When I look at the end of the log file the build seems to be stalled and showing the following output.

/bin/bash ../libtool --quiet --mode=install ../install-sh -o root -g root -m 755 xfs_db /usr/sbin
../install-sh -o root -g root -m 755 xfs_admin.sh /usr/sbin/xfs_admin
../install-sh -o root -g root -m 755 xfs_ncheck.sh /usr/sbin/xfs_ncheck
../install-sh -o root -g root -m 755 xfs_metadump.sh /usr/sbin/xfs_metadump
./install-sh -o root -g root -m 755 -d /usr/share/doc/xfsprogs
./install-sh -o root -g root -m 644 README /usr/share/doc/xfsprogs
>>> partclone 0.2.89 Downloading
--2019-03-08 09:46:43--  http://partclone.nchc.org.tw/download/stable/old/0.2.89/partclone_0.2.89.orig.tar.gz
Resolving partclone.nchc.org.tw (partclone.nchc.org.tw)... 140.110.240.54
Connecting to partclone.nchc.org.tw (partclone.nchc.org.tw)|140.110.240.54|:80... failed: Connection timed out.
Retrying.

--2019-03-08 09:48:53--  (try: 2)  http://partclone.nchc.org.tw/download/stable/old/0.2.89/partclone_0.2.89.orig.tar.gz
Connecting to partclone.nchc.org.tw (partclone.nchc.org.tw)|140.110.240.54|:80... failed: Connection timed out.
Retrying.

--2019-03-08 09:51:05--  (try: 3)  http://partclone.nchc.org.tw/download/stable/old/0.2.89/partclone_0.2.89.orig.tar.gz
Connecting to partclone.nchc.org.tw (partclone.nchc.org.tw)|140.110.240.54|:80... 

Thank you.

[mpmackenna]

@george1421 I think I would like to do both. If I could add the init.xz as a pxe boot option to boot to a live FOS install that would be pretty sweet. I would also like to make a bootable FOS USB stick for debug as well. If you already have a valid img for use with RUFUS I will happily take a download of that. I hope I am understanding the questions correctly. My current issue is related to video going black when I boot a Lenovo L380 but I am interested in the tools for various uses going forward. Thank you for your help.

[george1421]

@mpmackenna I think we are still talking about two different things here. This is what I’m referring to: https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image This process allows you to bypass the pxe/ipxe boot process and to launch the FOS (linux OS) directly from a usb stick. There are some uefi firmwares (especially on lenovo) that are buggy and don’t pxe boot well. So I came up with this unsupported process as outlined in the tutorial.

[Sebastian Roth]

@mpmackenna Definitely read through what George posted - same as I mentioned in my first answer. Might save you a lot of time.

About the build issue? Have you tried to manually download the source archive? http://partclone.nchc.org.tw/download/stable/old/0.2.89/partclone_0.2.89.orig.tar.gz
For me this link works perfectly fine!

[mpmackenna]

@Sebastian-Roth The issue with the link was by default my connection is not allowing traffic to Taiwan. I made an exception and was able to download from the link in the build process. The build is running now. Thank you.

[mpmackenna]

@george1421 Thanks so much for the bootable FOS image. I was able to boot to the debug option on the FOS stick and get an interactive shell in the FOS environment. Networking is working too. Which makes me wonder why the bootable USB is working on this drive but when I choose the debug option on the same model after PXE booting I see bzimage and init.xz download and then I just get a black screen. The only difference I noticed between the two environments is that my FOG installation lists the bzimage kernel as 4.19.1. When I run

uname -a

in the debug environment from your USB boot media it lists kernel version as 4.19.6. Do you have a recommendation on how I can further troubleshoot this issue? Are there changes to the kernel between 4.19.1 and 4.19.6 that contain items that might be making the difference for this host?
Thanks again for the bootable FOS image and all your help!

[mpmackenna]

@Sebastian-Roth Your build instructions work! I am the proud owner of a freshly built bzImage! Thank you.

Setup is 16124 bytes (padded to 16384 bytes).
System is 8169 kB
CRC 8c40e932
Kernel: arch/x86/boot/bzImage is ready  (#1)
mike@VLIN-018987 11:57:46 0 :~/fos/fos[master] $ find . -name bzImage
./kernelsourcex64/arch/x86/boot/bzImage
./kernelsourcex64/arch/x86_64/boot/bzImage
./dist/bzImage
mike@VLIN-018987 11:58:18 0 :~/fos/fos[master] $

[george1421]

@mpmackenna said in Unable to build FOS:

after PXE booting I see bzimage and init.xz download and then I just get a black screen.

We have seen this a few times before. What is happening is the hand-off from the iPXE kernel to uefi and the on to bzImage (FOS) is not working correctly. This is often because of bugs in the uefi firmware. One suggestion is to make sure the firmware is updated on your target computer. We have seen quite a few early in a model production cycle bios with problems that have been later fixed with updates.

You can also update the FOS usb image with the bzImage and inits from your fog server. That will put the boot usb on the same release as your FOG server.

[Sebastian Roth]

@mpmackenna Which model of PC do you have by the way?

Just to add to what George already said: Now that you know how to compile your own kernel you can start debugging the hang on PXE booting. Start by enabling so called “early printk” to get early kernel debug output:

cd kernelsourcex64
make menuconfig

Select Kernel hacking -> enable “Early printk” and “Early printk via the EFI framebuffer” just below that. Then Exit, Exit, Save yes

Now to get some more debug output I suggest adding a line to eboot.c. As far as I have seen this is one of the most common places where things go wrong with newish hardware. So edit arch/x86/boot/compressed/eboot.c and find function efi_main(...) (around line 730). Now 15 lines down that function you should see sys_table = _table;. Add a print statement just after that line:

...
        sys_table = _table;
        efi_printk(sys_table, "################## Hello World! ##################\n");
...

Save and exit and build a new bzimage.

make bzImage

No need to run the full ./build.sh ... again for this because re-compiling only the kernel here will go much quicker.

[mpmackenna]

@Sebastian-Roth I completed the changes to the kernel build as you suggested. I copied over the new bzImage file. I booted the machine over iPXE and chose “Debug Mode”. I am able to verify that the newly compiled kernel is active as the print statement that was defined is being printed to the screen. Unfortunately, that is the only output I receive. The screen just hangs with the print statement displayed and never progresses past that statement. The station Model is a Lenovo ThinkPad L380, here is some information from Lenovo on the model. How can I further troubleshoot this issue? Thank you!

[Sebastian Roth]

@mpmackenna said in Unable to build FOS:

Unfortunately, that is the only output I receive. The screen just hangs with the print statement displayed and never progresses past that statement.

Well that’s a perfect start to get into debugging where exactly it hangs. Simply add more of those efi_printk further down that function and possibly as well in sub functions called by efi_main. In other models we often saw issues with PCI initialization. You’ll see that code a bit further down.