Custom init.gz
-
Hello all,
I know I’ve been rather busy on fog these days, but I’ve just been trying to come up with things to help the whole project out.
I am, by no means, a developer, but I am a very avid computer enthusiast who likes to tinker around.
I’ve been trying to build my own init.gz file for use within the fog system, and I’ve been, for the most part, successful.
Only a couple of parts have confused me.Using the WIKI page, I see that we are supposed to be using buildroot2011.08 but I’ve never had success with that particular version. I’ve had, however, success with buildroot2013.05.
There’s one part, though, that keeps eluding me. That part is the pigz portion. If I use the init.gz that’s prebuilt, uploading of rec.img.000 and sys.img.000 works flawlessly. When using a my custom one, it uploads, just fine, the rec.img.000 file, but then it completes the upload.
Downloading is a whole different issue that I’ve yet to succeed with. With that aside, I’ve been trying to build a more up-to-date init system anyway as the headers currently in use are 3.0.4 and I think it’d be more suited for a more updated kernel tree.The item I have, for the developers, if you’re willing to share is how to get things to compile properly and what changes, I think, should have to be made for this all to work. Can anyone tell me what’s missing? I’ll give the link to my version of init.gz just to see if their’s anything that I can correct to get pigz to work for both rec and sys img files.
First things first, the device_table.txt file, in more recent versions of buildroot, is now located at system/device_table.txt rather than target/generic/device_table.txt
Also, the custom path for the fs is now:
system/skeleton rather than fs/skeletonI’m still unsure about pigz, but many of the customize scripts need the last line changed from:
[COLOR=#000000]$(eval $(call AUTOTARGETS,package/customize,parted))[/COLOR]
[COLOR=#000000]to:[/COLOR]
[COLOR=#000000]$(eval $(autotools-package))[/COLOR]
I don’t know what else to share from here. At least I have, for the most part, a working init.gz, though I’m unsure of how to build custom packages for buildroot. Maybe somebody can help me?
I know that partclone can do many things but depends upon the utilities and tools to operate, so we’d have to include them. At the very least, we’d have to include ntfsprogs, e2fsprogs, and maybe btrfs-tools and possibly more but I don’t know what I’m doing totally. I’ll try to tinker and play, but I can only do so much alone. -
Ran into a few issues. First, the kernel headers needs a few lines input into the sysinfo.h file. You can download the kernel and unpack it. Then do a search for sysinfo.h, there are multiple, but do the one in the ./include/uapi/linux/sysinfo.h folder of the kernel-dir. You’ll need to add, near the top:
#ifndef __kernel_long_t
typedef long __kernel_long_t;
typedef unsigned long __kernel_ulong_t;
#endif
below the define _LINUX_SYSINFO_H part.
Once done, re-tar it backinto linux-<kernelver>.tar.bz2
In selectcore, make a change to:
package/customize/selectcore/selectcore.mk file on line:
27 from
$(CONFIG_UPDATE) $(SELECTCORE_DIR)/build-aux TO:
$(call CONFIG_UPDATE,$$(SELECTCORE_DIR)/build-aux) -
I’ll post my config file once I get it all working, but more, or less, it’s the same as it was, with the changes from fs/skeleton to system/skeleton and with target/generic/device_table.txt to system/device_table.txt
I’m also attempting to remove ntfsprogsfog and parted as that is built into to system naturally. Partclone is added, but I don’t know if it’s compiling. It probably doesn’t help too much that fog/scripts is directly copying the files to the filesystem.
[url=“/_imported_xf_attachments/0/385_mybuildroot.txt?:”]mybuildroot.txt[/url]
-
Just for assurance,
uLibc is being built on:
0.9.33.2
gcc is being build on:
4.7.3
Kernel headers are:
3.10.9
Binutils are at 2.23.2
Busybox is still 1.21.0I don’t know what else to place.
-
One other change that needs to happen is:
in package/customize/chntpw/chntpw.mk
Change line 22 from: toolchain/patch-kernel.sh $(CHNTPW_DIR) package/customize/chntpw *.patch
TO:
support/scripts/apply-patches.sh $(CHNTPW_DIR) package/customize/chntpw *.patchAnd change any other files that may make the same reference. Of toolchain/patch-kernel.sh to support/scripts/apply-patches.sh (e.g. selectcore, pigz, dmidecode, and chntpw)
-
Alright, I had to do some, slightly heavy, modding to chntpw-source-110511.zip so I’ll upload it here in case any one else finds it useful.
As I had to mod the file, I removed the Makefile patch from the package/customize/chntpw folder as well, so I’m not trying to patch the files I already fixed to get working with buildroot2013.05
[url=“/_imported_xf_attachments/0/386_chntpw-source-110511.zip?:”]chntpw-source-110511.zip[/url]
-
My guess is the reason I’m not seeing both partitions get made is a lack of timing to register that it still has to process the sys partition.
Well, that’s not the case. I’ve added a statement to this upload process to watch when and what’s happening along with a couple of sleep statements in between. It seems to recognize the command and variables properly, just won’t run the system.
-
So I found out, today, why It isn’t creating the sys.img.000 file. It’s because, with my custom init.gz, it doesn’t recognize the /dev/sda2 partition. I don’t know why it doesn’t, but it doesn’t. I think it’s because the device_table.txt file, isn’t actually set in the config to use the static table versus the dynamic one, but for whatever reason, it only cares about it on the single disk, resizable option. I’ll keep testing to try to narrow it down. I’ve removed the references to partclone as it doesn’t build for some reason. It could be I haven’t got a clue (most likely) or because fog actually manually writes it’s partclone files to the system any way. I’m building now with static table reference.
Just for posterity, I think I need to add this here. To add the fog components to the menu system, after you copy customize to package in buildroot, don’t forget to add source “package/customize/Config.in” somewhere in package/Config.in file. I added mine at the top, right above busybox. Don’t place it outside of a menu reference, so my source line is the 2nd line in the file.
-
I’m still not having any luck with a custom init.gz file.
I’ve followed the instructions on the WIKI page, but it just doesn’t seem to work with even the direct copies from what the WIKI page states, or with my modified versions. If I fix all the minor issues with the absolute instructions given, or if I use my own, uploading an image will only copy the rec.img.000 file when I set the system to do a Single Disk, Resizable image. I’ve modified the fog script so output information and sleep at various times, at it seems, with the custom built init.gz system, after it finishes copying rec.img.000 it, seemingly, removes the ${hd}2 variable from availability, but right before it does the first partition copy, it does exist. If I use the init.gz file that FOG has already created, all works fine. I’m just trying to figure out what the main differences are.
Can anyone help?
-
I’ve tried a diff of my init.gz and the FOG built init.gz, and, really, the only main differences I’m finding are binaries that I’m using are slightly more modern. Things like pigz is 2.3, clamav is set for 0.97.8, my customized chntpw-source-110511.zip, parted-3.1, the buildroot packaged version of ntfs-3g and tools, and gptfdisk from the buildroot system. It just seems strange that a customized version of init.gz would lose a partition after partimage runs.
-
What’s up everybody?
Just wanted to post a little more info. I’m now going to give up, for now, on 2013.05 build root and try stepping back to 2011.08 just as a shot. I’m working towards building more modern software on top of it including 3.10.9 headers. This does pose a slight issue in that apparently there were all types of issue in busybox 1.18.5 and linux-2.4 where they had to create their own struct sysinfo portions to let the kernel work. This is important to know because when you’re setting up the toolchain and change header information from 3.0.X Headers to 2.6 Manual entry mode, it, from what I can tell, degrades to 2.4 specs and there’s quite a lot of editing to work on. So rather than type all of the editing I did, I’ll (once successful) post the busybox-1.18.5.tar.bz2 file with the modifications I’ve made to make this actually build.
When you can, in your buildroot directory edit the file:
output/host/usr/i386-unknown-linux-uclibc/sysroot/usr/include/sys/sysinfo.h
At the top of the file, place:
#include <sys/sysinfo.h>and
remove the lines referencing struct sysinfo.
There’s another caveat that I haven’t figure out quite yet in the error that keeps cancelling out is:
make[1]: Leaving directory `/home/buildroot-2011.08/output/build/busybox-1.18.5’
make: *** [/home/buildroot-2011.08/output/build/busybox-1.18.5/.stamp_built] Error 2
Seemingly when a part of the build is referencing libbb/lib.a
As far as I can tell, everything will continue on if you simply re-type make ARCH=i386, but it’s kind of a nuisance to have to do this many many times throughout busybox’s build period. -
So I’ve given up on my venture towards building a modernized custom init.gz. I’ll just modify the current one as needed. I know I’ve been busy with these all, but when I buildroot2013.05, it loses partitions, when I buildroot2011.08, the file’s too large. I don’t know where I’m going wrong, but I think the fog custom components (the scripts, etc, init) should just be a part of the skeleton rather than built as a package, but this would require modernized scripts as well.
-
It’s still in it’s prelimary, but I think I’ve finally created a modernized init.gz file that works. The only thing I’ve not tinkered with so far is lessening the packages available in it so we don’t have a 17MB init.gz file, though I suppose ultimately that doesn’t matter. Bash is included as well just as a safe measure from my reading a different post with dd and gzip relying upon bash existing.
Once I’ve tested the uploads, if it’s successful, I’ll tar the exact file up with DL’s and host it on my personal site as I don’t think this forum will store a file that large. If it can be here, I’ll put it here as well.
Thanks,
-
Alright, I spoke a little too soon. That doesn’t mean I haven’t made success. I won’t post the file here, just the link as when compressed with all dl files needed it’s about 300MB which I’m certain is too large for this forum.
I’ll get the upload working on both partitions, as soon as I figure out why it’s not exactly.
-
So here’s the buildroot that I use:
[url]https://mastacontrola.com/fogboot/buildroot-2013.08-rc2_FOG.tar.bz2[/url]
If you’re to wget the file, as always, use:
wget --no-check-certificate [url]http://mastacontrola.com/fogboot/buildroot-2013.08-rc2_FOG.tar.bz2[/url]It’s rather large, I know, but it’s preconfigured with everything we need including the buildroot .config and it builds on Centos 6.4 with no issues. I haven’t built it on other OS’s. You will need the normal development information for building this thing, but a couple things that I needed to add were glibc-static and perl-ExtUtils-ParseXS and all dependencies needed there.
The file is about 300MB in size, and it will take a while to download as I’m only sporting 5MB upload speeds, sorry guys, but it’s better than nothing right?
Maybe others can look at what I’m doing wrong to find out the partitioning issues?
You do the normal building with:
make ARCH=i386
If you want to edit the packages to install, that’s up to you, but you will need to download it. When you setup for this you do the normal:
make ARCH=i386 menuconfigHopefully people find this useful.
Also, my buildroot actually builds partclone and partimage natively so I’ve modified the fog package and rather than using customize within the package directory, I just placed the FOG components in the package directory.
Thanks guys,
-
Sorry,
There were a few components missing from this. I didn’t build my own busybox configuration or use the one that FOG graciously provides for us. I’m trying to limit size a little, so hopefully after this run, fdisk and all the bobs and bits from the filesystem for partclone is all available. The file will still be named the same once uploaded.
Thanks all,
-
So I’ve, seemingly, made progress on getting the upload to work. It seems that my builds just ran thru way too quickly for pigz to keep up with writing to the NFS. The only issue is that my VM is extremely slow on the upload and download process. It only pushes speeds as high as 758 MiB/min and as slow as 100 MiB/min, but this sounds likely to just be a driver issue in my kernel.
I found that by adding a sleep command to the partimage save of the 2 partition section seems to work. I’m tinkering now with stepping up the intervals for the sleep timer to find out which one works best, at least for my setup. I don’t know if it’s simply because I am making an image off of a VM or if this would be a common issue seen by all. Though I don’t know that many people are using this particular setup.
My init.gz size is 9.7MB versus the 12-13 MB of the original one. I don’t know what I changed, but it’s probably the busybox config more than anything.
I’m taking down the tarball for now as it will probably be a couple of days until I get the sleep timer just right. Hopefully this is okay for everybody.
-
Found out that, the problem I’m seeing with the image upload process (as much as I can tell) is halting (resizable) because it’s referencing as kB. This in and of itself isn’t necessarily wrong, but when I try to manual create the disk partitions using parted, it tells me that 105905 (one byte behind part2start) is not valid for the partition. If I leave the end alone, it completes, but it isn’t the right size. It’s much smaller (part 1 that is) when it’s created using this particular method of parted. However, it successfully recreates the partition if I call the sectors: specifically, the sectors referenced are 206847 (partition one end) and 206848 is the start sector on the drive.
This method I can get rid of the need for the sleep commands, and all runs as expected. This is really weird to me, but it’s working as far as I can tell.
-
To add to the above,
My upload process with using Sectors works beautifully. However, the deploy process (which I have trouble with in both the original and my custom builds) does not work with single partition resizeable. However, it’s also not working with MPS, but that might be because I created the images using my custom build as that’s what I’ve been testing most frequently. I’ll find out later on. Still making progress and it’s a slow right. If you all can’t tell, I’ve not worked too much on the web interface in quite some time due to this. I will, however, try to get work done on it once I got this all figured out. In the mean time, I’ll work on tweaking the OS portion so it can be as small as possible, while still giving all of the needed elements a chance for us. I’m contemplating removing the partclone parts for now as it doesn’t build all the needed modules any what. It only creates:
partclone.restore
partclone.ntfsfixboot
partclone.info
partclone.chkimg
partclone.dd
fail-mbr.binI don’t know how to get it to build the [fstypes] of partclone executables, but I’ll work that out, hopefully in the future as well. Though we’re not using it right now anyway, so it’s not a huge thing.
-
Just a status update,
I am still having the problems with deploying, essentially, any image, but this is the same for the original init.gz file as well as mine.
It only images, during a download process, about 20%-35% then the system restarts. Sometimes the % complete is less, but generally speaking it just quits for some reason. I’m not seeing any errors pop up but that doesn’t mean that there isn’t any either.
Other than that, my init.gz is sitting around 6 MB, versus the 12-13MB of the original. So I think I’m doing well in piecing it back together. I think it’s missing a calculator function at this point though. I feeling like Dory from Finding Nemo, just keep trying, just keep trying …
It’s been a fun ride so far, if only a little frustrating, but at least I got one kink figured out. Yeah uploading single disk resizable!
