Custom init.gz


  • Senior Developer

    Hello all,

    I know I’ve been rather busy on fog these days, but I’ve just been trying to come up with things to help the whole project out.

    I am, by no means, a developer, but I am a very avid computer enthusiast who likes to tinker around.

    I’ve been trying to build my own init.gz file for use within the fog system, and I’ve been, for the most part, successful.
    Only a couple of parts have confused me.

    Using the WIKI page, I see that we are supposed to be using buildroot2011.08 but I’ve never had success with that particular version. I’ve had, however, success with buildroot2013.05.

    There’s one part, though, that keeps eluding me. That part is the pigz portion. If I use the init.gz that’s prebuilt, uploading of rec.img.000 and sys.img.000 works flawlessly. When using a my custom one, it uploads, just fine, the rec.img.000 file, but then it completes the upload.
    Downloading is a whole different issue that I’ve yet to succeed with. With that aside, I’ve been trying to build a more up-to-date init system anyway as the headers currently in use are 3.0.4 and I think it’d be more suited for a more updated kernel tree.

    The item I have, for the developers, if you’re willing to share is how to get things to compile properly and what changes, I think, should have to be made for this all to work. Can anyone tell me what’s missing? I’ll give the link to my version of init.gz just to see if their’s anything that I can correct to get pigz to work for both rec and sys img files.

    First things first, the device_table.txt file, in more recent versions of buildroot, is now located at system/device_table.txt rather than target/generic/device_table.txt

    Also, the custom path for the fs is now:
    system/skeleton rather than fs/skeleton

    I’m still unsure about pigz, but many of the customize scripts need the last line changed from:
    [COLOR=#000000]$(eval $(call AUTOTARGETS,package/customize,parted))[/COLOR]
    [COLOR=#000000]to:[/COLOR]
    [COLOR=#000000]$(eval $(autotools-package))[/COLOR]
    I don’t know what else to share from here. At least I have, for the most part, a working init.gz, though I’m unsure of how to build custom packages for buildroot. Maybe somebody can help me?
    I know that partclone can do many things but depends upon the utilities and tools to operate, so we’d have to include them. At the very least, we’d have to include ntfsprogs, e2fsprogs, and maybe btrfs-tools and possibly more but I don’t know what I’m doing totally. I’ll try to tinker and play, but I can only do so much alone.


  • Developer

    THANK YOU GILOU!!!



  • And for those looking to avoid compiling it, and need a 32 bits init.gz , I made one: [url]https://gilouweb.com/bordel/init_fog_i686.xz[/url] based on buildroot of r1284.


  • Senior Developer

    [url]https://mastacontrola.com/fogboot/images/init.xz[/url]

    Forewarning though, it’s 64 bit, so you’ll likely need a 64 bit kernel.

    [url]https://mastacontrola.com/fogboot/kernel/bzImage[/url]



  • Any chance we could get a link to your compiled init.gz?


  • Senior Developer

    I’ve added a few tools to this. sed, rsync, librsync, resize2fs, gawk, and a few others that I can’t remember off the top of my head.

    The lzma compression is still used, and the init.gz size is now around 7.5 MB. I hope that’s okay with everyone.

    This has been really fun, if not frustrating at times, to play with. I haven’t figured out a way to send the output of Partclone to the screen and only send errors to /tmp/status.fog. I also haven’t, quite yet, figured out how to enable the progress bar on the Web GUI, but that can be for later, or never if it comes down to it. Having the screen tell is is good enough for me, though it wouldn’t hurt to know the progress information from time to time.

    I have saved quite a lot of loading time on the kernel though. I’ve figured out a method (though the configuration file isn’t updated with this new method quite yet) to get the kernel size down to just shy of 6MB. This means our total load out size is only around 14 MB versus the old 0.33 sizes of around 12-13MB for the kernel and another 12 MB for the init.gz. 14 vs. 24, you be the judge.


  • Senior Developer

    All,

    I think I’ve tweaked enough with the init.gz. Still in initial testing, but I think I’ve got it down enough that it was safe for me to update the src/buildroot files on the trunk of svn. We’re now at r1004 because of this. Now once I’m complete with testing the init.gz I’ll update svn with my latest and greatest one. It’s built around 3.12.1 headers so I may have to make a go at updating the kernel that we can download.

    Compression,

    I’m going to move mine away from gzip as it’s, to me , a rather large init.gz file if only gzipped. I’m compressing it with
    [code]lzma -z -9 -e < rootfs.ext2 > init.gz[/code]
    Thank you Fernando for giving me the insight for this compression medium.

    as it makes it from ~10-12 MB down to ~5-6 MB. This means less time waiting for it to load.

    I don’t mind the kernel being larger as that is to be expected due to it containing all the drivers for the systems to be imaged, but I think a smaller loading rootfs is a necessary thing.


  • Senior Developer

    It seems the upload process worked, but the deploy did not work properly. I think this is because the win7.mbr in use has something wrong with it and can’t get to the proper winload, or I’m a total idiot. However, I’m going to do the same test once again, but with a sysprepped system rather than just a generic test. As an aside to this all, it appears the multipart images nonsysprepped is working once again now that I’m using partclone. Maybe I’m overthinking it, but essentially we’re back to using a cloning utility that works in much the same way we’re all used to.


  • Senior Developer

    Well,

    I’ve now tested upload of ext partclone, my Debian box, and that seems to have taken well. Now to deploy the same methodology for ntfs test. I’ve actually got a real system to test this on, so I’m going to test on both. The real system will require generalized sysprep as it’s, obviously, different than my VM, but all and all, I will keep all posted.


  • Senior Developer

    Figured out a couple of things.

    The display of partclone wasn’t working because it was counting all output types as errors even when it wasn’t an error (in fog script it redirects errors to status.fog.)

    Removing the redirects seems to allow display to work.

    The next problem to overcome, after upload is complete, is download of resizeable images. I think this issue, as described earlier, was due to the sector/kB associations. First, ntfsprogs DOES NOT like 100% as a layout. -1s seems to work for the end point, but I prefer to use actual size of the disk. The method to this approach, assuming Windows 7 always uses the same information for the 1st partition, is to, rather than using kB as the startpoint, is to use B for the start point. Sectors will work as well, 206847 is the end sector of the first partition. However, the current method used kB. So the start kB of the second partition was/is always 105906kB, which for all purposes is the same end kB of the first partition. I was able to create the second partition properly by using the diskSize variable, slightly modified (added grep -v “Flags”) as the end point with kB and the start point as 105906176B.

    Hope this helps out. Partclone seems to be working on the upload side, now just waiting to play with download. Hopefully I’ll be able to post something soon.


  • Senior Developer

    So,

    In trying to get partclone working, I’ve noticed a couple of things.

    First,

    I forgot all about my old sector based partition creation to actually recreate /dev/sda2 during upload process. The only problem I forsee with using sector based partition creation is 4k disks. I can probably figure out a method to get the sector autonomously in the future so this wouldn’t necessarily be a problem anymore, but for now I’m back to using that.

    After the partition is resized, it needs (I think) to be fixed.

    Something like:
    [code]ntfsfix -b -d ${part}[/code]

    Of course you need the partition recreated and existing for this to work, but that should be trivial.



  • Thanks tom going to give this a go :rolleyes:


  • Senior Developer

    Eventually I’ll update the trunk/src/buildroot to contain the needed elements for the buildroot portions of the package, to include the Config.in file to make things work properly.

    For now, it’s just a hoopla of a lot of work.


  • Senior Developer

    There’s quite a few modifications from the 2011.08 package that FOG 0.33b is built on.

    First things first, if you’re going to try updating it, make sure you have a debian system (vm or not) to build partclone on.

    To build partclone I use:

    Download the partclone you want. I’m using partclone_0.2.58.orig.tar.gz

    Extract the file:
    [code]tar -xzf partclone_0.2.58.orig.tar.gz[/code]

    Prep the system.
    [code]echo “deb http://free.nchc.org.tw/drbl-core drbl dev stable testing unstable” >> /etc/apt/sources.list
    echo “deb-src http://free.nchc.org.tw/drbl-core drbl dev stable testing unstable” >> /etc/apt/sources.list
    apt-get update
    apt-get build-dep partclone
    cd partclone-0.2.58/[/code]

    Begin the Makefile creation: If you arch is already i386/i686 this will work.
    [code]./configure --enable-static --enable-xfs --enable-btrfs --enable-ntfs --enable-extfs --enable-fat --enable-hfsp --enable-static --enable-ncursesw[/code]

    I don’t know what kinds of problems you’ll run into, so I’m going to just assume for now that you’re on i386 debian.

    Build the binaries:
    [code]make[/code]

    Once complete, change to the src directory of partclone.
    [code]cd src[/code]

    Now copy the partclone files to your buildroot fog folder: (I think it something like below)
    [code]for i in ls|grep ^partclone\\\.|grep -v \\\.c$|grep -v \\\.h$|grep -v \\\.o$; do cp $i ~/buildroot-2013.08.1/package/fog/partclone/; done[/code]

    Your partclone binaries should be ready for use on the init.gz system now.



  • Hi Tom keeping a close eye on ur work on the init.gz I really like where ur going with it. give us a shout when it ready for testing.
    [SIZE=6][B]Thanks for all your hard work ;)[/B][/SIZE]


  • Senior Developer

    I’m running into some weird issues that is, I think, requiring me to perform a debian based build process. Partclone is almost entirely only supported (building the source this is) on debian. I think part of my issues so far is because it’s requiring shared libraries (being built without --enable-static) however if I try to run build with --enable-static, it fails miserably on CentOS 6.4 64 bit and i386 bit.

    I’m just trying to get a hand at what’s happening.


  • Senior Developer

    Now I’m attempting to rebuild my tarball and have integrated the scripts to install the libraries needed automatically. This way I can just build it. It contains the original .config file for 32 bit builds and will require make ARCH=i386 to operate. If you try to build it with x86_64, it will most likely fail on the uclibc steps as there’s no directory for lib64 libraries in uclibc. I’m still testing so don’t attempt it yet, but the fix seems to be switching from uclibc to eglibc (experimental) for this. I’m still testing though but thinking this is the best method so far.


  • Senior Developer

    Good news, sorta…

    I got the libraries needed and the partclone is working. It isn’t giving the ncurses menu (the menu that shows the progress) but it is creating the (at least the sys file) right now. I haven’t tested the abilities for single disk resizeable, but I do have ntfs working on a multipart single disk image so far. I haven’t tested deploy, but at least upload seems to be working.

    With this, the init.gz is very large comparitively to my previous version. By large, I mean it spiked from 6.1 mb from my original source to 30mb with all libraries attached. However, it has name resolution and even includes scp and ssh utilities.


  • Senior Developer

    Well,

    Good news/bad news time.

    As I don’t know fully what I’m doing in creating this, I got the init.gz to actually run and operate with 64 bit capabilities. The kernel and init.gz were fully compatible. I didn’t notice anything major, except… all the libraries could not be found :(


  • Senior Developer

    I’m attempting something dangerous I think.

    I’m attempting to build a 64 bit init.gz, which of course will require a 64 bit kernel to go along with it. I’ll try to keep it posted here.


Log in to reply
 

487
Online

38957
Users

10706
Topics

101581
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.