Include partclone 3.20?

[Sebastian Roth]

@george1421 I am sure we can get it compiled, either by switching buildroot to building all libs as static only (tried that but ran into other issues that would need to be fixed first) or we need to add some magic (patch, sed, buildroot-specific mk-foo) to make the linker happy.

Most probably going down the later path is way easier. I was able to manually link one of the binaries by adding -ldl to the linker command. So it’s more or less a matter of adding this to partclone’s compilation per se. I will look into this some time this week.

[george1421]

@sebastian-roth said in Include partclone 3.20?:

I was able to manually link one of the binaries by adding -ldl

This is the path I was going down too when researching the error they just said put -ldl as the last library in the linker. The question was where (ignorance of buildroot)? The other thing I thought is update the version of buildroot to current, maybe a later release of build root would have a compatible version of the crypto lib.

The other thing I found is that ubuntu has 0.3.20 with some kind of patch level beyond 0.3.20 but I couldn’t find the source files to see if building that had fixed the crypto lib errors.

[Fog_Newb]

@sebastian-roth Ok thanks, I will try it out and report back. These inits, they just go on the FOG server right? Or do I also need to add something to the mac OS Grub/FOS boot stick?

[Sebastian Roth]

@fog_newb said in Include partclone 3.20?:

Or do I also need to add something to the mac OS Grub/FOS boot stick?

Which way did you create that boot stick?

[Fog_Newb]

This post is deleted!

[Fog_Newb]

This post is deleted!

[Fog_Newb]

@sebastian-roth Uhm… @george1421 can probably answer that better than I can.

Sorry for deleting and reposting the replies. The way the thread looks after replying, it looks like I replied to the OP and not to another reply.

[george1421]

@sebastian-roth said in Include partclone 3.20?:

Which way did you create that boot stick?

The OP used the tutorial I created to boot into grub and then into fos linux on a usb stick. But it sill uses the standard bzImage and init.xz. So no difference there. We still need buildroot to compile correctly.

[Fog_Newb]

@george1421

Thanks, I wasn’t sure how this one was made. I downloaded the img and used rufus to write it onto a USB. It works great by the way.

[Sebastian Roth]

@Fog_Newb There you go: https://github.com/FOGProject/fos/releases/tag/testing

Just download init.xz and put in /var/www/html/fog/service/ipxe/ in your FOG server - rename the original file instead of overwriting it!

[george1421]

@fog_newb FWIW: the testing init.xz goes onto the flash drive if you are booting with that. /var/www/html/fog/service/ipxe/ is used when pxe booting via iPXE into the FOG menu.

[Fog_Newb]

@sebastian-roth @george1421 Good news!

So far so good. I was able to capture the APFS drive on the real mac using the USB stick and the updated init.xy. I put the updated init.xy on both the stick and the server.

I was also able to capture both drives APFS and NTFS on the hackintosh laptop. Now I have to figure out how to back up the data in case deploying the images goes sideways.

[Sebastian Roth]

@fog_newb Thanks for testing and the quick feedback.

Now I have to figure out how to back up the data in case deploying the images goes sideways.

Which data do you mean? On the device? Probably using Clonezilla to have a backup at hand?!?

[Fog_Newb]

@sebastian-roth Yes, Clonezilla but I need to get some more SSDs/HDDs.

[Fog_Newb]

@sebastian-roth Does 1.5.9.163 have these new inits?

I read the dev-branch info but wasn’t sure. I never know what most of that stuff means.

https://imgur.com/V9EZOmH.jpg

[george1421]

@fog_newb said in Include partclone 3.20?:

Does 1.5.9.163 have these new inits?

No these inits are still in testing phase. Since partclone is at the heart of FOG imaging and the versions jumped from 0.3.13 to 0.3.20 these inits must have more testing with people before they are added to the dev branch and then onto the 1.5.10 when its released. They might work great with your hardware but fail on some genericly assembled computer.

[Piotr86PL]

I’m currently testing init.xz images built by myself using the partclone-0320 branch. In addition, these images have, for my part, added these two commits of mine from github that fix bugs with BTRFS (https://github.com/FOGProject/fos/pull/47 https://github.com/FOGProject/fos/pull/45). The whole thing was built using Buildroot 2022.02.5, which fixes bugs related to udev (https://github.com/FOGProject/fos/issues/46). I know it’s too many changes to treat my experience as relevant to the addition of this particular partclone version, but I think it’s worth sharing anyway.

The FOG server on which this custom init.xz runs is based on Fedora 36 (/images on XFS), the latest (at the time of writing this) development version of FOG, compiled kernel 5.15.71, and an updated version of UDPcast to 20200328 (the latest does not work with FOG). This server, runs in production, so there are 30 images restored each day, and sometimes more (mostly Windows 10 and Ubuntu). But it happens to image other computers that have other systems, such as Fedora 36 Workstation with the BTRFS file system. I mainly use Multicast restore (computers in the school computer lab), but sometimes I need to restore one computer and I use Unicast.

The previous FOG server was based on Debian 10 (/images on EXT4), FOG 1.5.9 and the latest official kernel and init.xz. The situation has changed dramatically after the migration.

On the old system, using Unicast I was getting speeds of around 7-8GB/min (for both restore and capture). With Multicast, this speed increased to 12GB/min.

After I built the server on Fedora, I used init.xz from the partclone-0320 branch without patches and the official 5.15.68 kernel and the speed increased to 14GB/min using both Multicast and Unicast. The capture speed went up to 9GB/min. After changing init.xz to the current one, the speeds have not changed, but at least I can safely restore the BTRFS file system without any errors.

I don’t know how much of this is due to Partclone 3.20 and how much is due to migrating the system to Fedora with XFS, but I can say that so far Partclone 3.20 is running very stably and hasn’t crashed yet. And I have already restored images based on NTFS, XFS, EXT4 and BTRFS. If I only notice any flaws with the operation of the whole thing I will describe them. But so far I have no complaints about my system. If it continues to perform as well as it does now I will migrate the FOGs in other computer labs to what I am using in this particular one.

[george1421]

@piotr86pl Excellent feedback. While you have many accomplishments, it would be interesting in your buildroot configs to roll partclone back to 0.3.13 (only) and see if your side by side comparison still holds. This will kind of give the developers insight to if the partclone upgrade generated these improvements or it was updating buildroot to something current and optimized for linux 5.15.x,

Either well done rebuilding the FOG imaging environment and the performance gains you have.

[Piotr86PL]

I built init.xz with partclone 0.3.13, with no changes to buildroot or scripts, and ran a series of tests. The end result was a difference of 1GB/min in some cases in favour of partclone 0.3.20, but this is a marginal difference to take into account. I noticed that partclone 0.3.13 was a little slower to start than partclone 0.3.20, i.e. after the message “Attempting to deploy image”, it took longer to start partclone 0.3.13 than 0.3.20 - about half a minute.

In general, changing the partclone version here is unlikely to affect the speed of image restoration, but more on things like support for non-Linux filesystems like NTFS or support for more exotic filesystems like BTRFS.

Also, I wouldn’t rush to update partclone to version 0.3.20, but I would definitely see it already when FOG 1.5.10 is released, as it fixes a few bugs after all and doesn’t cause me any problems. I would also give buildroot in a newer version for the release of FOG 1.5.10, so as not to scare users away with those udev warnings that appear because of eudev. But it’s all up to the developers now. I’ll keep an eye on the operation of my setup and as soon as something happens I’ll investigate it and let you know.

[george1421]

@piotr86pl said in Include partclone 3.20?:

I wouldn’t rush to update partclone to version 0.3.20, but I would definitely see it already when FOG 1.5.10 is released

First of all, thank you for taking time to debug/test the setup with the old version of partclone and the current version. Its important to know exactly which change made a difference in the imaging times.

If we can get enough people to test the development ints (the ones issued by the developers not in the current dev branch) and prove them out, I’m sure that 0.3.20 will be in the 1.5.10 release. As noted in this thread partclone 0.3.20 addresses an issue with the apple file system so there is more pressure to get it into the 1.5.10 over a speed improvement.

@Developers please note the testing by @Piotr86PL