Development FOG not capturing image - PartClone update

Sebastian Roth

@ty900000 said in FOG/Apache PKI/Certificate Authentication:

I pulled the latest init and got a different error this time

Wait a second. Where did you pull it from? Did you use these ones? https://dev.fogproject.org/blue/organizations/jenkins/fos/detail/master/113/artifacts

Tom Elliott

@Sebastian-Roth He did, I can see the changes I created in the output.

ty900000

@Tom-Elliott

I stepped through everything until it halted. Pressing [Enter] here doesn’t do anything.

Sebastian Roth

@Tom-Elliott I really wonder why we don’t see other people report this error. Were you actually able to replicate this? Maybe this is just some RAM issue that causes binaries to fail on this particular machine!?

By the way, @ty900000 would you mind opening a new topic for this? Better to keep things sorted. I can move all the related messages over…

Tom Elliott

@Sebastian-Roth @ty900000

Would you mind trying the latest inits from: https://dev.fogproject.org/job/fos/job/master/lastSuccessfulBuild/

The init.xz and init_32.xz should be good.

Essentially I’m having a check on the partclone to be used and removing a couple of arguments as they are not built during the configuration and build of partclone.

ty900000

@Tom-Elliott

Yes! It worked perfectly. I’ve tested it a bunch of times and it works great. I do get this output after one of the partitions. It doesn’t affect anything it seems, but I’ve just noticed it.

Sebastian Roth

@Tom-Elliott Are you able to replicate the issue as seen in the pictures?
@ty900000 Does this happen on several machines? All the same model or different ones?

ty900000

@Sebastian-Roth

To start, I am using Hyper-V for everything. Yes, I do get that above image when I try to capture other images - either Windows or Linux. When I attempt to deploy the Windows image (the original image I’ve been trying to take), I get this error. But it does seem to complete. It does something similar for the the Linux image.

Tom Elliott

@ty900000 @Sebastian-Roth
I haven’t replicated, but to be fair I also haven’t watched that closely. We did image one machine yesterday and all seemed fine.

Looking at my images folder, however, I do notice that I’m missing the “imager” partition from my image. Luckily I had another image of the machine that did have the missing partition.

I pushed another fix and believe the issue was as @Quazz noted is the -c argument was missing. Strange as that is, as the -c argument doesn’t appear to be a part of the spec list (unless somebody already added that to the patch for partclone and I didn’t know it?)

This will take a while to build of course as I only just pushed it.

Sebastian Roth

@Tom-Elliott said in Development FOG not capturing image - PartClone update:

Strange as that is, as the -c argument doesn’t appear to be a part of the spec list

I think the -c is important to make partclone.imager actually use the parclone image format.

I haven’t replicated.

My guess is that this is something specific to Hyper-V or maybe even just @ty900000’s setup. Not saying we shouldn’t try to figure this out and eventually fix if it’s in the inits. My feeling is that this is not about partclone command line parameters or anything.

@ty900000 Please do me a favor and play with the the image’s setting Image Manager. Try Partclone Zstd if you have used Gzip since and even more so try out Partclone Uncompressed! Capture the image with these changed settings once more and let us know if it makes any difference.

shruggy

@Tom-Elliott said in Development FOG not capturing image - PartClone update:

I pushed another fix and believe the issue was as @Quazz noted is the -c argument was missing. Strange as that is, as the -c argument doesn’t appear to be a part of the spec list (unless somebody already added that to the patch for partclone and I didn’t know it?)

My understanding of it is this:

images captured with partclone.imager -c need to be restored with partclone.restore, raw images captured with partclone.dd or with partclone.imager without -c need to be restored with partclone.dd
the options for partclone.imager are basically the same as for any partclone.$ftype (including -c and -a), it’s partclone.dd that is a special case. The symbol DD only pertains to partclone.dd, the symbol IMG is specifically defined in Makefile for partclone.imager, but never gets used in upstream Partclone code.

ty900000

@Sebastian-Roth

I did test with several of the other compression types. As far as I can figure the Partclone Uncompressed is the only one that throws the above error with the 1 minute time out. And I also noticed it is not capturing sda3 - which is the raw partition on my particular Windows image. I think when it gets done with sda2 and attempts to move to sda3, it throws the Usage: unset_name [OPTIONS] error from farther above.

The only reason I am using Uncompressed is because the Gzip and Zstd image captures with 6 compression were slower than I remember them being in older (much older) versions of FOG - about 3 GB/min capture and deploy whereas the uncompressed was 6+GB/min, which I know is about line speed. We had a server at my last job and we were getting 6+GB/min when we captured and deployed a machine with Zstd level 6 compression. I need to do some testing tomorrow with a physical machine capture and deploy to see what’s going on with my network…

ty900000

I pulled the latest init just now, too. Using both uncompressed and Zstd

Quazz

@shruggy said in Development FOG not capturing image - PartClone update:

@Tom-Elliott said in Development FOG not capturing image - PartClone update:

I pushed another fix and believe the issue was as @Quazz noted is the -c argument was missing. Strange as that is, as the -c argument doesn’t appear to be a part of the spec list (unless somebody already added that to the patch for partclone and I didn’t know it?)

My understanding of it is this:

images captured with partclone.imager -c need to be restored with partclone.restore, raw images captured with partclone.dd or with partclone.imager without -c need to be restored with partclone.dd

the options for partclone.imager are basically the same as for any partclone.$ftype (including -c and -a), it’s partclone.dd that is a special case. The symbol DD only pertains to partclone.dd, the symbol IMG is specifically defined in Makefile for partclone.imager, but never gets used in upstream Partclone code.

Adding on to this, partclone.imager is generated using partclone.dd source files.

As per the makefile: https://github.com/Thomas-Tsai/partclone/blob/master/src/Makefile.am#L56

Making this commit: https://github.com/Thomas-Tsai/partclone/commit/2d1ee7c94139f390453ea8cb8675f67173bde33b problematic.

The special code for RAW is then locked to partclone.dd while partclone.imager is stuck using the “normal” code, which doesn’t work properly for its scenario.

Restoring the old code for partclone.imager should work. (there is no option to change the if condition main.c since there is no opt for imager…)

The stuff in https://forums.fogproject.org/topic/14078/1-5-7-89-partclone-doesn-t-capture-an-image-in-dd-mode-wrong-options-in-fog-upload/17 should work.

We just need to create a patch for it imo.

shruggy

@ty900000 Could you please try my init.xz with patched Partclone and say if it works for you?

ty900000

@shruggy

That seems to work better. It is pull the ‘raw’ partition 3 and works when using uncompressed capture and deploy, Zstd compression 6 worked well too. I did see a few errors that didn’t seem to affect anything. The first one was after imaging with complete and the second image was before it started.

shruggy

@Quazz has just fixed it in the source.

And I suppose the errors won’t show up if you set the image type to “Multiple Partition Image - Single Disk (Not Resizable)”, since the awk script gets only invoked during partition resizing.

ty900000

@shruggy

Okay, cool. I’ll wait for the build to complete and then pull it done for one more test. Thanks!

Quazz

@ty900000 That build does not yet include the partclone fixes, though. But we are getting close to that.

Sebastian Roth

@ty900000 said in Development FOG not capturing image - PartClone update:

I did test with several of the other compression types. As far as I can figure the Partclone Uncompressed is the only one that throws the above error with the 1 minute time out.

and…

I pulled the latest init just now, too. Using both uncompressed and Zstd
… picture with segmentation fault …

Those two answers don’t add up for me. Can you please do a simple test with the different compressions (none, Gzip, Zstd) and post which error exact you get for each.

Development FOG not capturing image - PartClone update

172

11.6k

17.1k

154.6k