Development FOG not capturing image - PartClone update

ty900000

To start, I am using Hyper-V for everything. Yes, I do get that above image when I try to capture other images - either Windows or Linux. When I attempt to deploy the Windows image (the original image I’ve been trying to take), I get this error. But it does seem to complete. It does something similar for the the Linux image.

Tom Elliott

@ty900000 @Sebastian-Roth
I haven’t replicated, but to be fair I also haven’t watched that closely. We did image one machine yesterday and all seemed fine.

Looking at my images folder, however, I do notice that I’m missing the “imager” partition from my image. Luckily I had another image of the machine that did have the missing partition.

I pushed another fix and believe the issue was as @Quazz noted is the -c argument was missing. Strange as that is, as the -c argument doesn’t appear to be a part of the spec list (unless somebody already added that to the patch for partclone and I didn’t know it?)

This will take a while to build of course as I only just pushed it.

Sebastian Roth

@Tom-Elliott said in Development FOG not capturing image - PartClone update:

Strange as that is, as the -c argument doesn’t appear to be a part of the spec list

I think the -c is important to make partclone.imager actually use the parclone image format.

I haven’t replicated.

My guess is that this is something specific to Hyper-V or maybe even just @ty900000’s setup. Not saying we shouldn’t try to figure this out and eventually fix if it’s in the inits. My feeling is that this is not about partclone command line parameters or anything.

@ty900000 Please do me a favor and play with the the image’s setting Image Manager. Try Partclone Zstd if you have used Gzip since and even more so try out Partclone Uncompressed! Capture the image with these changed settings once more and let us know if it makes any difference.

shruggy

@Tom-Elliott said in Development FOG not capturing image - PartClone update:

I pushed another fix and believe the issue was as @Quazz noted is the -c argument was missing. Strange as that is, as the -c argument doesn’t appear to be a part of the spec list (unless somebody already added that to the patch for partclone and I didn’t know it?)

My understanding of it is this:

images captured with partclone.imager -c need to be restored with partclone.restore, raw images captured with partclone.dd or with partclone.imager without -c need to be restored with partclone.dd
the options for partclone.imager are basically the same as for any partclone.$ftype (including -c and -a), it’s partclone.dd that is a special case. The symbol DD only pertains to partclone.dd, the symbol IMG is specifically defined in Makefile for partclone.imager, but never gets used in upstream Partclone code.

ty900000

@Sebastian-Roth

I did test with several of the other compression types. As far as I can figure the Partclone Uncompressed is the only one that throws the above error with the 1 minute time out. And I also noticed it is not capturing sda3 - which is the raw partition on my particular Windows image. I think when it gets done with sda2 and attempts to move to sda3, it throws the Usage: unset_name [OPTIONS] error from farther above.

The only reason I am using Uncompressed is because the Gzip and Zstd image captures with 6 compression were slower than I remember them being in older (much older) versions of FOG - about 3 GB/min capture and deploy whereas the uncompressed was 6+GB/min, which I know is about line speed. We had a server at my last job and we were getting 6+GB/min when we captured and deployed a machine with Zstd level 6 compression. I need to do some testing tomorrow with a physical machine capture and deploy to see what’s going on with my network…

ty900000

I pulled the latest init just now, too. Using both uncompressed and Zstd

Quazz

@shruggy said in Development FOG not capturing image - PartClone update:

@Tom-Elliott said in Development FOG not capturing image - PartClone update:

I pushed another fix and believe the issue was as @Quazz noted is the -c argument was missing. Strange as that is, as the -c argument doesn’t appear to be a part of the spec list (unless somebody already added that to the patch for partclone and I didn’t know it?)

My understanding of it is this:

images captured with partclone.imager -c need to be restored with partclone.restore, raw images captured with partclone.dd or with partclone.imager without -c need to be restored with partclone.dd

the options for partclone.imager are basically the same as for any partclone.$ftype (including -c and -a), it’s partclone.dd that is a special case. The symbol DD only pertains to partclone.dd, the symbol IMG is specifically defined in Makefile for partclone.imager, but never gets used in upstream Partclone code.

Adding on to this, partclone.imager is generated using partclone.dd source files.

As per the makefile: https://github.com/Thomas-Tsai/partclone/blob/master/src/Makefile.am#L56

Making this commit: https://github.com/Thomas-Tsai/partclone/commit/2d1ee7c94139f390453ea8cb8675f67173bde33b problematic.

The special code for RAW is then locked to partclone.dd while partclone.imager is stuck using the “normal” code, which doesn’t work properly for its scenario.

Restoring the old code for partclone.imager should work. (there is no option to change the if condition main.c since there is no opt for imager…)

The stuff in https://forums.fogproject.org/topic/14078/1-5-7-89-partclone-doesn-t-capture-an-image-in-dd-mode-wrong-options-in-fog-upload/17 should work.

We just need to create a patch for it imo.

shruggy

@ty900000 Could you please try my init.xz with patched Partclone and say if it works for you?

ty900000

@shruggy

That seems to work better. It is pull the ‘raw’ partition 3 and works when using uncompressed capture and deploy, Zstd compression 6 worked well too. I did see a few errors that didn’t seem to affect anything. The first one was after imaging with complete and the second image was before it started.

shruggy

@Quazz has just fixed it in the source.

And I suppose the errors won’t show up if you set the image type to “Multiple Partition Image - Single Disk (Not Resizable)”, since the awk script gets only invoked during partition resizing.

ty900000

@shruggy

Okay, cool. I’ll wait for the build to complete and then pull it done for one more test. Thanks!

Quazz

@ty900000 That build does not yet include the partclone fixes, though. But we are getting close to that.

Sebastian Roth

@ty900000 said in Development FOG not capturing image - PartClone update:

I did test with several of the other compression types. As far as I can figure the Partclone Uncompressed is the only one that throws the above error with the 1 minute time out.

and…

I pulled the latest init just now, too. Using both uncompressed and Zstd
… picture with segmentation fault …

Those two answers don’t add up for me. Can you please do a simple test with the different compressions (none, Gzip, Zstd) and post which error exact you get for each.

ty900000

@Sebastian-Roth

My apologies; I know my previous message was confusing.

A few days ago, the init from Jenkins and using Uncompressed would throw the error when it deployed - the image above about /dev/sda3. A few hours after that a new init was pushed to Jenkins that I tried. That one reintroduced the segmentation error during image capture, not matter what compression I tried.

I currently am using the init @shruggy suggested yesterday and that one seems to have fixed everything for me during image capture and deployment.

I haven’t pulled the latest init from the Jenkins. Would you like me to do that?

Thanks everyone!

Sebastian Roth

@ty900000 said:

A few days ago, the init from Jenkins and using Uncompressed would throw the error when it deployed - the image above about /dev/sda3.

I guess you mean this error: alt text

That would mean the capture did run fine and partclone did not segfault when capturing. Though we probably can’t figure out which version it was you were using back then as too many things were tried in a rush. Nevermind.

I currently am using the init @shruggy suggested yesterday …

@shruggy Would you mind telling us which version of partclone this is?

ty900000

@Sebastian-Roth

I’m sorry! I had to backtrace this information, if you still want it: FOS 112 had the segmentation fault, FOS 113 had the segmentation fault (also tried the debug with this one), FOS 114 captured without the segmentation fault and introduced the unset_name [OPTIONS] error during capture and subsequently the /dev/sda3 error during deployment, and FOS 115 reintroduced the segmentation error.

I can redownload FOS 114 files and test with those again, if need be.

Tom Elliott

@ty900000 The problem with using 114, is it’s missing an argument. So the Segfault is double’d due to -c/-a0 and not likely due to either of the arguments, but rather the fact partclone is segfaulting as it cannot determine file size, as it is literally commented out of the code. This means, a raw (imager) image file will not be generated at all. (/images/<imagename>/d1p<#ofparttion_to_be_cloned_in_imager_format>.img will be missing comletely.)

In my case the partition number was 2, and in yours it appears the partition number is 3.

I have created a fork of the partclone repository and implemented what I would think should address the issue by reimplementing the filesize display capabilities. As @Sebastian-Roth learned, simply uncommenting the line allowed partclone to work properly (as far as we could tell).

Until we patch the issue on our side (either by using our forked copy of the repository or directly implementing it as a part of the build process), this is still going to be a problem. I think we have a way to fix it, just haven’t had time to implement and have you (or anybody else who’d be willing) to test it.

shruggy

@Sebastian-Roth said in Development FOG not capturing image - PartClone update:

@shruggy Would you mind telling us which version of partclone this is?

It’s the latest git shapshot:
https://github.com/Thomas-Tsai/partclone/tree/58d138da6e0f473acd80a7c9b8544d104377d50f
with the fix you suggested:

$ cd git/fos/Buildroot/package/partclone
$ ls
Config.in  partclone-0.3.12.patch  partclone.imager.patch  partclone.mk
$ cat partclone.imager.patch
diff --git a/src/ddclone.c b/src/ddclone.c
index 04f8775..cab5386 100644
--- a/src/ddclone.c
+++ b/src/ddclone.c
@@ -30,7 +30,9 @@ void read_super_blocks(char* device, file_system_info* fs_info)
 	}
 	strncpy(fs_info->fs, raw_MAGIC, FS_MAGIC_SIZE);
 	fs_info->block_size  = PART_SECTOR_SIZE;
-	//fs_info->device_size = get_partition_size(&src);
+#ifdef IMG
+	fs_info->device_size = get_partition_size(&src);
+#endif
 	fs_info->totalblock  = fs_info->device_size / PART_SECTOR_SIZE;
 	fs_info->usedblocks  = fs_info->device_size / PART_SECTOR_SIZE;
 	close(src);

I haven’t yet pulled the latest FOS, however. So it doesn’t contain the changes to func.sh by @Tom-Elliott i.e. no special case for partclone.imager.

On the other note, when using unpatched Partclone 0.3.12 I also sometimes got segmentation fault, but didn’t report it. The case when Partclone finished successfully, but actually didn’t capture the partition was the more common outcome. I’d say, somewhere in the relation of 4 (empty) successes to 1 segfault, IIRC. And I didn’t notice any pattern to segfaults then (e.g. it could happen with or without compression).

Sebastian Roth

@ty900000 Sorry, forgot to update this topic as well. We have added a fix. Please update to the latest dev-branch and it should be all good to go again,

Development FOG not capturing image - PartClone update

205

11.6k

17.1k

154.6k