Error decrypting LUKS partition prior to capture/imaging

george1421

@humoss233 There are a couple of things at play here.

First of all (if everything else is setup) you can automate this with a fog post init scripts. These scripts are run just after the FOS Linux engine starts but before any imaging take place. These scripts are intended for bringing up raid cards, or any other hardware related activities before imaging starts. So once you can get things working manually then we can focus on automation.

Secondly, if you setup a debug capture or deploy you can debug or bring up hardware prior to imaging. Once the hardware is setup you would key in fog to start imaging (this would be done on the target computer). In debug mode the FOS scripts will pause between each step to wait for an enter key press. This allows you to read or react to error messages. If you need to break out of the imaging script just key in ctrl-c, fix what was needed then restart the imaging process by keying in fog again.

Now that we have some of the basic debugging processes out of the way we can think about the root of the problems.

In the linked article the FOS Linux kernel will need the dm-mod kernel driver loaded. FOS Linux doesn’t support dynamically linked modules, so it will need to be compiled in. The vchange command is part of LVM. I don’t know off the top of my head if vchange is part of FOS Linux. If not the inits will need to be recompiled to include lvm commands.

Understand I’m researching this as I write the post so it may seem a bit disjointed.
As for the LUKS code bits, those will probably need to be compiled into the inits using buildroot. This is not something that is native to FOS Linux, but I assume could be added. Looking at the FOS Linux buildroot compiler I see a “cryptsetup” package that is available. So that’s a good sign. Looking into the FOS Linux buildroot config file the cryptsetup option is enabled BR2_PACKAGE_CRYPTSETUP=y so the binaries should be in the inits.

So the only question then does the kernel have the required modules built in.
ref: https://cateee.net/lkddb/web-lkddb/DM_CRYPT.html
Looking into the FOS Linux kernel config, dm crypt is not enabled. So this is going to be a problem.

humoss233

@george1421 Thanks for your comprehensive treatment of this topic! Absolutely re: getting it working manually before automating with init scripts.

It sounds like dm crypt is not something that can be enabled with a flag? If not, I could try to rebuild the kernel if it’s a simple one-liner tweak somewhere here https://github.com/FOGProject/fos for example?

george1421

@humoss233 I recompiled the kernel last night after my post with the dm_crypt enabled. Give me a minute and I’ll upload it where you can get to it.

https://drive.google.com/open?id=1N6q6Oqmi7W7WkdtNPK1H0O8B1f-a4RFU
Note to future readers I may remove this file at any time so the link may not be valid in the future

To use this new kernel, download it from the link and save it in /var/www/html/fog/service/ipxe directory on the fog server as bzImageCrypt Then manually register one host and then go into the web ui in the host management for this target system. Update the kernel field with bzImageCrypt (watch the case because it IS important). Save the host management page and then schedule another debug capture. Then test your commands again with the modules loaded into the kernel.

ref Kernel patch file for differences between standard config and dm_crypt added config

--- kernelx64.config    2019-08-29 12:46:58.222184653 -0400
+++ .config     2019-10-20 00:20:29.579817034 -0400
@@ -1273,12 +1273,17 @@
 # CONFIG_BCACHE is not set
 CONFIG_BLK_DEV_DM_BUILTIN=y
 CONFIG_BLK_DEV_DM=y
-# CONFIG_DM_MQ_DEFAULT is not set
+CONFIG_DM_MQ_DEFAULT=y
 # CONFIG_DM_DEBUG is not set
-# CONFIG_DM_UNSTRIPED is not set
-# CONFIG_DM_CRYPT is not set
-# CONFIG_DM_SNAPSHOT is not set
-# CONFIG_DM_THIN_PROVISIONING is not set
+CONFIG_DM_BUFIO=y
+CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING=y
+CONFIG_DM_DEBUG_BLOCK_STACK_TRACING=y
+CONFIG_DM_BIO_PRISON=y
+CONFIG_DM_PERSISTENT_DATA=y
+CONFIG_DM_UNSTRIPED=y
+CONFIG_DM_CRYPT=y
+CONFIG_DM_SNAPSHOT=y
+CONFIG_DM_THIN_PROVISIONING=y
 # CONFIG_DM_CACHE is not set
 # CONFIG_DM_WRITECACHE is not set
 # CONFIG_DM_ERA is not set
@@ -3424,8 +3429,6 @@
 CONFIG_HAVE_ARCH_KASAN=y
 # CONFIG_KASAN is not set
 CONFIG_ARCH_HAS_KCOV=y
-CONFIG_CC_HAS_SANCOV_TRACE_PC=y
-# CONFIG_KCOV is not set
 # CONFIG_DEBUG_SHIRQ is not set

humoss233

@george1421 thanks for uploading this! I just got a chance to try it out. It fails but a different error this time, so I think we’re getting closer.

[Mon Oct 21 root@fogclient /]# cryptsetup luksOpen /dev/md126p3 crypt
Enter passphrase for /dev/md126p3:
device-mapper: table: 251:0: crypt: Error allocating crypto tfm
reload ioctl on    failed: No such file or directory
Failed to setup dm-crypt key mapping for device /dev/md126p3
Check that the kernel supports aes-xts-plain64 cipher (check syslog for more info).

From /var/log/messages:

Oct 21 21:03:16 fogclient user.err kernel: device-mapper: table: 251:0: crypt: Error allocating crypto tfm
Oct 21 21:03:16 fogclient user.warn kernel: device-mapper: ioctl: error adding target to table

Doing some research, it looks like a module may be missing (“No such file or directory”). Could it be that the module for the cipher is missing? I’m currently using cryptsetup default options (which uses aes-xts-plain64 as cipher). When I try cat crypto | grep aes on this FOS build, I only see aes-generic whereas I see ~20 options when doing the same on my Ubuntu server (including xts-aes-aesni).

Let me know if I can do anything to help debug further.

Some links referencing similar error messages out there:

https://www.raspberrypi.org/forums/viewtopic.php?t=60278
https://bugs.centos.org/view.php?id=9311

george1421

@humoss233 OK I do see some crypto parameters not enabled in the kernel.

CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_TI is not set
# CONFIG_CRYPTO_AES_X86_64 is not set
# CONFIG_CRYPTO_AES_NI_INTEL is not set
# CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64 is not set
# CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64 is not set

if you could run cat /proc/crypto | grep aes on both fos linux and the system where the it works. Or is that where you posted above the cat crypto | grep aes above?

Sebastian Roth

@humoss233 As well run cryptsetup --help and check the last couple of lines for cipher information (from https://superuser.com/questions/1039487/check-that-kernel-supports-aes-xts-plain64-cipher).

george1421

@humoss233 I added in aes ni and recompiled it here: https://drive.google.com/open?id=1N6q6Oqmi7W7WkdtNPK1H0O8B1f-a4RFU

--- kernelx64.config    2019-08-29 12:46:58.222184653 -0400
+++ .config     2019-10-20 00:20:29.579817034 -0400
@@ -1273,12 +1273,17 @@
 # CONFIG_BCACHE is not set
 CONFIG_BLK_DEV_DM_BUILTIN=y
 CONFIG_BLK_DEV_DM=y
-# CONFIG_DM_MQ_DEFAULT is not set
+CONFIG_DM_MQ_DEFAULT=y
 # CONFIG_DM_DEBUG is not set
-# CONFIG_DM_UNSTRIPED is not set
-# CONFIG_DM_CRYPT is not set
-# CONFIG_DM_SNAPSHOT is not set
-# CONFIG_DM_THIN_PROVISIONING is not set
+CONFIG_DM_BUFIO=y
+CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING=y
+CONFIG_DM_DEBUG_BLOCK_STACK_TRACING=y
+CONFIG_DM_BIO_PRISON=y
+CONFIG_DM_PERSISTENT_DATA=y
+CONFIG_DM_UNSTRIPED=y
+CONFIG_DM_CRYPT=y
+CONFIG_DM_SNAPSHOT=y
+CONFIG_DM_THIN_PROVISIONING=y
 # CONFIG_DM_CACHE is not set
 # CONFIG_DM_WRITECACHE is not set
 # CONFIG_DM_ERA is not set
@@ -3135,10 +3140,12 @@
 CONFIG_CRYPTO_NULL2=y
 # CONFIG_CRYPTO_PCRYPT is not set
 CONFIG_CRYPTO_WORKQUEUE=y
-# CONFIG_CRYPTO_CRYPTD is not set
+CONFIG_CRYPTO_CRYPTD=y
 # CONFIG_CRYPTO_MCRYPTD is not set
 CONFIG_CRYPTO_AUTHENC=y
 # CONFIG_CRYPTO_TEST is not set
+CONFIG_CRYPTO_SIMD=y
+CONFIG_CRYPTO_GLUE_HELPER_X86=y

 #
 # Authenticated Encryption with Associated Data
@@ -3220,8 +3227,8 @@
 #
 CONFIG_CRYPTO_AES=y
 # CONFIG_CRYPTO_AES_TI is not set
-# CONFIG_CRYPTO_AES_X86_64 is not set
-# CONFIG_CRYPTO_AES_NI_INTEL is not set
+CONFIG_CRYPTO_AES_X86_64=y
+CONFIG_CRYPTO_AES_NI_INTEL=y
 # CONFIG_CRYPTO_ANUBIS is not set
 CONFIG_CRYPTO_ARC4=y
 # CONFIG_CRYPTO_BLOWFISH is not set
@@ -3424,8 +3431,6 @@
 CONFIG_HAVE_ARCH_KASAN=y
 # CONFIG_KASAN is not set
 CONFIG_ARCH_HAS_KCOV=y
-CONFIG_CC_HAS_SANCOV_TRACE_PC=y
-# CONFIG_KCOV is not set
 # CONFIG_DEBUG_SHIRQ is not set

 #
@@ -3460,7 +3465,7 @@
 # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
 # CONFIG_LOCK_TORTURE_TEST is not set
 # CONFIG_WW_MUTEX_SELFTEST is not set
-# CONFIG_STACKTRACE is not set
+CONFIG_STACKTRACE=y
 # CONFIG_WARN_ALL_UNSEEDED_RANDOM is not set
 # CONFIG_DEBUG_KOBJECT is not set
 # CONFIG_DEBUG_BUGVERBOSE is not set

Sebastian Roth

@george1421 As well there should be CONFIG_CRYPTO_XTS (see https://cateee.net/lkddb/web-lkddb/CRYPTO_XTS.html) - but you need to enable CONFIG_EXPERIMENTAL for that option to show up.

george1421

@george1421 With XTS kernel module too: https://drive.google.com/open?id=1N6q6Oqmi7W7WkdtNPK1H0O8B1f-a4RFU

Edit: We may not be done yet depending on the password hash you used ref: https://lists.gt.net/gentoo/user/300718

humoss233

@george1421 I tried the version with the XTS kernel module and it works! luksOpen works without errors and I can also mount the decrypted partition and see my files. Now I just need to setup a post init script.

@george1421 @Sebastian-Roth
Debugging info on FOS (latest bzImageCrypt):
cryptsetup --help

loop-AES: aes, Key 256 bits
plain: aes-cbc-essiv:sha256, Key: 256 bits, Password hashing: ripemd160
LUKS1: aes-xts-plain64, Key: 256 bits, LUKS header hashing: sha256, RNG: /dev/urandom

on Ubuntu LTS 18.04:
cryptsetup --help

Default compiled-in device cipher parameters:
        loop-AES: aes, Key 256 bits
        plain: aes-cbc-essiv:sha256, Key: 256 bits, Password hashing: ripemd160
        LUKS1: aes-xts-plain64, Key: 256 bits, LUKS header hashing: sha256, RNG: /dev/urandom

user@server:~$ cat /proc/crypto  | grep aes
name         : cmac(aes)
driver       : cmac(aes-aesni)
name         : __xts(aes)
driver       : cryptd(__xts-aes-aesni)
name         : pcbc(aes)
driver       : pcbc-aes-aesni
module       : aesni_intel
name         : fpu(pcbc(__aes))
driver       : fpu(pcbc(__aes-aesni))
module       : aesni_intel
name         : pcbc(__aes)
driver       : pcbc(__aes-aesni)
name         : xts(aes)
driver       : xts-aes-aesni
module       : aesni_intel
name         : ctr(aes)
driver       : ctr-aes-aesni
module       : aesni_intel
name         : cbc(aes)
driver       : cbc-aes-aesni
module       : aesni_intel
name         : ecb(aes)
driver       : ecb-aes-aesni
module       : aesni_intel
name         : gcm(aes)
driver       : generic-gcm-aesni
module       : aesni_intel
name         : __generic-gcm-aes-aesni
driver       : __driver-generic-gcm-aes-aesni
module       : aesni_intel
name         : rfc4106(gcm(aes))
driver       : rfc4106-gcm-aesni
module       : aesni_intel
name         : __gcm-aes-aesni
driver       : __driver-gcm-aes-aesni
module       : aesni_intel
name         : __xts(aes)
driver       : __xts-aes-aesni
module       : aesni_intel
name         : __ctr(aes)
driver       : __ctr-aes-aesni
module       : aesni_intel
name         : __cbc(aes)
driver       : __cbc-aes-aesni
module       : aesni_intel
name         : __ecb(aes)
driver       : __ecb-aes-aesni
module       : aesni_intel
name         : __aes
driver       : __aes-aesni
module       : aesni_intel
name         : aes
driver       : aes-aesni
module       : aesni_intel
name         : aes
driver       : aes-asm
module       : aes_x86_64
driver       : drbg_nopr_ctr_aes256
driver       : drbg_nopr_ctr_aes192
driver       : drbg_nopr_ctr_aes128
driver       : drbg_pr_ctr_aes256
driver       : drbg_pr_ctr_aes192
driver       : drbg_pr_ctr_aes128
name         : aes
driver       : aes-generic

george1421

@humoss233 OK for the post init script, can you document the steps needed to activate that volume?

Maybe something before you create the postinit script is to pxe boot into a debug capture/ or deploy what ever action you want to do. Then manually activate that disk using your commands. And finally launch the imaging script with fog. You will have to press enter at each step, but this way you can capture any error messages if any. If it captures OK then you can take the steps to activate it and place it in a bash script in the /images/dev/postinit scripts directory. And then finally hook your bash script into the fog.postinit master script.

humoss233

@george1421 I followed your instructions, but I keep running into an error after typing in “fog.” Maybe it’s because I set Host Primary Disk to /dev/mapper/crypt (which I confirm exists after using cryptsetup). Error message, commands, and host/image settings below.

mdadm -D /dev/md126
cryptsetup luksOpen /dev/md126p3 crypt
fog

george1421

@humoss233 Well I guess a few things here.

The kernel parameters are not complete for some reason. There is a variable mode or something (like that) that should be up or down depending on if you are capturing or deploying.

so after running the cryptsetup, what does lsblk show? What happens if you manually try to mount that encrypted partition over /mnt can you read the partition contents?

Does this command cryptsetup luksOpen /dev/md126p3 crypt create a device called /dev/crypt?

If so /dev/crypt should represent an encrypted partition /dev/md126p3 and not the physical disk /dev/md126.

Understand we have not worked with encrypted partitions so we have to rely on your knowledge of the filesystem.

humoss233

@george1421 /dev/mapper/crypt is created, not /dev/crypt. Cryptsetup uses device mapper to create a mapped decrypted partition. I can mount this decrypted partition using mount /dev/mapper/crypt /mnt/temp and successfully view all the files on the partition. This is why I thought it’d work to use /dev/mapper/crypt in the “Host Primary Disk” field. Could FOS be confused because it expects to find a disk device and not a partition?

I’m not sure re: kernel parameters. This is a capture in debug mode. I’ve successfully completed captures of the full encrypted partition without debug mode (using /dev/md126 as “Host Primary Disk”). So, I’m not sure if missing parameters are contributing to the error.

See: https://wiki.archlinux.org/index.php/Dm-crypt/Device_encryption#Unlocking/Mapping_LUKS_partitions_with_the_device_mapper

See lsblk output below:

Sebastian Roth

@humoss233 Great to see George has come up with the correct set of kernel options for your crypto setup.

At this point I think we are hitting kind of a wall. We might find a hole through but I am not sure yet.

FOG is made to capture whole disks, so one of the first things it does is get a list of partitions from the device. This surely fails on /dev/mapper/crypto. There is an option in FOG that you can use to make it capture only one single partition (in the host’s settings you have Partition - defaults to Everything) but the script code as it exists right now would still try to enumerate the partitions and bail out.

So looking at your lsblk output my first idea was to set Host Primary Disk to /dev/md126 and create a symbolic link pointing from md126p3 to mapper/crypt. But that doesn’t work because /dev/md126p3 device file already exists. Hmmmm, well maybe you can delete it. It’s not an issue in the live FOS Linux because on reboot it will be restored. Try this:

mdadm -D /dev/md126
cryptsetup luksOpen /dev/md126p3 crypt
rm /dev/md126p3
ln -s /dev/mapper/crypt /dev/md126p3
fog

Quazz

It also looks like OP is using mdraid, not sure if specifying a disk will produce the desired results under those circumstances anyway. Though; I don’t know at all how that’s handled behind the screens so it could be no problem at all.

humoss233

@Sebastian-Roth clever hack! there was one more hurdle: blockdev --rereadpt in the runPartprobe function fails due to ioctl error on BLKRRPART: Device or resource busy because cryptsetup luksOpen appears to be locking the device. Luckily partprobe works fine, so I just replaced that part of the script. Here’s my final commands (the last line just shows that the line has been replaced successfully). After running fog, the decrypted partition/disk is successfully captured (with /dev/md126 as “Host Primary Disk”). 1 GB instead of 800 GB!

george1421

@humoss233 I’m not sure this will work, since you are linking the /dev/md126p3_crypt to /dev/md126p3 then deleted it and then recreating it as itself. You are kind of looping back to itself. I can see a circular link here.

I wonder if you can rename /dev/md126p3 right from the start to /dev/md126p3raw and then do your cryptsetup against the renamed raw device and linking.

humoss233

@george1421 That’s a good point and your method is safer, but the one that I’m using (from @Sebastian-Roth) also works - I unzipped and mounted the resulting .img file to make sure it’s good. It’s beyond me but cryptsetup must work in a way that once the decrypted partition is mapped, it’s no longer dependent on the device file representation.

Now I just need to think of a clever way of prompting for and transmitting the password over the network, as I’d rather not put the plaintext pass in the postinit script.

Both of you, thanks very much for your help!!

george1421

@humoss233 Hmm… pass-o-words…

How about an encrypted password passed as a kernel parameter to FOS Linux bzImage, then in your postinit script decode the password using local seed (same one used to encrypt the password).

Error decrypting LUKS partition prior to capture/imaging

176

11.6k

17.1k

154.6k