Error decrypting LUKS partition prior to capture/imaging

humoss233 · Oct 20, 2019, 2:29 PM

@george1421 Thanks for your comprehensive treatment of this topic! Absolutely re: getting it working manually before automating with init scripts.

It sounds like dm crypt is not something that can be enabled with a flag? If not, I could try to rebuild the kernel if it’s a simple one-liner tweak somewhere here https://github.com/FOGProject/fos for example?

george1421 · Oct 20, 2019, 9:49 AM

@humoss233 I recompiled the kernel last night after my post with the dm_crypt enabled. Give me a minute and I’ll upload it where you can get to it.

https://drive.google.com/open?id=1N6q6Oqmi7W7WkdtNPK1H0O8B1f-a4RFU
Note to future readers I may remove this file at any time so the link may not be valid in the future

To use this new kernel, download it from the link and save it in /var/www/html/fog/service/ipxe directory on the fog server as bzImageCrypt Then manually register one host and then go into the web ui in the host management for this target system. Update the kernel field with bzImageCrypt (watch the case because it IS important). Save the host management page and then schedule another debug capture. Then test your commands again with the modules loaded into the kernel.

ref Kernel patch file for differences between standard config and dm_crypt added config

--- kernelx64.config    2019-08-29 12:46:58.222184653 -0400
+++ .config     2019-10-20 00:20:29.579817034 -0400
@@ -1273,12 +1273,17 @@
 # CONFIG_BCACHE is not set
 CONFIG_BLK_DEV_DM_BUILTIN=y
 CONFIG_BLK_DEV_DM=y
-# CONFIG_DM_MQ_DEFAULT is not set
+CONFIG_DM_MQ_DEFAULT=y
 # CONFIG_DM_DEBUG is not set
-# CONFIG_DM_UNSTRIPED is not set
-# CONFIG_DM_CRYPT is not set
-# CONFIG_DM_SNAPSHOT is not set
-# CONFIG_DM_THIN_PROVISIONING is not set
+CONFIG_DM_BUFIO=y
+CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING=y
+CONFIG_DM_DEBUG_BLOCK_STACK_TRACING=y
+CONFIG_DM_BIO_PRISON=y
+CONFIG_DM_PERSISTENT_DATA=y
+CONFIG_DM_UNSTRIPED=y
+CONFIG_DM_CRYPT=y
+CONFIG_DM_SNAPSHOT=y
+CONFIG_DM_THIN_PROVISIONING=y
 # CONFIG_DM_CACHE is not set
 # CONFIG_DM_WRITECACHE is not set
 # CONFIG_DM_ERA is not set
@@ -3424,8 +3429,6 @@
 CONFIG_HAVE_ARCH_KASAN=y
 # CONFIG_KASAN is not set
 CONFIG_ARCH_HAS_KCOV=y
-CONFIG_CC_HAS_SANCOV_TRACE_PC=y
-# CONFIG_KCOV is not set
 # CONFIG_DEBUG_SHIRQ is not set

humoss233 · Oct 21, 2019, 9:18 PM

@george1421 thanks for uploading this! I just got a chance to try it out. It fails but a different error this time, so I think we’re getting closer.

[Mon Oct 21 root@fogclient /]# cryptsetup luksOpen /dev/md126p3 crypt
Enter passphrase for /dev/md126p3:
device-mapper: table: 251:0: crypt: Error allocating crypto tfm
reload ioctl on    failed: No such file or directory
Failed to setup dm-crypt key mapping for device /dev/md126p3
Check that the kernel supports aes-xts-plain64 cipher (check syslog for more info).

From /var/log/messages:

Oct 21 21:03:16 fogclient user.err kernel: device-mapper: table: 251:0: crypt: Error allocating crypto tfm
Oct 21 21:03:16 fogclient user.warn kernel: device-mapper: ioctl: error adding target to table

Doing some research, it looks like a module may be missing (“No such file or directory”). Could it be that the module for the cipher is missing? I’m currently using cryptsetup default options (which uses aes-xts-plain64 as cipher). When I try cat crypto | grep aes on this FOS build, I only see aes-generic whereas I see ~20 options when doing the same on my Ubuntu server (including xts-aes-aesni).

Let me know if I can do anything to help debug further.

Some links referencing similar error messages out there:

https://www.raspberrypi.org/forums/viewtopic.php?t=60278
https://bugs.centos.org/view.php?id=9311

george1421 · Oct 21, 2019, 9:46 PM

@humoss233 OK I do see some crypto parameters not enabled in the kernel.

CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_TI is not set
# CONFIG_CRYPTO_AES_X86_64 is not set
# CONFIG_CRYPTO_AES_NI_INTEL is not set
# CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64 is not set
# CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64 is not set

if you could run cat /proc/crypto | grep aes on both fos linux and the system where the it works. Or is that where you posted above the cat crypto | grep aes above?

Sebastian Roth · Oct 21, 2019, 9:49 PM

@humoss233 As well run cryptsetup --help and check the last couple of lines for cipher information (from https://superuser.com/questions/1039487/check-that-kernel-supports-aes-xts-plain64-cipher).

george1421 · Oct 21, 2019, 3:54 PM

@humoss233 I added in aes ni and recompiled it here: https://drive.google.com/open?id=1N6q6Oqmi7W7WkdtNPK1H0O8B1f-a4RFU

--- kernelx64.config    2019-08-29 12:46:58.222184653 -0400
+++ .config     2019-10-20 00:20:29.579817034 -0400
@@ -1273,12 +1273,17 @@
 # CONFIG_BCACHE is not set
 CONFIG_BLK_DEV_DM_BUILTIN=y
 CONFIG_BLK_DEV_DM=y
-# CONFIG_DM_MQ_DEFAULT is not set
+CONFIG_DM_MQ_DEFAULT=y
 # CONFIG_DM_DEBUG is not set
-# CONFIG_DM_UNSTRIPED is not set
-# CONFIG_DM_CRYPT is not set
-# CONFIG_DM_SNAPSHOT is not set
-# CONFIG_DM_THIN_PROVISIONING is not set
+CONFIG_DM_BUFIO=y
+CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING=y
+CONFIG_DM_DEBUG_BLOCK_STACK_TRACING=y
+CONFIG_DM_BIO_PRISON=y
+CONFIG_DM_PERSISTENT_DATA=y
+CONFIG_DM_UNSTRIPED=y
+CONFIG_DM_CRYPT=y
+CONFIG_DM_SNAPSHOT=y
+CONFIG_DM_THIN_PROVISIONING=y
 # CONFIG_DM_CACHE is not set
 # CONFIG_DM_WRITECACHE is not set
 # CONFIG_DM_ERA is not set
@@ -3135,10 +3140,12 @@
 CONFIG_CRYPTO_NULL2=y
 # CONFIG_CRYPTO_PCRYPT is not set
 CONFIG_CRYPTO_WORKQUEUE=y
-# CONFIG_CRYPTO_CRYPTD is not set
+CONFIG_CRYPTO_CRYPTD=y
 # CONFIG_CRYPTO_MCRYPTD is not set
 CONFIG_CRYPTO_AUTHENC=y
 # CONFIG_CRYPTO_TEST is not set
+CONFIG_CRYPTO_SIMD=y
+CONFIG_CRYPTO_GLUE_HELPER_X86=y

 #
 # Authenticated Encryption with Associated Data
@@ -3220,8 +3227,8 @@
 #
 CONFIG_CRYPTO_AES=y
 # CONFIG_CRYPTO_AES_TI is not set
-# CONFIG_CRYPTO_AES_X86_64 is not set
-# CONFIG_CRYPTO_AES_NI_INTEL is not set
+CONFIG_CRYPTO_AES_X86_64=y
+CONFIG_CRYPTO_AES_NI_INTEL=y
 # CONFIG_CRYPTO_ANUBIS is not set
 CONFIG_CRYPTO_ARC4=y
 # CONFIG_CRYPTO_BLOWFISH is not set
@@ -3424,8 +3431,6 @@
 CONFIG_HAVE_ARCH_KASAN=y
 # CONFIG_KASAN is not set
 CONFIG_ARCH_HAS_KCOV=y
-CONFIG_CC_HAS_SANCOV_TRACE_PC=y
-# CONFIG_KCOV is not set
 # CONFIG_DEBUG_SHIRQ is not set

 #
@@ -3460,7 +3465,7 @@
 # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
 # CONFIG_LOCK_TORTURE_TEST is not set
 # CONFIG_WW_MUTEX_SELFTEST is not set
-# CONFIG_STACKTRACE is not set
+CONFIG_STACKTRACE=y
 # CONFIG_WARN_ALL_UNSEEDED_RANDOM is not set
 # CONFIG_DEBUG_KOBJECT is not set
 # CONFIG_DEBUG_BUGVERBOSE is not set

Sebastian Roth · Oct 21, 2019, 9:52 PM

@george1421 As well there should be CONFIG_CRYPTO_XTS (see https://cateee.net/lkddb/web-lkddb/CRYPTO_XTS.html) - but you need to enable CONFIG_EXPERIMENTAL for that option to show up.

george1421 · Oct 21, 2019, 4:02 PM

@george1421 With XTS kernel module too: https://drive.google.com/open?id=1N6q6Oqmi7W7WkdtNPK1H0O8B1f-a4RFU

Edit: We may not be done yet depending on the password hash you used ref: https://lists.gt.net/gentoo/user/300718

humoss233 · Oct 21, 2019, 10:34 PM

@george1421 I tried the version with the XTS kernel module and it works! luksOpen works without errors and I can also mount the decrypted partition and see my files. Now I just need to setup a post init script.

@george1421 @Sebastian-Roth
Debugging info on FOS (latest bzImageCrypt):
cryptsetup --help

loop-AES: aes, Key 256 bits
plain: aes-cbc-essiv:sha256, Key: 256 bits, Password hashing: ripemd160
LUKS1: aes-xts-plain64, Key: 256 bits, LUKS header hashing: sha256, RNG: /dev/urandom

on Ubuntu LTS 18.04:
cryptsetup --help

Default compiled-in device cipher parameters:
        loop-AES: aes, Key 256 bits
        plain: aes-cbc-essiv:sha256, Key: 256 bits, Password hashing: ripemd160
        LUKS1: aes-xts-plain64, Key: 256 bits, LUKS header hashing: sha256, RNG: /dev/urandom

user@server:~$ cat /proc/crypto  | grep aes
name         : cmac(aes)
driver       : cmac(aes-aesni)
name         : __xts(aes)
driver       : cryptd(__xts-aes-aesni)
name         : pcbc(aes)
driver       : pcbc-aes-aesni
module       : aesni_intel
name         : fpu(pcbc(__aes))
driver       : fpu(pcbc(__aes-aesni))
module       : aesni_intel
name         : pcbc(__aes)
driver       : pcbc(__aes-aesni)
name         : xts(aes)
driver       : xts-aes-aesni
module       : aesni_intel
name         : ctr(aes)
driver       : ctr-aes-aesni
module       : aesni_intel
name         : cbc(aes)
driver       : cbc-aes-aesni
module       : aesni_intel
name         : ecb(aes)
driver       : ecb-aes-aesni
module       : aesni_intel
name         : gcm(aes)
driver       : generic-gcm-aesni
module       : aesni_intel
name         : __generic-gcm-aes-aesni
driver       : __driver-generic-gcm-aes-aesni
module       : aesni_intel
name         : rfc4106(gcm(aes))
driver       : rfc4106-gcm-aesni
module       : aesni_intel
name         : __gcm-aes-aesni
driver       : __driver-gcm-aes-aesni
module       : aesni_intel
name         : __xts(aes)
driver       : __xts-aes-aesni
module       : aesni_intel
name         : __ctr(aes)
driver       : __ctr-aes-aesni
module       : aesni_intel
name         : __cbc(aes)
driver       : __cbc-aes-aesni
module       : aesni_intel
name         : __ecb(aes)
driver       : __ecb-aes-aesni
module       : aesni_intel
name         : __aes
driver       : __aes-aesni
module       : aesni_intel
name         : aes
driver       : aes-aesni
module       : aesni_intel
name         : aes
driver       : aes-asm
module       : aes_x86_64
driver       : drbg_nopr_ctr_aes256
driver       : drbg_nopr_ctr_aes192
driver       : drbg_nopr_ctr_aes128
driver       : drbg_pr_ctr_aes256
driver       : drbg_pr_ctr_aes192
driver       : drbg_pr_ctr_aes128
name         : aes
driver       : aes-generic

george1421 · Oct 21, 2019, 10:44 PM

@humoss233 OK for the post init script, can you document the steps needed to activate that volume?

Maybe something before you create the postinit script is to pxe boot into a debug capture/ or deploy what ever action you want to do. Then manually activate that disk using your commands. And finally launch the imaging script with fog. You will have to press enter at each step, but this way you can capture any error messages if any. If it captures OK then you can take the steps to activate it and place it in a bash script in the /images/dev/postinit scripts directory. And then finally hook your bash script into the fog.postinit master script.

humoss233 · Oct 22, 2019, 1:54 AM

@george1421 I followed your instructions, but I keep running into an error after typing in “fog.” Maybe it’s because I set Host Primary Disk to /dev/mapper/crypt (which I confirm exists after using cryptsetup). Error message, commands, and host/image settings below.

mdadm -D /dev/md126
cryptsetup luksOpen /dev/md126p3 crypt
fog

george1421 · Oct 22, 2019, 2:10 AM

@humoss233 Well I guess a few things here.

The kernel parameters are not complete for some reason. There is a variable mode or something (like that) that should be up or down depending on if you are capturing or deploying.

so after running the cryptsetup, what does lsblk show? What happens if you manually try to mount that encrypted partition over /mnt can you read the partition contents?

Does this command cryptsetup luksOpen /dev/md126p3 crypt create a device called /dev/crypt?

If so /dev/crypt should represent an encrypted partition /dev/md126p3 and not the physical disk /dev/md126.

Understand we have not worked with encrypted partitions so we have to rely on your knowledge of the filesystem.

humoss233 · Oct 21, 2019, 8:32 PM

@george1421 /dev/mapper/crypt is created, not /dev/crypt. Cryptsetup uses device mapper to create a mapped decrypted partition. I can mount this decrypted partition using mount /dev/mapper/crypt /mnt/temp and successfully view all the files on the partition. This is why I thought it’d work to use /dev/mapper/crypt in the “Host Primary Disk” field. Could FOS be confused because it expects to find a disk device and not a partition?

I’m not sure re: kernel parameters. This is a capture in debug mode. I’ve successfully completed captures of the full encrypted partition without debug mode (using /dev/md126 as “Host Primary Disk”). So, I’m not sure if missing parameters are contributing to the error.

See: https://wiki.archlinux.org/index.php/Dm-crypt/Device_encryption#Unlocking/Mapping_LUKS_partitions_with_the_device_mapper

See lsblk output below:

Sebastian Roth · Oct 22, 2019, 6:18 AM

@humoss233 Great to see George has come up with the correct set of kernel options for your crypto setup.

At this point I think we are hitting kind of a wall. We might find a hole through but I am not sure yet.

FOG is made to capture whole disks, so one of the first things it does is get a list of partitions from the device. This surely fails on /dev/mapper/crypto. There is an option in FOG that you can use to make it capture only one single partition (in the host’s settings you have Partition - defaults to Everything) but the script code as it exists right now would still try to enumerate the partitions and bail out.

So looking at your lsblk output my first idea was to set Host Primary Disk to /dev/md126 and create a symbolic link pointing from md126p3 to mapper/crypt. But that doesn’t work because /dev/md126p3 device file already exists. Hmmmm, well maybe you can delete it. It’s not an issue in the live FOS Linux because on reboot it will be restored. Try this:

mdadm -D /dev/md126
cryptsetup luksOpen /dev/md126p3 crypt
rm /dev/md126p3
ln -s /dev/mapper/crypt /dev/md126p3
fog

Quazz · Oct 22, 2019, 3:50 AM

It also looks like OP is using mdraid, not sure if specifying a disk will produce the desired results under those circumstances anyway. Though; I don’t know at all how that’s handled behind the screens so it could be no problem at all.

humoss233 · Oct 22, 2019, 9:29 AM

@Sebastian-Roth clever hack! there was one more hurdle: blockdev --rereadpt in the runPartprobe function fails due to ioctl error on BLKRRPART: Device or resource busy because cryptsetup luksOpen appears to be locking the device. Luckily partprobe works fine, so I just replaced that part of the script. Here’s my final commands (the last line just shows that the line has been replaced successfully). After running fog, the decrypted partition/disk is successfully captured (with /dev/md126 as “Host Primary Disk”). 1 GB instead of 800 GB!

george1421 · Oct 22, 2019, 9:44 AM

@humoss233 I’m not sure this will work, since you are linking the /dev/md126p3_crypt to /dev/md126p3 then deleted it and then recreating it as itself. You are kind of looping back to itself. I can see a circular link here.

I wonder if you can rename /dev/md126p3 right from the start to /dev/md126p3raw and then do your cryptsetup against the renamed raw device and linking.

humoss233 · Oct 22, 2019, 1:11 PM

@george1421 That’s a good point and your method is safer, but the one that I’m using (from @Sebastian-Roth) also works - I unzipped and mounted the resulting .img file to make sure it’s good. It’s beyond me but cryptsetup must work in a way that once the decrypted partition is mapped, it’s no longer dependent on the device file representation.

Now I just need to think of a clever way of prompting for and transmitting the password over the network, as I’d rather not put the plaintext pass in the postinit script.

Both of you, thanks very much for your help!!

george1421 · Oct 22, 2019, 7:55 PM

@humoss233 Hmm… pass-o-words…

How about an encrypted password passed as a kernel parameter to FOS Linux bzImage, then in your postinit script decode the password using local seed (same one used to encrypt the password).

humoss233 · Oct 24, 2019, 4:42 PM

@george1421 that’s a good idea - I’ve been researching it, but it looks like openssl is not available in FOS. Is there another way available to decrypt a given cipher?

Error decrypting LUKS partition prior to capture/imaging

162

12.0k

17.3k

155.2k