BTRFS postdownloadscript

[mstabrin]

@Tom-Elliott Thank you for this amazing information I will directly try it tomorrow!

Regarding the 95%: Yes it seems that when I used 100% it could not resize it. 95% seemed to work in my case though. Possibly due to some overhead when the data is fragmented on the drive. This incremental test to resize was recommended in the post I linked.

Looking forward to work with you on the topic

@george1421 I asked for a debug task on the machine and it is booting into it.
I am using a virtual machine from vmware, maybe that is the reason for the missing information?

[Sebastian Roth]

@mstabrin @Tom-Elliott Seems like you have pushed this a lot forward. Well done.

Just allow me one comment about the code:

while ! btrfs filesystem resize ...

Having worked on those kind of scripts for a long time this looks really scary to me. Running such a resize command in a loop will be causing trouble sooner or later I am sure. I am wondering why this command should return with a non-zero return code but still should be called again. Does btrfs resize need to be done in steps. I really doubt so.

[Sebastian Roth]

@Tom-Elliott One more thing I just remembered. We do have a “global” setting to set the percentage of how small we size down the partition (code ref for NTFS and EXT) and I think we should go along the same line with btrfs as well.

The percentage can be set via FOG web UI -> FOG Configuration -> FOG Settings -> General Settings -> CAPTURERESIZEPCT

[Tom Elliott]

@sebastian-roth I knew about the capture resize pct issue, and just was verifying with mstabrin that everything was configured appropriately and the 95% was the use of free space to shrink to.

Modified with using the database capture resize percent is now used here:

diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
index c754273..4d069b4 100755
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
@@ -58,8 +58,9 @@ beginUpload() {
             dots "Getting Windows/Linux Partition Count"
             countPartTypes "$hd" "ntfs" "ntfscnt"
             countPartTypes "$hd" "extfs" "extfscnt"
+            countPartTypes "$hd" "btrfs" "btrfscnt"
             countPartTypes "$hd" "*" "partscnt"
-            if [[ $ntfscnt -eq 0 && $extfscnt -eq 0 ]]; then
+            if [[ $ntfscnt -eq 0 && $extfscnt -eq 0 && $btrfscnt -eq 0 ]]; then
                 echo "Failed"
                 debugPause
                 handleError "No resizable partitions found ($0)\n   Args Passed: $*"
@@ -70,6 +71,8 @@ beginUpload() {
             debugPause
             echo " * EXTFS Partition count of: $extfscnt"
             debugPause
+            echo " * BTRFS Partition count of: $btrfscnt"
+            debugPause
             echo " * Total Partition count of: $partscnt"
             debugPause
             case $osid in
diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
index 959be5f..b29e5ca 100644
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
@@ -258,6 +258,31 @@ expandPartition() {
                     ;;
             esac
             ;;
+        btrfs)
+            # Based on info from @mstabrin on forums.fogproject.org
+            if [[ ! -d /tmp/btrfs ]]; then
+                mkdir /tmp/btrfs >>/tmp/btfrslog.txt 2>&1
+                if [[$? -gt 0 ]]; then
+                    echo "Failed"
+                    debugPause
+                    handleError "Could not create /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+                fi
+            fi
+            mount $part /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
+            if [[ $? -gt 0 ]]; then
+                echo "Failed"
+                debugPause
+                handleError "Could not mount $part to /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+            fi
+            btrfs filesystem resize max /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
+            if [[ $? -gt 0 ]]; then
+                echo "Failed"
+                debugPause
+                handleError "Could not resize btrfs partition (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+            fi
+            umount $part /tmp/btrfs >/dev/null 2>&1
+            echo "Done"
+            ;;
         *)
             echo " * Not expanding ($part -- $fstype)"
             debugPause
@@ -597,6 +622,32 @@ shrinkPartition() {
                     ;;
             esac
             ;;
+        btrfs)
+            # Based on info from @mstabrin on forums.fogproject.org
+            # https://forums.fogproject.org/topic/15159/btrfs-postdownloadscript/3
+            if [[ ! -d /tmp/btrfs ]]; then
+                mkdir /tmp/btrfs >>/tmp/btfrslog.txt 2>&1
+                if [[$? -gt 0 ]]; then
+                    echo "Failed"
+                    debugPause
+                    handleError "Could not create /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+                fi
+            fi
+            mount $part /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
+            if [[ $? -gt 0 ]]; then
+                echo "Failed"
+                debugPause
+                handleError "Could not mount $part to /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+            fi
+            local free_size_original=$(btrfs filesystem usage -b /tmp/btrfs | grep unallocated | grep -Eo '[0-9]+')
+            local fsize_pct=$(calculate "${percent}/100")
+            local mult_val=$(calculate "1-${fsize_pct}")
+            local free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
+            while ! btrfs filesystem resize -${free_size} /tmp/btrfs; do
+                [[ ${mult_val} -le 0 ]] && break || mult_val=$(${mult_val} - ${fsize_pct})
+                free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
+            done
+            ;;
         *)
             echo " * Not shrinking ($part $fstype)"
             ;;

Then for the while loop, I’m with you, but this is according to the following article:
https://unix.stackexchange.com/questions/424758/resize-btrfs-filesystem-to-the-minimum-size-in-a-single-step

Basically it shrinks and continues to shrink in steps until it cannot shrink. BTRFS is not very nice about shrinking. You can expand until your drive is in the middle of the yard, but shrinking, while now possible, is still relatively new to BTRFS. It is one of the reasons we didn’t add BTRFS to the shrinking when BTRFS started to become popular. It literally could not be shrunk.

[Sebastian Roth]

@tom-elliott @mstabrin In that stackexchange topic they mention while sudo btrfs filesystem resize .... Why is while ! btrfs filesystem resize ... used in this script - reversed logic?

@mstabrin said:

Populating /dev using udev: udevd[3160]: failed to execute ‘/lib/udev/${exec_prefix}/bin/udevadm’ ‘${exec_prefix}/bin/udevadm trigger -s block -p ID_BTRFS_READY=0’: No such file or directory.

Good you posted the pictures. I didn’t expect that message to come from our FOS Linux. Haven’t seen this message before I think. It’s probably to do with us using partprobe a couple of times. No idea which file or directory is not found. I think it’s save to ignore those for now. Seems to be a known issue in udev rules provided by eudev package used by buildroot which we build our inits from.

[Tom Elliott]

@sebastian-roth I don’t know, @mstabrin had already tested it. I don’t have any BTRFS to test with, so I just used what was working (with minor adjustments for error controls) I don’t really know why while ! btrfs filesystem resize … is used, I just knew it was already working at least in this case.

[mstabrin]

Hello @Sebastian-Roth ,

Populating /dev using udev: udevd[3160]: failed to execute ‘/lib/udev/${exec_prefix}/bin/udevadm’ ‘${exec_prefix}/bin/udevadm trigger -s block -p ID_BTRFS_READY=0’: No such file or directory.

I realised that there are similiar udev related messages in the logs of our VMs and I checked the other VMs and this seems to be a vmware related issue but should not cause too much trouble

Feb 10 08:22:40 fog-server2021 multipathd[745]: sda: add missing path
Feb 10 08:22:40 fog-server2021 multipathd[745]: sda: failed to get udev uid: Invalid argument
Feb 10 08:22:40 fog-server2021 multipathd[745]: sda: failed to get sysfs uid: Invalid argument
Feb 10 08:22:40 fog-server2021 multipathd[745]: sda: failed to get sgio uid: No such file or directory

Maybe it is related?

Regarding while ! btrfs filesystem resize ...:

My reasoning was simply based on the assumption that it happens more often that the unallocated value is not too far away from the truth than the other way around.
Therefore, it would require less steps to reduce the value of provided unallocated free space to a value that it works, than incrementally decreasing the overall free space until it no longer works, i.e., I am asking for: do you have 4GB free? no? Do you have 3 GB free? instead of: do you still have 200MB free? yes? do you still have 200 MB free? As suggested in the post.

While the latter method ensures that you really shrink the partition to its minimum (with a 200MB margin in this case) for my Ubuntu golden image with 5GB occupied on an at least 50GB hard drive I thought that even 10GB occupied should be enough for imaging (and later deployment on other at least 50GB hard drives) since it is expanded anyway during deployment.
As @Tom-Elliott said, shrinking is usually not what btrfs is good at due to fragmentation of snapshots in the images as far as I know. Therefore, I fear that an incremental search is the way to go in this case and a loop unavoidable.

But of course it makes sense to optimise the space usage since 95% of a huge drive can of course be larger than the total capacity of a small one.
I did not know about the global CAPTURERESIZEPCT value though and that makes things more consistent
So we should be able to refine the method once things are working properly from the shrink / expand perspective to optimise the workflow

@Tom-Elliott Thank you for the coding, I will give it a try now fingers crossed

[mstabrin]

Hello everybody,
I needed to get myself into the FOS environment yesterday, BUT it works now (At least for my usecase )

@george1421

Are you using a FOS Linux USB boot drive? I would expect to only see the “Unknown request type :: Null” when booted from a USB stick into FOS Linux without having a corresponding task scheduled on the FOG server. I also noticed that your kernel parameters are incomplete for a capture or deploy (which would also lead to that error message).

This happened because I used the normal Debug task and not the Capture - Debug task.
Using the Caputer - Debug made things work as you described them

@Tom-Elliott
I needed to make some minor changes to your suggested changes, but it was amazingly helpful to know which parts to look at

funcs.sh (based on what you send me):

--- funcs.sh_ori	2021-02-10 13:25:08.000000000 +0100
+++ funcs.sh	2021-02-11 09:08:31.660652070 +0100
@@ -255,15 +255,15 @@
                     fi
                     echo "Done"
                     ;;
             esac
             ;;
-        extfs)
+        btrfs)
             # Based on info from @mstabrin on forums.fogproject.org
             if [[ ! -d /tmp/btrfs ]]; then
                 mkdir /tmp/btrfs >>/tmp/btfrslog.txt 2>&1
-                if [[$? -gt 0 ]]; then
+                if [[ $? -gt 0 ]]; then
                     echo "Failed"
                     debugPause
                     handleError "Could not create /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
                 fi
             fi
@@ -625,11 +625,11 @@
         btrfs)
             # Based on info from @mstabrin on forums.fogproject.org
             # https://forums.fogproject.org/topic/15159/btrfs-postdownloadscript/3
             if [[ ! -d /tmp/btrfs ]]; then
                 mkdir /tmp/btrfs >>/tmp/btfrslog.txt 2>&1
-                if [[$? -gt 0 ]]; then
+                if [[ $? -gt 0 ]]; then
                     echo "Failed"
                     debugPause
                     handleError "Could not create /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
                 fi
             fi
@@ -638,16 +638,18 @@
                 echo "Failed"
                 debugPause
                 handleError "Could not mount $part to /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
             fi
             local free_size_original=$(btrfs filesystem usage -b /tmp/btrfs | grep unallocated | grep -Eo '[0-9]+')
-            local mult_val=0.95
-            local free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
-            while ! btrfs filesystem resize -${free_size} /tmp/btrfs; do
-                [[ ${mult_val} -le 0 ]] && break || mult_val=$(${mult_val} - 0.05)
-                free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
+            local fsize_pct=$(calculate_float "${percent}/100")
+            local mult_val=$(calculate_float "1-${fsize_pct}")
+            local free_size=$(calculate "${mult_val}*${free_size_original}")
+            while ! btrfs filesystem resize -${free_size} /tmp/btrfs >>/tmp/btrfslog.txt 2>&1; do
+            [[ $(echo "${mult_val} <= 0" | bc -l) -gt 0 ]] && break || mult_val=$(calculate_float "${mult_val}-0.05")
+                free_size=$(calculate "${mult_val}*${free_size_original}")
             done
+            umount /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
             ;;
         *)
             echo " * Not shrinking ($part $fstype)"
             ;;
     esac
@@ -2421,5 +2423,8 @@
 }
 # Calculates information
 calculate() {
     echo $(awk 'BEGIN{printf "%.0f\n", '$*'}')
 }
+calculate_float() {
+    echo $(awk 'BEGIN{printf "%f\n", '$*'}')
+}

Small changelog:

• I needed to add a new calculate_float function to make the float calculations work
• Minor syntax problems in the if statements and missing whitespaces
• btrfs instead of extfs in shrinkPartition
• Fixed the if condition inside the loop.
• Used the calculate and calculate_float instead of my suggested bc -l
• Used a fixed decrement of 5%, as we try to be as close to the percent value as possible. Otherwise the mechanism would break if the percent value is large.
• Unmount the btrfs file system after shrinking to allow for partition shrinking later.

fog.sh (based on what you send me):

--- fog.upload_ori	2021-02-10 13:22:20.000000000 +0100
+++ fog.upload	2021-02-11 08:44:34.415100613 +0100
@@ -42,11 +42,12 @@
             part_number=0
             for part in $parts; do
                 fsTypeSetting "$part"
                 getPartitionNumber "$part"
                 case $fstype in
-                    ntfs|extfs)
+                    ntfs|extfs|btrfs)
+                    #ntfs|extfs)
                         continue
                         ;;
                     *)
                         fixed_size_partitions="$fixed_size_partitions:$part_number"
                         ;;

• Added btrfs to the shrinkable partition list

Thank you for all the help
If you need more information from me, feel free to ask

[Tom Elliott]

@mstabrin While this won’t make the inits readily available quite yet, I have decided to push the code into github. Thanks for the assist and finding of bugs, it made adding this feature quite a lot faster.

[mstabrin]

@Tom-Elliott I have to thank you for the fast response
I will keep the modifications intact for now and when I encounter problems using it I will come back to you

[Sebastian Roth]

@mstabrin said in BTRFS postdownloadscript:

I will keep the modifications intact for now and when I encounter problems using it I will come back to you

Just wondering if you had time to test this and encountered any issue so far?

It’s part of the official code and will be in the next release. So it would be good to have some confidence this is working as expected.

Thanks in advance!

[mstabrin]

Hello @sebastian-roth ,

So far I did not encounter any problems imaging and deploying resizable BTRFS images

However, I “only” tested simple examples, i.e., one SSD with EFI, BTRFS, SWAP partitioning.

But no problems with this setup.
However, more complex examples will follow soon and I will come back to you in case I encounter some problems.

[mstabrin]

@Tom-Elliott @george1421 @Sebastian-Roth A quick follow up, so I deployed the image to many different PCs now and so far I did only have minor issues which might not actually be related to BTRFS in the first place.

I used to have the partition order:

1. EFI
2. BTRFS
3. SWAP 2GB
   on a VM with a 50GB HDD.

Image options: Single Disk - Resizable - Everything

Everything worked great if I deployed to HDDs with >=50GB.
However, it failed when I tried the deployment to an HDD with 40GB.

It seemed to be something like: The SWAP starting block on the original HDD is on a starting block that is not even present on the 40GB HDD.

So I moved the partition order to:

1. EFI
2. SWAP 2GB
3. BTRFS

And everything works as intended.

I will play around with different approaches though.
Like separate the partitioning from the actual image deployment.
We have many different systems requiring different amounts of SWAP space and/or dual/triple/many boot.
So my idea would be to deploy the partitioning first and afterwards only image the BTRFS partition itself and distribute it to a specified aim partition on the aim PC.

Thanks again for the help!

[Sebastian Roth]

@mstabrin Good to hear the BTRFS resize code is working for you! Thanks again for your work on this.

For a long time FOG would not move partitions if they were recognized as “fixed” because moving e.g. a recovery partition in the old MBR days would render it useless because of static sector addressing in boot loaders and such. Now as we all move towards UEFI and GPT partition layouts we have updated this logic to handle “fixed” partitions as fixed in size but not fixed in position anymore. For more details read this topic: https://forums.fogproject.org/topic/15025/move-partitions-on-gpt-layouts-need-people-to-test

I did a test with Linux partition layout having SWAP at the end and it was able to shrink it down and deploy to a smaller size disk: https://forums.fogproject.org/post/141373

To make a long story short, download the latest inits from our webserver (64 bit, 32 bit) and you should have this feature included as well as your BTRFS resize.