https://news.fogproject.org/fog-1-3-5-and-client-0-11-11-officially-released/

https://news.fogproject.org/fog-1-5-0-rc-11/

TLDR; Rerun the fog installer if you have lost “Database Connectivity” to your fog server, or run the ALTER USER syntax shown below.

So Ubuntu 16, among others I suppose, enable a “security updates” to be applied automatically as a “default” to things. Why, well it makes it simpler to ensure your Ubuntu systems are in compliance and patched for any potential exploits. This causes unknown and unexpected issues.

I figured it’d be a safe thing to express that there could be problems (as many of you have already experienced) that when these updates go up (with or without your knowledge) it can break functionality in unexpected and inopportune ways.

The quickest fix is to simply rerun the fog installer which should correct the problem.

As a note, it seems this problem is specific only when the mysql account is the 'root' user AND the password is blank.

The “fix” if you must do it manually is to open a terminal and obtain root:
Super (Windows Key) + T then sudo -i (in most cases).

From there, open mysql with mysql -u root

NOTE: MySQL MUST be run with ROOT.

Run:

ALTER USER 'root'@'127.0.0.1' IDENTIFIED WITH mysql_native_password BY ''; AND
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '';

It’s okay if one of them fails. This is going to fix Most people’s issues.

I would highly recommend removing the unattended-upgrades as many of these “sudden” issues came as a security patch ubuntu pushed out. By default Ubuntu typically set’s this for you as enabled and it can cause havoc on you as you (the admin) may not have “done” anything.

To prevent this problem from happening in the future you could run:

apt-get -y remove unattended-upgrades (AS Root again).

FOG is still actively being developed. It’s not necessarily readily apparent, but we can assure you things are still being worked on. These updates may not be communicated in a way that everybody just knows, but can easily be seen if one were to look at our repository site.

Between our own schedules and lives, we can get very busy. We try to keep things updated and help out on the forums even during lull periods. This might mean we aren’t pushing an RC or release as frequently. It may mean we’re working on other things for the project, such as can be seen if looking at our github site.

Our forums are heavily active, and this should point as an indicator to our “status” as well.

If anybody would like to see an increase in developers donating their time to making this free software, consider donating either with monetary support or by spending personal time to help with development.

FOG is an open source project - it’s even in the name. It is driven by people donating their time and resources. The releases of FOG revolve around when developers can spare a few hours throughout the week. Sometimes that will mean releases will be further, sometimes that will mean releases will be faster. That’s just the nature of our project, and many other open source projects.

Hey everybody,

I know you see me here on occasion from time to time. Life decisions have made it more difficult for me to do things I would normally be doing. Rest assured, I am still around, and while I’m not quite as active as I was in the past, it’s not because I don’t want to be.

I had to move, and as part of that I have none of my normal development stuff readily available. Part of the move made me not have a laptop, until today.

I need to setup my dev environment again, so it may take a little bit, but I will be back up.

That’s correct. The main reason fog is constantly moving forward is because the codebase is improved upon. Major bugs tend to be addressed for the next release. We don’t do an LTS because there’s really two main people working on fog in a consistent manor. Those two are @Joe-Schmitt and myself. Debian and Libreoffice have the team too be able to perform such a feat. Their product is Opensource but they have an employment team which can afford them that luxury. FOG has a team but we make no money and as such are required to work full time jobs. We work on FOG in our free time. I’ve had the ability to even work on it from work because we used the software.

Maintaining many different versions is difficult. And we don’t have a support team. WYSIWYG and I think we’ve done pretty well on support, even if we don’t have the ability to do dedicated support for our product. 1.5 was a major step toward modernizing the GUI. 1.6 will vastly improve on this. It was only recently we kind of came up with a road map on how best to proceed. Of note, 1.5 will be maintained until 1.6 is released. 1.6 is focused on making he GUI much more modern. 1.7 will be focused mostly toward fixing and refactoring the FOG client. 1.8 will focus on making the FOS system more modular and usable. I don’t know yet for 1.9. 2.0 will bridge the gap for our rewrite based on the work from 1.5 and up. While we do plan to try to do backports where possible, it’s much easier to ask people to update to the latest version than it is to try to maintain many different versions with backports in mind. At least for what FOG does.

I doubt this will appease anybody, but it’s what I think needs to be said. We are working hard and provide support for our product as best we can. The community makes fogs support system, I think, one of the best around. Add to that and you can almost always have a developer working side by side to help and fix issues as they come up, I don’t think it’s unfair to ask users to update to a specific version. Even if there are bugs, we will always try to correct what we can, when we can. (And normally it’s a pretty quick turn around).

I’m not perfect and I’ll give you that. We don’t even have a test suite to know if things are working as intended. We have to rely on the community and suggestions are great, just understand our answers won’t always be what people want to hear.

https://news.fogproject.org/fog-1-4-0-officially-released/

https://news.fogproject.org/fog-1-4-4-officially-released/

https://news.fogproject.org/fog-1-5-0-rc-12/

https://news.fogproject.org/fog-1-5-3/

@ttrammell the bitlocker issue is strange because we do see this from time to time. The fix? manage-bde -off c:

This isn’t a tpm issue though. Typically its because Microsoft is essentially encrypting the free space.

@mstabrin While this won’t make the inits readily available quite yet, I have decided to push the code into github. Thanks for the assist and finding of bugs, it made adding this feature quite a lot faster.

@sebastian-roth I don’t know, @mstabrin had already tested it. I don’t have any BTRFS to test with, so I just used what was working (with minor adjustments for error controls) I don’t really know why while ! btrfs filesystem resize … is used, I just knew it was already working at least in this case.

@sebastian-roth I knew about the capture resize pct issue, and just was verifying with mstabrin that everything was configured appropriately and the 95% was the use of free space to shrink to.

Modified with using the database capture resize percent is now used here:

diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
index c754273..4d069b4 100755
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
@@ -58,8 +58,9 @@ beginUpload() {
             dots "Getting Windows/Linux Partition Count"
             countPartTypes "$hd" "ntfs" "ntfscnt"
             countPartTypes "$hd" "extfs" "extfscnt"
+            countPartTypes "$hd" "btrfs" "btrfscnt"
             countPartTypes "$hd" "*" "partscnt"
-            if [[ $ntfscnt -eq 0 && $extfscnt -eq 0 ]]; then
+            if [[ $ntfscnt -eq 0 && $extfscnt -eq 0 && $btrfscnt -eq 0 ]]; then
                 echo "Failed"
                 debugPause
                 handleError "No resizable partitions found ($0)\n   Args Passed: $*"
@@ -70,6 +71,8 @@ beginUpload() {
             debugPause
             echo " * EXTFS Partition count of: $extfscnt"
             debugPause
+            echo " * BTRFS Partition count of: $btrfscnt"
+            debugPause
             echo " * Total Partition count of: $partscnt"
             debugPause
             case $osid in
diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
index 959be5f..b29e5ca 100644
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
@@ -258,6 +258,31 @@ expandPartition() {
                     ;;
             esac
             ;;
+        btrfs)
+            # Based on info from @mstabrin on forums.fogproject.org
+            if [[ ! -d /tmp/btrfs ]]; then
+                mkdir /tmp/btrfs >>/tmp/btfrslog.txt 2>&1
+                if [[$? -gt 0 ]]; then
+                    echo "Failed"
+                    debugPause
+                    handleError "Could not create /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+                fi
+            fi
+            mount $part /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
+            if [[ $? -gt 0 ]]; then
+                echo "Failed"
+                debugPause
+                handleError "Could not mount $part to /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+            fi
+            btrfs filesystem resize max /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
+            if [[ $? -gt 0 ]]; then
+                echo "Failed"
+                debugPause
+                handleError "Could not resize btrfs partition (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+            fi
+            umount $part /tmp/btrfs >/dev/null 2>&1
+            echo "Done"
+            ;;
         *)
             echo " * Not expanding ($part -- $fstype)"
             debugPause
@@ -597,6 +622,32 @@ shrinkPartition() {
                     ;;
             esac
             ;;
+        btrfs)
+            # Based on info from @mstabrin on forums.fogproject.org
+            # https://forums.fogproject.org/topic/15159/btrfs-postdownloadscript/3
+            if [[ ! -d /tmp/btrfs ]]; then
+                mkdir /tmp/btrfs >>/tmp/btfrslog.txt 2>&1
+                if [[$? -gt 0 ]]; then
+                    echo "Failed"
+                    debugPause
+                    handleError "Could not create /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+                fi
+            fi
+            mount $part /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
+            if [[ $? -gt 0 ]]; then
+                echo "Failed"
+                debugPause
+                handleError "Could not mount $part to /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+            fi
+            local free_size_original=$(btrfs filesystem usage -b /tmp/btrfs | grep unallocated | grep -Eo '[0-9]+')
+            local fsize_pct=$(calculate "${percent}/100")
+            local mult_val=$(calculate "1-${fsize_pct}")
+            local free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
+            while ! btrfs filesystem resize -${free_size} /tmp/btrfs; do
+                [[ ${mult_val} -le 0 ]] && break || mult_val=$(${mult_val} - ${fsize_pct})
+                free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
+            done
+            ;;
         *)
             echo " * Not shrinking ($part $fstype)"
             ;;

Then for the while loop, I’m with you, but this is according to the following article:
https://unix.stackexchange.com/questions/424758/resize-btrfs-filesystem-to-the-minimum-size-in-a-single-step

Basically it shrinks and continues to shrink in steps until it cannot shrink. BTRFS is not very nice about shrinking. You can expand until your drive is in the middle of the yard, but shrinking, while now possible, is still relatively new to BTRFS. It is one of the reasons we didn’t add BTRFS to the shrinking when BTRFS started to become popular. It literally could not be shrunk.

@mstabrin Here’s what I have, though I have not pushed it into the tree yet:

diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
index c754273..4d069b4 100755
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
@@ -58,8 +58,9 @@ beginUpload() {
             dots "Getting Windows/Linux Partition Count"
             countPartTypes "$hd" "ntfs" "ntfscnt"
             countPartTypes "$hd" "extfs" "extfscnt"
+            countPartTypes "$hd" "btrfs" "btrfscnt"
             countPartTypes "$hd" "*" "partscnt"
-            if [[ $ntfscnt -eq 0 && $extfscnt -eq 0 ]]; then
+            if [[ $ntfscnt -eq 0 && $extfscnt -eq 0 && $btrfscnt -eq 0 ]]; then
                 echo "Failed"
                 debugPause
                 handleError "No resizable partitions found ($0)\n   Args Passed: $*"
@@ -70,6 +71,8 @@ beginUpload() {
             debugPause
             echo " * EXTFS Partition count of: $extfscnt"
             debugPause
+            echo " * BTRFS Partition count of: $btrfscnt"
+            debugPause
             echo " * Total Partition count of: $partscnt"
             debugPause
             case $osid in
diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
index 959be5f..0f1d64e 100644
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
@@ -258,6 +258,31 @@ expandPartition() {
                     ;;
             esac
             ;;
+        btrfs)
+            # Based on info from @mstabrin on forums.fogproject.org
+            if [[ ! -d /tmp/btrfs ]]; then
+                mkdir /tmp/btrfs >>/tmp/btfrslog.txt 2>&1
+                if [[$? -gt 0 ]]; then
+                    echo "Failed"
+                    debugPause
+                    handleError "Could not create /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+                fi
+            fi
+            mount $part /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
+            if [[ $? -gt 0 ]]; then
+                echo "Failed"
+                debugPause
+                handleError "Could not mount $part to /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+            fi
+            btrfs filesystem resize max /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
+            if [[ $? -gt 0 ]]; then
+                echo "Failed"
+                debugPause
+                handleError "Could not resize btrfs partition (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+            fi
+            umount $part /tmp/btrfs >/dev/null 2>&1
+            echo "Done"
+            ;;
         *)
             echo " * Not expanding ($part -- $fstype)"
             debugPause
@@ -597,6 +622,31 @@ shrinkPartition() {
                     ;;
             esac
             ;;
+        btrfs)
+            # Based on info from @mstabrin on forums.fogproject.org
+            # https://forums.fogproject.org/topic/15159/btrfs-postdownloadscript/3
+            if [[ ! -d /tmp/btrfs ]]; then
+                mkdir /tmp/btrfs >>/tmp/btfrslog.txt 2>&1
+                if [[$? -gt 0 ]]; then
+                    echo "Failed"
+                    debugPause
+                    handleError "Could not create /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+                fi
+            fi
+            mount $part /tmp/btrfs >>/tmp/btrfslog.txt 2>&1
+            if [[ $? -gt 0 ]]; then
+                echo "Failed"
+                debugPause
+                handleError "Could not mount $part to /tmp/btrfs (${FUNCNAME[0]})\n   Info: $(cat /tmp/btrfslog.txt)\n   Args Passed: $*"
+            fi
+            local free_size_original=$(btrfs filesystem usage -b /tmp/btrfs | grep unallocated | grep -Eo '[0-9]+')
+            local mult_val=0.95
+            local free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
+            while ! btrfs filesystem resize -${free_size} /tmp/btrfs; do
+                [[ ${mult_val} -le 0 ]] && break || mult_val=$(${mult_val} - 0.05)
+                free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
+            done
+            ;;
         *)
             echo " * Not shrinking ($part $fstype)"
             ;;

From there, you can test these changes by updating the relevant files and placing using postinit scripts or you can extract the init with something like:

cd /var/www/fog/service/ipxe
mkdir tmp
cp init.xz init_orig.xz
xz -d init.xz
mount -o loop init tmp

Edit the file tmp/bin/fog.upload
Edit the file in tmp/usr/share/fog/lib/funcs.sh

Then:

umount tmp
xz -C crc32 -9 init

Let us know if this works as intended or any changes as needed.

Thanks again!

@mstabrin said in BTRFS postdownloadscript:

Hello @Sebastian-Roth and @george1421 ,
Thank you for all the information!

I am using FOG version 1.5.9.
Before I did any further testing I resolved my partition mess:
The allocation of the discs sectors was actually in the correct order, just the naming was messed up by a random allocation order during installation.

sda1 - EFI
sda2 - btrfs
sda3 - swap

However I still get a similar error with - No resizable partitions found):

(I looked at a lot of related threads and those people also posted screenshots of their problems and solutions. However, most of them are unavailable nowadays and therefore the thread not informative anymore. Is posting a screenshot here actually the best method in this case or do you prefer another solution?)

So I played around with btrfs resizing and came up with the following inside of the debug mode based on https://unix.stackexchange.com/questions/424758/resize-btrfs-filesystem-to-the-minimum-size-in-a-single-step

mount /dev/sda2 /mnt/bt

free_size_original=$(btrfs filesystem usage -b /mnt/bt | grep unallocated | grep -Eo '[0-9]+')

mult_val=0.95
free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
while ! btrfs filesystem resize -${free_size} /mnt/bt
do
    if [[ ${mult_val} -le 0 ]]
    then
        break
    else
        mult_val=$(${mult_val} - 0.05)
    fi
    free_size=$(echo "scale=0;(${mult_val}*${free_size_original})/1" | bc -l)
done

To bring it back to full size it only needs a:

btrfs filesystem resize max /mnt/bt

I guess the tricky part now is to allocate the correct partition sizes after the shrink and before the max.

@george1421 I tried to use fthe fog command within the debug mode but only got:

And the next enter sends me back to the command line.

Best,
Markus

So I’m working on adding this into the FOS code base and have a lot of code written already.

I’m assuming you have mult_val 0.95 as a percentage to leave an excess room of about 5% during the shrinking? This way the shrink has enough room for any files and what not?

Just trying to understand as we have code for calculating side. I will, for now, have it built for the 0.95 as we know this works. Just want to see if we can use the automated parts already.

Thank you,

@julianc as this is being captured have you tried the instructions it’s recommending? Run chkdsk /f reboot machine. Let it complete reboot and boot machine into windows. Restart and let it boot into windows. Reboot and begin capture.

@robpomeroy As far as I can see, RAID 1 is part of the issue here.

So, and I realize you’re likely aware of these things already, but just so others might better understand:

The RAID, in and of itself, has nothing to do with the imaging speed. This is because of how FOG handles imaging. You should be able to capture an image regardless of how it’s setup as long as it meets FOGs requirement. That said, Linux RAID as presented to the system is not a typical scenario FOG would prefer to use. What do I mean?

FOG handles particular filesystems. NTFS, EXT (2,3,4) for resizing. As SDA2 is partitioned as Linux Raid (Type = FD), there would be no method of capturing the 2nd partition as resizable. As far as fog is concerned, it’s dealing with 2 partition filesystem or which both would present as raw.

So 1tb capture at 5-6 hours seems pretty good. (Long but it makes sense as it’s likely capturing raw anyway.)

Now on the deploy, you are not just writing to 1 disk. Typically a deploy goes MUCH faster than capture (for obvious reasons I think). In your particular case, however, you have RAID 1. So when the image is writing to the disk it’s also writing to the redundant disk. This the basic premise of RAID 1. So, quite literally, you’re writing to the disk 2 times for one file (or block in this case?)

If you had a proper raid system (Hardware or software based, that presents as a single disk) in RAID 1, you would probably have no issues in capturing the image in Resizable mode. Of course, with RAID 1, you’d still likely see a bit of a time deploying the image as, it again, would be deploying 2 writes for 1 block (or whatever).

@sebastian-roth thank you!

@sebastian-roth and @o71 x-send file was enabled so snapins could be downloaded from the web server without needing to load the File entirely into php, lessening memory and what not. I believe the correct usage would be more what Sebastian suggested, giving the absolute path, not the actual file, and likely an over site from me when coding the section. Back when I put it in. I was pretty new to php, so forgive me. I never coded to purposely expose anything. I will make sure it’s fixed properly in working-1.6.

Thanks for letting us know. Even if Apache isn’t supporting it, it doesn’t hurt anything and if somebody wants to use an engine that does support it, all the better.