Eric Johnson

Eric Johnson

Reading the man page eh? Silly me downloading util-linux and tracing through source… LOL. (Hadn’t really started that except the download part )

Three times with flock $disk sfdisk $disk < $file now, no resize errors.

Gonna remove all the debug code and let em run to completion now.

I think ya got it though! Thanks!

Eric Johnson

Im just joining in here. But we are seeing the something like the same problem here. Fog 1.5.6. We have Dell 3430’s we are getting ready for deployment this fall.

A 3430 is a new model for us, and the first we have that doesn’t let you have a MBR boot disk, just GPT. Got everything working with a GPT clonemaster which for various ugly reasons has partitions like this:

[root@fog clonemaster10-lab-gpt]# cat d1.minimum.partitions
label: gpt
label-id: 701D9ABD-7D9A-11E9-B9AE-5254009E1079
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 257228766

/dev/sda1 : start= 2048, size= 1124352, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=701D9AB9-7D9A-11E9-B9AE-5254009E1079
/dev/sda2 : start= 1126400, size= 234728416, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=701D9ABA-7D9A-11E9-B9AE-5254009E1079
/dev/sda3 : start= 255332352, size= 204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=701D9ABB-7D9A-11E9-B9AE-5254009E1079, name=“attrs=\x22GUID:63”

sda2 is the real windows 10 partition…

cat d1.partitions
label: gpt
label-id: 701D9ABD-7D9A-11E9-B9AE-5254009E1079
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 257228766

/dev/sda1 : start= 2048, size= 1124352, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=701D9AB9-7D9A-11E9-B9AE-5254009E1079
/dev/sda2 : start= 1126400, size= 254204148, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=701D9ABA-7D9A-11E9-B9AE-5254009E1079
/dev/sda3 : start= 255332352, size= 204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=701D9ABB-7D9A-11E9-B9AE-5254009E1079, attrs=“GUID:63”

cat d1.fixed_size_partitions
:3:3

All seemed great till we tried it on some new machines to deploy and after fog/oobe/namechange/domainjoin SOME of them wouldn’t let anyone log in. Turns out the middle partition didn’t get extended correctly in some cases so windows was out of disk.

I can multicast to 4 identical machines and on 3 of them /dev/sda2 gets resized correctly, but one it doesn’t. And the one it fails on is not always the same… Funky eh?

When I do a debug deploy with ismajordebug=9 it always works…

Was going to go digging into my memory to rebuild a init.xz that has ismajordebug=9 next. See if that makes 4 host multicast work. Or points to the problem.

Oh, and a manual run of /usr/share/fog/lib/procsfdisk.awk in debug mode seems to be producing the correct output.

vaguely wondering if $tmp_file2 is getting hosed some how before fillSfdiskWithPartitions calls applySfdiskPartitions… But like i said I can not get problem to replicate in majordebug mode yet.

Would be glad to instrument out fog.download in any way you suggest.

More tomorrow if I find anything useful.

E

Eric Johnson

Current status:

Tried clever suggestion from @Quazz … it didn’t pan out.

But inspired me to think outside the box. Tried using /root/sfblahblah.$$ instead of /tmp (since tmpfs is different than ramdisk…) That was just as unsuccessful.

Just adding a “sync;sync;sync” (cause anything I tell you three times is true) seems successful. On two tests so far.

Maybe?

Current working diff…

[root@fog ipxe]# diff -u /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh /mnt/init/usr/share/fog/lib/partition-funcs.sh
--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh	2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh	2019-07-11 18:10:20.000000000 -0400
@@ -401,8 +401,17 @@
     #    majorDebugPause
     #fi
     #[[ $status -eq 0 ]] && applySfdiskPartitions "$disk" "$tmp_file1" "$tmp_file2"
+#    processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" 
     processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" > "$tmp_file2"
     status=$?
+sync;sync;sync
+#	echo $tmp_file2
+#	ls -l $tmp_file2
+#        cat $tmp_file2
+#echo "after"
+#cat /proc/sys/kernel/random/entropy_avail
+#Pause #ESJ
+	
     if [[ $ismajordebug -gt 0 ]]; then
         echo "Debug"
         majorDebugEcho "Trying to fill with the disk with these partititions:"
[root@fog ipxe]#

Eric Johnson

Bad news. On the 3rd deploy with sync;sync;sync one failed.

Good news:

I moved all the metering to AFTER the sfdisk (and didn’t send the sfdisk output to /dev/null as was suggested by @Sebastian-Roth ). Found something interesting. Twice on one of the 4 machines the sfdisk is failing with a disk in use error. The input is good, no issue with the tmp file. But sfdisk seems to think the disk is in use. I think we have the smoking gun.

Adding some df and swapon -s now to see if this I can see the disk in use by some other tools. Next step if to dig up sfdisk source and see what makes it think it is use.

Maybe --force or --no-reread as suggested by the error mesage as a BFMI workaround?

--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh 2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh      2019-07-12 08:37:10.000000000 -0400
@@ -73,7 +73,14 @@
     local file="$2"
     [[ -z $disk ]] && handleError "No disk passed (${FUNCNAME[0]})\n   Args Passed: $*"
     [[ -z $file ]] && handleError "No file to receive from passed (${FUNCNAME[0]})\n   Args Passed: $*"
-    sfdisk $disk < $file >/dev/null 2>&1
+#    sfdisk $disk < $file >/dev/null 2>&1
+    sfdisk $disk < $file 
+echo " "
+echo $file
+ls -l $file
+cat $file
+Pause #ESJ
+
     [[ ! $? -eq 0 ]] && majorDebugEcho "sfdisk failed in (${FUNCNAME[0]})"
 }
 # $1 is the name of the disk drive
@@ -403,6 +410,8 @@
     #[[ $status -eq 0 ]] && applySfdiskPartitions "$disk" "$tmp_file1" "$tmp_file2"
     processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" > "$tmp_file2"
     status=$?
+#Pause #ESJ
+
     if [[ $ismajordebug -gt 0 ]]; then
         echo "Debug"
         majorDebugEcho "Trying to fill with the disk with these partititions:"

alt text

Eric Johnson

Two 4 stations multicasts. No resize problems. Below diff is only change. I think this one is done!

diff -u /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh /mnt/init/usr/share/fog/lib/partition-funcs.sh 
--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh 2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh      2019-07-12 10:26:47.000000000 -0400
@@ -73,7 +73,7 @@
     local file="$2"
     [[ -z $disk ]] && handleError "No disk passed (${FUNCNAME[0]})\n   Args Passed: $*"
     [[ -z $file ]] && handleError "No file to receive from passed (${FUNCNAME[0]})\n   Args Passed: $*"
-    sfdisk $disk < $file >/dev/null 2>&1
+    flock $disk sfdisk $disk < $file >/dev/null 2>&1
     [[ ! $? -eq 0 ]] && majorDebugEcho "sfdisk failed in (${FUNCNAME[0]})"
 }
 # $1 is the name of the disk drive

Eric Johnson

Found a couple more sfdisk that might need protecting…

[root@fog ipxe]# diff -ur /mnt/init-orig/usr/share/fog/lib /mnt/init/usr/share/fog/lib
diff -ur /mnt/init-orig/usr/share/fog/lib/funcs.sh /mnt/init/usr/share/fog/lib/funcs.sh
--- /mnt/init-orig/usr/share/fog/lib/funcs.sh   2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/funcs.sh        2019-07-12 12:57:23.000000000 -0400
@@ -1977,7 +1977,7 @@
         sfdiskLegacyOriginalPartitionFileName "$imagePath" "$disk_number"
         if [[ -r $sfdiskoriginalpartitionfilename ]]; then
             dots "Inserting Extended partitions (Original)"
-            sfdisk $disk < $sfdiskoriginalpartitionfilename >/dev/null 2>&1
+            flock $disk sfdisk $disk < $sfdiskoriginalpartitionfilename >/dev/null 2>&1
             case $? in
                 0)
                     echo "Done"
@@ -1988,7 +1988,7 @@
             esac
         elif [[ -e $sfdisklegacyoriginalpartitionfilename ]]; then
             dots "Inserting Extended partitions (Legacy)"
-            sfdisk $disk < $sfdisklegacyoriginalpartitionfilename >/dev/null 2>&1
+            flock $disk sfdisk $disk < $sfdisklegacyoriginalpartitionfilename >/dev/null 2>&1
             case $? in
                 0)
                     echo "Done"
diff -ur /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh /mnt/init/usr/share/fog/lib/partition-funcs.sh
--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh 2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh      2019-07-12 10:26:47.000000000 -0400
@@ -73,7 +73,7 @@
     local file="$2"
     [[ -z $disk ]] && handleError "No disk passed (${FUNCNAME[0]})\n   Args Passed: $*"
     [[ -z $file ]] && handleError "No file to receive from passed (${FUNCNAME[0]})\n   Args Passed: $*"
-    sfdisk $disk < $file >/dev/null 2>&1
+    flock $disk sfdisk $disk < $file >/dev/null 2>&1
     [[ ! $? -eq 0 ]] && majorDebugEcho "sfdisk failed in (${FUNCNAME[0]})"
 }
 # $1 is the name of the disk drive

Eric Johnson

Hi Folks,

Fog version 1.5.9

Have a sort of problem/request. A lot of our fog’ed hosts have virtual box installed on them. This leads to fog system seeing the virtual box interface mac addresses and making them “pending”. These macs are all OUI 0A:00:27 (usually follow by 00:00:XX where XX is often 10 or 09 or 07) But many hosts wind up having the same mac address pending for these virtual box interfaces.

Sometimes (not all the times, in fact only occasionally, and I haven’t figured out how or why) this seems to break hostname changer when more than one host has the same mac. This happens occasionally. I don’t have the exact error from the fog.log handy though, will update this note when it happens again.

Deleting the pending mac list will fix this problem, and hostname changer does it’s job again.

To effect a quick fix I tried looking through the web php for a convenient place to stick some sort of “if mac starts with 0A:00:27 please ignore” code, but my OO php skills are not up to this finding this.

Thoughts?
E

Eric Johnson

Found a couple more sfdisk that might need protecting…

[root@fog ipxe]# diff -ur /mnt/init-orig/usr/share/fog/lib /mnt/init/usr/share/fog/lib
diff -ur /mnt/init-orig/usr/share/fog/lib/funcs.sh /mnt/init/usr/share/fog/lib/funcs.sh
--- /mnt/init-orig/usr/share/fog/lib/funcs.sh   2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/funcs.sh        2019-07-12 12:57:23.000000000 -0400
@@ -1977,7 +1977,7 @@
         sfdiskLegacyOriginalPartitionFileName "$imagePath" "$disk_number"
         if [[ -r $sfdiskoriginalpartitionfilename ]]; then
             dots "Inserting Extended partitions (Original)"
-            sfdisk $disk < $sfdiskoriginalpartitionfilename >/dev/null 2>&1
+            flock $disk sfdisk $disk < $sfdiskoriginalpartitionfilename >/dev/null 2>&1
             case $? in
                 0)
                     echo "Done"
@@ -1988,7 +1988,7 @@
             esac
         elif [[ -e $sfdisklegacyoriginalpartitionfilename ]]; then
             dots "Inserting Extended partitions (Legacy)"
-            sfdisk $disk < $sfdisklegacyoriginalpartitionfilename >/dev/null 2>&1
+            flock $disk sfdisk $disk < $sfdisklegacyoriginalpartitionfilename >/dev/null 2>&1
             case $? in
                 0)
                     echo "Done"
diff -ur /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh /mnt/init/usr/share/fog/lib/partition-funcs.sh
--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh 2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh      2019-07-12 10:26:47.000000000 -0400
@@ -73,7 +73,7 @@
     local file="$2"
     [[ -z $disk ]] && handleError "No disk passed (${FUNCNAME[0]})\n   Args Passed: $*"
     [[ -z $file ]] && handleError "No file to receive from passed (${FUNCNAME[0]})\n   Args Passed: $*"
-    sfdisk $disk < $file >/dev/null 2>&1
+    flock $disk sfdisk $disk < $file >/dev/null 2>&1
     [[ ! $? -eq 0 ]] && majorDebugEcho "sfdisk failed in (${FUNCNAME[0]})"
 }
 # $1 is the name of the disk drive

Eric Johnson

Two 4 stations multicasts. No resize problems. Below diff is only change. I think this one is done!

diff -u /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh /mnt/init/usr/share/fog/lib/partition-funcs.sh 
--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh 2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh      2019-07-12 10:26:47.000000000 -0400
@@ -73,7 +73,7 @@
     local file="$2"
     [[ -z $disk ]] && handleError "No disk passed (${FUNCNAME[0]})\n   Args Passed: $*"
     [[ -z $file ]] && handleError "No file to receive from passed (${FUNCNAME[0]})\n   Args Passed: $*"
-    sfdisk $disk < $file >/dev/null 2>&1
+    flock $disk sfdisk $disk < $file >/dev/null 2>&1
     [[ ! $? -eq 0 ]] && majorDebugEcho "sfdisk failed in (${FUNCNAME[0]})"
 }
 # $1 is the name of the disk drive

Eric Johnson

@Quazz

Reading the man page eh? Silly me downloading util-linux and tracing through source… LOL. (Hadn’t really started that except the download part )

Three times with flock $disk sfdisk $disk < $file now, no resize errors.

Gonna remove all the debug code and let em run to completion now.

I think ya got it though! Thanks!

Eric Johnson

Bad news. On the 3rd deploy with sync;sync;sync one failed.

Good news:

I moved all the metering to AFTER the sfdisk (and didn’t send the sfdisk output to /dev/null as was suggested by @Sebastian-Roth ). Found something interesting. Twice on one of the 4 machines the sfdisk is failing with a disk in use error. The input is good, no issue with the tmp file. But sfdisk seems to think the disk is in use. I think we have the smoking gun.

Adding some df and swapon -s now to see if this I can see the disk in use by some other tools. Next step if to dig up sfdisk source and see what makes it think it is use.

Maybe --force or --no-reread as suggested by the error mesage as a BFMI workaround?

--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh 2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh      2019-07-12 08:37:10.000000000 -0400
@@ -73,7 +73,14 @@
     local file="$2"
     [[ -z $disk ]] && handleError "No disk passed (${FUNCNAME[0]})\n   Args Passed: $*"
     [[ -z $file ]] && handleError "No file to receive from passed (${FUNCNAME[0]})\n   Args Passed: $*"
-    sfdisk $disk < $file >/dev/null 2>&1
+#    sfdisk $disk < $file >/dev/null 2>&1
+    sfdisk $disk < $file 
+echo " "
+echo $file
+ls -l $file
+cat $file
+Pause #ESJ
+
     [[ ! $? -eq 0 ]] && majorDebugEcho "sfdisk failed in (${FUNCNAME[0]})"
 }
 # $1 is the name of the disk drive
@@ -403,6 +410,8 @@
     #[[ $status -eq 0 ]] && applySfdiskPartitions "$disk" "$tmp_file1" "$tmp_file2"
     processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" > "$tmp_file2"
     status=$?
+#Pause #ESJ
+
     if [[ $ismajordebug -gt 0 ]]; then
         echo "Debug"
         majorDebugEcho "Trying to fill with the disk with these partititions:"

alt text

Eric Johnson

Current status:

Tried clever suggestion from @Quazz … it didn’t pan out.

But inspired me to think outside the box. Tried using /root/sfblahblah.$$ instead of /tmp (since tmpfs is different than ramdisk…) That was just as unsuccessful.

Just adding a “sync;sync;sync” (cause anything I tell you three times is true) seems successful. On two tests so far.

Maybe?

Current working diff…

[root@fog ipxe]# diff -u /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh /mnt/init/usr/share/fog/lib/partition-funcs.sh
--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh	2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh	2019-07-11 18:10:20.000000000 -0400
@@ -401,8 +401,17 @@
     #    majorDebugPause
     #fi
     #[[ $status -eq 0 ]] && applySfdiskPartitions "$disk" "$tmp_file1" "$tmp_file2"
+#    processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" 
     processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" > "$tmp_file2"
     status=$?
+sync;sync;sync
+#	echo $tmp_file2
+#	ls -l $tmp_file2
+#        cat $tmp_file2
+#echo "after"
+#cat /proc/sys/kernel/random/entropy_avail
+#Pause #ESJ
+	
     if [[ $ismajordebug -gt 0 ]]; then
         echo "Debug"
         majorDebugEcho "Trying to fill with the disk with these partititions:"
[root@fog ipxe]#

Eric Johnson

@Quazz grrr. no luck. entropy started pretty high and was only slightly higher after the call to processSfdisk in 8 out of 8 cases… 2258-2261 before and after in all cases.

Was a great idea, worth trying though!

Eric Johnson

@Quazz That was a crazy brilliant idea!

I did the below, first cat before anything, then the processSfdisk to the screen then a second cat, then a pause so I can read it and it probably wasn’t important at that point (I think that would have shown the entropy increase, please let me know if I am miss-understanding what you wanted to see) but on two unicast tests so far… 2260 before / 2261 after (identical)

Going to do a multicast test to all 4 in a sec. See if there is any variation.

diff -u /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh /mnt/init/usr/share/fog/lib/partition-funcs.sh
--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh 2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh      2019-07-11 15:31:10.000000000 -0400
@@ -401,8 +401,19 @@
     #    majorDebugPause
     #fi
     #[[ $status -eq 0 ]] && applySfdiskPartitions "$disk" "$tmp_file1" "$tmp_file2"
+cat /proc/sys/kernel/random/entropy_avail
+    processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" 
+echo "after"
+cat /proc/sys/kernel/random/entropy_avail
+
+        Pause #ESJ
     processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" > "$tmp_file2"
     status=$?
+       echo $tmp_file2
+       ls -l $tmp_file2
+        cat $tmp_file2
+#        Pause ESJ
+
     if [[ $ismajordebug -gt 0 ]]; then
         echo "Debug"
         majorDebugEcho "Trying to fill with the disk with these partititions:"

Eric Johnson

@Sebastian-Roth Doh. Good idea, should have thought of that. But since even just doing some other output (see my recent post) seems to make the problem vanish…

Dis one is very odd.

Eric Johnson

So… current status.

Since it worked all the time with my diff with a Pause I just removed my Pause in non-debug mode… Seriously.

I have done 6 multicast deploys to 4 machines with this init and had no resize problems. Running with the original init showed 1-2 out of 4 resize failures.

This of course is NOT what I would call a fix. Kinda a workaround. And I don’t know if this is fixing the problem or just making it less likely to happen, as this should not be actually fixing anything. I through it out here for folks who might run into the same problem as a workaround till we figure out what the hell is going on…

diff -u /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh /mnt/init/usr/share/fog/lib/partition-funcs.sh
--- /mnt/init-orig/usr/share/fog/lib/partition-funcs.sh 2019-05-04 17:58:07.000000000 -0400
+++ /mnt/init/usr/share/fog/lib/partition-funcs.sh      2019-07-11 09:22:49.000000000 -0400
@@ -401,8 +401,15 @@
     #    majorDebugPause
     #fi
     #[[ $status -eq 0 ]] && applySfdiskPartitions "$disk" "$tmp_file1" "$tmp_file2"
+    processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" 
+#        Pause ESJ
     processSfdisk "$minf" filldisk "$disk" "$disk_size" "$fixed" "$orig" > "$tmp_file2"
     status=$?
+       echo $tmp_file2
+       ls -l $tmp_file2
+        cat $tmp_file2
+#        Pause ESJ
+
     if [[ $ismajordebug -gt 0 ]]; then
         echo "Debug"
         majorDebugEcho "Trying to fill with the disk with these partititions:"

Eric Johnson

@Eric Johnson

Best posts made by Eric Johnson

Latest posts made by Eric Johnson