Posts made by shruggy

@jhumpf I don’t use the FOG client on Linux, but these are my scripts for CentOS 6:
/etc/rc.d/rc.local:

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
. root/ckhostname
exit 0

/root/ckhostname:

#!/bin/sh
network=/etc/sysconfig/network
ipaddr=$(hostname -I)
shostname=$(hostname)
rhostname="$(dig +noall +answer +short -x $ipaddr)"
rhostname=${rhostname%.}
if [ "$rhostname" != "$shostname" ]; then
  sed -i "/^HOSTNAME=/s/=.*/=$rhostname/" "$network"
  hostname $rhostname
fi

I plan to retire them as I’m migrating the hosts to CentOS 8.

hostname -I may not give you what you want depending on how your system is configured, instead you can try something like ip -4 -br addr|awk -F'[ /]+' '/UP/,$0=$3' or, more cautiously worded:

ip -4 -br addr | awk '$2=="UP" {sub(/\/.*/,"",$3); print $3}'

@Sebastian-Roth said in FOG v1.5.7 on Ubuntu 18.04 random problem multicast:

@Tom-Elliott Have you ever seen a PC failing to proceed from one partition to the other in multicast?

IIRC, I had this same problem once or twice, after one of FOG upgrades, probably half a year ago. I guess that prompted me to try out the dev-branch back then, the problem went away and I stayed on the dev-branch since then.

@tec618 What happens if you deploy unicast to that PC that failed to pick up partition 8?

In my case, it never happened in the unicast mode. I have a setup with 4 legacy primary MBR partitions. NTFS (500 MB) - NTFS (237 GB) - ext2 (500 MB) - LVM2 (237 GB). d1.partitions:

label: dos
label-id: 0x871158f2
device: /dev/sda
unit: sectors

/dev/sda1 : start=        2048, size=     1024000, type=7, bootable
/dev/sda2 : start=     1026048, size=   499081216, type=7
/dev/sda3 : start=   500107264, size=     1024000, type=83
/dev/sda4 : start=   501131264, size=   499083264, type=8e

IIRC, the freeze up happened after restoring the 3rd partition.

@george1421 said in [dev-branch] multicast: for some hosts DB not updated after restore:

@shruggy How many systems do you typically image at the same time with multicast? How much memory do you have on the fog server?

Usually, it’s 36 systems at once. The setup is similar to @tec618’s: FOG on a VM with 4GB RAM, but both the VM and the hosting server run CentOS 7, and it’s Xen, not KVM. PHP 7.3 from Remi’s repo.

@Quazz @Sebastian-Roth
The two awk lines changed in this commit are underquoted:
https://github.com/FOGProject/fos/commit/5f0e7b5f7d77986b0e0019dd23c875c6d40ec869

@Sebastian-Roth You can mark this as solved now. I didn’t go with the adjustments you suggested, though: just wanted to try first the configuration suggested at https://www.sitepoint.com/php-fpm-tuning-using-pm-static-max-performance and it worked.

Here is an excerpt from my current /etc/php-fpm.d/www.conf (the changed lines are the first two and the last):

pm = static
pm.max_children = 40
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
pm.max_requests = 500

I have a pool of 38 identical hosts.

See this post.
Apache logs don’t contain anything of note.
PHP-FPM log during (or shortly after) multicast restore sessions sometimes contains these warnings:

[04-Jan-2020 16:38:01] NOTICE: [pool www] child 29241 started
[05-Jan-2020 02:54:37] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 17 total children
[05-Jan-2020 02:54:38] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 22 total children
...
[25-Jan-2020 18:00:58] NOTICE: [pool www] child 9916 started
[25-Jan-2020 18:54:59] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 17 total children
[25-Jan-2020 18:55:00] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 22 total children
[25-Jan-2020 18:55:01] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 27 total children

Output of egrep '^pm\.(start|min|max)' /etc/php-fpm.d/www.conf

pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 2000

Output of free -h on FOG server

              total        used        free      shared  buff/cache   available
Mem:           3,9G        362M        372M        8,6M        3,2G        3,5G
Swap:          4,0G          0B        4,0G

Output of lscpu | egrep '^Core|^Socket'

Core(s) per socket:    2
Socket(s):             1

Output of ps --no-headers -o rss,cmd -C php-fpm|awk '{sum+=$1}END{print sum/NR/1024"M"}'

21.2727M

Output of mysql -u root fog -e 'select l.taskID,s.tsName from taskLog as l,taskStates as s where l.taskStateID=s.tsID and l.id between 3282 and 3348 order by l.taskID'

+--------+-------------+
| taskID | tsName      |
+--------+-------------+
| 1701   | In-Progress |
| 1701   | Complete    |
| 1702   | In-Progress |
| 1702   | Complete    |
| 1703   | In-Progress |
| 1703   | Complete    |
| 1704   | In-Progress |
| 1704   | Complete    |
| 1705   | In-Progress |
| 1706   | In-Progress |
| 1706   | Complete    |
| 1707   | In-Progress |
| 1707   | Complete    |
| 1708   | In-Progress |
| 1708   | Complete    |
| 1709   | In-Progress |
| 1709   | Complete    |
| 1710   | In-Progress |
| 1710   | Complete    |
| 1711   | In-Progress |
| 1711   | Complete    |
| 1712   | In-Progress |
| 1712   | Complete    |
| 1713   | In-Progress |
| 1713   | Complete    |
| 1714   | In-Progress |
| 1714   | Complete    |
| 1715   | In-Progress |
| 1715   | Complete    |
| 1716   | In-Progress |
| 1716   | Complete    |
| 1717   | In-Progress |
| 1717   | Complete    |
| 1718   | In-Progress |
| 1718   | Complete    |
| 1719   | In-Progress |
| 1719   | Complete    |
| 1720   | In-Progress |
| 1721   | In-Progress |
| 1721   | Complete    |
| 1722   | In-Progress |
| 1722   | Complete    |
| 1723   | In-Progress |
| 1723   | Complete    |
| 1724   | In-Progress |
| 1724   | Complete    |
| 1725   | In-Progress |
| 1725   | Complete    |
| 1726   | In-Progress |
| 1726   | Complete    |
| 1727   | In-Progress |
| 1727   | Complete    |
| 1728   | In-Progress |
| 1728   | Complete    |
| 1729   | In-Progress |
| 1730   | In-Progress |
| 1730   | Complete    |
| 1731   | In-Progress |
| 1732   | In-Progress |
| 1732   | Complete    |
| 1733   | In-Progress |
| 1733   | Complete    |
| 1734   | In-Progress |
| 1735   | In-Progress |
| 1736   | In-Progress |
| 1736   | Complete    |
+--------+-------------+

@shruggy said in 1.5.7.89: partclone doesn't capture an image in dd mode: wrong options in fog.upload:

After the coming Microsoft Patch Day (probably over the next weekend) I am planning to capture another disk image with this and deploy it to my pool in multi-cast mode.

I did it last weekend and the results are mixed. Yes, the image was successfully captured and then restored to 36 PCs in multi-cast. But: On five hosts I got this error message after restoring the image:

Reattempting to update database: Failed

The image was restored successfully on those hosts nevertheless. Only the FOG database wasn’t updated. All 36 PCs are identical hardware.

In the Imaging Log the End column for those five hosts says:

-0001-11-30 00:00:00

while the Duration column says:

2020 years 1 month 18 days 15 hours 35 minutes 43 seconds

It looks like somehow the data for Start timestamp got written into Duration?

@Sebastian-Roth said in Development FOG not capturing image - PartClone update:

@shruggy Would you mind telling us which version of partclone this is?

It’s the latest git shapshot:
https://github.com/Thomas-Tsai/partclone/tree/58d138da6e0f473acd80a7c9b8544d104377d50f
with the fix you suggested:

$ cd git/fos/Buildroot/package/partclone
$ ls
Config.in  partclone-0.3.12.patch  partclone.imager.patch  partclone.mk
$ cat partclone.imager.patch
diff --git a/src/ddclone.c b/src/ddclone.c
index 04f8775..cab5386 100644
--- a/src/ddclone.c
+++ b/src/ddclone.c
@@ -30,7 +30,9 @@ void read_super_blocks(char* device, file_system_info* fs_info)
 	}
 	strncpy(fs_info->fs, raw_MAGIC, FS_MAGIC_SIZE);
 	fs_info->block_size  = PART_SECTOR_SIZE;
-	//fs_info->device_size = get_partition_size(&src);
+#ifdef IMG
+	fs_info->device_size = get_partition_size(&src);
+#endif
 	fs_info->totalblock  = fs_info->device_size / PART_SECTOR_SIZE;
 	fs_info->usedblocks  = fs_info->device_size / PART_SECTOR_SIZE;
 	close(src);

I haven’t yet pulled the latest FOS, however. So it doesn’t contain the changes to func.sh by @Tom-Elliott i.e. no special case for partclone.imager.

On the other note, when using unpatched Partclone 0.3.12 I also sometimes got segmentation fault, but didn’t report it. The case when Partclone finished successfully, but actually didn’t capture the partition was the more common outcome. I’d say, somewhere in the relation of 4 (empty) successes to 1 segfault, IIRC. And I didn’t notice any pattern to segfaults then (e.g. it could happen with or without compression).

@Quazz has just fixed it in the source.

And I suppose the errors won’t show up if you set the image type to “Multiple Partition Image - Single Disk (Not Resizable)”, since the awk script gets only invoked during partition resizing.

@ty900000 Could you please try my init.xz with patched Partclone and say if it works for you?

@Tom-Elliott said in Development FOG not capturing image - PartClone update:

I pushed another fix and believe the issue was as @Quazz noted is the -c argument was missing. Strange as that is, as the -c argument doesn’t appear to be a part of the spec list (unless somebody already added that to the patch for partclone and I didn’t know it?)

My understanding of it is this:

• images captured with partclone.imager -c need to be restored with partclone.restore, raw images captured with partclone.dd or with partclone.imager without -c need to be restored with partclone.dd
• the options for partclone.imager are basically the same as for any partclone.$ftype (including -c and -a), it’s partclone.dd that is a special case. The symbol DD only pertains to partclone.dd, the symbol IMG is specifically defined in Makefile for partclone.imager, but never gets used in upstream Partclone code.

I tested and uploaded another version: this time it’s the latest Partclone with the fix suggested by @Sebastian-Roth, i.e.

diff --git a/src/ddclone.c b/src/ddclone.c
index 04f8775..cab5386 100644
--- a/src/ddclone.c
+++ b/src/ddclone.c
@@ -30,7 +30,9 @@ void read_super_blocks(char* device, file_system_info* fs_info)
        }
        strncpy(fs_info->fs, raw_MAGIC, FS_MAGIC_SIZE);
        fs_info->block_size  = PART_SECTOR_SIZE;
-       //fs_info->device_size = get_partition_size(&src);
+#ifdef IMG
+       fs_info->device_size = get_partition_size(&src);
+#endif
        fs_info->totalblock  = fs_info->device_size / PART_SECTOR_SIZE;
        fs_info->usedblocks  = fs_info->device_size / PART_SECTOR_SIZE;
        close(src);

The symbol IMG is defined in Makefile when building partclone.imager and prevents this change from being built into partclone.dd. For now, I’ve successfully captured a disk from one machine and restored it to another one. After the coming Microsoft Patch Day (probably over the next weekend) I am planning to capture another disk image with this and deploy it to my pool in multi-cast mode.

This time, I also built the 64-bit kernel 4.19.94 and used it together with this image. It’s here.

Uploaded new version of the FOS image in my post: replaced partclone 0.2.89 with partclone 0.2.91 (aka the latest commit in release branch).

• checks for image version (currently 0001) and displays it on progress screen
• updates translations
• fixes compile issue on CentOS 6.8
• fixes --disable-* options behavior for configure
• updates Makefile
• additional checks for malloc as well as different sizes recorded in an image (used blocks, total blocks, device size)

@sascha735 According to this post, CentOS 8 is not supported by FOG 1.5.7.

@Sebastian-Roth

Processor architecture set to Westmere

Do you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

It definitely wouldn’t: Westmere are Intel Core processors from year 2010 IIRC.

Partclone at 0.2.89

You might want to use the proper patch file for that version for the web UI status bar to work properly.

Yes, i found and applied it, just failed to mention (or rather did it very clumsily: that bit about misleading diff).

I’ve built a new (or rather old) init.xz. It’s here. I didn’t have time and wanted to get things done quickly, so as a stop-gap measure I just replaced partclone 0.3.12 with the old version 0.2.89 0.2.91. Besides, the image also has some other differences (that hopefully will speed up testing it):

1. Buildroot 2019.11
2. Processor architecture set to Westmere
3. Partclone at 0.2.89 0.2.91
4. Zstd 1.4.4 (it’s really fast just as release notes promised)
5. MDadm 4.1
6. GPTfdisk 1.0.4

The whole diff is somewhat misleading, but nevertheless can be found here.

With it, I captured the disk from my master (still using FOG 1.5.7.89) and successfully restored it to another machine.

As @Sebastian-Roth has already found the root of the problem it’s probably unneeded anymore, but after this I was planning to update partclone on the image to the latest git snapshot on the release branch (that reports itself as 0.2.91). Done.

@JonesDK Sorry for the delay. You can try it out now (as described below).

Update. Image updated and can be tested now.

@JonesDK I’m the topic starter from the post cross-linked by @george1421. Here is the FOS image I built. Could you please test it and report if it works for you?

I’ve only built an 64-bit image. If your FOG server is on CentOS put it into the /var/www/fog/service/ipxe directory (backup the old init.xz beforehand). Change the ownership to be the same as of init_32.xz file in there and don’t forget to restore the SELinux context:

cd /var/www/fog/service/ipxe
sudo mv init.xz init.xz.bak
sudo mv ~/Downloads/init.xz . 
sudo chown --reference=init_32.xz init.xz
sudo restorecon init.xz

Now try to capture the image with FOG. On my system I was using the image type “Multiple Partition Image - Single Disk (Not Resizable)” and as image manager “Partclone Zstd” with compression level 7.

OK, partclone.imager seems to not have the -c option either. My trick works when capturing just one partition (or the whole disk) in dd mode, but to capture only the LVM volume in dd mode while allowing other modes for the rest of the disk, we’ll probably need something like:

diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
index 5d6054f..050ace9 100644
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
@@ -2049,6 +2049,13 @@ savePartition() {
     getPartType "$part"
     local ebrfilename=""
     local swapuuidfilename=""
+    local clone="-c"
+    local checksum_mode="-a0"
+    if [ "0x8e" == "$parttype" ]; then
+        fstype=imager
+       clone=
+       checksum_mode=
+    fi
     case $fstype in
         swap)
             echo " * Saving swap partition UUID"
@@ -2068,7 +2075,7 @@ savePartition() {
                     debugPause
                     imgpart="$imagePath/d${disk_number}p${part_number}.img"
                     uploadFormat "$fifoname" "$imgpart"
-                    partclone.$fstype -n "Storage Location $storage, Image name $img" -cs $part -O $fifoname -Nf 1 -a0
+                    partclone.$fstype -n "Storage Location $storage, Image name $img" $clone -s $part -O $fifoname -N -f1 $checksum_mode
                     exitcode=$?
                     case $exitcode in
                         0)

Will test it tomorrow.

I’m not sure, but I believe that partclone.imager does have the -c option. It’s probably -a0 that got it tripped. (I haven’t checked it though). But partclone.dd has neither -c nor -a. Despite partclone.dd -h happily listing the last option. This is from the current partclone.c source:

if CHKIMG
	static const char *sopt = "-hvd::L:s:f:CFiBz:Nn:";
#elif RESTORE
	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
#elif DD
	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
#else
	static const char *sopt = "-hvd::L:cx:brDo:O:s:f:RCFINiqWBz:E:a:k:Kn:Tt";
#endif