@Tom-Elliott Thanks Tom! I will test out the funcs.sh change to see how the upload results change.
@george1421 Yes, a clonezilla image directly uploaded will dedupe (with no compression.) The file is chunked into pieces and deduped based on those chunks. The art is to match the dedupe chunk size with the data inside the image and to match the chunk boundaries between the algorithm and the incoming data.
Hi George,
I’ve tested extensively with ‘compression 0’ for pigz but cannot get any deduplication since (I assume) the data is still being chunked at 128K and placed in a gzip wrapper. Once the .img is written, the file command shows the data as ‘gzip compressed data.’ As shown in the uploadFormat code, I have hard-set pigz to -0 compression (as well as set the same within the FOG client properties.)
Also, this dedupe storage is used for other data storage successfully (VM datastores, general fileserver data, etc.) (I’m an ‘old-hand’ with dedupe storage systems.)
Thanks for the info about dev review/focus on dedupe filesystems. So with that in mind, I am only looking for suggestions/guidance on how best to bypass pigz (or if that is a large task affecting huge parts of FOG.)
In working with a deduplication storage location for images, I’ve found that pigz prevents any realistic dedupe results. (Even the same image backed up twice is not recognized by the dedupe engine.)
I’ve attempted to modify the uploadFormat function in init’s usr/share/fog/lib/funcs.sh file to change pigz behavior (-i and -0 hardcoded) without any useful results. I’m now considering how best to remove pigz from from uploadFormat. But this will potentially impact image pushes, since pigz is called for decompression.
Thanks!
uploadFormat()
{
if [ ! -n "$1" ]; then
echo "Missing Cores";
return;
elif [ ! -n "$2" ]; then
echo "Missing file in file out";
return;
elif [ ! -n "$3" ]; then
echo "Missing file name to store";
return;
fi
if [ "$imgFormat" == "2" ]; then
# pigz -p $1 $PIGZ_COMP < $2 | split -a 3 -d -b 200m - ${3}. &
pigz -i -p 1 -0 < $2 | split -a 3 -d -b 200m - ${3}. &
else
if [ "$imgType" == "n" ]; then
# pigz -p $1 $PIGZ_COMP < $2 > ${3}.000 &
pigz -i -p 1 -0 < $2 > ${3}.000 &
else
# pigz -p $1 $PIGZ_COMP < $2 > $3 &
pigz -i -p 1 -0 < $2 > $3 &
fi
fi
}
@Sebastian-Roth The attached tgz (0_1455389711392_fog_debug_RHEL6.tgz) is from 6269 and was run on RHEL6. I’ll have to check to see if commenting clearScreen changes the output.
@Tom-Elliott I just pulled the latest git clone and the OS menu is now displayed. Since the redhat-release file isn’t missing, there must be another cause for strSuggestedOS to be unset. I will move forward with testing this version install. Thanks for your response/help!
Version: 6271 Installer/Updater
What version of Linux would you like to run the installation for?
1) Redhat Based Linux (Redhat, CentOS, Mageia)
2) Debian Based Linux (Debian, Ubuntu, Kubuntu, Edubuntu)
3) Arch Linux
@Sebastian-Roth Neither system (RHEL6 or RHEL7) is missing the redhat-release file. Example from 6:
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.7 (Santiago)
I can provide the full debug text via e-mail if you’d like.
I’m sure the versions have changed, but 1.2.0 installed without this issue on RHEL7. I then attempted an upgrade using the git clone, which had an issue with ‘backupDatabase.’ Rather than modify the install script, I deployed a clean RHEL7 and attempted an install from git clone - this ‘hung’ with the loop mentioned. I then deployed a RHEL6 system believing it would be smoother. This then hung with the same loop at the start of the install. (To clarify, when the RHEL7 clean install hung, I did not enable -x debug on bash. I moved to RHEL6 and saw the same hang. After enabling -x and seeing the loop, I assumed it is the same as what occurred on RHEL7. I will go back to that system to see if I can verify the same bash loop.)
Here is the output in the debug text for RHEL7 - it appears to loop at the same point:
448: strSuggestedDNS=
449: [[ -f /etc/resolv.conf ]]
cat /etc/resolv.conf | grep “nameserver” | head -n 1 | tr -d “nameserver” | tr -d [:blank:] | grep “^[0-9].[0-9].[0-9].[0-9]$”)
cat /etc/resolv.conf | grep “nameserver” | head -n 1 | tr -d “nameserver” | tr -d [:blank:] | grep “^[0-9].[0-9].[0-9].[0-9]$”
5550: cat /etc/resolv.conf
5550: grep nameserver
5550: tr -d nameserver
5550: grep ‘^[0-9].[0-9].[0-9].[0-9]$’
5550: head -n 1
5550: tr -d ‘[:blank:]’
550: strSuggestedDNS=172.16.4.10
552: [[ -z 172.16.4.10 ]]
557: strSuggestedSNUser=fogstorage
displayOSChoices
559: displayOSChoices
6645: blFirst=1
6646: [[ -z ‘’ ]]
6647: [[ ‘’ -eq 1 ]]
6650: osid=
6651: [[ -z ‘’ ]]
6651: [[ ! -z ‘’ ]]
6646: [[ -z ‘’ ]]
6647: [[ ‘’ -eq 1 ]]
6650: osid=
The release file:
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo)
@Sebastian-Roth I made the updates to installfog.sh, then ran the command with:
./installfog > /tmp/fog_debug.txt
This looped only for a few seconds, but was enough to create a 16MB text file. I’ve pasted the last few lines before the loop starts (I’ll attach the full tgz if I can figure out how on this forum…)
448: strSuggestedDNS=
449: [[ -f /etc/resolv.conf ]]
5550: cat /etc/resolv.conf
5550: grep ‘^[0-9].[0-9].[0-9].[0-9]$’
5550: tr -d nameserver
5550: tr -d ‘[:blank:]’
5550: head -n 1
5550: grep nameserver
550: strSuggestedDNS=172.16.4.10
552: [[ -z 172.16.4.10 ]]
557: strSuggestedSNUser=fogstorage
displayOSChoices
559: displayOSChoices
6645: blFirst=1
6646: [[ -z ‘’ ]]
6647: [[ ‘’ -eq 1 ]]
6650: osid=
6651: [[ -z ‘’ ]]
6651: [[ ! -z ‘’ ]]
6646: [[ -z ‘’ ]]
6647: [[ ‘’ -eq 1 ]]
6650: osid=
6651: [[ -z ‘’ ]]
6651: [[ ! -z ‘’ ]]
6646: [[ -z ‘’ ]]
6647: [[ ‘’ -eq 1 ]]
6650: osid=
6651: [[ -z ‘’ ]]
6651: [[ ! -z ‘’ ]]
6646: [[ -z ‘’ ]]
6647: [[ ‘’ -eq 1 ]]
6650: osid=
6651: [[ -z ‘’ ]]
6651: [[ ! -z ‘’ ]]
6646: [[ -z ‘’ ]]
6647: [[ ‘’ -eq 1 ]]
No prompt to enter anything - the FOG ASCII and the revision number are displayed and then it “hangs.” The debug output shows the osid loop.
I just pulled revision 6267 (and later 6269 in troubleshooting) using git. I have tested on both RHEL 7 and RHEL 6. This is a clean install. When the installfog.sh script is launched, it shows the FOG ASCII art and the version number and then hangs indefinitely. I edited installfog.sh and added -x to the /bin/bash statement and re-ran. The result was a never-ending loop stuck on checking osid - snippet below:
++ osid=
++ [[ -z '' ]]
++ [[ ! -z '' ]]
++ [[ -z '' ]]
++ [[ '' -eq 1 ]]
++ osid=
++ [[ -z '' ]]
++ [[ ! -z '' ]]
++ [[ -z '' ]]
++ [[ '' -eq 1 ]]
++ osid=
++ [[ -z '' ]]
++ [[ ! -z '' ]]
++ [[ -z '' ]]
++ [[ '' -eq 1 ]]
++ osid=
++ [[ -z '' ]]
++ [[ ! -z '' ]]
++ [[ -z '' ]]
++ [[ '' -eq 1 ]]
++ osid=
Did I miss this somewhere as a known issue of some type…?
Mod edited to use code box.