[SCRIPT] FOG tools

ch3i

Hi,

I use a script to compress images (inspired by https://forums.fogproject.org/topic/5141/compression-change/6) and display configuration of my FOG ( using https://forums.fogproject.org/topic/4581/let-s-make-scripts).

I have terminated the compression of images present where I launch the script (to compress an image you have to launch the script on the master node of the group where the image is stored) :

Backup old image
Compress new image
Update database
Remove backup

In futur release I want to add :

A progress bar on pigz (using pv maybe) >> Done
Checking if pigz is installed
function to move image in another folder/storage.

Script (you need to have pigz installed) :

#!/bin/bash
##############################################
# Tools for FOG
# Last change 2015-06-15
#
# /!\ pigz and pv are required /!\
#
# Ch3i
##############################################

##############################################
# Functions
##############################################

FUNC_TOP(){
echo -e "\033[42m#########################################################
#                 FOG Images tools                      #
#########################################################\033[0m\n\n"
}

##############################################
# Get FOG configuration
##############################################

source /opt/fog/.fogsettings
snmysqlhost=${snmysqlhost#p:}
logfile="FOGtroubleshoot.log"

clear
FUNC_TOP

echo -e "/!\ pigz and pv are required for image compression /!\ \n"
echo "Press Enter to continue..."; read

clear
FUNC_TOP

echo -e "1 - Change image compression
2 - Export configuration in `pwd`/$logfile
3 - Exit\n"
read -e fog_operation

clear
FUNC_TOP

if [ "$fog_operation" == "1" ]
then
	##############################################
	# List all images (id/name/compression)
	##############################################

	echo -e "Choose the image to resize\n"
	echo -e "|ID\t| Image name (compression)"
	echo "------------------------------------------"
	# Display all images >> mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.imageID, images.imageName, images.imageCompress FROM images ORDER BY imageID" | while read image_id image_name image_compression
	mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.imageID, images.imageName, images.imageCompress FROM images INNER JOIN imageGroupAssoc ON images.imageID = imageGroupAssoc.igaImageID INNER JOIN nfsGroups ON imageGroupAssoc.igaStorageGroupID = nfsGroups.ngID INNER JOIN nfsGroupMembers ON nfsGroupMembers.ngmGroupID = nfsGroups.ngID WHERE ngmHostname = '$ipaddress' ORDER BY imageID" | while read image_id image_name image_compression #Display images on the node where the script is running
	do
		echo -e "|$image_id\t| $image_name ($image_compression)"
		echo "------------------------------------------"
	done
	echo -en "\nEnter the Image ID : "; read image_id_selected
	echo -n "Enter the compression level (0-9) : "; read image_compression_selected

	##############################################
	# Get image selected informations
	##############################################

	image_name_selected=$(mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.imageName FROM images WHERE images.imageID='$image_id_selected'")
	image_compression_original=$(mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.imageCompress FROM images WHERE images.imageID='$image_id_selected'")
	image_path=$(mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.ImagePath FROM images WHERE images.imageID='$image_id_selected'")
	
	if [ "$image_compression_selected" -ge 0 -a "$image_compression_selected" -le 9 -a "$image_name_selected" != "" ]
	then
		echo -n "Do you want to change compression of $image_name_selected from $image_compression_original to $image_compression_selected ? (y/n) : "; read go_compression
	else
		echo "error"
		echo "Exiting..."
		sleep 1
		clear
		exit
	fi
	
	##############################################
	# Resize the image and update database
	##############################################
	if [ "$go_compression" = "Y" -o "$go_compression" = "y" ]
	then
		clear
		FUNC_TOP
		echo -e "Changing compression of $image_name_selected from $image_compression_original to $image_compression_selected : \n"
		echo -n "Backup img...In progress"
		for f in "$storageLocation/$image_path/"*.img; do mv "$f" "${f%.img}-bak.img"; done
		echo -e "\033[1K"
		echo "Backup img...Done"
		echo "Compress new img...In progress"
		for f in "$storageLocation/$image_path/"*bak.img; do pigz -d -c $f | pv $f | pigz -"$image_compression_selected" > "${f%-bak.img}.img"; done
		echo -e "\033[1K"
		echo "Compress new img...Done"
		echo -n "Remove backups...In progress"
		rm "$storageLocation/$image_path/"*bak.img
		chmod 777 -R "$storageLocation/$image_path/"
		echo -e "\033[1K"
		echo "Remove backups...Done"
		echo -n "Update database...In progress"
		mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "UPDATE images SET images.imageCompress='$image_compression_selected' WHERE images.imageID='$image_id_selected'"
		echo -e "\033[1K"
		echo "Update database...Done"
		echo "Exiting..."
		sleep 1
		exit
	else
		echo "Exiting..."
		sleep 1
		clear
		exit
	fi
elif [ "$fog_operation" == "2" ]
then
	##############################################
	# Export Configuration
	##############################################

	(( EUID != 0 )) && exec sudo -- "$0" "$@"
	linuxReleaseName=`lsb_release -a 2> /dev/null | grep "Distributor ID" | awk '{print $3,$4,$5,$6,$7,$8,$9}' | tr -d " "`;
	if [ -z "$linuxReleaseName" ]; then
		# Fall back incase lsb_release does not exist / fails - use /etc/issue over /etc/*release*
		linuxReleaseName=`cat /etc/issue /etc/*release* 2>/dev/null | head -n1 | awk '{print $1}'`;
	fi
	#logfile="FOGtroubleshoot.log"
	# Delete logfile if it exists
	if [ -f $logfile ]; then
		rm $logfile
	fi
	#argument 1 is the logfile name
	getStatus() {
		i=1;
		if [ ! -z "`echo $linuxReleaseName | grep -Ei 'Fedora|Redhat|CentOS|Mageia'`" ]; then
			RHVER=`rpm -qa | grep release | xargs rpm -q --queryformat '%{VERSION}' | cut -c -1`;
		fi
		if [ "`echo $RHVER`" -gt 6 ]; then
			services="xinetd rpcbind nfs-server vsftpd firewalld FOGMulticastManager FOGSnapinReplicator FOGImageReplicator FOGScheduler";
			nicename="TFTP RPCBind NFS FTP Firewall FOGMulticastManager FOGSnapinReplicator FOGImageReplicator FOGScheduler";
			for service in $services; do
				niceval=`echo $nicename|awk '{print $'$i'}' |sed 's/_/ /'`;
				echo "----------------------$niceval status below" >> $1
				systemctl status $service >> $1;
				i=`expr $i '+' 1`;
			done
		else
			services="xinetd rpcbind nfs vsftpd iptables FOGMulticastManager FOGSnapinReplicator FOGImageReplicator FOGScheduler";
			nicename="TFTP RPCBind NFS FTP Firewall FOGMulticastManager FOGSnapinReplicator FOGImageReplicator FOGScheduler";
			for service in $services; do
				niceval=`echo $nicename|awk '{print $'$i'}' |sed 's/_/ /'`;
				echo "----------------------$niceval status below" >> $1
				service $service status >> $1;
				i=`expr $i '+' 1`;
			done
		fi
	}
	#argument 1 is the logfile name
	getLogs() {
		if [ ! -z "`echo $linuxReleaseName | grep -Ei 'Fedora|Redhat|CentOS|Mageia'`" ]; then
			httploc='httpd';
			httpsep='_';
		else
			httploc='apache2';
			httpsep='.';
		fi
		logs="/var/log/foginstall.log /var/log/${httploc}/error${httpsep}log /var/log/${httploc}/access${httpsep}log";
		nicename="Installation Apache_Error Apache_Access";
		i=1;
		for lfname in $logs; do
			if [ -f "$lfname" ]; then
				niceval=`echo $nicename|awk '{print $'$i'}' |sed 's/_/ /'`;
				echo ----------------------$niceval log below >> $1;
				tail -30 $lfname >> $1
			fi
			i=`expr $i '+' 1`;
		done
	}
	#argument 1 is the logfile name
	getConfs() {
		conffiles="/etc/dhcp/dhcpd.conf /etc/dnsmasq.d/ltsp.conf /etc/xinetd.d/tftp /etc/vsftpd/vsftpd.conf /etc/rc.d/rc.local";
		for confname in $conffiles; do
			if [ -f "$confname" ]; then
				echo "----------------------$confname file below" >> $1;
				cat $confname >> $1;
			fi
		done
	}
	echo '----------------------'$linuxReleaseName' version below' >> $logfile
	cat /etc/issue >> $logfile
	if [ ! -z "`echo $linuxReleaseName | grep -Ei 'Fedora|Redhat|CentOS|Mageia'`" ]; then
		echo '----------------------SELinux status below' >> $logfile;
		sestatus >> $logfile
	fi
	echo '----------------------IP Configuration below' >> $logfile
	ip addr >> $logfile
	getStatus $logfile;
	getLogs $logfile;
	getConfs $logfile;
	storageLocation="/images";
	if [ -d "/opt/fog" -a -f "/opt/fog/.fogsettings" ]; then
		. /opt/fog/.fogsettings;
	fi
	if [ -d "$storageLocation" ]; then
		echo '----------------------Check for /images/.mntcheck' >> $logfile
		ls $storageLocation -a |grep "mntcheck" >> $logfile
		echo '----------------------Check for /images/dev/.mntcheck' >> $logfile
		ls ${storageLocation}/dev -a |grep "mntcheck" >> $logfile
		echo '----------------------/images & file permissions below' >> $logfile
		ls -lR $storageLocation >> $logfile
	fi
	if [ -d "/tftpboot" ]; then
		echo '----------------------/tftpboot & file permissions below' >> $logfile
		ls -lR /tftpboot >> $logfile
	fi
	sed -i 's/$/
	/g' $logfile
	echo -n "Script Completed ";
	date
	echo "Your logfile can be found in `pwd`/$logfile";
	echo "Exiting..."
	sleep 1
	exit
else
	echo "Exiting..."
	sleep 1
	clear
	exit
fi

It’s not perfect but functional, I used it without problem, if you want to test it make a backup of the target image before

Ch3i.

Wayne Workman

That is super sweet.

Wayne Workman

Now that we know we can change compression settings post-upload,

Would it make sense to always upload with compression set at 0 and then compress server-side afterwards?

Then, we wouldn’t need FTP client side to move images from /images/dev to /images because after compression completes, the image could be moved?

Maybe even a progress bar for the compression progress could be displayed under “Tasks” ?

ch3i

@Wayne-Workman said:

Then, we wouldn’t need FTP client side to move images from /images/dev to /images because after compression completes, the image could be moved?

Depends the server performance, if you have multiple uploads you will have a high cpu utilization.

ch3i

Updated :

Correct rights problem on new imgs
Adding progression bar using PV

With progression :

#########################################################
#                 FOG Images tools                      #
#########################################################

Changing compression of MASTER_WB from 0 to 5 :

                         
Backup img...Done
 
Compress new img...In progress
24,2MO 0:00:00 [  78MB/s] [=======================================================>] 100%            
73,3GO 0:18:12 [68,7MB/s] [=======================================================>] 100%            
37,5GO 0:09:58 [64,1MB/s] [=======================================================>] 100%            
 5,1kO 0:00:00 [ 394kB/s] [=======================================================>] 100%            
 142MO 0:00:00 [ 163MB/s] [=======================================================>] 100%            
 
Compress new img...Done
                             
Remove backups...Done
                              
Update database...Done
Exiting...

The size isn’t the compressed size, but size before.

Joseph Hales

I would think multiple uploads would be an edge case I would love to have this as a feature especially if it could compress a temp copy so you could deploy immediately.

Wayne Workman

@Joseph-Hales said:

I would think multiple uploads would be an edge case I would love to have this as a feature especially if it could compress a temp copy so you could deploy immediately.

Well, I suppose the uncompressed copy could serve as the “temp” copy.

And, I suppose FTP could go ahead and just move that over from /dev to /images like normal

and in the background, the server could compress the /images image to /dev and when it’s done (and waiting for active tasks for that image to complete), it could replace it with the compressed copy?

Joseph Hales

That would be perfect.

Wayne Workman

I think this is easily accomplished.

Using Ch3i’s script for compression, and making that automated, and scheduling a Cron task on the server to run that script every hour or so should do the trick.

The script will need a default compression set, and it’ll need to check the DB for any images that aren’t set to that default (say, compression 1). Then, it would check to see if that image exists in it’s storage node, if so, recompress it to /node/dev , and then check the DB if there are active tasks associated with that image… and just wait till they get done, then replace the old image with the new.

I think I can do this, I’ll try tonight or tomorrow night.

Wayne Workman

@Wayne-Workman said:

I think I can do this, I’ll try tonight or tomorrow night.

Didn’t get to this tonight… passed out straight away after work. It was a long and tough day.
And my laptop is acting up… I replaced the fan in it yesterday and the fan was literally the very last component I could take out of the plastic frame… I think I’m going to have to open it up again and re-do the thermal compound, might have used too much. It’s the worst designed laptop I’ve ever seen… It’s like they wanted to make cleaning/replacing the fan & radiator as difficult as possible.

Wayne Workman

Bumping this thread.

[SCRIPT] FOG tools

200

12.0k

17.3k

155.2k