[SCRIPT] FOG tools


  • Moderator

    Hi,

    I use a script to compress images (inspired by https://forums.fogproject.org/topic/5141/compression-change/6) and display configuration of my FOG ( using https://forums.fogproject.org/topic/4581/let-s-make-scripts).

    I have terminated the compression of images present where I launch the script (to compress an image you have to launch the script on the master node of the group where the image is stored) :

    • Backup old image
    • Compress new image
    • Update database
    • Remove backup

    In futur release I want to add :

    • A progress bar on pigz (using pv maybe) >> Done
    • Checking if pigz is installed
    • function to move image in another folder/storage.

    Script (you need to have pigz installed) :

    #!/bin/bash
    ##############################################
    # Tools for FOG
    # Last change 2015-06-15
    #
    # /!\ pigz and pv are required /!\
    #
    # Ch3i
    ##############################################
    
    ##############################################
    # Functions
    ##############################################
    
    FUNC_TOP(){
    echo -e "\033[42m#########################################################
    #                 FOG Images tools                      #
    #########################################################\033[0m\n\n"
    }
    
    ##############################################
    # Get FOG configuration
    ##############################################
    
    source /opt/fog/.fogsettings
    snmysqlhost=${snmysqlhost#p:}
    logfile="FOGtroubleshoot.log"
    
    clear
    FUNC_TOP
    
    echo -e "/!\ pigz and pv are required for image compression /!\ \n"
    echo "Press Enter to continue..."; read
    
    clear
    FUNC_TOP
    
    echo -e "1 - Change image compression
    2 - Export configuration in `pwd`/$logfile
    3 - Exit\n"
    read -e fog_operation
    
    clear
    FUNC_TOP
    
    if [ "$fog_operation" == "1" ]
    then
    	##############################################
    	# List all images (id/name/compression)
    	##############################################
    
    	echo -e "Choose the image to resize\n"
    	echo -e "|ID\t| Image name (compression)"
    	echo "------------------------------------------"
    	# Display all images >> mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.imageID, images.imageName, images.imageCompress FROM images ORDER BY imageID" | while read image_id image_name image_compression
    	mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.imageID, images.imageName, images.imageCompress FROM images INNER JOIN imageGroupAssoc ON images.imageID = imageGroupAssoc.igaImageID INNER JOIN nfsGroups ON imageGroupAssoc.igaStorageGroupID = nfsGroups.ngID INNER JOIN nfsGroupMembers ON nfsGroupMembers.ngmGroupID = nfsGroups.ngID WHERE ngmHostname = '$ipaddress' ORDER BY imageID" | while read image_id image_name image_compression #Display images on the node where the script is running
    	do
    		echo -e "|$image_id\t| $image_name ($image_compression)"
    		echo "------------------------------------------"
    	done
    	echo -en "\nEnter the Image ID : "; read image_id_selected
    	echo -n "Enter the compression level (0-9) : "; read image_compression_selected
    
    	##############################################
    	# Get image selected informations
    	##############################################
    
    	image_name_selected=$(mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.imageName FROM images WHERE images.imageID='$image_id_selected'")
    	image_compression_original=$(mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.imageCompress FROM images WHERE images.imageID='$image_id_selected'")
    	image_path=$(mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "SELECT images.ImagePath FROM images WHERE images.imageID='$image_id_selected'")
    	
    	if [ "$image_compression_selected" -ge 0 -a "$image_compression_selected" -le 9 -a "$image_name_selected" != "" ]
    	then
    		echo -n "Do you want to change compression of $image_name_selected from $image_compression_original to $image_compression_selected ? (y/n) : "; read go_compression
    	else
    		echo "error"
    		echo "Exiting..."
    		sleep 1
    		clear
    		exit
    	fi
    	
    	##############################################
    	# Resize the image and update database
    	##############################################
    	if [ "$go_compression" = "Y" -o "$go_compression" = "y" ]
    	then
    		clear
    		FUNC_TOP
    		echo -e "Changing compression of $image_name_selected from $image_compression_original to $image_compression_selected : \n"
    		echo -n "Backup img...In progress"
    		for f in "$storageLocation/$image_path/"*.img; do mv "$f" "${f%.img}-bak.img"; done
    		echo -e "\033[1K"
    		echo "Backup img...Done"
    		echo "Compress new img...In progress"
    		for f in "$storageLocation/$image_path/"*bak.img; do pigz -d -c $f | pv $f | pigz -"$image_compression_selected" > "${f%-bak.img}.img"; done
    		echo -e "\033[1K"
    		echo "Compress new img...Done"
    		echo -n "Remove backups...In progress"
    		rm "$storageLocation/$image_path/"*bak.img
    		chmod 777 -R "$storageLocation/$image_path/"
    		echo -e "\033[1K"
    		echo "Remove backups...Done"
    		echo -n "Update database...In progress"
    		mysql --host="$snmysqlhost" --user="$snmysqluser" --password="$snmysqlpass" "fog" -Bse "UPDATE images SET images.imageCompress='$image_compression_selected' WHERE images.imageID='$image_id_selected'"
    		echo -e "\033[1K"
    		echo "Update database...Done"
    		echo "Exiting..."
    		sleep 1
    		exit
    	else
    		echo "Exiting..."
    		sleep 1
    		clear
    		exit
    	fi
    elif [ "$fog_operation" == "2" ]
    then
    	##############################################
    	# Export Configuration
    	##############################################
    
    	(( EUID != 0 )) && exec sudo -- "$0" "$@"
    	linuxReleaseName=`lsb_release -a 2> /dev/null | grep "Distributor ID" | awk '{print $3,$4,$5,$6,$7,$8,$9}' | tr -d " "`;
    	if [ -z "$linuxReleaseName" ]; then
    		# Fall back incase lsb_release does not exist / fails - use /etc/issue over /etc/*release*
    		linuxReleaseName=`cat /etc/issue /etc/*release* 2>/dev/null | head -n1 | awk '{print $1}'`;
    	fi
    	#logfile="FOGtroubleshoot.log"
    	# Delete logfile if it exists
    	if [ -f $logfile ]; then
    		rm $logfile
    	fi
    	#argument 1 is the logfile name
    	getStatus() {
    		i=1;
    		if [ ! -z "`echo $linuxReleaseName | grep -Ei 'Fedora|Redhat|CentOS|Mageia'`" ]; then
    			RHVER=`rpm -qa | grep release | xargs rpm -q --queryformat '%{VERSION}' | cut -c -1`;
    		fi
    		if [ "`echo $RHVER`" -gt 6 ]; then
    			services="xinetd rpcbind nfs-server vsftpd firewalld FOGMulticastManager FOGSnapinReplicator FOGImageReplicator FOGScheduler";
    			nicename="TFTP RPCBind NFS FTP Firewall FOGMulticastManager FOGSnapinReplicator FOGImageReplicator FOGScheduler";
    			for service in $services; do
    				niceval=`echo $nicename|awk '{print $'$i'}' |sed 's/_/ /'`;
    				echo "----------------------$niceval status below" >> $1
    				systemctl status $service >> $1;
    				i=`expr $i '+' 1`;
    			done
    		else
    			services="xinetd rpcbind nfs vsftpd iptables FOGMulticastManager FOGSnapinReplicator FOGImageReplicator FOGScheduler";
    			nicename="TFTP RPCBind NFS FTP Firewall FOGMulticastManager FOGSnapinReplicator FOGImageReplicator FOGScheduler";
    			for service in $services; do
    				niceval=`echo $nicename|awk '{print $'$i'}' |sed 's/_/ /'`;
    				echo "----------------------$niceval status below" >> $1
    				service $service status >> $1;
    				i=`expr $i '+' 1`;
    			done
    		fi
    	}
    	#argument 1 is the logfile name
    	getLogs() {
    		if [ ! -z "`echo $linuxReleaseName | grep -Ei 'Fedora|Redhat|CentOS|Mageia'`" ]; then
    			httploc='httpd';
    			httpsep='_';
    		else
    			httploc='apache2';
    			httpsep='.';
    		fi
    		logs="/var/log/foginstall.log /var/log/${httploc}/error${httpsep}log /var/log/${httploc}/access${httpsep}log";
    		nicename="Installation Apache_Error Apache_Access";
    		i=1;
    		for lfname in $logs; do
    			if [ -f "$lfname" ]; then
    				niceval=`echo $nicename|awk '{print $'$i'}' |sed 's/_/ /'`;
    				echo ----------------------$niceval log below >> $1;
    				tail -30 $lfname >> $1
    			fi
    			i=`expr $i '+' 1`;
    		done
    	}
    	#argument 1 is the logfile name
    	getConfs() {
    		conffiles="/etc/dhcp/dhcpd.conf /etc/dnsmasq.d/ltsp.conf /etc/xinetd.d/tftp /etc/vsftpd/vsftpd.conf /etc/rc.d/rc.local";
    		for confname in $conffiles; do
    			if [ -f "$confname" ]; then
    				echo "----------------------$confname file below" >> $1;
    				cat $confname >> $1;
    			fi
    		done
    	}
    	echo '----------------------'$linuxReleaseName' version below' >> $logfile
    	cat /etc/issue >> $logfile
    	if [ ! -z "`echo $linuxReleaseName | grep -Ei 'Fedora|Redhat|CentOS|Mageia'`" ]; then
    		echo '----------------------SELinux status below' >> $logfile;
    		sestatus >> $logfile
    	fi
    	echo '----------------------IP Configuration below' >> $logfile
    	ip addr >> $logfile
    	getStatus $logfile;
    	getLogs $logfile;
    	getConfs $logfile;
    	storageLocation="/images";
    	if [ -d "/opt/fog" -a -f "/opt/fog/.fogsettings" ]; then
    		. /opt/fog/.fogsettings;
    	fi
    	if [ -d "$storageLocation" ]; then
    		echo '----------------------Check for /images/.mntcheck' >> $logfile
    		ls $storageLocation -a |grep "mntcheck" >> $logfile
    		echo '----------------------Check for /images/dev/.mntcheck' >> $logfile
    		ls ${storageLocation}/dev -a |grep "mntcheck" >> $logfile
    		echo '----------------------/images & file permissions below' >> $logfile
    		ls -lR $storageLocation >> $logfile
    	fi
    	if [ -d "/tftpboot" ]; then
    		echo '----------------------/tftpboot & file permissions below' >> $logfile
    		ls -lR /tftpboot >> $logfile
    	fi
    	sed -i 's/$/
    	/g' $logfile
    	echo -n "Script Completed ";
    	date
    	echo "Your logfile can be found in `pwd`/$logfile";
    	echo "Exiting..."
    	sleep 1
    	exit
    else
    	echo "Exiting..."
    	sleep 1
    	clear
    	exit
    fi
    

    It’s not perfect but functional, I used it without problem, if you want to test it make a backup of the target image before :D

    Ch3i.


  • Moderator

    Bumping this thread.


  • Moderator

    @Wayne-Workman said:

    I think I can do this, I’ll try tonight or tomorrow night.

    Didn’t get to this tonight… passed out straight away after work. It was a long and tough day.
    And my laptop is acting up… I replaced the fan in it yesterday and the fan was literally the very last component I could take out of the plastic frame… I think I’m going to have to open it up again and re-do the thermal compound, might have used too much. It’s the worst designed laptop I’ve ever seen… It’s like they wanted to make cleaning/replacing the fan & radiator as difficult as possible.


  • Moderator

    I think this is easily accomplished.

    Using Ch3i’s script for compression, and making that automated, and scheduling a Cron task on the server to run that script every hour or so should do the trick.

    The script will need a default compression set, and it’ll need to check the DB for any images that aren’t set to that default (say, compression 1). Then, it would check to see if that image exists in it’s storage node, if so, recompress it to /node/dev , and then check the DB if there are active tasks associated with that image… and just wait till they get done, then replace the old image with the new.

    I think I can do this, I’ll try tonight or tomorrow night.


  • Testers

    That would be perfect.


  • Moderator

    @Joseph-Hales said:

    I would think multiple uploads would be an edge case I would love to have this as a feature especially if it could compress a temp copy so you could deploy immediately.

    Well, I suppose the uncompressed copy could serve as the “temp” copy.

    And, I suppose FTP could go ahead and just move that over from /dev to /images like normal

    and in the background, the server could compress the /images image to /dev and when it’s done (and waiting for active tasks for that image to complete), it could replace it with the compressed copy?


  • Testers

    I would think multiple uploads would be an edge case I would love to have this as a feature especially if it could compress a temp copy so you could deploy immediately.


  • Moderator

    Updated :

    • Correct rights problem on new imgs
    • Adding progression bar using PV

    With progression :

    #########################################################
    #                 FOG Images tools                      #
    #########################################################
    
    Changing compression of MASTER_WB from 0 to 5 :
    
                             
    Backup img...Done
     
    Compress new img...In progress
    24,2MO 0:00:00 [  78MB/s] [=======================================================>] 100%            
    73,3GO 0:18:12 [68,7MB/s] [=======================================================>] 100%            
    37,5GO 0:09:58 [64,1MB/s] [=======================================================>] 100%            
     5,1kO 0:00:00 [ 394kB/s] [=======================================================>] 100%            
     142MO 0:00:00 [ 163MB/s] [=======================================================>] 100%            
     
    Compress new img...Done
                                 
    Remove backups...Done
                                  
    Update database...Done
    Exiting...
    

    The size isn’t the compressed size, but size before.


  • Moderator

    @Wayne-Workman said:

    Then, we wouldn’t need FTP client side to move images from /images/dev to /images because after compression completes, the image could be moved?

    Depends the server performance, if you have multiple uploads you will have a high cpu utilization.


  • Moderator

    Now that we know we can change compression settings post-upload,

    Would it make sense to always upload with compression set at 0 and then compress server-side afterwards?

    Then, we wouldn’t need FTP client side to move images from /images/dev to /images because after compression completes, the image could be moved?

    Maybe even a progress bar for the compression progress could be displayed under “Tasks” ?


  • Moderator

    That is super sweet.


Log in to reply
 

360
Online

39.3k
Users

11.0k
Topics

104.4k
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.