Useful Scripts

Tom Elliott · May 23, 2016, 3:46 PM

Hey all,

I know this probably seems rather random, but this is just a place for me to put some of my scripts during programming for FOG.

While these are specifically setup for my environment, simple edits can make them portable.

If you all want to try your hand on these for your own projects, feel free.

Copy to svn
– All this does is take my files as I have edited them and copies them to my trunk directory for subversion/svn.
– I call it copytosvn
– The extra “trunk2” stuff you see is for btsync when that was running.
– if I call it as copytosvn git it will do the same as for trunk, but to only the git. This was added due to sourceforge being down last year.

#!/bin/bash
dots() {
    local pad=$(printf "%0.1s" "."{1..60})
    printf " * %s%*.*s" "$1" 0 $((60-${#1})) "$pad"
}
errorStat() {
    if [ "$1" != 0 ]; then
        echo "Failed"
        exit 1
    fi
    echo "OK"
}
trunkUpdate() {
    cwd=`pwd`
    if [[ -z $1 || $1 == svn ]]; then
        cd /root/trunk
        dots "Updating SVN Directory"
        /usr/bin/svn update >/dev/null 2>&1
        cd /root/fogproject
    elif [[ $1 == git ]]; then
        dots "Updating GIT Directory"
        cd /root/fogproject
        /usr/bin/git checkout dev-branch >/dev/null 2>&1
        /usr/bin/git pull >/dev/null 2>&1
    fi
    errorStat $?
}
versionUpdate() {
    dots "Updating Version in File"
    trunkver=`git describe --tags | awk -F'-[^0-9]*' '{value=$2+$3+1; print value}'` >/dev/null 2>&1
    sed -i "s/define('FOG_VERSION'.*);/define('FOG_VERSION', '$trunkver');/g" /var/www/fog/lib/fog/system.class.php >/dev/null 2>&1
//g" /var/www/fog/lib/fog/system.class.php >/dev/null 2>&1
    errorStat $?
}
copyFilesToTrunk() {
    dots "Copying files to trunk"
    for filename in `find /var/www/fog -type d`; do
        if [[ -z $1 || $1 == svn ]]; then
            cp -r $filename/* /root/trunk/packages/web/${filename#/var/www/fog} >/dev/null 2>&1
        elif [[ $1 == git ]]; then
            cp -r $filename/* /root/fogproject/packages/web/${filename#/var/www/fog} >/dev/null 2>&1
        fi
    done
    if [[ -z $1 || $1 == svn ]]; then
        rm -rf /root/trunk/packages/web/lib/fog/config.class.php >/dev/null 2>&1
        rm -rf /root/trunk/packages/web/management/other/cache/* >/dev/null 2>&1
        rm -rf /root/trunk/packages/web/management/other/ssl >/dev/null 2>&1
        rm -rf /root/trunk/packages/web/status/injectHosts.php >/dev/null 2>&1
        rm -rf /root/trunk2/* >/dev/null 2>&1
        cp -r /root/trunk/* /root/trunk2/ >/dev/null 2>&1
    elif [[ $1 == git ]]; then
        rm -rf /root/fogproject/packages/web/lib/fog/config.class.php >/dev/null 2>&1
        rm -rf /root/fogproject/packages/web/management/other/cache/* >/dev/null 2>&1
        rm -rf /root/fogproject/packages/web/management/other/ssl >/dev/null 2>&1
        rm -rf /root/fogproject/packages/web/status/injectHosts.php >/dev/null 2>&1
        rm -rf /root/trunk2/* >/dev/null 2>&1
        cp -r /root/fogproject/* /root/trunk2/ >/dev/null 2>&1
    fi
    errorStat $?
}
makefogtar() {
    dots "Creating FOG Tar File"
    rm -rf /opt/fog_trunk
    if [[ -z $1 || $1 == svn ]]; then
        cp -r /root/trunk /opt/fog_trunk
        find /opt/fog_trunk -name .svn -exec rm -rf {} \; >/dev/null 2>&1
    elif [[ $1 == git ]]; then
        cp -r /root/fogproject /opt/fog_trunk
        find /opt/fog_trunk -name .git -exec rm -rf {} \; >/dev/null 2>&1
    fi
    cd /opt
    tar -cjf /var/www/html/fog_trunk.tar.bz2 fog_trunk
    rm -rf /opt/fog_trunk
    cd $cwd
    unset cwd
    errorStat $?
}
trunkUpdate $1
versionUpdate $1
copyFilesToTrunk $1
makefogtar $1

Copy back trunk
– Copies the file in the trunk folder back to the base folder. Useful if i’m making changes that seem to break things and just need to start fresh, or if something needs to be looked at as its broken, but known working in older version. Allows me to revert the folder and copy back the files in the trunk to the base so I can see things more clearly.
– Create a backup of your original config.class.php in the root directory.
– I named it copybacktrunk

#!/bin/bash
rm -rf /var/www/fog;cp -r ~/trunk/packages/web /var/www/fog; cp ~/config.class.php /var/www/fog/lib/fog/config.class.php;chown -R apache:apache /var/www/fog; chown -R 500:apache /var/www/fog/service/ipxe

Tom Elliott · May 23, 2016, 9:43 PM

Build buildroot
– Copies the files from the main trunk directory of a remote system (my dev box) and places the files where needed.
– Does require pre-environment setup (config files etc… already in place.
– I call it buildBuildroot
– Copies the init’s back to my other servers as needed.

#!/bin/bash
arguments="$*"
dots() {
    local pad=$(printf "%0.1s" "."{1..60})
    printf " * %s%*.*s" "$1" 0 $((60-${#1})) "$pad"
}
errorStat() {
    local status=$1
    if [[ $status != 0 ]]; then
        echo "Failed"
        return 1
    fi
    echo "Done"
    return 0
}
getCustomBuildroot() {
    dots "Copying customized 64 bit init files"
    scp -rP 10002 mastaweb:/root/trunk/src/buildroot/* /root/buildroot/ >/dev/null 2>&1
    errorStat $?
    dots "Copying customized 32 bit init files"
    scp -rP 10002 mastaweb:/root/trunk/src/buildroot/* /root/32build/buildroot/ >/dev/null 2>&1
    errorStat $?
}
build64bit() {
    dots "Moving to 64 bit build environment"
    if [[ -d /root/buildroot ]]; then
        cd /root/buildroot
    fi
    errorStat $?
    [[ $? != 0 ]] && exit 1
    dots "Removing fog build directory to start fresh"
    if [[ -d "/root/buildroot/output/build/fog-1" ]]; then
        rm -rf "/root/buildroot/output/build/fog-1" >/dev/null 2>&1
    fi
    errorStat $?
    [[ $? != 0 ]] && exit 1
    dots "Building 64 bit inits"
    make -j $(nproc) >/root/buildroot/buildroot64.log 2>&1
    errorStat $?
    [[ $? != 0 ]] && exit 1
    dots "Copying 64 bit inits to remotes"
    scp output/images/rootfs.ext2.xz ubuntu64:/var/www/fog/service/ipxe/init.xz >/dev/null 2>&1 && \
    scp -P 10002 output/images/rootfs.ext2.xz mastaweb:/var/www/html/init.xz >/dev/null 2>&1 && \
    scp -P 10008 output/images/rootfs.ext2.xz fogstorage:/var/www/html/fog/service/ipxe/init.xz >/dev/null 2>&1
    errorStat $?
}
build32bit() {
    dots "Moving to 32 bit build environment"
    if [[ -d /root/32build/buildroot ]]; then
        cd /root/32build/buildroot >/dev/null 2>&1
    fi
    errorStat $?
    [[ $? != 0 ]] && exit 1
    dots "Removing fog build directory to start fresh"
    if [[ -d "/root/32build/buildroot/output/build/fog-1" ]]; then
        rm -rf "/root/32build/buildroot/output/build/fog-1" >/dev/null 2>&1
    fi
    errorStat $?
    [[ $? != 0 ]] && exit 1
    dots "Building 32 bit inits"
    make -j $(nproc) ARCH=i486 >/root/32build/buildroot/buildroot32.log 2>&1
    errorStat $?
    [[ $? != 0 ]] && exit 1
    dots "Copying 32 bit inits to remotes"
    scp output/images/rootfs.ext2.xz ubuntu64:/var/www/fog/service/ipxe/init_32.xz >/dev/null 2>&1 && \
    scp -P 10002 output/images/rootfs.ext2.xz mastaweb:/var/www/html/init_32.xz >/dev/null 2>&1 && \
    scp -P 10008 output/images/rootfs.ext2.xz fogstorage:/var/www/html/fog/service/ipxe/init_32.xz >/dev/null 2>&1
    errorStat $?
}
if [[ $arguments == +(*'customfiles'*) ]]; then
    getCustomBuildroot
    exit
fi
getCustomBuildroot
build64bit
build32bit

Tom Elliott · May 23, 2016, 3:46 PM

Build iPXE
– Build’s the ipxe files and updates the trunk version via remote methods now.
– I call it buildIpxe
– Adding arguments just add iPXE arguments to build with extra args.
For example: buildIpxe DEBUG=dhcp will build the ipxe files with the DEBUG=dhcp enabled on the built files.

#!/bin/bash
cwd=`pwd`
cd /root/ipxe
git pull
cd /root/ipxe/src/
make -j $(nproc) bin/ipxe.iso bin/{undionly,ipxe,intel,realtek}.{,k,kk}pxe bin/ipxe.lkrn EMBED=ipxescript $*
scp -P 10002 /root/ipxe/src/bin/{undionly,ipxe,intel,realtek}.{,k,kk}pxe mastaweb:/root/trunk/packages/tftp/
scp -P 10002 /root/ipxe/src/bin/ipxe.lkrn mastaweb:/root/trunk/packages/tftp/ipxe.krn
scp -P 10002 /root/ipxe/src/bin/ipxe.iso mastaweb:/root/trunk/packages/tftp/ipxe.iso
scp -P 10002 /root/ipxe/src/ipxescript mastaweb:/root/trunk/src/ipxe/src/ipxescript
scp -P 10002 /root/ipxe/src/config/general.h mastaweb:/root/trunk/src/ipxe/src/config/general.h
scp -P 10002 /root/ipxe/src/config/settings.h mastaweb:/root/trunk/src/ipxe/src/config/settings.h
scp -P 10002 /root/ipxe/src/config/console.h mastaweb:/root/trunk/src/ipxe/src/config/console.h
cd /root/ipxe-efi
git pull
cd /root/ipxe-efi/src/
make -j $(nproc) bin-{i386,x86_64}-efi/{snp{,only},ipxe,intel,realtek}.efi EMBED=ipxescript $*
scp -P 10002 /root/ipxe-efi/src/bin-i386-efi/{snp{,only},ipxe,intel,realtek}.efi mastaweb:/root/trunk/packages/tftp/i386-efi/
scp -P 10002 /root/ipxe-efi/src/bin-x86_64-efi/{snp{,only},ipxe,intel,realtek}.efi mastaweb:/root/trunk/packages/tftp/
scp -P 10002 /root/ipxe-efi/src/ipxescript mastaweb:/root/trunk/src/ipxe/src-efi/ipxescript
scp -P 10002 /root/ipxe-efi/src/config/general.h mastaweb:/root/trunk/src/ipxe/src-efi/config/general.h
scp -P 10002 /root/ipxe-efi/src/config/settings.h mastaweb:/root/trunk/src/ipxe/src-efi/config/settings.h
scp -P 10002 /root/ipxe-efi/src/config/console.h mastaweb:/root/trunk/src/ipxe/src-efi/config/console.h
cd $cwd
unset cwd

Tom Elliott · May 23, 2016, 9:49 PM

Init redo
– Allows me to redo the inits without having to do a full rebuild
– I call it init_redo.sh
– Takes one argument, for now. The argument will be the webroot to look for the ipxe files. If it is not set or cannot be found it will try to find the webroot under the more common locations"

#!/bin/bash
# Check for webroot
if [ -n "$1" -a -e "$1" ]; then
    webroot="$1";
elif [ -e "/srv/http/fog" ]; then
    webroot="/srv/http/fog";
elif [ -e "/var/www/html/fog" ]; then
    webroot="/var/www/html/fog";
elif [ -e "/var/www/fog" ]; then
    webroot="/var/www/fog";
fi
if [ ! -e "$webroot/service" ]; then
    echo "No service directory to work from"
    exit 1
fi
if [ ! -e "$webroot/service/ipxe" ]; then
    echo "No ipxe directory to work from"
    exit 1
fi
ipxeroot="$webroot/service/ipxe"
currDirectory=`pwd`
init64=''
init32=''
if [ ! -f "$ipxeroot/init.xz" -a ! -f "$ipxeroot/init" ]; then
    echo "No 64 bit init to process"
    init64='no'
fi
if [ ! -f "$ipxeroot/init_32.xz" -a ! -f "$ipxeroot/init_32" ]; then
    echo "No 32 bit init to process"
    init32='no'
fi
if [ -n "$init64" -a -n "$init32" ]; then
    echo "No init files to process"
    exit 1
fi
if [ ! -e "$ipxeroot/tmp" ]; then
    mkdir $ipxeroot/tmp >/dev/null 2>&1
fi
copyTrunkFiles() {
    svn up /root/trunk/ >/dev/null 2>&1
    cp -r /root/trunk/src/buildroot/package/fog/scripts/bin/* $ipxeroot/tmp/bin/ >/dev/null 2>&1
    cp -r /root/trunk/src/buildroot/package/fog/scripts/usr/* $ipxeroot/tmp/usr/ >/dev/null 2>&1
    cp -r /root/trunk/src/buildroot/package/fog/scripts/etc/* $ipxeroot/tmp/etc/ >/dev/null 2>&1
    cp -r /root/trunk/src/buildroot/system/skeleton/etc/* $ipxeroot/tmp/etc/ >/dev/null 2>&1
    errorStat $?
}
mountTmpFolder() {
    mount -o loop $ipxeroot/$1 $ipxeroot/tmp >/dev/null 2>&1
    if [ -z "$2" ]; then
        errorStat $?
    elif [ "$?" != "0" -a -f "$ipxeroot/$1" ]; then
        unmountTmpFolder "true"
        mountTmpFolder "$1"
    fi
}
unmountTmpFolder() {
    umount $ipxeroot/tmp >/dev/null 2>&1
    if [ -z "$1" ]; then
        errorStat $?
    fi
}
initFSCheck() {
    fsck.ext2 -a $ipxeroot/$1 >/dev/null 2>&1
    errorStat $?
}
recompressInit() {
    xz -9 -C crc32 >/dev/null 2>&1 < $ipxeroot/$1 > $ipxeroot/${1}.xz
    errorStat $?
}
decompressInit() {
    xz -d >/dev/null 2>&1 < $ipxeroot/${1}.xz > $ipxeroot/$1
    errorStat $?
}
errorStat() {
    if [ "$1" != "0" ]; then
        echo "Failed"
        exit 1
    fi
    echo "OK"
}
if [ -z "$init64" ]; then
    if [ -f "$ipxeroot/init.xz" -a ! -f "$ipxeroot/init" ]; then
        echo -n " * Decompressing 64 bit init..."
        decompressInit "init"
    else
        echo " * 64 bit init already extracted"
    fi
    echo -n " * Mounting 64 bit init..."
    mountTmpFolder "init" "yes"
    echo -n " * Copying trunk files..."
    copyTrunkFiles
    echo -n " * Unmounting init..."
    unmountTmpFolder
    echo -n " * Checking ext2 filesystem..."
    initFSCheck "init"
    echo -n " * Recompressing 64 bit Init..."
    recompressInit "init"
fi
if [ -z "$init_32" ]; then
    if [ -f "$ipxeroot/init_32.xz" -a ! -f "$ipxeroot/init_32" ]; then
        echo -n " * Decompressing 32 bit init..."
        decompressInit "init_32"
    else
        echo " * 32 bit init already extracted"
    fi
    echo -n " * Mounting 32 bit init..."
    mountTmpFolder "init_32" "yes"
    echo -n " * Copying trunk files..."
    copyTrunkFiles
    echo -n " * Unmounting init..."
    unmountTmpFolder
    echo -n " * Checking ext2 filesystem..."
    initFSCheck "init_32"
    echo -n " * Recompressing 32 bit Init..."
    recompressInit "init_32"
fi

Tom Elliott · May 23, 2016, 9:50 PM

Fog git sync
– Syncs the git respository with the svn repository information
– I call it foggitsync.sh
– Only used to sync/merge the dev-branch information

#!/bin/bash

cd /root/fogproject
# Sync SVN with git repo
git checkout svn 
git svn -A authors.txt rebase
git commit -a --allow-empty-message -m ''
# Merge changes from svn to dev-branch
git checkout dev-branch
git pull origin dev-branch
git merge --strategy-option=theirs svn -m 'Merge svn with dev-branch'
# Push changes to remote
git push origin svn dev-branch

Wayne Workman · Oct 27, 2016, 8:26 PM

Just quickly copy/pasting some scripts I made for managing 15 storage nodes without killing myself.

I’m going to take all these and other stuff community has made and pile them all up into one big git repo called “Fog Tools” or something.

All of the below scripts in this post are dependent on ssh-pki being setup from the server to all nodes and having aliases created for them all, I have created a tool that helps you do this: https://github.com/wayneworkman/ssh-pki-setup

installPackage.sh

#!/bin/bash
array=( aifog annex bmfog clfog cvfog ckfog dufog fmfog hffog jwfog lhfog prfog rofog wgfog )

#Packages are space delimited if there are multiple ones.
packages="lsof"

for i in "${array[@]}"
do
    printf "Installing $packages at: $i..."
    successCheck=$(ssh $i "yum install $packages -y > /dev/null 2>&1;echo \$?")
    if [[ "$successCheck" -eq 0 ]]; then
        printf "Success!\n"
    else
        printf "Failed!\n"
    fi
    printf "\n"


done

removePackage.sh

#!/bin/bash
array=( aifog annex bmfog clfog cvfog ckfog dufog fmfog hffog jwfog lhfog prfog rofog wgfog )

#Packages are space delimited if there are multiple ones.
packages="mod_evasive"

for i in "${array[@]}"
do
    printf "Removing $packages at: $i..."
    successCheck=$(ssh $i "yum remove $packages -y > /dev/null 2>&1;echo \$?")
    if [[ "$successCheck" -eq 0 ]]; then
        printf "Success!\n"
    else
        printf "Failed!\n"
    fi
    printf "\n"


done

rebootNodes.sh

#!/bin/bash
ssh aifog "reboot"
ssh annex "reboot"
ssh bmfog "reboot"
ssh clfog "reboot"
ssh cvfog "reboot"
ssh ckfog "reboot"
ssh dufog "reboot"
ssh fmfog "reboot"
ssh hffog "reboot"
ssh jwfog "reboot"
ssh lhfog "reboot"
ssh prfog "reboot"
ssh rofog "reboot"
ssh wgfog "reboot"

updateNodesOS.sh

#!/bin/bash
array=( aifog annex bmfog clfog cvfog ckfog dufog fmfog hffog jwfog lhfog prfog rofog wgfog )

clear
echo
echo
echo "Updating defined systems."
echo
for i in "${array[@]}"
do
    printf "Updating installed packages at: $i..."
    successCheck=$( ssh $i "yum update -y > /dev/null 2>&1;echo \$?")
    if [[ "$successCheck" -eq 0 ]]; then
        printf "Success!\n"
    else
        printf "Failed!\n"
    fi
    printf "\n"
done

updateNodesFOG.sh

#!/bin/bash
array=( aifog annex bmfog clfog cvfog ckfog dufog fmfog hffog jwfog lhfog prfog rofog wgfog )

clear
echo
echo
echo "Updating FOG on defined systems."
echo
for i in "${array[@]}"
do
    printf "Updating FOG at: $i..."
    successCheck=$(ssh $i "cd /root/git/fogproject > /dev/null 2>&1;git reset --hard > /dev/null 2>&1;git pull > /dev/null 2>&1;cd bin > /dev/null 2>&1;./installfog.sh -y > /dev/null 2>&1;echo \$?")
    if [[ "$successCheck" -eq 0 ]]; then
        printf "Success!\n"
    else
        printf "Failed!\n"
    fi
    printf "\n"


done

Joe Schmitt · Oct 30, 2016, 5:34 PM

We have created a dedicated repository for community scripts: https://github.com/FOGProject/fog-community-scripts. All scripts in this repository are under the MIT license, and contain a brief README describing basic usage.

Useful Scripts

197

12.0k

17.3k

155.1k