Useful Scripts


  • Senior Developer

    Hey all,

    I know this probably seems rather random, but this is just a place for me to put some of my scripts during programming for FOG.

    While these are specifically setup for my environment, simple edits can make them portable.

    If you all want to try your hand on these for your own projects, feel free.

    1. Copy to svn
      – All this does is take my files as I have edited them and copies them to my trunk directory for subversion/svn.
      – I call it copytosvn
      – The extra “trunk2” stuff you see is for btsync when that was running.
      – if I call it as copytosvn git it will do the same as for trunk, but to only the git. This was added due to sourceforge being down last year.
    #!/bin/bash
    dots() {
        local pad=$(printf "%0.1s" "."{1..60})
        printf " * %s%*.*s" "$1" 0 $((60-${#1})) "$pad"
    }
    errorStat() {
        if [ "$1" != 0 ]; then
            echo "Failed"
            exit 1
        fi
        echo "OK"
    }
    trunkUpdate() {
        cwd=`pwd`
        if [[ -z $1 || $1 == svn ]]; then
            cd /root/trunk
            dots "Updating SVN Directory"
            /usr/bin/svn update >/dev/null 2>&1
            cd /root/fogproject
        elif [[ $1 == git ]]; then
            dots "Updating GIT Directory"
            cd /root/fogproject
            /usr/bin/git checkout dev-branch >/dev/null 2>&1
            /usr/bin/git pull >/dev/null 2>&1
        fi
        errorStat $?
    }
    versionUpdate() {
        dots "Updating Version in File"
        trunkver=`git describe --tags | awk -F'-[^0-9]*' '{value=$2+$3+1; print value}'` >/dev/null 2>&1
        sed -i "s/define('FOG_VERSION'.*);/define('FOG_VERSION', '$trunkver');/g" /var/www/fog/lib/fog/system.class.php >/dev/null 2>&1
    //g" /var/www/fog/lib/fog/system.class.php >/dev/null 2>&1
        errorStat $?
    }
    copyFilesToTrunk() {
        dots "Copying files to trunk"
        for filename in `find /var/www/fog -type d`; do
            if [[ -z $1 || $1 == svn ]]; then
                cp -r $filename/* /root/trunk/packages/web/${filename#/var/www/fog} >/dev/null 2>&1
            elif [[ $1 == git ]]; then
                cp -r $filename/* /root/fogproject/packages/web/${filename#/var/www/fog} >/dev/null 2>&1
            fi
        done
        if [[ -z $1 || $1 == svn ]]; then
            rm -rf /root/trunk/packages/web/lib/fog/config.class.php >/dev/null 2>&1
            rm -rf /root/trunk/packages/web/management/other/cache/* >/dev/null 2>&1
            rm -rf /root/trunk/packages/web/management/other/ssl >/dev/null 2>&1
            rm -rf /root/trunk/packages/web/status/injectHosts.php >/dev/null 2>&1
            rm -rf /root/trunk2/* >/dev/null 2>&1
            cp -r /root/trunk/* /root/trunk2/ >/dev/null 2>&1
        elif [[ $1 == git ]]; then
            rm -rf /root/fogproject/packages/web/lib/fog/config.class.php >/dev/null 2>&1
            rm -rf /root/fogproject/packages/web/management/other/cache/* >/dev/null 2>&1
            rm -rf /root/fogproject/packages/web/management/other/ssl >/dev/null 2>&1
            rm -rf /root/fogproject/packages/web/status/injectHosts.php >/dev/null 2>&1
            rm -rf /root/trunk2/* >/dev/null 2>&1
            cp -r /root/fogproject/* /root/trunk2/ >/dev/null 2>&1
        fi
        errorStat $?
    }
    makefogtar() {
        dots "Creating FOG Tar File"
        rm -rf /opt/fog_trunk
        if [[ -z $1 || $1 == svn ]]; then
            cp -r /root/trunk /opt/fog_trunk
            find /opt/fog_trunk -name .svn -exec rm -rf {} \; >/dev/null 2>&1
        elif [[ $1 == git ]]; then
            cp -r /root/fogproject /opt/fog_trunk
            find /opt/fog_trunk -name .git -exec rm -rf {} \; >/dev/null 2>&1
        fi
        cd /opt
        tar -cjf /var/www/html/fog_trunk.tar.bz2 fog_trunk
        rm -rf /opt/fog_trunk
        cd $cwd
        unset cwd
        errorStat $?
    }
    trunkUpdate $1
    versionUpdate $1
    copyFilesToTrunk $1
    makefogtar $1
    
    1. Copy back trunk
      – Copies the file in the trunk folder back to the base folder. Useful if i’m making changes that seem to break things and just need to start fresh, or if something needs to be looked at as its broken, but known working in older version. Allows me to revert the folder and copy back the files in the trunk to the base so I can see things more clearly.
      – Create a backup of your original config.class.php in the root directory.
      – I named it copybacktrunk
    #!/bin/bash
    rm -rf /var/www/fog;cp -r ~/trunk/packages/web /var/www/fog; cp ~/config.class.php /var/www/fog/lib/fog/config.class.php;chown -R apache:apache /var/www/fog; chown -R 500:apache /var/www/fog/service/ipxe
    

  • Senior Developer

    We have created a dedicated repository for community scripts: https://github.com/FOGProject/fog-community-scripts. All scripts in this repository are under the MIT license, and contain a brief README describing basic usage.


  • Moderator

    Just quickly copy/pasting some scripts I made for managing 15 storage nodes without killing myself.

    I’m going to take all these and other stuff community has made and pile them all up into one big git repo called “Fog Tools” or something.

    All of the below scripts in this post are dependent on ssh-pki being setup from the server to all nodes and having aliases created for them all, I have created a tool that helps you do this: https://github.com/wayneworkman/ssh-pki-setup

    installPackage.sh

    #!/bin/bash
    array=( aifog annex bmfog clfog cvfog ckfog dufog fmfog hffog jwfog lhfog prfog rofog wgfog )
    
    #Packages are space delimited if there are multiple ones.
    packages="lsof"
    
    for i in "${array[@]}"
    do
        printf "Installing $packages at: $i..."
        successCheck=$(ssh $i "yum install $packages -y > /dev/null 2>&1;echo \$?")
        if [[ "$successCheck" -eq 0 ]]; then
            printf "Success!\n"
        else
            printf "Failed!\n"
        fi
        printf "\n"
    
    
    done
    

    removePackage.sh

    #!/bin/bash
    array=( aifog annex bmfog clfog cvfog ckfog dufog fmfog hffog jwfog lhfog prfog rofog wgfog )
    
    #Packages are space delimited if there are multiple ones.
    packages="mod_evasive"
    
    for i in "${array[@]}"
    do
        printf "Removing $packages at: $i..."
        successCheck=$(ssh $i "yum remove $packages -y > /dev/null 2>&1;echo \$?")
        if [[ "$successCheck" -eq 0 ]]; then
            printf "Success!\n"
        else
            printf "Failed!\n"
        fi
        printf "\n"
    
    
    done
    

    rebootNodes.sh

    #!/bin/bash
    ssh aifog "reboot"
    ssh annex "reboot"
    ssh bmfog "reboot"
    ssh clfog "reboot"
    ssh cvfog "reboot"
    ssh ckfog "reboot"
    ssh dufog "reboot"
    ssh fmfog "reboot"
    ssh hffog "reboot"
    ssh jwfog "reboot"
    ssh lhfog "reboot"
    ssh prfog "reboot"
    ssh rofog "reboot"
    ssh wgfog "reboot"
    

    updateNodesOS.sh

    #!/bin/bash
    array=( aifog annex bmfog clfog cvfog ckfog dufog fmfog hffog jwfog lhfog prfog rofog wgfog )
    
    clear
    echo
    echo
    echo "Updating defined systems."
    echo
    for i in "${array[@]}"
    do
        printf "Updating installed packages at: $i..."
        successCheck=$( ssh $i "yum update -y > /dev/null 2>&1;echo \$?")
        if [[ "$successCheck" -eq 0 ]]; then
            printf "Success!\n"
        else
            printf "Failed!\n"
        fi
        printf "\n"
    done
    

    updateNodesFOG.sh

    #!/bin/bash
    array=( aifog annex bmfog clfog cvfog ckfog dufog fmfog hffog jwfog lhfog prfog rofog wgfog )
    
    clear
    echo
    echo
    echo "Updating FOG on defined systems."
    echo
    for i in "${array[@]}"
    do
        printf "Updating FOG at: $i..."
        successCheck=$(ssh $i "cd /root/git/fogproject > /dev/null 2>&1;git reset --hard > /dev/null 2>&1;git pull > /dev/null 2>&1;cd bin > /dev/null 2>&1;./installfog.sh -y > /dev/null 2>&1;echo \$?")
        if [[ "$successCheck" -eq 0 ]]; then
            printf "Success!\n"
        else
            printf "Failed!\n"
        fi
        printf "\n"
    
    
    done
    

  • Senior Developer

    Fog git sync
    – Syncs the git respository with the svn repository information
    – I call it foggitsync.sh
    – Only used to sync/merge the dev-branch information

    #!/bin/bash
    
    cd /root/fogproject
    # Sync SVN with git repo
    git checkout svn 
    git svn -A authors.txt rebase
    git commit -a --allow-empty-message -m ''
    # Merge changes from svn to dev-branch
    git checkout dev-branch
    git pull origin dev-branch
    git merge --strategy-option=theirs svn -m 'Merge svn with dev-branch'
    # Push changes to remote
    git push origin svn dev-branch
    

  • Senior Developer

    Init redo
    – Allows me to redo the inits without having to do a full rebuild
    – I call it init_redo.sh
    – Takes one argument, for now. The argument will be the webroot to look for the ipxe files. If it is not set or cannot be found it will try to find the webroot under the more common locations"

    #!/bin/bash
    # Check for webroot
    if [ -n "$1" -a -e "$1" ]; then
        webroot="$1";
    elif [ -e "/srv/http/fog" ]; then
        webroot="/srv/http/fog";
    elif [ -e "/var/www/html/fog" ]; then
        webroot="/var/www/html/fog";
    elif [ -e "/var/www/fog" ]; then
        webroot="/var/www/fog";
    fi
    if [ ! -e "$webroot/service" ]; then
        echo "No service directory to work from"
        exit 1
    fi
    if [ ! -e "$webroot/service/ipxe" ]; then
        echo "No ipxe directory to work from"
        exit 1
    fi
    ipxeroot="$webroot/service/ipxe"
    currDirectory=`pwd`
    init64=''
    init32=''
    if [ ! -f "$ipxeroot/init.xz" -a ! -f "$ipxeroot/init" ]; then
        echo "No 64 bit init to process"
        init64='no'
    fi
    if [ ! -f "$ipxeroot/init_32.xz" -a ! -f "$ipxeroot/init_32" ]; then
        echo "No 32 bit init to process"
        init32='no'
    fi
    if [ -n "$init64" -a -n "$init32" ]; then
        echo "No init files to process"
        exit 1
    fi
    if [ ! -e "$ipxeroot/tmp" ]; then
        mkdir $ipxeroot/tmp >/dev/null 2>&1
    fi
    copyTrunkFiles() {
        svn up /root/trunk/ >/dev/null 2>&1
        cp -r /root/trunk/src/buildroot/package/fog/scripts/bin/* $ipxeroot/tmp/bin/ >/dev/null 2>&1
        cp -r /root/trunk/src/buildroot/package/fog/scripts/usr/* $ipxeroot/tmp/usr/ >/dev/null 2>&1
        cp -r /root/trunk/src/buildroot/package/fog/scripts/etc/* $ipxeroot/tmp/etc/ >/dev/null 2>&1
        cp -r /root/trunk/src/buildroot/system/skeleton/etc/* $ipxeroot/tmp/etc/ >/dev/null 2>&1
        errorStat $?
    }
    mountTmpFolder() {
        mount -o loop $ipxeroot/$1 $ipxeroot/tmp >/dev/null 2>&1
        if [ -z "$2" ]; then
            errorStat $?
        elif [ "$?" != "0" -a -f "$ipxeroot/$1" ]; then
            unmountTmpFolder "true"
            mountTmpFolder "$1"
        fi
    }
    unmountTmpFolder() {
        umount $ipxeroot/tmp >/dev/null 2>&1
        if [ -z "$1" ]; then
            errorStat $?
        fi
    }
    initFSCheck() {
        fsck.ext2 -a $ipxeroot/$1 >/dev/null 2>&1
        errorStat $?
    }
    recompressInit() {
        xz -9 -C crc32 >/dev/null 2>&1 < $ipxeroot/$1 > $ipxeroot/${1}.xz
        errorStat $?
    }
    decompressInit() {
        xz -d >/dev/null 2>&1 < $ipxeroot/${1}.xz > $ipxeroot/$1
        errorStat $?
    }
    errorStat() {
        if [ "$1" != "0" ]; then
            echo "Failed"
            exit 1
        fi
        echo "OK"
    }
    if [ -z "$init64" ]; then
        if [ -f "$ipxeroot/init.xz" -a ! -f "$ipxeroot/init" ]; then
            echo -n " * Decompressing 64 bit init..."
            decompressInit "init"
        else
            echo " * 64 bit init already extracted"
        fi
        echo -n " * Mounting 64 bit init..."
        mountTmpFolder "init" "yes"
        echo -n " * Copying trunk files..."
        copyTrunkFiles
        echo -n " * Unmounting init..."
        unmountTmpFolder
        echo -n " * Checking ext2 filesystem..."
        initFSCheck "init"
        echo -n " * Recompressing 64 bit Init..."
        recompressInit "init"
    fi
    if [ -z "$init_32" ]; then
        if [ -f "$ipxeroot/init_32.xz" -a ! -f "$ipxeroot/init_32" ]; then
            echo -n " * Decompressing 32 bit init..."
            decompressInit "init_32"
        else
            echo " * 32 bit init already extracted"
        fi
        echo -n " * Mounting 32 bit init..."
        mountTmpFolder "init_32" "yes"
        echo -n " * Copying trunk files..."
        copyTrunkFiles
        echo -n " * Unmounting init..."
        unmountTmpFolder
        echo -n " * Checking ext2 filesystem..."
        initFSCheck "init_32"
        echo -n " * Recompressing 32 bit Init..."
        recompressInit "init_32"
    fi
    

  • Senior Developer

    Build iPXE
    – Build’s the ipxe files and updates the trunk version via remote methods now.
    – I call it buildIpxe
    – Adding arguments just add iPXE arguments to build with extra args.
    For example: buildIpxe DEBUG=dhcp will build the ipxe files with the DEBUG=dhcp enabled on the built files.

    #!/bin/bash
    cwd=`pwd`
    cd /root/ipxe
    git pull
    cd /root/ipxe/src/
    make -j $(nproc) bin/ipxe.iso bin/{undionly,ipxe,intel,realtek}.{,k,kk}pxe bin/ipxe.lkrn EMBED=ipxescript $*
    scp -P 10002 /root/ipxe/src/bin/{undionly,ipxe,intel,realtek}.{,k,kk}pxe mastaweb:/root/trunk/packages/tftp/
    scp -P 10002 /root/ipxe/src/bin/ipxe.lkrn mastaweb:/root/trunk/packages/tftp/ipxe.krn
    scp -P 10002 /root/ipxe/src/bin/ipxe.iso mastaweb:/root/trunk/packages/tftp/ipxe.iso
    scp -P 10002 /root/ipxe/src/ipxescript mastaweb:/root/trunk/src/ipxe/src/ipxescript
    scp -P 10002 /root/ipxe/src/config/general.h mastaweb:/root/trunk/src/ipxe/src/config/general.h
    scp -P 10002 /root/ipxe/src/config/settings.h mastaweb:/root/trunk/src/ipxe/src/config/settings.h
    scp -P 10002 /root/ipxe/src/config/console.h mastaweb:/root/trunk/src/ipxe/src/config/console.h
    cd /root/ipxe-efi
    git pull
    cd /root/ipxe-efi/src/
    make -j $(nproc) bin-{i386,x86_64}-efi/{snp{,only},ipxe,intel,realtek}.efi EMBED=ipxescript $*
    scp -P 10002 /root/ipxe-efi/src/bin-i386-efi/{snp{,only},ipxe,intel,realtek}.efi mastaweb:/root/trunk/packages/tftp/i386-efi/
    scp -P 10002 /root/ipxe-efi/src/bin-x86_64-efi/{snp{,only},ipxe,intel,realtek}.efi mastaweb:/root/trunk/packages/tftp/
    scp -P 10002 /root/ipxe-efi/src/ipxescript mastaweb:/root/trunk/src/ipxe/src-efi/ipxescript
    scp -P 10002 /root/ipxe-efi/src/config/general.h mastaweb:/root/trunk/src/ipxe/src-efi/config/general.h
    scp -P 10002 /root/ipxe-efi/src/config/settings.h mastaweb:/root/trunk/src/ipxe/src-efi/config/settings.h
    scp -P 10002 /root/ipxe-efi/src/config/console.h mastaweb:/root/trunk/src/ipxe/src-efi/config/console.h
    cd $cwd
    unset cwd
    

  • Senior Developer

    Build buildroot
    – Copies the files from the main trunk directory of a remote system (my dev box) and places the files where needed.
    – Does require pre-environment setup (config files etc… already in place.
    – I call it buildBuildroot
    – Copies the init’s back to my other servers as needed.

    #!/bin/bash
    arguments="$*"
    dots() {
        local pad=$(printf "%0.1s" "."{1..60})
        printf " * %s%*.*s" "$1" 0 $((60-${#1})) "$pad"
    }
    errorStat() {
        local status=$1
        if [[ $status != 0 ]]; then
            echo "Failed"
            return 1
        fi
        echo "Done"
        return 0
    }
    getCustomBuildroot() {
        dots "Copying customized 64 bit init files"
        scp -rP 10002 mastaweb:/root/trunk/src/buildroot/* /root/buildroot/ >/dev/null 2>&1
        errorStat $?
        dots "Copying customized 32 bit init files"
        scp -rP 10002 mastaweb:/root/trunk/src/buildroot/* /root/32build/buildroot/ >/dev/null 2>&1
        errorStat $?
    }
    build64bit() {
        dots "Moving to 64 bit build environment"
        if [[ -d /root/buildroot ]]; then
            cd /root/buildroot
        fi
        errorStat $?
        [[ $? != 0 ]] && exit 1
        dots "Removing fog build directory to start fresh"
        if [[ -d "/root/buildroot/output/build/fog-1" ]]; then
            rm -rf "/root/buildroot/output/build/fog-1" >/dev/null 2>&1
        fi
        errorStat $?
        [[ $? != 0 ]] && exit 1
        dots "Building 64 bit inits"
        make -j $(nproc) >/root/buildroot/buildroot64.log 2>&1
        errorStat $?
        [[ $? != 0 ]] && exit 1
        dots "Copying 64 bit inits to remotes"
        scp output/images/rootfs.ext2.xz ubuntu64:/var/www/fog/service/ipxe/init.xz >/dev/null 2>&1 && \
        scp -P 10002 output/images/rootfs.ext2.xz mastaweb:/var/www/html/init.xz >/dev/null 2>&1 && \
        scp -P 10008 output/images/rootfs.ext2.xz fogstorage:/var/www/html/fog/service/ipxe/init.xz >/dev/null 2>&1
        errorStat $?
    }
    build32bit() {
        dots "Moving to 32 bit build environment"
        if [[ -d /root/32build/buildroot ]]; then
            cd /root/32build/buildroot >/dev/null 2>&1
        fi
        errorStat $?
        [[ $? != 0 ]] && exit 1
        dots "Removing fog build directory to start fresh"
        if [[ -d "/root/32build/buildroot/output/build/fog-1" ]]; then
            rm -rf "/root/32build/buildroot/output/build/fog-1" >/dev/null 2>&1
        fi
        errorStat $?
        [[ $? != 0 ]] && exit 1
        dots "Building 32 bit inits"
        make -j $(nproc) ARCH=i486 >/root/32build/buildroot/buildroot32.log 2>&1
        errorStat $?
        [[ $? != 0 ]] && exit 1
        dots "Copying 32 bit inits to remotes"
        scp output/images/rootfs.ext2.xz ubuntu64:/var/www/fog/service/ipxe/init_32.xz >/dev/null 2>&1 && \
        scp -P 10002 output/images/rootfs.ext2.xz mastaweb:/var/www/html/init_32.xz >/dev/null 2>&1 && \
        scp -P 10008 output/images/rootfs.ext2.xz fogstorage:/var/www/html/fog/service/ipxe/init_32.xz >/dev/null 2>&1
        errorStat $?
    }
    if [[ $arguments == +(*'customfiles'*) ]]; then
        getCustomBuildroot
        exit
    fi
    getCustomBuildroot
    build64bit
    build32bit
    

Log in to reply
 

782
Online

38748
Users

10575
Topics

100103
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.