SVN 6513 Suddenly having couple hundred (already known) hosts showing up under pending...

[Hanz]

I have been having multiple issues with hosts being dropped randomly without any updates or anything being touched. My goups were still there but all of them had 0 for number of hosts while most have around 30 (labs). Having to import saved database to correct the issue.

I have over 1700 hosts as well

Fedora 22
PHP 7.0 on server
SVN 6513
1.7G Ram used out of 8G at idle

[Wayne Workman]

This happens overnight?

[Hanz]

@Wayne-Workman This last instance happened last night while away from all machines.

[Wayne Workman]

@Hanz Can we setup some CRON tasks to monitor your DB so we have some sort of idea about what’s happening at the moment the hosts are lost?

[Hanz]

@Wayne-Workman that’s fine

[Wayne Workman]

This script is CRON ready. Set it up to run under root’s crontab event, for every minute. I can help set this up, but basically,

Switch to root.
Ubuntu/Debian: sudo su
Fedora/CentOS/RHEL: su root

I’d recommend putting the script into /root so:
cd /root

Make a new file called monitor.sh with:
vi monitor.sh
Copy/paste this into the file, save and exit.

Then make that file executable with chmod +x monitor.sh

then enter root’s crontab script with:
crontab -e

and make a entry that looks like:
* * * * * /path/to/the/script/monitor.sh
Then save that.

#----- MySQL Credentials -----#
snmysqluser=""
snmysqlpass=""
snmysqlhost=""
# If user and pass is blank, leave just a set of double quotes like ""
# if the db is local, set the host to just double quotes "" or "127.0.0.1" or "localhost"

#----- Test & Environment Specific Variables -----#
minimumExpectedApprovedHosts=1744
minimumExpectedMacs=3511
log=/root/monitor.log
logHighlights=/root/monitorHighlights.log

#Set this to your $PATH variable.
#You can get your $PATH variable by executing the following command:
# echo $PATH
myPATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin"





#----- Set paths of all used programs -----#

previousIFS=$IFS


IFS=:
  for p in $myPATH; do

        #Get mysql path
        if [[ -f $p/mysql ]]; then
                mysql=$p/mysql
        fi

        #Get echo path
        if [[ -f $p/echo ]]; then
                echo=$p/echo
        fi

        #Get date path
        if [[ -f $p/date ]]; then
                date=$p/date
        fi

        #Get uptime path
        if [[ -f $p/uptime ]]; then
                uptime=$p/uptime
        fi

        #Get free path
        if [[ -f $p/free ]]; then
                free=$p/free
        fi

        #Get ss path
        if [[ -f $p/ss ]]; then
                ss=$p/ss
        fi

  done

#Restore previous contents of IFS
IFS=$previousIFS











#----- Begin Program -----#

sqlCountApprovedHosts="SELECT COUNT(*) FROM hosts WHERE hostPending <> 1"
sqlCountTotalHosts="SELECT COUNT(*) FROM hosts"
sqlCountMacs="SELECT COUNT(*) FROM hostMAC"
sqlShowProcesses="SHOW FULL PROCESSLIST"
NOW=$($date '+%d/%m/%Y %H:%M:%S')
options="-sN"
if [[ $snmysqlhost != "" ]]; then
        options="$options -h$snmysqlhost"
fi
if [[ $snmysqluser != "" ]]; then
        options="$options -u$snmysqluser"
fi
if [[ $snmysqlpass != "" ]]; then
        options="$options -p$snmysqlpass"
fi
options="$options -D fog -e"


$echo "----------------------------------------------------------" >> $log
$echo "----------------------------------------------------------" >> $log
$echo "----------------------------------------------------------" >> $log
$echo "----------------------------------------------------------" >> $log
$echo "----------------------------------------------------------" >> $log
$echo "----------------------------------------------------------" >> $log


$echo $NOW >> $log
$echo " " >> $log

$echo "Number of Approved Hosts:" >> $log
NumberOfApprovedHosts=$($mysql $options "$sqlCountApprovedHosts")
$echo "$NumberOfApprovedHosts" >> $log
$echo " " >> $log

$echo "Number of Total Hosts:" >> $log
NumberOfTotalHosts=$($mysql $options "$sqlCountTotalHosts")
$echo "$NumberOfTotalHosts" >> $log
$echo " " >> $log

$echo "Number of Total MAC addresses:" >> $log
NumberOfTotalMacAddresses=$($mysql $options "$sqlCountMacs")
$echo "$NumberOfTotalMacAddresses" >> $log
$echo " " >> $log

$echo "uptime Output:" >> $log
UptimeOutput=$($uptime)
$echo "$UptimeOutput" >> $log
$echo " " >> $log

$echo "free Output:" >> $log
FreeOutput=$($free)
$echo "$FreeOutput" >> $log
$echo " " >> $log

$echo "Connected TCP Socket Connections:" >> $log
ConnectedTCPSocketConnections=$($ss --tcp)
$echo "$ConnectedTCPSocketConnections" >> $log
$echo " " >> $log

$echo "All current MySQL Processes:" >> $log
AllCurrentMysqlProcesses=$($mysql $options "$sqlShowProcesses")
$echo "$AllCurrentMysqlProcesses" >> $log
$echo " " >> $log


#Below is the check for seperating the interesting logs.
#This can be changed as needed for other purposes.

if [[ $NumberOfApprovedHosts -le $minimumExpectedApprovedHosts ]]; then


        $echo "----------------------------------------------------------" >> $logHighlights
        $echo "----------------------------------------------------------" >> $logHighlights
        $echo "----------------------------------------------------------" >> $logHighlights
        $echo "----------------------------------------------------------" >> $logHighlights
        $echo "----------------------------------------------------------" >> $logHighlights
        $echo "----------------------------------------------------------" >> $logHighlights


        $echo $NOW >> $logHighlights
        $echo " " >> $logHighlights

        $echo "Number of Approved Hosts:" >> $logHighlights
        $echo "$NumberOfApprovedHosts" >> $logHighlights
        $echo " " >> $logHighlights

        $echo "Number of Total Hosts:" >> $logHighlights
        $echo "$NumberOfTotalHosts" >> $logHighlights
        $echo " " >> $logHighlights

        $echo "Number of Total MAC addresses:" >> $logHighlights
        $echo "$NumberOfTotalMacAddresses" >> $logHighlights
        $echo " " >> $logHighlights

        $echo "uptime Output:" >> $logHighlights
        $echo "$UptimeOutput" >> $logHighlights
        $echo " " >> $logHighlights

        $echo "free Output:" >> $logHighlights
        $echo "$FreeOutput" >> $logHighlights
        $echo " " >> $logHighlights

        $echo "Connected TCP Socket Connections:" >> $logHighlights
        $echo "$ConnectedTCPSocketConnections" >> $logHighlights
        $echo " " >> $logHighlights

        $echo "All current MySQL Processes:" >> $logHighlights
        $echo "$AllCurrentMysqlProcesses" >> $logHighlights
        $echo " " >> $logHighlights


fi

[Wayne Workman]

Here’s the github project for that:
https://github.com/wayneworkman/MonitorFOGHosts

[Wayne Workman]

Sample output:

----------------------------------------------------------
----------------------------------------------------------
----------------------------------------------------------
----------------------------------------------------------
----------------------------------------------------------
----------------------------------------------------------
02/03/2016 14:33:01

Pending Hosts:
837     R0219508346WDMB
838     r0219508713WDMB

Number of Approved Hosts:
450

Number of Total Hosts:
452

Number of Total MAC addresses:
493

uptime Output:
 14:33:01 up 20 days,  5:54,  1 user,  load average: 0.77, 0.90, 1.00

free Output:
              total        used        free      shared  buff/cache   available
Mem:        4038628      807812      452320        2652     2778496     3145004
Swap:       4063228       15872     4047356

Connected TCP Socket Connections:
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
LAST-ACK   1      1               10.2.1.11:50359        54.164.24.149:http
ESTAB      0      0               10.2.1.11:50760        54.164.24.149:http
ESTAB      0      64              10.2.1.11:ssh               10.2.3.9:49455
CLOSE-WAIT 1      0               10.2.1.11:50372        54.164.24.149:http
ESTAB      0      0               10.2.1.11:43108       54.209.230.199:http
ESTAB      0      0               10.2.1.11:58418       54.209.230.199:http
SYN-SENT   0      1               10.2.1.11:41105            10.2.3.28:microsoft-ds
LAST-ACK   1      1               10.2.1.11:42702       54.209.230.199:http
CLOSE-WAIT 1      0               10.2.1.11:58319          52.6.165.90:http

All current MySQL Processes:
37      root    localhost       fog     Sleep   7               NULL    0.000
38      root    localhost       fog     Sleep   48              NULL    0.000
39      root    localhost       fog     Sleep   51              NULL    0.000
40      root    localhost       fog     Sleep   52              NULL    0.000
41      root    localhost       fog     Sleep   0               NULL    0.000
7589    root    localhost       fog     Sleep   2               NULL    0.000
8459    root    localhost       fog     Sleep   3               NULL    0.000
9621    root    localhost       fog     Sleep   2               NULL    0.000
11409   root    localhost       fog     Sleep   3               NULL    0.000
12354   root    localhost       fog     Sleep   2               NULL    0.000
12465   root    localhost       fog     Sleep   2               NULL    0.000
12510   root    localhost       fog     Sleep   2               NULL    0.000
12529   root    localhost       fog     Sleep   2               NULL    0.000
12535   root    localhost       fog     Sleep   2               NULL    0.000
12555   root    localhost       fog     Sleep   2               NULL    0.000
12694   root    localhost       fog     Query   0       init    SHOW FULL PROCESSLIST   0.000

[Sebastian Roth]

@Hanz Any news on this yet? I find it very concerning to hear that hosts “drop” into pending state out of nowhere… Hope we can get some more details on this.

[Hanz]

@Sebastian-Roth Sry, it just hasn’t happened again…I actually thought my database kept failing or something, but @Wayne-Workman thought it reminded him of a bug where hosts were disappearing, hence the title. I’ve setup the monitoring script to watch database, but nothing has occurred yet. I will post if/when something happens.

[Sebastian Roth]

@Hanz Searching through the FOG code for ‘hostPending’ and ‘pending’ (minus pending MAC stuff) I can’t really find any place where FOG would set hostPending other than in lib/client/registerclient.class.php - but this is only called when the new client auto-registers an unknown client. In the clients description you’d see ‘Pending Registration created by FOG_CLIENT’ then! Can you please check what the descriptions of those hosts look like?

[Hanz]

@Sebastian-Roth I’m thinking the database crashed honestly, it happened before, but I was trying to fix an issue with hosts that had been named incorrectly and fighting with fog client (I didn’t know was set to re-domain and name…tough fight until I figured out what was happening)… but it just recently happened overnight with nothing touched. All groups went to zero hosts and I had no hosts period. After reinstalling the database it was fixed both times.

[Sebastian Roth]

@Hanz Thanks a lot for your answer! I’ll mark this solved for now but please keep us posted if you see it happen again!!