RC10 Broken Items on upgrade

Tom Elliott

@adukes40 is the GUI becoming responsive?

adukes40

@Tom-Elliott Stopping them doesnt seem to stop the traffic. All sites are back up to high usage, with master site 100% outgoing traffic pegged. This is after stopping those two services on all nodes

Tom Elliott

@adukes40 then can you get the latest ten lines of the Apache access logs?

adukes40

@Tom-Elliott

10.104.12.83 - - [13/Sep/2016:15:45:49 -0400] “-” 408 0 “-” “-”
10.104.12.196 - - [13/Sep/2016:15:47:02 -0400] “-” 408 0 “-” “-”
10.104.12.13 - - [13/Sep/2016:15:49:13 -0400] “-” 408 0 “-” “-”
10.104.12.103 - - [13/Sep/2016:15:49:15 -0400] “-” 408 0 “-” “-”
10.104.12.59 - - [13/Sep/2016:15:50:04 -0400] “-” 408 0 “-” “-”
10.104.12.105 - - [13/Sep/2016:15:50:20 -0400] “-” 408 0 “-” “-”
10.105.13.139 - - [13/Sep/2016:15:51:16 -0400] “-” 408 0 “-” “-”
10.104.12.20 - - [13/Sep/2016:15:55:27 -0400] “-” 408 0 “-” “-”
10.102.12.74 - - [13/Sep/2016:15:57:08 -0400] “-” 408 0 “-” “-”
10.104.12.96 - - [13/Sep/2016:15:57:12 -0400] “-” 408 0 “-” “-”

Tom Elliott

@adukes40 do you have a bunch of snapins?

adukes40

@Tom-Elliott

about 700ish powershell scripts. Thats it, but they are all at the nodes. I did see in the Snapin Replication log it mentions these.

Tom Elliott

@adukes40 Are they large? Are they all being deployed?

adukes40

@Tom-Elliott
In all it says size is 244,133 bytes of the snapins directory

Tom Elliott

@adukes40 so about 250 kilo bytes

Tom Elliott

Remember how you cleared encryption data

adukes40

@Tom-Elliott

I can find the command again yea

Tom Elliott

@Tom-Elliott I need you to clear shash for om snapin table

Tom Elliott

Update snapins set sHash=‘’

adukes40

@Tom-Elliott

mysql> update snapins set sHash=‘’;
Query OK, 0 rows affected (0.01 sec)
Rows matched: 746 Changed: 0 Warnings: 0

I have to go for now, keep the message coming and I will respond once I get a minutes, I have to take my son to soccer practice.

Tom Elliott

@adukes40 any better on GUI?

Tom Elliott

@Tom-Elliott hell try update snapins set sHash='abc';
Please

adukes40

@Tom-Elliott As discussed last night, around 6pm bandwidth ded down and the GUI ran fine. This morning it is back to what it was doing yesterday. As requested here is the tail -f of the access.log

root@MSDCATS09:/var/log/apache2# tail -f access.log
::1 - - [14/Sep/2016:08:05:39 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
10.102.12.182 - - [14/Sep/2016:08:05:39 -0400] “-” 408 0 “-” “-”
::1 - - [14/Sep/2016:08:05:40 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:41 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:42 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:43 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:44 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:45 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:46 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:47 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”

Also here is a screenshot of what I see on the bandwidth monitor:

You can see the 6PM, and then this morning when I disabled the NIC, other than that something is being pushed from this server, but no clue what.

Wayne Workman

@adukes40 Please install apachetop.
Ubuntu/Debian:
sudo apt-get install apachetop -y
Fedora/RHEL/CentOS:
sudo yum install apachetop -y

Then run it:
apachetop

Stop running it with ctrl+c

It’ll give you counters for page requests, and file requests. Let it run for a few minutes and then take a screenshot of it, and post here please.

Also, you can install iftop as well, to monitor bandwidth being sent to individual IP addresses. Install as normal, i.e. yum install iftop or apt-get install iftop and then run with:
iftop -n

Stop running it with ctrl+c

I’d like a screenshot of that too.