RC10 Broken Items on upgrade

[adukes40]

@Tom-Elliott Stopping them doesnt seem to stop the traffic. All sites are back up to high usage, with master site 100% outgoing traffic pegged. This is after stopping those two services on all nodes

[Tom Elliott]

@adukes40 then can you get the latest ten lines of the Apache access logs?

[adukes40]

@Tom-Elliott

10.104.12.83 - - [13/Sep/2016:15:45:49 -0400] “-” 408 0 “-” “-”
10.104.12.196 - - [13/Sep/2016:15:47:02 -0400] “-” 408 0 “-” “-”
10.104.12.13 - - [13/Sep/2016:15:49:13 -0400] “-” 408 0 “-” “-”
10.104.12.103 - - [13/Sep/2016:15:49:15 -0400] “-” 408 0 “-” “-”
10.104.12.59 - - [13/Sep/2016:15:50:04 -0400] “-” 408 0 “-” “-”
10.104.12.105 - - [13/Sep/2016:15:50:20 -0400] “-” 408 0 “-” “-”
10.105.13.139 - - [13/Sep/2016:15:51:16 -0400] “-” 408 0 “-” “-”
10.104.12.20 - - [13/Sep/2016:15:55:27 -0400] “-” 408 0 “-” “-”
10.102.12.74 - - [13/Sep/2016:15:57:08 -0400] “-” 408 0 “-” “-”
10.104.12.96 - - [13/Sep/2016:15:57:12 -0400] “-” 408 0 “-” “-”

[Tom Elliott]

@adukes40 do you have a bunch of snapins?

[adukes40]

@Tom-Elliott

about 700ish powershell scripts. Thats it, but they are all at the nodes. I did see in the Snapin Replication log it mentions these.

[Tom Elliott]

@adukes40 Are they large? Are they all being deployed?

[adukes40]

@Tom-Elliott
In all it says size is 244,133 bytes of the snapins directory

[Tom Elliott]

@adukes40 so about 250 kilo bytes

[Tom Elliott]

Remember how you cleared encryption data

[adukes40]

@Tom-Elliott

I can find the command again yea

[Tom Elliott]

@Tom-Elliott I need you to clear shash for om snapin table

[Tom Elliott]

Update snapins set sHash=‘’

[adukes40]

@Tom-Elliott

mysql> update snapins set sHash=‘’;
Query OK, 0 rows affected (0.01 sec)
Rows matched: 746 Changed: 0 Warnings: 0

I have to go for now, keep the message coming and I will respond once I get a minutes, I have to take my son to soccer practice.

[Tom Elliott]

@adukes40 any better on GUI?

[Tom Elliott]

@Tom-Elliott hell try update snapins set sHash='abc';
Please

[adukes40]

@Tom-Elliott As discussed last night, around 6pm bandwidth ded down and the GUI ran fine. This morning it is back to what it was doing yesterday. As requested here is the tail -f of the access.log

root@MSDCATS09:/var/log/apache2# tail -f access.log
::1 - - [14/Sep/2016:08:05:39 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
10.102.12.182 - - [14/Sep/2016:08:05:39 -0400] “-” 408 0 “-” “-”
::1 - - [14/Sep/2016:08:05:40 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:41 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:42 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:43 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:44 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:45 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:46 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”
::1 - - [14/Sep/2016:08:05:47 -0400] “OPTIONS * HTTP/1.0” 200 125 “-” “Apache/2. 4.7 (Ubuntu) OpenSSL/1.0.1f (internal dummy connection)”

Also here is a screenshot of what I see on the bandwidth monitor:

You can see the 6PM, and then this morning when I disabled the NIC, other than that something is being pushed from this server, but no clue what.

[Wayne Workman]

@adukes40 Please install apachetop.
Ubuntu/Debian:
sudo apt-get install apachetop -y
Fedora/RHEL/CentOS:
sudo yum install apachetop -y

Then run it:
apachetop

Stop running it with ctrl+c

It’ll give you counters for page requests, and file requests. Let it run for a few minutes and then take a screenshot of it, and post here please.

Also, you can install iftop as well, to monitor bandwidth being sent to individual IP addresses. Install as normal, i.e. yum install iftop or apt-get install iftop and then run with:
iftop -n

Stop running it with ctrl+c

I’d like a screenshot of that too.

[Tom Elliott]

@adukes40 can you disable ipv6?

[adukes40]

@Wayne-Workman

I know this only shows 40 seconds, but i let atop run for 3 minutes, and nothing changed. when I hit ctrl C. its just dumps me back to the terminal

And here is the bandwidth usage. As you can see I re-enabled the NIC at 11:15… instantly starts heading to maxed out.

[Wayne Workman]

@adukes40 For one of those IP addresses, can you remote into one of them and get the c:\fog.log and post the last couple pages of it here?