RC10 Broken Items on upgrade

[adukes40]

@Tom-Elliott mysql> update hosts set hostPubKey=‘’, hostSecToken=‘’, hostSecTime=‘0000-00-00 00:00:00’;
Query OK, 2590 rows affected (0.16 sec)
Rows matched: 6222 Changed: 2590 Warnings: 0

[Wayne Workman]

https://wiki.fogproject.org/wiki/index.php?title=FOG_Client#Manually_Reset_Encryption_On_ALL_Hosts

[Tom Elliott]

@adukes40 Now it’s a waiting game. I suspect you have some older new clients who were stuck in a loop trying to update their clients.

[adukes40]

should I still continue on going up to RC-11?

[Wayne Workman]

@adukes40 just chill for 10 minutes. Then check top again, and the dashboard again.

[adukes40]

@Wayne-Workman Ok, well I will go grab some lunch and then check on it.

[adukes40]

Ok back from lunch. it seems to be hit or miss i fit loads, which is better than it was. Starts to look like the normal pages. Still having timeouts, and slow performance. Checkin while trying to image still takes time. and here is TOP

[adukes40]

Something noticed. So I updated the servers to RC-10 this morning around 7:30am. Since then we have had three sites fully maxed out on bandwidth. a 10mb site, and 2 100mb sites. Complete 100% full pipe. I just disabled the virtual NIC on the master FOG server, and wala bandwidth dropped majorly.

What can cause this to happen? it also happened last month when I upgraded to RC-7. In that case it start replicated images for what ever reason. But since that time I turned off replication on the images so it would not happen next time. So from 7:30 this morning until 3:00 this afternoon, the outbound traffic of the master site has been maxed out while other sites incoming have been maxed or extremely high, and it was resolved by disabling the NIC.

[Tom Elliott]

@adukes40 can you check the replication logs?

[adukes40]

@Tom-Elliott Trying to, but GUI is getting bogged down again. Where on the server itself can I find the logs for that?

[Tom Elliott]

@adukes40 /var /log forward slash fog

[Tom Elliott]

@Tom-Elliott in the interim please stop fog image replicator and snapin

[adukes40]

@Tom-Elliott I stopped them, but I am not seeing anything in the replicator.log, seems pretty empty. And that was from 8:47 this morning. anything with newer timestamps doesn’t make sense or isn’t one of the sites in question.

[Tom Elliott]

@adukes40 pleSe stop the same services on all nodes?

[adukes40]

@Tom-Elliott ok give me a few minutes

[adukes40]

@Tom-Elliott so far I receive this on the nodes:

root@MSDBETS09:~# service FOGImageReplicator stop

• Stopping FOG Computer Imaging Solution: FOGImageReplicator start-stop-daemon: warning: failed to kill 1148: No such process
  [ OK ]
  root@MSDBETS09:~# service FOGSnapinReplicator stop
• Stopping FOG Computer Imaging Solution: FOGSnapinReplicator start-stop-daemon: warning: failed to kill 1179: No such process
  [ OK ]

[Tom Elliott]

@adukes40 is the GUI becoming responsive?

[adukes40]

@Tom-Elliott Stopping them doesnt seem to stop the traffic. All sites are back up to high usage, with master site 100% outgoing traffic pegged. This is after stopping those two services on all nodes

[Tom Elliott]

@adukes40 then can you get the latest ten lines of the Apache access logs?

[adukes40]

@Tom-Elliott

10.104.12.83 - - [13/Sep/2016:15:45:49 -0400] “-” 408 0 “-” “-”
10.104.12.196 - - [13/Sep/2016:15:47:02 -0400] “-” 408 0 “-” “-”
10.104.12.13 - - [13/Sep/2016:15:49:13 -0400] “-” 408 0 “-” “-”
10.104.12.103 - - [13/Sep/2016:15:49:15 -0400] “-” 408 0 “-” “-”
10.104.12.59 - - [13/Sep/2016:15:50:04 -0400] “-” 408 0 “-” “-”
10.104.12.105 - - [13/Sep/2016:15:50:20 -0400] “-” 408 0 “-” “-”
10.105.13.139 - - [13/Sep/2016:15:51:16 -0400] “-” 408 0 “-” “-”
10.104.12.20 - - [13/Sep/2016:15:55:27 -0400] “-” 408 0 “-” “-”
10.102.12.74 - - [13/Sep/2016:15:57:08 -0400] “-” 408 0 “-” “-”
10.104.12.96 - - [13/Sep/2016:15:57:12 -0400] “-” 408 0 “-” “-”