RC10 Broken Items on upgrade

[adukes40]

@Tom-Elliott mysql> describe hosts \G
*************************** 1. row ***************************
Field: hostID
Type: int(11)
Null: NO
Key: PRI
Default: NULL
Extra: auto_increment
*************************** 2. row ***************************
Field: hostName
Type: varchar(16)
Null: NO
Key: UNI
Default: NULL
Extra:
*************************** 3. row ***************************
Field: hostDesc
Type: longtext
Null: NO
Key:
Default: NULL
Extra:
*************************** 4. row ***************************
Field: hostIP
Type: varchar(25)
Null: NO
Key: MUL
Default: NULL
Extra:
*************************** 5. row ***************************
Field: hostImage
Type: int(11)
Null: NO
Key:
Default: NULL
Extra:
*************************** 6. row ***************************
Field: hostBuilding
Type: int(11)
Null: NO
Key:
Default: NULL
Extra:
*************************** 7. row ***************************
Field: hostCreateDate
Type: timestamp
Null: NO
Key:
Default: CURRENT_TIMESTAMP
Extra:
*************************** 8. row ***************************
Field: hostLastDeploy
Type: datetime
Null: NO
Key:
Default: NULL
Extra:
*************************** 9. row ***************************
Field: hostCreateBy
Type: varchar(50)
Null: NO
Key:
Default: NULL
Extra:
*************************** 10. row ***************************
Field: hostUseAD
Type: char(1)
Null: NO
Key: MUL
Default: NULL
Extra:
*************************** 11. row ***************************
Field: hostADDomain
Type: varchar(250)
Null: NO
Key:
Default: NULL
Extra:
*************************** 12. row ***************************
Field: hostADOU
Type: longtext
Null: NO
Key:
Default: NULL
Extra:
*************************** 13. row ***************************
Field: hostADUser
Type: varchar(250)
Null: NO
Key:
Default: NULL
Extra:
*************************** 14. row ***************************
Field: hostADPass
Type: varchar(250)
Null: NO
Key:
Default: NULL
Extra:
*************************** 15. row ***************************
Field: hostADPassLegacy
Type: longtext
Null: NO
Key:
Default: NULL
Extra:
*************************** 16. row ***************************
Field: hostProductKey
Type: longtext
Null: YES
Key:
Default: NULL
Extra:
*************************** 17. row ***************************
Field: hostPrinterLevel
Type: varchar(2)
Null: NO
Key:
Default: NULL
Extra:
*************************** 18. row ***************************
Field: hostKernelArgs
Type: varchar(250)
Null: NO
Key:
Default: NULL
Extra:
*************************** 19. row ***************************
Field: hostKernel
Type: varchar(250)
Null: NO
Key:
Default: NULL
Extra:
*************************** 20. row ***************************
Field: hostDevice
Type: varchar(250)
Null: NO
Key:
Default: NULL
Extra:
*************************** 21. row ***************************
Field: hostInit
Type: longtext
Null: YES
Key:
Default: NULL
Extra:
*************************** 22. row ***************************
Field: hostPending
Type: enum(‘0’,‘1’)
Null: NO
Key:
Default: NULL
Extra:
*************************** 23. row ***************************
Field: hostPubKey
Type: longtext
Null: NO
Key:
Default: NULL
Extra:
*************************** 24. row ***************************
Field: hostSecToken
Type: longtext
Null: NO
Key:
Default: NULL
Extra:
*************************** 25. row ***************************
Field: hostSecTime
Type: timestamp
Null: NO
Key:
Default: 0000-00-00 00:00:00
Extra:
*************************** 26. row ***************************
Field: hostPingCode
Type: varchar(20)
Null: YES
Key:
Default: NULL
Extra:
*************************** 27. row ***************************
Field: hostExitBios
Type: longtext
Null: YES
Key:
Default: NULL
Extra:
*************************** 28. row ***************************
Field: hostExitEfi
Type: longtext
Null: YES
Key:
Default: NULL
Extra:
*************************** 29. row ***************************
Field: hostEnforce
Type: enum(‘0’,‘1’)
Null: NO
Key:
Default: 1
Extra:
29 rows in set (0.00 sec)

[Tom Elliott]

@adukes40 Run:
update hosts set hostPubKey='', hostSecToken='', hostSecTime='0000-00-00 00:00:00';

[adukes40]

@Tom-Elliott mysql> update hosts set hostPubKey=‘’, hostSecToken=‘’, hostSecTime=‘0000-00-00 00:00:00’;
Query OK, 2590 rows affected (0.16 sec)
Rows matched: 6222 Changed: 2590 Warnings: 0

[Wayne Workman]

https://wiki.fogproject.org/wiki/index.php?title=FOG_Client#Manually_Reset_Encryption_On_ALL_Hosts

[Tom Elliott]

@adukes40 Now it’s a waiting game. I suspect you have some older new clients who were stuck in a loop trying to update their clients.

[adukes40]

should I still continue on going up to RC-11?

[Wayne Workman]

@adukes40 just chill for 10 minutes. Then check top again, and the dashboard again.

[adukes40]

@Wayne-Workman Ok, well I will go grab some lunch and then check on it.

[adukes40]

Ok back from lunch. it seems to be hit or miss i fit loads, which is better than it was. Starts to look like the normal pages. Still having timeouts, and slow performance. Checkin while trying to image still takes time. and here is TOP

[adukes40]

Something noticed. So I updated the servers to RC-10 this morning around 7:30am. Since then we have had three sites fully maxed out on bandwidth. a 10mb site, and 2 100mb sites. Complete 100% full pipe. I just disabled the virtual NIC on the master FOG server, and wala bandwidth dropped majorly.

What can cause this to happen? it also happened last month when I upgraded to RC-7. In that case it start replicated images for what ever reason. But since that time I turned off replication on the images so it would not happen next time. So from 7:30 this morning until 3:00 this afternoon, the outbound traffic of the master site has been maxed out while other sites incoming have been maxed or extremely high, and it was resolved by disabling the NIC.

[Tom Elliott]

@adukes40 can you check the replication logs?

[adukes40]

@Tom-Elliott Trying to, but GUI is getting bogged down again. Where on the server itself can I find the logs for that?

[Tom Elliott]

@adukes40 /var /log forward slash fog

[Tom Elliott]

@Tom-Elliott in the interim please stop fog image replicator and snapin

[adukes40]

@Tom-Elliott I stopped them, but I am not seeing anything in the replicator.log, seems pretty empty. And that was from 8:47 this morning. anything with newer timestamps doesn’t make sense or isn’t one of the sites in question.

[Tom Elliott]

@adukes40 pleSe stop the same services on all nodes?

[adukes40]

@Tom-Elliott ok give me a few minutes

[adukes40]

@Tom-Elliott so far I receive this on the nodes:

root@MSDBETS09:~# service FOGImageReplicator stop

• Stopping FOG Computer Imaging Solution: FOGImageReplicator start-stop-daemon: warning: failed to kill 1148: No such process
  [ OK ]
  root@MSDBETS09:~# service FOGSnapinReplicator stop
• Stopping FOG Computer Imaging Solution: FOGSnapinReplicator start-stop-daemon: warning: failed to kill 1179: No such process
  [ OK ]

[Tom Elliott]

@adukes40 is the GUI becoming responsive?

[adukes40]

@Tom-Elliott Stopping them doesnt seem to stop the traffic. All sites are back up to high usage, with master site 100% outgoing traffic pegged. This is after stopping those two services on all nodes