SVN 4972 to SVN 5046 server load

[Joseph Hales]

I have one of the largest deployments running SVN but like I said this was fine under a previous SVN one of the earlier changes that was made was to remove some sql validation checks that were used previously to clean incomplete records from the db it seems innodb didn’t like the function calls I wonder if these have been readded since?

[Tom Elliott]

@Joseph-Hales I have not added back any invalid entry checks.

[Sebastian Roth]

@Joseph-Hales Thank you so much for the lsof output text file. I feel like this is kind of proofing that it’s the clients causing this massive storm of apache childs and probably mysql load as well. This is why Tom and I are a bit helpless debugging this issue as we don’t have such a big test environment. I just grepped through the output and found roughly 120 established connections to distinct hosts/clients. And another 50 or so in FIN_WAIT state.

I think Tom is pretty convinced that it is something he’s done in the code. Maybe it is but I still wonder why clients keep open connections and even don’t close them properly. Maybe this was introduced with the new client? Sorry if I am heading the wrong way here. It’s just me guessing…

[Joseph Hales]

I am only running the legacy client on my hosts.

[Joseph Hales]

After updating SVN to 5054 and restarting the server completly it still seems high to me.

System Overview
Username	jhales
Web Server	10.200.10.150
TFTP Server	10.200.10.150
Load Average	29.00, 31.61, 20.87
System Uptime	13 min, 0 users```

[Joseph Hales]

FYI I can load the pages but they are extremely slow and sometimes fail to load at all.

[Sebastian Roth]

@Joseph-Hales said:

I am only running the legacy client on my hosts.

Thanks for clarifying!! So this idea was really a dead end. That’s good news as I think we can rule out the possibility of it being an issue introduced by the client. Must be the PHP code then I reckon.

[Wayne Workman]

How about making the checkin time even greater?

I mean, there is no reason for 10,000 hosts in the middle of the school year to be checking for tasks every 2 minutes… that’s kind of absurd.

Why not set it to 2 hours?

[Joseph Hales]

Do they still checkin on boot? The reason I ask is that I would like a newly imaged machine to pull snapins as soon as possible.

[Joseph Hales]

SVN 5058 fixes the main issue or at least makes it manageable.

Username	jhales
Web Server	10.200.10.150
TFTP Server	10.200.10.150
Load Average	6.79, 13.52, 13.55
System Uptime	1:28, 1 user

[Wayne Workman]

@Joseph-Hales Not sure…

but is it worth 10,000 hosts hitting one server every few minutes just so the occasional imaged computer gets snapins fast? To me it’s not worth it.

[Joseph Hales]

we image 20 to 50 pc’s a day on average.

[Wayne Workman]

@Joseph-Hales said:

we image 20 to 50 pc’s a day on average.

Well that’s a lot…

You would benefit a lot from having multiple “Full installations” of FOG, configuring them all to point to the main server for MySQL (for one master DB), and then just setup each one as a storage node. Then, you could slowly migrate all the clients to point to their local server. Long as you have a copy of your SSL key on all the servers (from the main server), you should be good to go.

Dispersing this massive load is ultimately going to be the best way to solve the performance issues.

[Joseph Hales]

Possibly but we are about to update the server hardware and add a storage node but up till now most of the performance issues appear to be database related not client checkin but I will test it next week.

[Wayne Workman]

@Joseph-Hales said:

Possibly but we are about to update the server hardware and add a storage node but up till now most of the performance issues appear to be database related not client checkin but I will test it next week.

It’s not hard to do, and is quite simple to do, but it’s not straight forward.

Install each additional server as a full installation. Notate your ftp user and pass in /opt/fog/.fogsettings for each server.

Point all servers to the main DB via their /opt/fog/.fogsettings file. the mysqlhost, mysqluser, and mysqlpass should be what the main needs.

Enable remote access on the main server’s mysql.

Copy /opt/fog/snapins/ssl from the main to all the other servers, delete what the other servers have and put the new files in place with the right permissions for fog:apache

then,

You now should have the convenience to access the fog web interface from any of the servers, and just set up each server as a storage node. Make the main server the master node.

There are a great number of ways to organize storage nodes/storage groups but since you have one massive server now that does everything, it’d operate the same if you made that one server a master node, then make all the storage nodes in the same group as the master node.

With this setup, you could then point each site’s DHCP Scopes to give out the local fog server for options 066 and 067 also

[Joseph Hales]

Current checkin time was set to 600 I now moved it to 60000 and restarted the server this is the result. Also what is the increment for the checkin value is it seconds or minutes. Can we update the tooltips to indicate the values for the config fields in fog?

Username	jhales
Web Server	10.200.10.150
TFTP Server	10.200.10.150
Load Average	34.96, 26.32, 11.34
System Uptime	5 min, 0 users```

No change or even worse preformance?

[Wayne Workman]

@Joseph-Hales How is it possible for the load averages to go up when you increase the checkin time span ???

[Tom Elliott]

@Joseph-Hales 600 is 10 minutes, so 60000 is 1000 minutes?

Where did you change this? You are running legacy client, so you would have to update all of the clients to receive these new configs?

[Wayne Workman]

@Tom-Elliott said:

You are running legacy client, so you would have to update all of the clients to receive these new configs?

set the interval for checkin
reinstall the legacy client on a computer.
copy C:\Program Files (x86)\FOG\etc\config.ini
push that file out as an update…

There is an option here:
Service Configuration -> Client Updater

This area will allow you to push out the new config.