Apache Issue

[Avaryan]

Ubuntu 14.04
FOG 1.4.2

Apache is starting a ton of processes… needless to say, it’s slowing everything down significantly.

MySQL cpu usage was also very high.

The CPU usage for this VM is capped at 3 Ghz. At idle with no tasks running it was at 2.9.
There’s a few thousand clients registered.

[Avaryan]

Changed the checking time from 60 to 180. After a few minutes, the web interface became responsive again. I can switch between pages easily now.

[george1421]

I guess I would have to ask you can you update to FOG 1.4.4?

Also lets collect some background information here

How many client computers are contacting this fog server?
What is your client check in time/interval?

[Sebastian Roth]

Just wanna add, what fog-client version do you use?

[Avaryan]

I’ll look into updating to 1.4.4 tomorrow.

The FOG client is version 0.11.12. It’s running on all our Windows computers. The check in time is the default setting.

As for number of devices, maybe around 1,000 online at once. Hard to say. Sent an email to our network admin regarding that.

[Tom Elliott]

@avaryan is it possible the fog server has run out of disk space? This would, potentially, cause high cpu as it gets caught waiting for a new file to be opened up. Normally things just crash hard in these cases, but that doesn’t mean it happens all the time.

[Avaryan]

No, there’s about 700GB of free space on it.

[Wayne Workman]

@avaryan Please be sure to check all the partitions, not just the big one. Use this command: df -h and look for partitions that have 100% usage.

[Sebastian Roth]

@avaryan said in Apache Issue:

The FOG client is version 0.11.12. It’s running on all our Windows computers. The check in time is the default setting.

That’s good you have the latest client version running.

As for number of devices, maybe around 1,000 online at once.

Well then there are always a dozen or so clients talking to the FOG server at any given time. So I don’t really wonder about the top output you posted. I am wondering if it got to the point where the number of concurrent clients do actually cause the server to slow down. One solution would be to increase checkin time for the clients. But that would be to the cost of having a longer time to wait till clients do the jobs scheduled for them.

Possibly start by looking at the apache access log to see if this is really clients causing this: tail -f /var/log/apache/access.log

As well you wanna read this excellent post: https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast

[Avaryan]

@wayne-workman said in Apache Issue:

@avaryan Please be sure to check all the partitions, not just the big one. Use this command: df -h and look for partitions that have 100% usage.

@sebastian-roth said in Apache Issue:

Possibly start by looking at the apache access log to see if this is really clients causing this: tail -f /var/log/apache/access.log

I get no output. I let it sit for about 10 minutes.

edit: Updated to 1.4.4. Still increadibly laggy, but tail is showing some output now… kinda.

127.0.0.1 - - [05/Oct/2017:09:26:27 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
127.0.0.1 - - [05/Oct/2017:09:26:29 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
127.0.0.1 - - [05/Oct/2017:09:26:30 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
127.0.0.1 - - [05/Oct/2017:09:26:31 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
127.0.0.1 - - [05/Oct/2017:09:26:32 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
127.0.0.1 - - [05/Oct/2017:09:26:33 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
127.0.0.1 - - [05/Oct/2017:09:26:51 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
127.0.0.1 - - [05/Oct/2017:09:29:13 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
127.0.0.1 - - [05/Oct/2017:09:29:16 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"

^ Notice that it gets a hit every 1-2 seconds, then stops for about 20 seconds, then over 2 minutes before the next to. Haven’t gotten anymore since that last one.

[Avaryan]

Changed the checking time from 60 to 180. After a few minutes, the web interface became responsive again. I can switch between pages easily now.

[george1421]

@avaryan Interesting, how many computers do you have checking into this FOG server?

[Avaryan]

@george1421 said in Apache Issue:

@avaryan Interesting, how many computers do you have checking into this FOG server?

I don’t know the exact number, but likely more than a thousand. There’s over 2,500 Windows computers in AD that would have the client on it.

Anyway, it’s been about 4 hours now and the web ui is still running well. I’d say that solved the issue.
Bonus: Got the FOG server updated.