Apache Issue



  • Ubuntu 14.04
    FOG 1.4.2

    Apache is starting a ton of processes… needless to say, it’s slowing everything down significantly.
    alt text

    MySQL cpu usage was also very high.

    The CPU usage for this VM is capped at 3 Ghz. At idle with no tasks running it was at 2.9.
    There’s a few thousand clients registered.



  • @george1421 said in Apache Issue:

    @avaryan Interesting, how many computers do you have checking into this FOG server?

    I don’t know the exact number, but likely more than a thousand. There’s over 2,500 Windows computers in AD that would have the client on it.

    Anyway, it’s been about 4 hours now and the web ui is still running well. I’d say that solved the issue.
    Bonus: Got the FOG server updated.


  • Moderator

    @avaryan Interesting, how many computers do you have checking into this FOG server?



  • Changed the checking time from 60 to 180. After a few minutes, the web interface became responsive again. I can switch between pages easily now.



  • @wayne-workman said in Apache Issue:

    @avaryan Please be sure to check all the partitions, not just the big one. Use this command: df -h and look for partitions that have 100% usage.

    alt text

    @sebastian-roth said in Apache Issue:

    Possibly start by looking at the apache access log to see if this is really clients causing this: tail -f /var/log/apache/access.log

    I get no output. I let it sit for about 10 minutes.
    alt text

    edit: Updated to 1.4.4. Still increadibly laggy, but tail is showing some output now… kinda.

    127.0.0.1 - - [05/Oct/2017:09:26:27 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
    127.0.0.1 - - [05/Oct/2017:09:26:29 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
    127.0.0.1 - - [05/Oct/2017:09:26:30 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
    127.0.0.1 - - [05/Oct/2017:09:26:31 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
    127.0.0.1 - - [05/Oct/2017:09:26:32 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
    127.0.0.1 - - [05/Oct/2017:09:26:33 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
    127.0.0.1 - - [05/Oct/2017:09:26:51 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
    127.0.0.1 - - [05/Oct/2017:09:29:13 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
    127.0.0.1 - - [05/Oct/2017:09:29:16 -0400] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.27 (Ubuntu) OpenSSL/1.1.0f (internal dummy connection)"
    

    ^ Notice that it gets a hit every 1-2 seconds, then stops for about 20 seconds, then over 2 minutes before the next to. Haven’t gotten anymore since that last one.


  • Developer

    @avaryan said in Apache Issue:

    The FOG client is version 0.11.12. It’s running on all our Windows computers. The check in time is the default setting.

    That’s good you have the latest client version running.

    As for number of devices, maybe around 1,000 online at once.

    Well then there are always a dozen or so clients talking to the FOG server at any given time. So I don’t really wonder about the top output you posted. I am wondering if it got to the point where the number of concurrent clients do actually cause the server to slow down. One solution would be to increase checkin time for the clients. But that would be to the cost of having a longer time to wait till clients do the jobs scheduled for them.

    Possibly start by looking at the apache access log to see if this is really clients causing this: tail -f /var/log/apache/access.log

    As well you wanna read this excellent post: https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast


  • Moderator

    @avaryan Please be sure to check all the partitions, not just the big one. Use this command: df -h and look for partitions that have 100% usage.



  • No, there’s about 700GB of free space on it.


  • Senior Developer

    @avaryan is it possible the fog server has run out of disk space? This would, potentially, cause high cpu as it gets caught waiting for a new file to be opened up. Normally things just crash hard in these cases, but that doesn’t mean it happens all the time.



  • I’ll look into updating to 1.4.4 tomorrow.

    The FOG client is version 0.11.12. It’s running on all our Windows computers. The check in time is the default setting.

    As for number of devices, maybe around 1,000 online at once. Hard to say. Sent an email to our network admin regarding that.


  • Developer

    Just wanna add, what fog-client version do you use?


  • Moderator

    I guess I would have to ask you can you update to FOG 1.4.4?

    Also lets collect some background information here

    How many client computers are contacting this fog server?
    What is your client check in time/interval?


Log in to reply
 

443
Online

39179
Users

10827
Topics

103019
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.