• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

FOG Server CPU usage 100%

Scheduled Pinned Locked Moved
Linux Problems
7
64
19.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    Sebastian Roth Moderator
    last edited by Nov 14, 2017, 5:44 PM

    Possibly it’s just that apache hits a wall with too many client requests and too little worker threads?

    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

    T 1 Reply Last reply Nov 14, 2017, 5:49 PM Reply Quote 0
    • T
      Tom Elliott @Sebastian Roth
      last edited by Nov 14, 2017, 5:49 PM

      @sebastian-roth It’s possible, but less likely. While a ton of machines hitting the system at the same time can cause issues, I think the communication is so minuscule that I’m going to chalk this up to constant reloading of the Database, which is likely in a HUGE state.

      @Fernando-Gietz You could try testing this theory by getting a backup of the current database, then clearing out the larger table sets, taskLog, historyLog, snapinJobs, imagingLog, etc…

      Things that are “logging” or have no real value in being extremely large.

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      1 Reply Last reply Reply Quote 0
      • W
        Wayne Workman @Tom Elliott
        last edited by Nov 14, 2017, 5:54 PM

        @tom-elliott said in FOG Server CPU usage 100%:

        He sort of replicated this problem by accidentally uploading a binary data file (like teamviewer installer) into the CSV importer.

        But you fixed that problem by verifying that the CSV uploaded was an actual CSV.

        I would recommend cleaning out the DB though. Just truncate the history table, the tasks table, etc.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
        Daily Clean Installation Results:
        https://fogtesting.fogproject.us/
        FOG Reporting:
        https://fog-external-reporting-results.fogproject.us/

        T F 2 Replies Last reply Nov 14, 2017, 5:59 PM Reply Quote 0
        • T
          Tom Elliott @Wayne Workman
          last edited by Nov 14, 2017, 5:59 PM

          @wayne-workman I know what was fixed, but I also know WHY it was causing issues, it wasn’t the “bad” data persay, but rather the “amount” of bad data in the database.

          Sifting through 5000 shreds of paper (let’s say a sheet of paper is 1000 shreds each) is much harder than sifting through 5 pieces of paper, right?

          So size may not be the issue, rather the quantity held within the size.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

          1 Reply Last reply Reply Quote 0
          • F
            Fernando Gietz Developer @Wayne Workman
            last edited by Nov 14, 2017, 6:13 PM

            @wayne-workman The database size is not very big

            +------------------------+---------------+---------------+
            | Table Name             | Quant of Rows | Total Size Kb |
            +------------------------+---------------+---------------+
            | LDAPServers            |             1 |         14.32 |
            | clientUpdates          |             0 |          4.00 |
            | dirCleaner             |             0 |          1.00 |
            | globalSettings         |           180 |         73.30 |
            | greenFog               |             0 |          1.00 |
            | groupMembers           |          6881 |        546.79 |
            | groups                 |           313 |         53.92 |
            | history                |          4268 |       1011.34 |
            | hookEvents             |           277 |         31.68 |
            | hostAutoLogOut         |           502 |         34.20 |
            | hostMAC                |          7113 |        826.06 |
            | hostScreenSettings     |           502 |         38.18 |
            | hosts                  |          7008 |       1088.59 |
            | imageGroupAssoc        |           279 |         18.03 |
            | imagePartitionTypes    |            12 |          3.34 |
            | imageTypes             |             4 |          3.22 |
            | images                 |           279 |         72.12 |
            | imagingLog             |          2537 |        273.77 |
            | inventory              |           618 |        257.35 |
            | ipxeTable              |          2097 |        171.71 |
            | keySequence            |            35 |          2.79 |
            | moduleStatusByHost     |          4932 |        403.68 |
            | modules                |            13 |          5.66 |
            | multicastSessions      |           181 |         26.43 |
            | multicastSessionsAssoc |          2130 |        154.86 |
            | nfsFailures            |             0 |          1.00 |
            | nfsGroupMembers        |             1 |         20.22 |
            | nfsGroups              |             1 |         13.05 |
            | notifyEvents           |             5 |          9.14 |
            | os                     |            12 |          3.26 |
            +------------------------+---------------+---------------+
            

            This server have 7000 entries in the hosts table but only is used by 400-500 computers

            T 1 Reply Last reply Nov 14, 2017, 6:50 PM Reply Quote 0
            • F
              Fernando Gietz Developer
              last edited by Nov 14, 2017, 6:16 PM

              In the develope environment I have other server, in this one I deleted 6500 entries and It works better.

              +------------------------+---------------+---------------+
              | Table Name             | Quant of Rows | Total Size Kb |
              +------------------------+---------------+---------------+
              | LDAPServers            |             1 |         14.32 |
              | clientUpdates          |             0 |          1.00 |
              | dirCleaner             |             0 |          1.00 |
              | globalSettings         |           183 |         73.87 |
              | greenFog               |             0 |          1.00 |
              | groupMembers           |           279 |        534.84 |
              | groups                 |            28 |         48.79 |
              | history                |         25895 |       5894.44 |
              | hookEvents             |           278 |         31.72 |
              | hostAutoLogOut         |           108 |         12.11 |
              | hostMAC                |          6938 |        787.91 |
              | hostScreenSettings     |           109 |         13.51 |
              | hosts                  |           480 |       1036.53 |
              | imageGroupAssoc        |           292 |         18.15 |
              | imagePartitionTypes    |            12 |          3.34 |
              | imageTypes             |             4 |          3.22 |
              | images                 |           292 |         71.45 |
              | imagingLog             |          1735 |        165.28 |
              | inventory              |           259 |         99.80 |
              | ipxeTable              |          1620 |        123.98 |
              | keySequence            |            35 |          2.79 |
              | moduleStatusByHost     |          6097 |        489.08 |
              | modules                |            13 |          5.66 |
              | multicastSessions      |          1208 |        138.77 |
              | multicastSessionsAssoc |         24929 |       1417.48 |
              | nfsFailures            |             0 |          1.00 |
              | nfsGroupMembers        |             1 |         12.18 |
              | nfsGroups              |             1 |          7.05 |
              | notifyEvents           |             5 |          9.14 |
              | os                     |            12 |          3.26 |
              +------------------------+---------------+---------------+
              
              1 Reply Last reply Reply Quote 0
              • W
                Wayne Workman @Fernando Gietz
                last edited by Nov 14, 2017, 6:18 PM

                @fernando-gietz said in FOG Server CPU usage 100%:

                The problem is the apache and his activity, it reaches to 100% of CPU.

                Are you using image replication?

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                Daily Clean Installation Results:
                https://fogtesting.fogproject.us/
                FOG Reporting:
                https://fog-external-reporting-results.fogproject.us/

                F 1 Reply Last reply Nov 14, 2017, 6:19 PM Reply Quote 0
                • F
                  Fernando Gietz Developer @Wayne Workman
                  last edited by Nov 14, 2017, 6:19 PM

                  @wayne-workman No. I have only one server and one node. But the service is running.

                  W 1 Reply Last reply Nov 14, 2017, 6:19 PM Reply Quote 0
                  • W
                    Wayne Workman @Fernando Gietz
                    last edited by Nov 14, 2017, 6:19 PM

                    @fernando-gietz Turn it off please and see what happens. systemctl stop FOGImageReplicator or something like that.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                    Daily Clean Installation Results:
                    https://fogtesting.fogproject.us/
                    FOG Reporting:
                    https://fog-external-reporting-results.fogproject.us/

                    F 1 Reply Last reply Nov 14, 2017, 6:28 PM Reply Quote 0
                    • F
                      Fernando Gietz Developer @Wayne Workman
                      last edited by Nov 14, 2017, 6:28 PM

                      @wayne-workman Nothing. the behavour is the same. I have restarted the apache too.

                      W 2 Replies Last reply Nov 14, 2017, 6:33 PM Reply Quote 0
                      • W
                        Wayne Workman @Fernando Gietz
                        last edited by Wayne Workman Nov 14, 2017, 12:35 PM Nov 14, 2017, 6:33 PM

                        @fernando-gietz Please read through this: https://stackoverflow.com/questions/32006559/show-which-php-scripts-apache-is-currently-running

                        We need to figure out what’s consuming all of your CPU. You say it’s httpd, but we don’t know anything further than that. Using the methods described in the thread above, you should be able to pin-point exactly which PHP script is consuming the CPU.

                        Thread from stackoverflow copy/pasted below:

                        You can get the current working directory and opened files using lsof. Sadly this does not include the script being run, but if scripts are in different directories, or open different files, you can distinguish between them. Eg the following shows a script in /var/www/stuff/php is running:

                        sudo lsof -c http | grep cwd
                        httpd   24475  id  cwd DIR 8,3     4096 1123403 /var/www/stuff/php
                        

                        You can configure apache to let you have the information. Add an entry like

                        <Location /server-status>
                            SetHandler server-status
                            ...
                        </Location>
                        

                        in your conf ensuring it is only made available to restricted remotes. Then browse to http://localhost/server-status and you will have a “ps” of running jobs. For example, this output shows gg.php is running (I’ve removed some columns):

                        Srv PID     Acc     M CPU     SS Slot Client  VHost  Request
                        0-0 23688   0/0/262 W 0.50    2  4.48 ::1     xxx:80 GET /php/gg.php HTTP/1.1
                        1-0 23743   0/0/187 W 0.00    0  3.30 ::1     xxx:80 GET /server-status HTTP/1.1
                        

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                        Daily Clean Installation Results:
                        https://fogtesting.fogproject.us/
                        FOG Reporting:
                        https://fog-external-reporting-results.fogproject.us/

                        1 Reply Last reply Reply Quote 0
                        • W
                          Wayne Workman @Fernando Gietz
                          last edited by Nov 14, 2017, 6:45 PM

                          @fernando-gietz There is also the apachetop command that you might be able to use too. I almost forgot about this article but this shows how to use it: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_Web_Interface Along with a list of historic threads where people have had high CPU problems in the past. (if we solve yours, it will be added to the list too).

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                          Daily Clean Installation Results:
                          https://fogtesting.fogproject.us/
                          FOG Reporting:
                          https://fog-external-reporting-results.fogproject.us/

                          1 Reply Last reply Reply Quote 0
                          • T
                            Tom Elliott @Fernando Gietz
                            last edited by Nov 14, 2017, 6:50 PM

                            @fernando-gietz I’m going to guess the tasks table is a bit on the high side maybe?

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                            F 1 Reply Last reply Nov 14, 2017, 7:03 PM Reply Quote 0
                            • F
                              Fernando Gietz Developer
                              last edited by Wayne Workman Nov 14, 2017, 12:56 PM Nov 14, 2017, 6:51 PM

                              Done.

                              [root@fog7 conf]# lsof -c http | grep cwd
                              httpd   13036   root  cwd       DIR              253,0      4096      128 /
                              httpd   13037 apache  cwd       DIR              253,2       133 23069131 /var/www/html/fog/management
                              httpd   13038 apache  cwd       DIR              253,0      4096      128 /
                              httpd   13039 apache  cwd       DIR              253,0      4096      128 /
                              httpd   13040 apache  cwd       DIR              253,0      4096      128 /
                              httpd   13041 apache  cwd       DIR              253,0      4096      128 /
                              httpd   13044 apache  cwd       DIR              253,2      4096  8388946 /var/www/html/fog/status
                              httpd   13118 apache  cwd       DIR              253,0      4096      128 /
                              httpd   13144 apache  cwd       DIR              253,0      4096      128 /
                              httpd   13145 apache  cwd       DIR              253,0      4096      128 /
                              

                              And now the output of server status

                              Server Version: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.6.31
                              Server MPM: prefork
                              Server Built: Jul 26 2017 04:45:44
                              
                              Current Time: Tuesday, 14-Nov-2017 19:49:35 CET
                              Restart Time: Tuesday, 14-Nov-2017 19:44:15 CET
                              Parent Server Config. Generation: 1
                              Parent Server MPM Generation: 0
                              Server uptime: 5 minutes 20 seconds
                              Server load: 1.01 0.85 0.65
                              Total accesses: 557 - Total Traffic: 3.4 MB
                              CPU Usage: u167.78 s11.08 cu0 cs0 - 55.9% CPU load
                              1.74 requests/sec - 10.8 kB/second - 6.2 kB/request
                              3 requests currently being processed, 6 idle workers
                              
                              W____WW__.......................................................
                              ................................................................
                              ................................................................
                              ................................................................
                              
                              Scoreboard Key:
                              "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
                              "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
                              "C" Closing connection, "L" Logging, "G" Gracefully finishing,
                              "I" Idle cleanup of worker, "." Open slot with no current process
                              
                              Srv	PID	Acc	M	CPU 	SS	Req	Conn	Child	Slot	Client	VHost	Request
                              0-0	13037	0/58/58	W 	5.22	120	0	0.0	0.19	0.19 	158.227.4.135	10.0.15.4:80	GET /fog/management/index.php?node=group&sub=membership&id=334
                              1-0	13038	0/75/75	_ 	6.88	2	107	0.0	0.27	0.27 	158.227.138.124	10.0.15.4:80	GET /fog/management/index.php?sub=requestClientInfo&configure&n
                              2-0	13039	0/76/76	_ 	6.59	2	0	0.0	0.23	0.23 	158.227.4.135	10.0.15.4:80	GET /server-status HTTP/1.1
                              3-0	13040	0/39/39	_ 	4.78	2	232	0.0	0.07	0.07 	158.227.138.124	10.0.15.4:80	GET /fog/management/index.php?sub=requestClientInfo&mac=94:57:A
                              4-0	13041	0/75/75	_ 	6.74	1	150	0.0	0.23	0.23 	158.227.115.42	10.0.15.4:80	GET /fog/management/index.php?node=task&sub=active&_=1510657597
                              5-0	13044	0/19/19	W 	131.09	87	0	0.0	1.69	1.69 	158.227.4.135	10.0.15.4:80	POST /fog/status/getservertime.php HTTP/1.1
                              6-0	13118	0/72/72	W 	7.25	0	0	0.0	0.23	0.23 	158.227.4.135	10.0.15.4:80	GET /server-status HTTP/1.1
                              7-0	13144	0/72/72	_ 	6.11	1	73	0.0	0.23	0.23 	158.227.138.124	10.0.15.4:80	GET /fog/service/getversion.php?newService&json HTTP/1.1
                              8-0	13145	0/71/71	_ 	4.20	2	75	0.0	0.24	0.24 	158.227.138.124	10.0.15.4:80	GET /fog/service/getversion.php?clientver&newService&json HTTP/
                              Srv	Child Server number - generation
                              PID	OS process ID
                              Acc	Number of accesses this connection / this child / this slot
                              M	Mode of operation
                              CPU	CPU usage, number of seconds
                              SS	Seconds since beginning of most recent request
                              Req	Milliseconds required to process most recent request
                              Conn	Kilobytes transferred this connection
                              Child	Megabytes transferred this child
                              Slot	Total megabytes transferred this slot
                              SSL/TLS Session Cache Status:
                              cache type: SHMCB, shared memory: 512000 bytes, current entries: 0
                              subcaches: 32, indexes per subcache: 88
                              index usage: 0%, cache usage: 0%
                              total entries stored since starting: 0
                              total entries replaced since starting: 0
                              total entries expired since starting: 0
                              total (pre-expiry) entries scrolled out of the cache: 0
                              total retrieves since starting: 0 hit, 0 miss
                              total removes since starting: 0 hit, 0 miss
                              
                              F 1 Reply Last reply Nov 14, 2017, 6:53 PM Reply Quote 0
                              • F
                                Fernando Gietz Developer @Fernando Gietz
                                last edited by Nov 14, 2017, 6:53 PM

                                @fernando-gietz

                                /fog/management/index.php?node=group&sub=membership&id=334
                                
                                1 Reply Last reply Reply Quote 0
                                • W
                                  Wayne Workman
                                  last edited by Nov 14, 2017, 6:56 PM

                                  Actually this line is using a ton of CPU:

                                  131.09	87	0	0.0	1.69	1.69 	158.227.4.135	10.0.15.4:80	POST /fog/status/getservertime.php HTTP/1.1
                                  

                                  131 is a LOT.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                  Daily Clean Installation Results:
                                  https://fogtesting.fogproject.us/
                                  FOG Reporting:
                                  https://fog-external-reporting-results.fogproject.us/

                                  S 1 Reply Last reply Nov 15, 2017, 12:07 AM Reply Quote 0
                                  • F
                                    Fernando Gietz Developer @Tom Elliott
                                    last edited by Nov 14, 2017, 7:03 PM

                                    @tom-elliott Tasks table?

                                    W 1 Reply Last reply Nov 14, 2017, 7:08 PM Reply Quote 0
                                    • W
                                      Wayne Workman @Fernando Gietz
                                      last edited by Nov 14, 2017, 7:08 PM

                                      @fernando-gietz You should have a table called tasks in MySQL.

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                      Daily Clean Installation Results:
                                      https://fogtesting.fogproject.us/
                                      FOG Reporting:
                                      https://fog-external-reporting-results.fogproject.us/

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        Sebastian Roth Moderator @Wayne Workman
                                        last edited by Nov 15, 2017, 12:07 AM

                                        @wayne-workman said:

                                        Actually this line is using a ton of CPU:

                                        131.09	87	0	0.0	1.69	1.69 	158.227.4.135	10.0.15.4:80	POST /fog/status/getservertime.php HTTP/1.1
                                        

                                        131 is a LOT.

                                        Yeah that looks like there is something strange going on with this script. Though I have no idea what could be causing this. On the other hand the PHP code in that file looks kind of like it might take more time than usual. I guess only @Tom-Elliott can tell us what’s those calls to ignore_user_abort() and set_time_limit() are used for.

                                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                        T 1 Reply Last reply Nov 15, 2017, 3:15 PM Reply Quote 0
                                        • F
                                          Fernando Gietz Developer
                                          last edited by Nov 15, 2017, 12:30 PM

                                          @Tom-Elliott about the tasks table:

                                          MariaDB [fog]> show table status from fog where Name = 'tasks';
                                          +-------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-----------------+----------+--------------------+---------+
                                          | Name  | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time         | Check_time          | Collation       | Checksum | Create_options     | Comment |
                                          +-------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-----------------+----------+--------------------+---------+
                                          | tasks | MyISAM |      10 | Dynamic    | 5217 |            125 |      653940 | 281474976710655 |       602112 |         0 |           5225 | 2017-06-14 18:23:02 | 2017-11-15 13:10:31 | 2017-09-07 15:31:22 | utf8_general_ci |     NULL | row_format=DYNAMIC |         |
                                          +-------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-----------------+----------+--------------------+---------+
                                          
                                          W 1 Reply Last reply Nov 15, 2017, 1:04 PM Reply Quote 0
                                          • 1
                                          • 2
                                          • 3
                                          • 4
                                          • 4 / 4
                                          4 / 4
                                          • First post
                                            42/64
                                            Last post

                                          272

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project