• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    HIGH CPU Fog Services after update r5029 v6759

    Scheduled Pinned Locked Moved Solved
    FOG Problems
    6
    73
    28.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Raymond BellR
      Raymond Bell Testers @Wayne Workman
      last edited by

      @Wayne-Workman i will try this i think this is what Tom is talking about

      Raymond Bell
      WiKi: https://wiki.fogproject.org/wiki/index.php/Main_Page

      1 Reply Last reply Reply Quote 0
      • Tom ElliottT
        Tom Elliott @Raymond Bell
        last edited by

        @Raymond-Bell said:

        Too many connections, Message: Check that database is running

        Too many connections, Message: Check that database is running.

        Can you edit the /etc/mysql/my.cnf file and add in the [mysqld] portion (or change) max_connections = 500 then restart the mysql service? (sudo service mysql restart)

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        Raymond BellR 1 Reply Last reply Reply Quote 0
        • Raymond BellR
          Raymond Bell Testers @Tom Elliott
          last edited by

          @Tom-Elliott said:

          @Raymond-Bell said:

          Too many connections, Message: Check that database is running

          Too many connections, Message: Check that database is running.

          Can you edit the /etc/mysql/my.cnf file and add in the [mysqld] portion (or change) max_connections = 500 then restart the mysql service? (sudo service mysql restart)

          Yes that helped with the to many connections
          Also this helped on the server but not the nodes

          sudo update-rc.d FOG*service* disable```
          
          Try disabling the FOG Services from starting at boot.
          
          Then, edit the /etc/rc.local file.
          
          Add:
          
          sleep 30
          /etc/init.d/FOGPingHosts stop
          /etc/init.d/FOGScheduler stop
          /etc/init.d/FOGImageReplicator stop
          /etc/init.d/FOGSnapinReplicator stop
          /etc/init.d/FOGMulticastManager stop
          /etc/init.d/FOGPingHosts start
          /etc/init.d/FOGScheduler start
          /etc/init.d/FOGImageReplicator start
          /etc/init.d/FOGSnapinReplicator start
          /etc/init.d/FOGMulticastManager start

          Raymond Bell
          WiKi: https://wiki.fogproject.org/wiki/index.php/Main_Page

          Tom ElliottT 1 Reply Last reply Reply Quote 0
          • B
            boeleke
            last edited by boeleke

            Guys , @Raymond-Bell & @Tom-Elliott

            do you got an patch-solution for this?
            After the patch from this afternoon i also get HIGH CPU services (now got 6777)

            alt text

            Server is freaking out 😞
            Ubuntu 14.04

            1 Reply Last reply Reply Quote 0
            • Tom ElliottT
              Tom Elliott @Raymond Bell
              last edited by

              @Raymond-Bell The rc.local stuff is also on the Storage Nodes?

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              Raymond BellR 1 Reply Last reply Reply Quote 0
              • Raymond BellR
                Raymond Bell Testers @Tom Elliott
                last edited by

                @Tom-Elliott said:

                @Raymond-Bell The rc.local stuff is also on the Storage Nodes?

                yes i edited rc.local on Storage Nodes also

                Raymond Bell
                WiKi: https://wiki.fogproject.org/wiki/index.php/Main_Page

                Tom ElliottT 1 Reply Last reply Reply Quote 0
                • Tom ElliottT
                  Tom Elliott @Raymond Bell
                  last edited by

                  @Raymond-Bell Did you restart the services after adding the rc.local? I’m sorry if I’m asking obvious questions, but I really just need to make sure. Current update should help alleviate CPU usage as well though.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  Raymond BellR 2 Replies Last reply Reply Quote 0
                  • Raymond BellR
                    Raymond Bell Testers @Tom Elliott
                    last edited by

                    @Tom-Elliott said:

                    @Raymond-Bell Did you restart the services after adding the rc.local? I’m sorry if I’m asking obvious questions, but I really just need to make sure. Current update should help alleviate CPU usage as well though.

                    yes restarted and rebooted and also up to r5038 6777
                    will do it again to make sure

                    Raymond Bell
                    WiKi: https://wiki.fogproject.org/wiki/index.php/Main_Page

                    1 Reply Last reply Reply Quote 0
                    • Raymond BellR
                      Raymond Bell Testers @Tom Elliott
                      last edited by

                      @Tom-Elliott Yes same result after

                      sudo service FOGMulticastManager stop
                      sudo service FOGImageReplicator stop
                      sudo service FOGSnapinReplicator stop
                      sudo service FOGScheduler stop
                      sudo service FOGPingHosts stop
                      
                      
                      sudo update-rc.d FOGMulticastManager disable
                      sudo update-rc.d FOGImageReplicator disable
                      sudo update-rc.d FOGSnapinReplicator disable
                      sudo update-rc.d FOGScheduler disable
                      sudo update-rc.d FOGPingHosts disable
                      
                      sudo service FOGMulticastManager start
                      sudo service FOGImageReplicator start
                      sudo service FOGSnapinReplicator start
                      sudo service FOGScheduler start
                      sudo service FOGPingHosts start
                      
                      top - 10:48:18 up  1:48,  2 users,  load average: 4.14, 4.92, 5.02
                      Tasks: 202 total,   6 running, 196 sleeping,   0 stopped,   0 zombie
                      %Cpu(s): 66.6 us, 32.7 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
                      KiB Mem:   1014980 total,   966204 used,    48776 free,    19852 buffers
                      KiB Swap:  1037308 total,    12316 used,  1024992 free.   529972 cached Mem
                      
                        PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                       5970 root      20   0   41920  20164  15184 R  45.6  2.0   0:07.37 FOGSnapinReplic
                       6026 root      20   0   41920  20392  15408 R  41.3  2.0   0:05.85 FOGPingHosts
                       5990 root      20   0   41920  20240  15256 R  39.6  2.0   0:06.34 FOGTaskSchedule
                       5950 root      20   0   41920  20488  15504 R  39.3  2.0   0:06.90 FOGImageReplica
                       5930 root      20   0   41920  20116  15132 R  34.3  2.0   0:05.32 FOGMulticastMan
                       5779 fog       20   0   11284   3900   3128 S   0.3  0.4   0:00.04 sshd
                          1 root      20   0    4732   3788   2548 S   0.0  0.4   0:02.83 init
                          2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
                          3 root      20   0       0      0      0 S   0.0  0.0   0:00.78 ksoftirqd/0
                          5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
                          7 root      20   0       0      0      0 S   0.0  0.0   0:05.27 rcu_sched
                          8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
                          9 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/0
                         10 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 watchdog/0
                         11 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 watchdog/1
                         12 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/1
                         13 root      20   0       0      0      0 S   0.0  0.0   0:00.12 ksoftirqd/1
                      

                      Raymond Bell
                      WiKi: https://wiki.fogproject.org/wiki/index.php/Main_Page

                      1 Reply Last reply Reply Quote 1
                      • B
                        boeleke
                        last edited by

                        alt text

                        Something weird. After the new update , my CPU performance from my server was going to 10-20%. After a reboot , he started again @100%

                        Pretty anoying

                        1 Reply Last reply Reply Quote 0
                        • B
                          baggar11
                          last edited by

                          I just realized after going through another git pull upgrade that the CPU spike will probably happen every time since the services are reinstalled. Here’s how I fixed my problem again, hopefully in a cleaner way. After upgrading to 6791, it seems to be working so far with a couple reboot tests.

                          Add this to your rc.local. Adjust sleep time based on your setup. My FOG setup is virtual, so 5 seconds seems to work well.

                          sleep 5
                          service FOGImageReplicator restart
                          service FOGMulticastManager restart
                          service FOGPingHosts restart
                          service FOGScheduler restart
                          service FOGSnapinReplicator restart
                          
                          Tom ElliottT 1 Reply Last reply Reply Quote 0
                          • Tom ElliottT
                            Tom Elliott @baggar11
                            last edited by

                            @baggar11, @Raymond-Bell I believe I may have fixed this now.

                            My issue – in theory?

                            I am moving almost EVERYTHING to static form (where I can). This allows me to get items back without having to initiate a whole class object. The problem, the DB has to be available for FOG to read its info. The services are checking for the item before DB is established. This was totally an over site and for that I’m sorry to all.

                            With any luck, this issue will be gone now. Please update and let me know.

                            Thanks.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                            B 1 Reply Last reply Reply Quote 0
                            • Tom ElliottT
                              Tom Elliott
                              last edited by

                              Why this might affect the CPU? It looks to the DB to get timeout values (and logs). The DB is fully operational, but it’s not initiated by fog meaning the DB is not available. It returns an int of 0 for the sleep time. Put that into an infinite loop, (there’s at least 2 that manage start/restart of the services. It is told to sleep for some period of time already within the looping I’m referring to. However it’s set to 0 (meaning 0 seconds), so it get’s stuck in an infinite loop without ever actually starting.

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              1 Reply Last reply Reply Quote 0
                              • B
                                baggar11 @Tom Elliott
                                last edited by

                                @Tom-Elliott said:

                                @baggar11, @Raymond-Bell I believe I may have fixed this now.
                                With any luck, this issue will be gone now. Please update and let me know.

                                Just updated to 6795, removed my “restart” lines from rc.local and rebooted. CPU was pegged again. Restarting the services manually brought cpu back down to idle.

                                Tom ElliottT 1 Reply Last reply Reply Quote 0
                                • Tom ElliottT
                                  Tom Elliott @baggar11
                                  last edited by

                                  @baggar11 as I understand it, this is still potentially due to starting the services before its dependencies have started. This is different from the pegged on update, I think.

                                  Can you simply try reinstall and see if it’s still happening? Or did update always work, only reboot caused these issues?

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  B 1 Reply Last reply Reply Quote 0
                                  • B
                                    baggar11 @Tom Elliott
                                    last edited by

                                    @Tom-Elliott said:

                                    @baggar11 as I understand it, this is still potentially due to starting the services before its dependencies have started. This is different from the pegged on update, I think.

                                    Can you simply try reinstall and see if it’s still happening? Or did update always work, only reboot caused these issues?

                                    For my upgrades, anything after 6753 would peg my cpu after a reboot. During and after the upgrade process using the install.sh, everything was fine. It’s only after the reboot that the services would max out the cpu. Probably the infinite loop issue as you wrote about below.

                                    Tom ElliottT 1 Reply Last reply Reply Quote 0
                                    • Tom ElliottT
                                      Tom Elliott @baggar11
                                      last edited by

                                      @baggar11 Okay, I’ve added a sleep time of 10 seconds, just in case of these situations where the int returned is 0. We must have at least 1 second sleeptime ( I believe ), and this just isn’t happening.

                                      The only word of caution I can think of, now, is while this might help with CPU Cycles, it will not make the FOG Services actually work properly as from Ubuntu’s standpoint the service is already running. While I could, potentially, come up with a way to restart the services more appropriately, I don’t know where to start at the moment.

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                      B Raymond BellR 2 Replies Last reply Reply Quote 0
                                      • B
                                        baggar11 @Tom Elliott
                                        last edited by

                                        @Tom-Elliott said:

                                        @baggar11 Okay, I’ve added a sleep time of 10 seconds, just in case of these situations where the int returned is 0. We must have at least 1 second sleeptime ( I believe ), and this just isn’t happening.

                                        The only word of caution I can think of, now, is while this might help with CPU Cycles, it will not make the FOG Services actually work properly as from Ubuntu’s standpoint the service is already running. While I could, potentially, come up with a way to restart the services more appropriately, I don’t know where to start at the moment.

                                        I’m not a coder so take this with a grain of salt. The following is a similar approach to another open source project I use.

                                        Make a script to run as a cron job every 5 minutes. On 1st boot, the script essentially checks for things like MySQL and network being up, and if all checks out starts the FOG services. All other times it runs, it would really only check for FOG services running and then exit. heh… 🙂

                                        Of course, it’s probably a waste of time as I assume systemd will probably take care of these things more intelligently than regular init and/or upstart. Which Fedora is already using and Ubuntu’s next LTS will be running.

                                        I really appreciate you taking a look into these things Tom. FOG is a very cool project.

                                        B 1 Reply Last reply Reply Quote 0
                                        • B
                                          boeleke @baggar11
                                          last edited by boeleke

                                          @baggar11 & @Tom-Elliott

                                          Like Baggar is saying. Thanks for taking your time to looking at these things.
                                          Its just anoying that it is taking 100% CPU & not working properly.
                                          Take your time but its probably a problem for all the ubuntu 14.04 - users after one update.

                                          1 Reply Last reply Reply Quote 0
                                          • Raymond BellR
                                            Raymond Bell Testers @Tom Elliott
                                            last edited by

                                            @Tom-Elliott After update to r5054 this morning
                                            Looks like it is fixed now but i have not rebooted any of them…
                                            Server

                                            top - 07:57:51 up 23:03,  3 users,  load average: 134.00, 122.87, 122.12
                                            Tasks: 351 total,  32 running, 312 sleeping,   0 stopped,   7 zombie
                                            %Cpu(s): 55.3 us, 42.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  2.7 si,  0.0 st
                                            KiB Mem:   4355692 total,  3461944 used,   893748 free,    69372 buffers
                                            KiB Swap:  1046524 total,    16896 used,  1029628 free.  2231712 cached Mem
                                            
                                              PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                                            17801 mysql     20   0  386360  85396   9744 S  19.5  2.0   3:07.93 mysqld
                                            20119 www-data  20   0  111272  19556  13116 R  11.3  0.4   0:04.27 apache2
                                            21537 www-data  20   0       0      0      0 Z   9.9  0.0   0:01.06 apache2
                                            21239 www-data  20   0  111188  19952  13344 R   8.6  0.5   0:02.65 apache2
                                            19002 www-data  20   0  111044  19584  13116 R   7.0  0.4   0:08.43 apache2
                                            18759 www-data  20   0  111296  19812  13092 R   6.6  0.5   0:13.16 apache2
                                            19062 www-data  20   0  110988  19788  13752 R   6.6  0.5   0:07.04 apache2
                                            20017 www-data  20   0  110996  18988  12572 S   6.6  0.4   0:04.30 apache2
                                            18845 www-data  20   0  111300  19240  12688 R   6.3  0.4   0:13.16 apache2
                                            18875 www-data  20   0  111256  20412  13732 R   6.3  0.5   0:12.60 apache2
                                            
                                            

                                            Storage Node Master

                                            top - 07:58:14 up 19:25,  2 users,  load average: 0.01, 0.06, 0.12
                                            Tasks: 201 total,   1 running, 195 sleeping,   0 stopped,   5 zombie
                                            %Cpu(s):  0.2 us,  0.2 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
                                            KiB Mem:   1014980 total,   975276 used,    39704 free,   106172 buffers
                                            KiB Swap:  1037308 total,    22532 used,  1014776 free.   443116 cached Mem
                                            
                                              PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                                                1 root      20   0    4616   3552   2432 S   0.0  0.3   0:02.85 init
                                                2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
                                                3 root      20   0       0      0      0 S   0.0  0.0   0:03.88 ksoftirqd/0
                                                5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
                                                7 root      20   0       0      0      0 S   0.0  0.0   0:26.13 rcu_sched
                                                8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
                                                9 root      rt   0       0      0      0 S   0.0  0.0   0:00.02 migration/0
                                               10 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 watchdog/0
                                               11 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 watchdog/1
                                               12 root      rt   0       0      0      0 S   0.0  0.0   0:00.02 migration/1
                                            
                                            

                                            Storage Node 2nd

                                            top - 07:58:31 up 22:59,  3 users,  load average: 0.01, 0.02, 0.05
                                            Tasks: 227 total,   1 running, 225 sleeping,   0 stopped,   1 zombie
                                            %Cpu(s):  0.1 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
                                            KiB Mem:   7735460 total,  7466792 used,   268668 free,    69996 buffers
                                            KiB Swap:  7828476 total,        0 used,  7828476 free.  6881904 cached Mem
                                            
                                              PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                                                7 root      20   0       0      0      0 S   0.3  0.0   0:29.70 rcu_sched
                                            11185 fog       20   0    5572   2852   2352 R   0.3  0.0   0:02.77 top
                                                1 root      20   0    4732   3844   2580 S   0.0  0.0   0:01.60 init
                                                2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
                                                3 root      20   0       0      0      0 S   0.0  0.0   0:04.09 ksoftirqd/0
                                                5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
                                                8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
                                                9 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/0
                                               10 root      rt   0       0      0      0 S   0.0  0.0   0:00.30 watchdog/0
                                               11 root      rt   0       0      0      0 S   0.0  0.0   0:00.32 watchdog/1
                                               12 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/1
                                            
                                            

                                            Raymond Bell
                                            WiKi: https://wiki.fogproject.org/wiki/index.php/Main_Page

                                            B 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 3 / 4
                                            • First post
                                              Last post

                                            172

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project