• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Storage Nodes stop reporting after a while

    Scheduled Pinned Locked Moved Solved
    FOG Problems
    2
    28
    4.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Wayne WorkmanW
      Wayne Workman
      last edited by

      FOG 1.3.0 RC-8
      CentOS 7 for every storage node and for the main server.

      This is a weird one, for sure.

      I set a custom fogstorage password last week, this SEEMS to be when the problems started. I changed the password for the mysql fogstorage user, and set this new password in every storage node’s /opt/fog/.fogsettings file, and re-ran the installer on them all. I looked to be fine then.

      Checking today, none of the nodes were reporting in the dashboard with a version or with an interface name on the bandwidth chart. I thought this was odd, and I re-ran the installer on all nodes without changing anything, and all started reporting again.

      I also manually checked the fogstorage credentials from one of the storage nodes with the command mysql -u fogstorage -h x.x.x.x -p and supplied the new password, it let me into the remote Database on the FOG Web Host just fine - so it’s not bad credentials - at least not directly, and I don’t think it’s a crazy character in my password either, my password has no single quotes (') or slanted quotes (`) marks or semicolons (;).

      Minutes later, more and more are not reporting anymore.

      0_1471375636396_upload-5e0e3115-a298-48c1-9346-87f8c47c09b9

      0_1471375646331_upload-fdb0b7c7-8db8-4be7-9025-7c76608adfa3

      Here are recent apache errors from the main server:

      [Tue Aug 16 08:40:31.115058 2016] [:error] [pid 24618] [client 10.2.32.5:50604] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=95
[Tue Aug 16 08:37:54.252228 2016] [:error] [pid 24009] [client 10.2.32.5:50502] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/printer.class.php on line 51, referer: http://10.51.1.53/fog/management/index.php?node=printer&sub=membership&id=100
[Tue Aug 16 08:36:33.385722 2016] [:error] [pid 2400] [client 10.2.32.5:50444] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/printer.class.php on line 51, referer: http://10.51.1.53/fog/management/index.php?node=printer&sub=membership&id=9
[Tue Aug 16 08:35:49.386900 2016] [:error] [pid 24009] [client 10.2.32.5:50408] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=95
[Tue Aug 16 08:33:40.235778 2016] [:error] [pid 20526] [client 10.2.32.5:50318] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=95
[Tue Aug 16 08:29:06.833715 2016] [:error] [pid 23622] [client 10.13.32.69:56729] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=38
[Tue Aug 16 08:28:03.143760 2016] [:error] [pid 2400] [client 10.13.32.69:56712] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=37
[Tue Aug 16 08:27:15.926313 2016] [:error] [pid 20526] [client 10.13.32.69:56701] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=36
[Tue Aug 16 07:53:54.451746 2016] [:error] [pid 15816] [client 10.13.32.69:56403] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=90
[Tue Aug 16 07:52:29.853261 2016] [:error] [pid 14449] [client 10.13.32.69:56387] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=85
[Tue Aug 16 07:09:42.465814 2016] [:error] [pid 3609] [client 10.33.8.27:50218] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=44
[Tue Aug 16 05:00:35.886373 2016] [core:notice] [pid 996] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Tue Aug 16 05:00:35.886349 2016] [mpm_prefork:notice] [pid 996] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.6.24 configured -- resuming normal operations
[Tue Aug 16 05:00:35.646696 2016] [ssl:warn] [pid 996] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Aug 16 05:00:35.645533 2016] [lbmethod_heartbeat:notice] [pid 996] AH02282: No slotmem from mod_heartmonitor
[Tue Aug 16 05:00:35.644728 2016] [auth_digest:notice] [pid 996] AH01757: generating secret for digest authentication ...
[Tue Aug 16 05:00:35.597089 2016] [ssl:warn] [pid 996] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Aug 16 05:00:35.484364 2016] [suexec:notice] [pid 996] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Aug 16 05:00:35.385443 2016] [core:notice] [pid 996] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Tue Aug 16 05:00:01.593539 2016] [mpm_prefork:notice] [pid 1004] AH00170: caught SIGWINCH, shutting down gracefully
[Mon Aug 15 15:31:23.139578 2016] [:error] [pid 6074] [client 10.31.32.32:49606] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://10.51.1.53/fog/management/index.php?node=group&sub=edit&id=65
[Mon Aug 15 13:12:00.113891 2016] [:error] [pid 18312] [client 10.34.8.35:50782] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/printer.class.php on line 51, referer: http://10.51.1.53/fog/management/index.php?node=printer&sub=membership&id=67
[Mon Aug 15 12:50:29.548034 2016] [:error] [pid 19212] [client 10.13.32.19:52086] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/fogpage.class.php on line 1076, referer: http://10.51.1.53/fog/management/index.php?node=snapin&sub=import
[Mon Aug 15 12:50:29.547727 2016] [:error] [pid 19212] [client 10.13.32.19:52086] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/fog/fogpage.class.php on line 1076, referer: http://10.51.1.53/fog/management/index.php?node=snapin&sub=import

      In some of the error logs on some of the nodes, this line is appearing:

      IP Passed is incorrect

      I’ve also verified that the password set in /opt/fog/.fogsettings matches what’s in Storage Management for that particular storage node.

      I’m kinda stumped.

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
      Daily Clean Installation Results:
      https://fogtesting.fogproject.us/
      FOG Reporting:
      https://fog-external-reporting-results.fogproject.us/

      1 Reply Last reply Reply Quote 0
      • Wayne WorkmanW
        Wayne Workman
        last edited by Wayne Workman

        Tom worked on a patch quietly since I started this thread - and I’ve tested it - and it works.

        My storage node at home that I tested on is an old Pentium 4 with an IDE drive in it - so it’s very slow.
        Results at home:

        I could open 12 pages and let them sit on the fog dashboard - CPU usage on the storage node stayed below 4, there weren’t uncontrollable httpd processes spawning, Open files by the apache user stayed below 4k - and the bandwidth chart not only reported - but it’s now reporting more smoothly than ever, is actually accurate now, and doesn’t have spells of mental disability anymore. It’s like butter.

        The issue, as it was explained to me, is that the JS which renders the bandwidth chart and does the polling of bandwidth.php on all nodes enabled for bandwidth reporting - it wasn’t waiting for a response, it would just re-issue another poll before the response was received. The more dashboards open, the worse it got, until eventually apache was unable to do anything do to it’s tremendous load of ‘stacked’ processes that were back-logged. I didn’t tell anyone but Tom - but this (now solved) issue would break imaging at remote locations, too - because we use the location plugin and apache on the remote nodes is relied upon to get imaging done in this setup. That too is solved now.

        These fixes will all be in RC-9.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
        Daily Clean Installation Results:
        https://fogtesting.fogproject.us/
        FOG Reporting:
        https://fog-external-reporting-results.fogproject.us/

        1 Reply Last reply Reply Quote 1
        • Wayne WorkmanW
          Wayne Workman
          last edited by

          And just found out - sometimes, just sometimes, when I click on a node that isn’t reporting in the Storage Management selector - it briefly displays the value and then disappears, sometimes the value stays for 2 or 3 seconds, sometimes it’s just a quick blink.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
          Daily Clean Installation Results:
          https://fogtesting.fogproject.us/
          FOG Reporting:
          https://fog-external-reporting-results.fogproject.us/

          1 Reply Last reply Reply Quote 0
          • Wayne WorkmanW
            Wayne Workman
            last edited by

            So I’ve been keeping an eye on the storage node version reporting.

            I’m 100% convinced this has to do with time somehow, like the amount of time since the node was installed.

            When the first node went blank, Airport, I re-ran the installer simply thinking something went wrong, and then it came back.

            When I update nodes, I just go down the list from top to bottom, but came back to Airport and re-did it. Lo-and-behold, the last ones I ran the installer on are still reporting, and others are dropping off like flies.

            0_1471379077808_upload-c9a5a8eb-c58b-43f7-bda0-f353c0e8d774

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
            Daily Clean Installation Results:
            https://fogtesting.fogproject.us/
            FOG Reporting:
            https://fog-external-reporting-results.fogproject.us/

            1 Reply Last reply Reply Quote 0
            • Wayne WorkmanW
              Wayne Workman
              last edited by

              And moments after that last post, Robinwood fell off…

              I expect them all to be not-reporting in the morning if the issue is consistent.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
              Daily Clean Installation Results:
              https://fogtesting.fogproject.us/
              FOG Reporting:
              https://fog-external-reporting-results.fogproject.us/

              1 Reply Last reply Reply Quote 0
              • Wayne WorkmanW
                Wayne Workman
                last edited by Wayne Workman

                The main and one of the nodes I checked does not have mod_evasive installed.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                Daily Clean Installation Results:
                https://fogtesting.fogproject.us/
                FOG Reporting:
                https://fog-external-reporting-results.fogproject.us/

                1 Reply Last reply Reply Quote 0
                • Wayne WorkmanW
                  Wayne Workman
                  last edited by

                  Any ideas on this?

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                  Daily Clean Installation Results:
                  https://fogtesting.fogproject.us/
                  FOG Reporting:
                  https://fog-external-reporting-results.fogproject.us/

                  1 Reply Last reply Reply Quote 0
                  • Wayne WorkmanW
                    Wayne Workman
                    last edited by

                    The apache logs on one of the remote storage nodes is full of the below errors. @Tom-Elliott

                    [Wed Aug 17 13:36:15.502689 2016] [:error] [pid 34177] [client 10.51.1.53:46758] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/commons/init.php:14\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/commons/init.php(14): iterator_to_array(Object(RegexIterator))\n#4 /var/www/html/fog/commons/init.php(121): Initiator->__construct()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/commons/init.php on line 14
[Wed Aug 17 13:36:15.710230 2016] [:error] [pid 30710] [client 10.51.1.53:46788] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/commons/init.php:14\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/commons/init.php(14): iterator_to_array(Object(RegexIterator))\n#4 /var/www/html/fog/commons/init.php(121): Initiator->__construct()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/commons/init.php on line 14
[Wed Aug 17 13:36:19.494413 2016] [:error] [pid 30706] [client 10.51.1.53:46826] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/commons/init.php:14\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/commons/init.php(14): iterator_to_array(Object(RegexIterator))\n#4 /var/www/html/fog/commons/init.php(121): Initiator->__construct()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/commons/init.php on line 14
[Wed Aug 17 13:36:19.617840 2016] [:error] [pid 30707] [client 10.51.1.53:46856] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/commons/init.php:14\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/commons/init.php(14): iterator_to_array(Object(RegexIterator))\n#4 /var/www/html/fog/commons/init.php(121): Initiator->__construct()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/commons/init.php on line 14
[Wed Aug 17 13:36:23.516636 2016] [:error] [pid 30709] [client 10.51.1.53:46900] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/commons/init.php:14\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/commons/init.php(14): iterator_to_array(Object(RegexIterator))\n#4 /var/www/html/fog/commons/init.php(121): Initiator->__construct()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/commons/init.php on line 14
[Wed Aug 17 13:36:23.699457 2016] [:error] [pid 30708] [client 10.51.1.53:46932] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/commons/init.php:14\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/commons/init.php(14): iterator_to_array(Object(RegexIterator))\n#4 /var/www/html/fog/commons/init.php(121): Initiator->__construct()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/commons/init.php on line 14

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                    Daily Clean Installation Results:
                    https://fogtesting.fogproject.us/
                    FOG Reporting:
                    https://fog-external-reporting-results.fogproject.us/

                    1 Reply Last reply Reply Quote 0
                    • Wayne WorkmanW
                      Wayne Workman
                      last edited by

                      I rebooted that particular Storage Node. For the moment, it appears to be reporting its version correctly. I expect it to quit again after some time passes.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                      Daily Clean Installation Results:
                      https://fogtesting.fogproject.us/
                      FOG Reporting:
                      https://fog-external-reporting-results.fogproject.us/

                      1 Reply Last reply Reply Quote 0
                      • Wayne WorkmanW
                        Wayne Workman
                        last edited by

                        Update on this.

                        The two storage nodes I’ve rebooted are sticking around with version reporting… Man this is weird.

                        I guess now the next step is to reboot them all and see what happens.

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                        Daily Clean Installation Results:
                        https://fogtesting.fogproject.us/
                        FOG Reporting:
                        https://fog-external-reporting-results.fogproject.us/

                        1 Reply Last reply Reply Quote 0
                        • Wayne WorkmanW
                          Wayne Workman
                          last edited by Wayne Workman

                          I checked the nodes version reporting and interface reporting this morning - it’s still working.

                          So - I have no idea why but - the moral of the story is if you change your fogstorage password on the main database, after changing the snmysqlpass field in /opt/fog/.fogsettings on each node and re-running the installer, reboot the node.

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                          Daily Clean Installation Results:
                          https://fogtesting.fogproject.us/
                          FOG Reporting:
                          https://fog-external-reporting-results.fogproject.us/

                          1 Reply Last reply Reply Quote 0
                          • Wayne WorkmanW
                            Wayne Workman
                            last edited by

                            I spoke too soon.

                            They are all not reporting again. Considering - they stayed reporting all night long into the morning before anyone got here - this must be load related somehow.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                            Daily Clean Installation Results:
                            https://fogtesting.fogproject.us/
                            FOG Reporting:
                            https://fog-external-reporting-results.fogproject.us/

                            1 Reply Last reply Reply Quote 0
                            • Wayne WorkmanW
                              Wayne Workman
                              last edited by

                              I wrote a script that reboots them all, and ran that. They are all now reporting again…

                              This wasn’t an issue until I changed the password for fogstorage - Why would that cause this?

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                              Daily Clean Installation Results:
                              https://fogtesting.fogproject.us/
                              FOG Reporting:
                              https://fog-external-reporting-results.fogproject.us/

                              1 Reply Last reply Reply Quote 0
                              • Wayne WorkmanW
                                Wayne Workman
                                last edited by Wayne Workman

                                Below is the apache error log from one of the nodes, snipped from when the server rebooted to when the issue happened again. The server operated for exactly 37 minutes before falling off again. All storage nodes fell off at the same time - since they were all rebooted at the same time.

                                [Thu Aug 18 08:29:09.737517 2016] [mpm_prefork:notice] [pid 1504] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Aug 18 08:29:22.823456 2016] [core:notice] [pid 1503] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Aug 18 08:29:22.907692 2016] [suexec:notice] [pid 1503] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Aug 18 08:29:23.127056 2016] [ssl:warn] [pid 1503] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using fe80::215:5dff:fe01:a00. Set the 'ServerName' directive globally to suppress this message
[Thu Aug 18 08:29:23.149139 2016] [auth_digest:notice] [pid 1503] AH01757: generating secret for digest authentication ...
[Thu Aug 18 08:29:23.149672 2016] [lbmethod_heartbeat:notice] [pid 1503] AH02282: No slotmem from mod_heartmonitor
[Thu Aug 18 08:29:23.150447 2016] [ssl:warn] [pid 1503] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Aug 18 08:29:23.230529 2016] [mpm_prefork:notice] [pid 1503] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.6.24 configured -- resuming normal operations
[Thu Aug 18 08:29:23.230555 2016] [core:notice] [pid 1503] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Aug 18 09:07:34.373250 2016] [:error] [pid 2727] [client 10.51.1.53:44934] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/lib/fog/eventmanager.class.php:62\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/lib/fog/eventmanager.class.php(62): iterator_to_array(Object(RegexIterator), false)\n#4 /var/www/html/fog/commons/init.php(137): EventManager->load()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/lib/fog/eventmanager.class.php on line 62
[Thu Aug 18 09:07:38.374827 2016] [:error] [pid 2727] [client 10.51.1.53:45122] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/commons/init.php:14\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/commons/init.php(14): iterator_to_array(Object(RegexIterator))\n#4 /var/www/html/fog/commons/init.php(121): Initiator->__construct()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/commons/init.php on line 14
[Thu Aug 18 09:07:42.356020 2016] [:error] [pid 2727] [client 10.51.1.53:45330] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/commons/init.php:14\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/commons/init.php(14): iterator_to_array(Object(RegexIterator))\n#4 /var/www/html/fog/commons/init.php(121): Initiator->__construct()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/commons/init.php on line 14
[Thu Aug 18 09:07:46.249272 2016] [:error] [pid 2727] [client 10.51.1.53:45518] PHP Fatal error:  Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/www/html/fog/lib/plugins/capone/html/images): failed to open dir: Too many open files' in /var/www/html/fog/commons/init.php:14\nStack trace:\n#0 [internal function]: RecursiveDirectoryIterator->__construct('/var/www/html/f...', 4096)\n#1 [internal function]: RecursiveDirectoryIterator->getChildren()\n#2 [internal function]: FilterIterator->next()\n#3 /var/www/html/fog/commons/init.php(14): iterator_to_array(Object(RegexIterator))\n#4 /var/www/html/fog/commons/init.php(121): Initiator->__construct()\n#5 /var/www/html/fog/commons/base.inc.php(10): require('/var/www/html/f...')\n#6 /var/www/html/fog/status/bandwidth.php(2): require_once('/var/www/html/f...')\n#7 {main}\n  thrown in /var/www/html/fog/commons/init.php on line 14

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                Daily Clean Installation Results:
                                https://fogtesting.fogproject.us/
                                FOG Reporting:
                                https://fog-external-reporting-results.fogproject.us/

                                1 Reply Last reply Reply Quote 1
                                • Wayne WorkmanW
                                  Wayne Workman
                                  last edited by

                                  Found a blurb about this error on Stack Overflow:

                                  http://stackoverflow.com/questions/14748499/fatal-error-too-many-open-files

                                  Quote:

                                  Don’t store DirectoryIterator objects for later; you will get an error saying “too many open files” when you store more than the operating system limit (usually 256 or 1024).

                                  For example, this will yield an error if the directory has too many files:

                                  <?php 
$files = array(); 
foreach (new DirectoryIterator('myDir') as $file) { 
    $files[] = $file; 
} 
?>

                                  Presumably, this approach is memory intensive as well.

                                  source: http://php.net/manual/pt_BR/directoryiterator.construct.php#87425

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                  Daily Clean Installation Results:
                                  https://fogtesting.fogproject.us/
                                  FOG Reporting:
                                  https://fog-external-reporting-results.fogproject.us/

                                  1 Reply Last reply Reply Quote 0
                                  • Wayne WorkmanW
                                    Wayne Workman
                                    last edited by

                                    Here is a more detailed post about it:

                                    http://superuser.com/questions/871276/failed-to-open-stream-too-many-open-files-php-1024-maximum-limit-redhat-wh

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                    Daily Clean Installation Results:
                                    https://fogtesting.fogproject.us/
                                    FOG Reporting:
                                    https://fog-external-reporting-results.fogproject.us/

                                    1 Reply Last reply Reply Quote 0
                                    • Wayne WorkmanW
                                      Wayne Workman
                                      last edited by Wayne Workman

                                      I’ve found this site that shows how to set new limits properly on Ubuntu.

                                      https://www.drupal.org/node/474152

                                      I’ve converted the commands for CentOS 7/Fedora/RHEL - and made it into an SSH command ran against a remote server.

                                      ssh aifog "echo 'apache soft nofile 1600' >> /etc/security/limits.conf;echo 'apache hard nofile 65535' >> /etc/security/limits.conf;systemctl restart httpd"

                                      Note that the above is an ssh command that uses aliases and pki-based certificates - that my project called ssh-pki-setup has already established for me. That command executes the commands in double quotes against the remote server, and it successfully added the options to the remote server’s file and restarts httpd.

                                      If successful, I’ll do this for all storage nodes - and post commands that can be ran manually too if you don’t have so many servers to manage as I do and prefer manual.

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                      Daily Clean Installation Results:
                                      https://fogtesting.fogproject.us/
                                      FOG Reporting:
                                      https://fog-external-reporting-results.fogproject.us/

                                      1 Reply Last reply Reply Quote 0
                                      • Wayne WorkmanW
                                        Wayne Workman
                                        last edited by

                                        After the last reboot - all storage nodes are still reporting. But I assume they will eventually fail. I’m just waiting for that to happen - to see if the one I modified hangs in there.

                                        In the mean time, I’ve been watching the number of files open by apache with this command I put together: lsof -l -u apache | wc -l

                                        Open files have been steadily growing:

                                        [root@wgfog ~]# lsof -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6316
[root@wgfog ~]# lsof -u apache -l | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6412
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6424
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6432
[root@wgfog ~]# clear
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6444
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6448
[root@wgfog ~]#
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6448
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6456
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6456
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6464
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6472
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
7010
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
7538
[root@wgfog ~]# [root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6444
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6448
[root@wgfog ~]#
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6448
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6456
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6456
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6464
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
6472
[root@wgfog ~]# lsof -l -u apache | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/42/gvfs
      Output information may be incomplete.
7010

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                        Daily Clean Installation Results:
                                        https://fogtesting.fogproject.us/
                                        FOG Reporting:
                                        https://fog-external-reporting-results.fogproject.us/

                                        1 Reply Last reply Reply Quote 0
                                        • Wayne WorkmanW
                                          Wayne Workman
                                          last edited by

                                          Ok, the node I’ve been monitoring open files on - just fell off.

                                          And - the magical limit is… 8006 files open by Apache - causes everything to just stop.

                                          And… the one storage node I modified… it stopped reporting too… 😞

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                          Daily Clean Installation Results:
                                          https://fogtesting.fogproject.us/
                                          FOG Reporting:
                                          https://fog-external-reporting-results.fogproject.us/

                                          1 Reply Last reply Reply Quote 0
                                          • Wayne WorkmanW
                                            Wayne Workman
                                            last edited by

                                            This article seems relevant but I don’t see how it could help in the case of FOG. https://httpd.apache.org/docs/current/vhosts/fd-limits.html

                                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                            Daily Clean Installation Results:
                                            https://fogtesting.fogproject.us/
                                            FOG Reporting:
                                            https://fog-external-reporting-results.fogproject.us/

                                            1 Reply Last reply Reply Quote 0
                                            • Wayne WorkmanW
                                              Wayne Workman
                                              last edited by

                                              I think this must be a file descriptors leak in the storage node configuration.

                                              I’m going to patch it myself after lunch.

                                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                              Daily Clean Installation Results:
                                              https://fogtesting.fogproject.us/
                                              FOG Reporting:
                                              https://fog-external-reporting-results.fogproject.us/

                                              1 Reply Last reply Reply Quote 0
                                              • 1
                                              • 2
                                              • 1 / 2
                                              • First post
                                                Last post

                                              322

                                              Online

                                              11.7k

                                              Users

                                              17.2k

                                              Topics

                                              154.8k

                                              Posts
                                              Copyright © 2012-2024 FOG Project