• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

GUI crashing every 30 minutes

Scheduled Pinned Locked Moved
FOG Problems
2
10
759
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • L
    LPetelik
    last edited by LPetelik Sep 30, 2022, 10:41 AM Sep 30, 2022, 4:36 PM

    During work hours, our FOG GUI is crashing about every 30 minutes or so. We are using it daily with a pretty heavy client load and maybe 3 to 6 machines imaging at a time. Imaging seems to still work when this happens. Running “service mysql start and stop” seems to temporarily fix it. The server is a Lenovo S30 so old but a workhorse. Master running Centos 3.10.0-1160.76.1.e17 Nodes are running on VM on desktops at each site. 20~ish sites.

    What we have tried/checked:
    We have updated the main server and all the nodes.
    We found something about updating the kernels online. We did that and it seemed better yesterday.
    (This item has been ongoing for a while, a nuisance that we keep hoping to work on as we have time but may be related.) I also suspect we might have some data corruption (I think we might have a duplicate mac address situation). I can’t look at all the hosts on the host page. We can only search. I was able to export from the Configurations page. I can’t export hosts from the hosts’ page though. The csv is hard to look through just because of the sheer volume of clients.
    We checked memory usage, it seems good.
    We have run the database clean-up commands. They come back pretty clean.
    We have checked passwords on the server and nodes.
    We have looked through logs and checked on any errors that seem like they could be the cause.

    Any suggestions or places we should start looking?

    G 1 Reply Last reply Sep 30, 2022, 4:50 PM Reply Quote 0
    • G
      george1421 Moderator @LPetelik
      last edited by george1421 Sep 30, 2022, 10:55 AM Sep 30, 2022, 4:50 PM

      @lpetelik For a bit of clarity I have a few more questions.

      1. Is the sql server process crashing (no longer in memory) or is it just not responding to requests.
      2. What version of FOG are you running?
      3. It sounds like you have a fog master node and multiple storage nodes running on VMs? Is that accurate? How many storage nodes do you have in this configuration?
      4. How many target computers are in your environment that have the fog client installed?
      5. When inspecting syslog/messages/sql server logs do you see any error messages relating to sql server, like too many connections?
      6. What is the fog server host operating system?

      Just for clarity in definition updating “the kernel” relates to the FOS Linux kernel [bzImage] that gets transferred to the target computer during imaging. This reference has nothing to do with the FOG server host OS kernel.

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

      L 2 Replies Last reply Sep 30, 2022, 8:34 PM Reply Quote 1
      • L
        LPetelik @george1421
        last edited by Sep 30, 2022, 8:34 PM

        @george1421

        Checking on the other answers, might be Monday before I can reply.
        2. 1.5.9
        3 Master on regular hardware, nodes on VM. 24 nodes
        4 I will get a more exact answer. Should be near 2500-3000

        I should also mention that the GUI dropping just started about a week ago.

        G 1 Reply Last reply Sep 30, 2022, 9:23 PM Reply Quote 0
        • G
          george1421 Moderator @LPetelik
          last edited by Sep 30, 2022, 9:23 PM

          @lpetelik said in GUI crashing every 30 minutes:

          3 Master on regular hardware, nodes on VM. 24 nodes
          4 I will get a more exact answer. Should be near 2500-3000

          I’m thinking mysql is complaining about max open connections and with 2500+ fog workstations you might be running into a myisam database performance issue. Both issues can be resolved. The myisam/innodb issue would be displayed by looking at top and then sorting by Processor. If mysql is having high cpu with many php-fpm workers then we need to switch over the database engine.

          I should also mention that the GUI dropping just started about a week ago.

          Could be an update issue or you just got lucky. A broken/slowed down sql server would make the web ui unresponsive.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

          1 Reply Last reply Reply Quote 0
          • L
            LPetelik @george1421
            last edited by LPetelik Oct 10, 2022, 10:21 AM Oct 10, 2022, 4:19 PM

            @george1421
            Sorry for the late reply

            1. I will need to check on this.
            2. 1.5.9 current stable on all nodes and master.
            3. Master is on regular hardware. One node is also. The rest of the nodes are on VM. 24 nodes
            4. A bit over 4000.
            5. Not “too many connections” errors. I am seeing a memory error and will try to address that today.
            6. A mix between Centos and Rocky. We are trying to move all of the nodes to Rocky.
            G L 2 Replies Last reply Oct 10, 2022, 5:14 PM Reply Quote 0
            • G
              george1421 Moderator @LPetelik
              last edited by Oct 10, 2022, 5:14 PM

              @lpetelik Will you submit a screen shot of top sorted by Procesor utilization? This should be captured under normal load imaging or not, but with all or as many as possible clients contacting the FOG server.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

              1 Reply Last reply Reply Quote 0
              • L
                LPetelik @LPetelik
                last edited by Oct 10, 2022, 7:03 PM

                @george1421

                Here is the top screenshot. It does look like a lot of processes running using quite a bit of CPU.

                9b9d1655-01e9-4b7e-aa17-6ed56fe1e28d-image.png

                G 2 Replies Last reply Oct 10, 2022, 7:13 PM Reply Quote 0
                • G
                  george1421 Moderator @LPetelik
                  last edited by Oct 10, 2022, 7:13 PM

                  @lpetelik I thought I had a tutorial on fixing this. The root of the issue is your fog database is still using myisam for the data engine and not the innodb engine.

                  I kind of outlined the steps here: https://forums.fogproject.org/topic/15856/web-gui-error-after-clicking-list-hosts

                  Let me write it up a bit clearer tonight, but all of the steps are in that thread. Just read the thread from the bottom up to make it understandable.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                  1 Reply Last reply Reply Quote 0
                  • G
                    george1421 Moderator @LPetelik
                    last edited by Oct 10, 2022, 11:49 PM

                    @lpetelik Well, blind I must be. Here is the tutorial I thought I created but could not find earlier today: https://forums.fogproject.org/topic/16099/configure-fog-database-to-use-innodb-engine

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                    L 1 Reply Last reply Oct 12, 2022, 3:49 PM Reply Quote 0
                    • L
                      LPetelik @george1421
                      last edited by LPetelik Oct 12, 2022, 9:50 AM Oct 12, 2022, 3:49 PM

                      @george1421 We just ran the engine update. Thanks for those links. We’ll watch it for a bit to see how it looks. The GUI is a bit slow at the moment, hoping that clears up.

                      On a side note, while working with this issue, we ran .installfog and during the install it picked a new STORAGENODE MYSQLPASS password that had a closing bracket in it. It broke the node’s connections to the master even with the password updated in all the spots it should be. Once we created a new SQL password (on a whim to test, without special characters) and updated it in .fogsettings, reran the .installfog file, the nodes started talking to the master again.

                      I think we are in much better shape. We need to fix our node password issue and start imaging again to know for sure.

                      1 Reply Last reply Reply Quote 0
                      • 1 / 1
                      • First post
                        Last post

                      202

                      Online

                      12.1k

                      Users

                      17.3k

                      Topics

                      155.3k

                      Posts
                      Copyright © 2012-2024 FOG Project