• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Web GUI, Fog login problem

    Scheduled Pinned Locked Moved Solved
    Bug Reports
    4
    26
    9.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Wayne WorkmanW
      Wayne Workman
      last edited by

      Hey,

      This is probably not a common thing but, yesterday we were having serious issues with our ISP. Things still had connectivity to the internet, but were just dog, dog slow.

      I jumped onto our Fog server’s Web GUI to do some imaging… Typed in the credentials as usual to log in and it just sat there… and sat there.

      The little counter at the bottom:

      Estimated FOG Sites: 4826
      Latest Version: 1.2.0

      That wasn’t displaying any info, it was just spinning (due to our dog, dog slow speeds yesterday).

      So, I suspect that since it detected an internet connection, it tries to pull those two pieces of data (from where ever?) and it doesn’t process a log in attempt until that data is loaded.
      Anyone else experience this?

      Thanks,
      Wayne

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
      Daily Clean Installation Results:
      https://fogtesting.fogproject.us/
      FOG Reporting:
      https://fog-external-reporting-results.fogproject.us/

      1 Reply Last reply Reply Quote 0
      • Tom ElliottT
        Tom Elliott
        last edited by

        The steps work almost like that. It’s not so much that it detects the internet or not, but attempts to go there. You are “logged” in at that point, but it’s waiting return which isn’t happening fast enough.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        1 Reply Last reply Reply Quote 0
        • Wayne WorkmanW
          Wayne Workman
          last edited by

          Maybe a time-out can be added for that piece?
          Something like 2 seconds, maybe?

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
          Daily Clean Installation Results:
          https://fogtesting.fogproject.us/
          FOG Reporting:
          https://fog-external-reporting-results.fogproject.us/

          1 Reply Last reply Reply Quote 0
          • cspenceC
            cspence Developer
            last edited by

            That’s something I wanted to look at. It’s something that happens when you run blocking system commands on a single-threaded server. It halts all other requests until the command completes. The same thing happens if you allow host listings to ping hosts on your network (I’m glad that’s disabled by default on the Post-1.2.0 builds.).

            That’s just my guess as to what is happening since I’ve seen similar problems before that fit my assumption. I haven’t dug into the code yet since I’m still working on other objectives for deploying FOG into my network.

            [U]Possible solution[/U]
            [LIST=1]
            []Have a background service that checks periodically and edits a static element of the website (asynchronous process method).
            [
            ]Handle the server check with client-side code. Client-side checks won’t halt the entire server waiting for them to complete.
            [/LIST]

            1 Reply Last reply Reply Quote 0
            • cspenceC
              cspence Developer
              last edited by

              I just confirmed that my guess was correct. The site uses PHP’s blocking function file_get_contents(). If you are like the original poster and have a slow connection or you are like me and have no Internet access at all, you will see the PHP server waiting for the website to resolve since the command is blocking. This is also why I noticed that the site became unresponsive temporarily upon visiting the configuration settings page.

              This is much more serious for users like me that provide an imaging service in an isolated/air-gapped network. An attacker could initiate a denial of service attack by sending requests to http(s)://foginstallation.tld/fog/management/index.php?node=client&sub=loginInfo. This type of attack wouldn’t set off normal load balancing equipment either since you could send these requests once in a while (with those timeouts, you could do a request every 10 seconds to really hurt performance).

              If you want to continue pulling data from a foreign website (fogproject.org) for tracking purposes, you’ll need to look into creating a service for checking the data that’s causing this trouble. The data pulled can either be placed directly on the database, or it could be placed on a static page.

              If you don’t care about tracking data, you can look into a few tricks to get around cross-domain request restrictions. One such way is by including a javascript file with the version and server data hard-coded in. But again, you lose all your tracking data if that’s why you do it (you would get client GET requests, not server ones).

              Finally, I would highly suggest replacing any fetchURL() invocations (which appear to be associated with fetching version information) with the service solution I suggested.

              Additionally, I would highly suggest providing the ability to turn off version checking for people who are running in isolated environments.

              1 Reply Last reply Reply Quote 0
              • Tom ElliottT
                Tom Elliott
                last edited by

                You want, I’m attempting a non-blocking method, now, to get contents from the URL.

                Please let me know if you notice improvement and if things are still functional.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                1 Reply Last reply Reply Quote 0
                • cspenceC
                  cspence Developer
                  last edited by

                  Good to hear. I’ll be able to test it tomorrow.

                  1 Reply Last reply Reply Quote 0
                  • Wayne WorkmanW
                    Wayne Workman
                    last edited by

                    [quote=“cspence, post: 43021, member: 28749”]Good to hear. I’ll be able to test it tomorrow.[/quote]

                    Bump.

                    Any updates?

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                    Daily Clean Installation Results:
                    https://fogtesting.fogproject.us/
                    FOG Reporting:
                    https://fog-external-reporting-results.fogproject.us/

                    1 Reply Last reply Reply Quote 0
                    • cspenceC
                      cspence Developer
                      last edited by

                      …work load was a bit high the last two days.

                      I just installed 3080. I see what you tried, but it still relies on a blocking operation: [URL=‘http://php.net/manual/en/function.fread.php’]fread/URL.

                      The fopen() function followed by stream_set_blocking($fp,0) is great for sending stream data to a non-blocking receiver. The fopen() function is blocking by nature. It sits and waits for any of the four conditions that causes the function to fail or complete (in our case, it waits for the socket timeout which is the same problem again).

                      If you want to do local testing for this problem, you can temporarily put the following line into your /etc/hosts file:

                      0.0.0.0 fogproject.org

                      Be sure to remove it when you want to end testing and see this site again. 😉

                      This is not a problem you can solve using server calls (whether or not they come from client-side code or internal functions). You’ll need to set up a service that can update your database or provide a file to check locally.

                      1 Reply Last reply Reply Quote 0
                      • Wayne WorkmanW
                        Wayne Workman
                        last edited by

                        Perhaps we can be given the option to change the timeout length?

                        Is there a PHP edit we can do to change it? Maybe the FOG web GUI can pull this value from a file, instead of it being hard-coded into the PHP ?

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                        Daily Clean Installation Results:
                        https://fogtesting.fogproject.us/
                        FOG Reporting:
                        https://fog-external-reporting-results.fogproject.us/

                        1 Reply Last reply Reply Quote 0
                        • cspenceC
                          cspence Developer
                          last edited by

                          [quote=“Wayne Workman, post: 43326, member: 28155”]Perhaps we can be given the option to change the timeout length?

                          Is there a PHP edit we can do to change it? Maybe the FOG web GUI can pull this value from a file, instead of it being hard-coded into the PHP ?[/quote]

                          You can change the socket timeout, but it is still going to eat performance with more users. Like I said before, it’s easy to cause a DoS by requesting that Ajax call repeatedly. This could also happen if you have three people on the website at the same time. Any time a user visits the login or views other pages that leverage fetchURL() like the configuration settings page, there is going to be a server freeze.

                          Considering that the only usage of fetchURL is for getting latest version or server count data, that could be handled outside of the web server. I’ll even write a service script to help with this.

                          1 Reply Last reply Reply Quote 0
                          • Wayne WorkmanW
                            Wayne Workman
                            last edited by

                            [quote=“cspence, post: 43329, member: 28749”]You can change the socket timeout, but it is still going to eat performance with more users. Like I said before, it’s easy to cause a DoS by requesting that Ajax call repeatedly. This could also happen if you have three people on the website at the same time. Any time a user visits the login or views other pages that leverage fetchURL() like the configuration settings page, there is going to be a server freeze.

                            Considering that the only usage of fetchURL is for getting latest version or server count data, that could be handled outside of the web server. I’ll even write a service script to help with this.[/quote]

                            On a serious note, how likely is a DoS to happen from within your own network? Nobody is going to build a FOG server inside a de-militarized zone, with a public IP address…

                            FOG is for internal use. 😉

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                            Daily Clean Installation Results:
                            https://fogtesting.fogproject.us/
                            FOG Reporting:
                            https://fog-external-reporting-results.fogproject.us/

                            1 Reply Last reply Reply Quote 0
                            • cspenceC
                              cspence Developer
                              last edited by

                              [quote=“Wayne Workman, post: 43330, member: 28155”]On a serious note, how likely is a DoS to happen from within your own network? Nobody is going to build a FOG server inside a de-militarized zone, with a public IP address…

                              FOG is for internal use. ;-)[/quote]

                              Unless you’re running an air-gapped hacking proving ground for Information Assurance (Cyber Security) students at a University where you need to deploy fresh machine images in short amounts of time. Then it’s definitely a threat. 😉

                              In that case, FOG is not for internal use. It’s for anyone who needs a copy of a machine image deployed.

                              1 Reply Last reply Reply Quote 0
                              • Tom ElliottT
                                Tom Elliott
                                last edited by

                                Forgive my ignorance I suppose, but how, exactly, is fread a blocking function? I’ve explicitly set the file descriptor to be done in a non-blocking mode. This means, if there’s data to return, it returns it, if not it returns none at the time it’s being read.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                1 Reply Last reply Reply Quote 0
                                • cspenceC
                                  cspence Developer
                                  last edited by

                                  No problem. I’ll just paste a bit of the documentation on that function here:

                                  [COLOR=#0000ff][B]fread()[/B] reads up to length bytes from the file pointer referenced by handle. Reading stops as soon as one of the following conditions is met: [/COLOR]
                                  [LIST]
                                  [][COLOR=#0000ff] length bytes have been read [/COLOR]
                                  [
                                  ][COLOR=#0000ff] EOF (end of file) is reached [/COLOR]
                                  [][COLOR=#0000ff] a packet becomes available or the [URL=‘http://php.net/manual/en/function.socket-set-timeout.php’][COLOR=#0000ff] socket timeout[/COLOR][/URL] occurs (for network streams) [/COLOR]
                                  [
                                  ][COLOR=#0000ff] if the stream is read buffered and it does not represent a plain file, at most one read of up to a number of bytes equal to the chunk size (usually 8192) is made; depending on the previously buffered data, the size of the returned data may be larger than the chunk size. [/COLOR]
                                  [/LIST]
                                  So let’s walk through the bullets:

                                  1. If the file descriptor is waiting for input, it will definitely not reach the length.
                                  2. There is no EOF because the socket is still waiting for input.
                                  3. The socket will ultimately fail out because it can’t contact the server. In the meantime, the command still waits.
                                  4. I’m not exactly sure how this behaves with a network socket, but it is still waiting for buffered data to arrive.

                                  One more thing to think about when trying to handle this process: If you were able to make fread() non-blocking (or multi-threaded), what affect does that have? Instant failure to grab any input. Additionally, since the function getting the data is not reading from readily available source (like a database entry), it has to wait or fail. This is why you need to have something like a service that does this outside of the server. The service can update the database or a file that the webserver could check. Instead of waiting for something that is not there at the instant it checks, it will always be there or not be there (EOF).

                                  The fopen() function can be non-blocking because it can be redirected somewhere like a log or database entry. But be careful, this doesn’t mean you should redirect this to a database each time there is a call to fetchURL(). You will get data before it is overwritten by the latest fetchURL() call. In other words, stuff will wack out. If a lock is placed on that database entry while trying to write to it, you’ll have another source of I/O blocking. This is yet another reason to just set up a service that will only update periodically and not every time you visit pages.

                                  If I made a mistake somewhere, feel free to point it out.

                                  1 Reply Last reply Reply Quote 0
                                  • Tom ElliottT
                                    Tom Elliott
                                    last edited by

                                    I’ll step through my knowledge, though I’m sure I am more likely wrong, but according to PHP’s own documentation:

                                    [url]http://php.net/manual/en/function.stream-set-blocking.php[/url]

                                    The first few lines:

                                    [quote]mode
                                    If mode is 0, the given stream will be switched to non-blocking mode, and if 1, it will be switched to blocking mode. This affects calls like fgets() and fread() that read from the stream. In non-blocking mode an fgets() call will always return right away while in blocking mode it will wait for data to become available on the stream.[/quote]

                                    The “mode” is to set the stream that’s being read directly into non-blocking.

                                    Add to this that we’re using an AJAX call to even perform the fetchURL phase, and now a completely non-blocking stream that simply checks for data, this means it should be entirely nonblocking. For further test, I’ve even added the ajax call a specific async: true, call to ensure that if for whatever reason it can’t make it to the local ajax caller, it should return false. The ONLY think I believe could be potentially causing the “block” at this point would be the foreach loop at this point.

                                    So maybe I’m thinking wrong, but when the only documentation I have to work off of states that something is non-blocking, I haven’t much choice but to entrust the official sources. I’m more on the line that the fread isn’t our blocking point there, but rather the for each loop itself.

                                    I can see if removing the loop is better, but still don’t know EXACTLY what’s causing the block itself. It just seems odd that now that (from my perspective) we’re using ajax and an explicitly set non-blocking stream, that it’s blocking at all.[/quote]

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                    1 Reply Last reply Reply Quote 0
                                    • cspenceC
                                      cspence Developer
                                      last edited by

                                      It’s definitely the server-side though. You can lock up the server by just visiting the URL the Ajax call uses. After all, client-side scripts should not lock up the server.

                                      1 Reply Last reply Reply Quote 0
                                      • Tom ElliottT
                                        Tom Elliott
                                        last edited by

                                        Alright,

                                        So I’m attempting more things. It is only using the stream to read the contents of the data at any given time. From my testing, the hwinfo (which also uses the same fetchURL command) doesn’t block the page anymore which was occurring with the fread. While potentially not perfect, I don’t know what else to try. The whole idea of the current version and number of sites is simply to give the normal fog user an idea of what is in use. Hopefully this new approach may work better.

                                        Thank you,

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                        1 Reply Last reply Reply Quote 0
                                        • cspenceC
                                          cspence Developer
                                          last edited by

                                          Before you try to code around an issue we don’t quite understand (I would hate to see you waste time on the wrong chunk of code.), I have some more test results.

                                          I tried commenting out the fetchURL() function contents, but it still froze up on the ajax call. So that was unexpected from what I know so far from analyzing the code. So I made sure that the apache server had been restarted and that you guys didn’t do something crazy with PHP (I had to make sure you weren’t running something like HipHop or some other compiled PHP project).

                                          Next I installed Wireshark to see how it was potentially doing on the network. FOG was sending DNS requests like crazy to my internal DNS (which of course is also isolated in my network). The requests would fail out naturally.

                                          By this point, I realized that this was a good time to follow my advice from earlier. I added 0.0.0.0 fogproject.org to my /etc/hosts. Instant page loads!

                                          This leaves us with two+ questions:
                                          [LIST=1]
                                          []Where in execution outside of fetchURL() does the server attempt to resolve fogproject.org when http(s)://foginstallation.tld/fog/management/index.php?node=client&sub=loginInfo?
                                          [
                                          ]Are those other queries being run non-blocking? (probably not)
                                          [*]If everything is running non-blocking (if, remember), does that mean that the way PHP does DNS is still blocking? (Unlikely from what I would expect so far.)
                                          [/LIST]

                                          1 Reply Last reply Reply Quote 0
                                          • Tom ElliottT
                                            Tom Elliott
                                            last edited by

                                            There’s only two calls to fogproject.org during the login phase. They’re both, now, and even before, using the fetchURL method.

                                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                            1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post

                                            203

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project