Web GUI, Fog login problem
-
Hey,
This is probably not a common thing but, yesterday we were having serious issues with our ISP. Things still had connectivity to the internet, but were just dog, dog slow.
I jumped onto our Fog server’s Web GUI to do some imaging… Typed in the credentials as usual to log in and it just sat there… and sat there.
The little counter at the bottom:
Estimated FOG Sites: 4826
Latest Version: 1.2.0That wasn’t displaying any info, it was just spinning (due to our dog, dog slow speeds yesterday).
So, I suspect that since it detected an internet connection, it tries to pull those two pieces of data (from where ever?) and it doesn’t process a log in attempt until that data is loaded.
Anyone else experience this?Thanks,
Wayne -
The steps work almost like that. It’s not so much that it detects the internet or not, but attempts to go there. You are “logged” in at that point, but it’s waiting return which isn’t happening fast enough.
-
Maybe a time-out can be added for that piece?
Something like 2 seconds, maybe? -
That’s something I wanted to look at. It’s something that happens when you run blocking system commands on a single-threaded server. It halts all other requests until the command completes. The same thing happens if you allow host listings to ping hosts on your network (I’m glad that’s disabled by default on the Post-1.2.0 builds.).
That’s just my guess as to what is happening since I’ve seen similar problems before that fit my assumption. I haven’t dug into the code yet since I’m still working on other objectives for deploying FOG into my network.
[U]Possible solution[/U]
[LIST=1]
[]Have a background service that checks periodically and edits a static element of the website (asynchronous process method).
[]Handle the server check with client-side code. Client-side checks won’t halt the entire server waiting for them to complete.
[/LIST] -
I just confirmed that my guess was correct. The site uses PHP’s blocking function file_get_contents(). If you are like the original poster and have a slow connection or you are like me and have no Internet access at all, you will see the PHP server waiting for the website to resolve since the command is blocking. This is also why I noticed that the site became unresponsive temporarily upon visiting the configuration settings page.
This is much more serious for users like me that provide an imaging service in an isolated/air-gapped network. An attacker could initiate a denial of service attack by sending requests to http(s)://foginstallation.tld/fog/management/index.php?node=client&sub=loginInfo. This type of attack wouldn’t set off normal load balancing equipment either since you could send these requests once in a while (with those timeouts, you could do a request every 10 seconds to really hurt performance).
If you want to continue pulling data from a foreign website (fogproject.org) for tracking purposes, you’ll need to look into creating a service for checking the data that’s causing this trouble. The data pulled can either be placed directly on the database, or it could be placed on a static page.
If you don’t care about tracking data, you can look into a few tricks to get around cross-domain request restrictions. One such way is by including a javascript file with the version and server data hard-coded in. But again, you lose all your tracking data if that’s why you do it (you would get client GET requests, not server ones).
Finally, I would highly suggest replacing any fetchURL() invocations (which appear to be associated with fetching version information) with the service solution I suggested.
Additionally, I would highly suggest providing the ability to turn off version checking for people who are running in isolated environments.
-
You want, I’m attempting a non-blocking method, now, to get contents from the URL.
Please let me know if you notice improvement and if things are still functional.
-
Good to hear. I’ll be able to test it tomorrow.
-
[quote=“cspence, post: 43021, member: 28749”]Good to hear. I’ll be able to test it tomorrow.[/quote]
Bump.
Any updates?
-
…work load was a bit high the last two days.
I just installed 3080. I see what you tried, but it still relies on a blocking operation: [URL=‘http://php.net/manual/en/function.fread.php’]fread/URL.
The fopen() function followed by stream_set_blocking($fp,0) is great for sending stream data to a non-blocking receiver. The fopen() function is blocking by nature. It sits and waits for any of the four conditions that causes the function to fail or complete (in our case, it waits for the socket timeout which is the same problem again).
If you want to do local testing for this problem, you can temporarily put the following line into your /etc/hosts file:
0.0.0.0 fogproject.org
Be sure to remove it when you want to end testing and see this site again.
This is not a problem you can solve using server calls (whether or not they come from client-side code or internal functions). You’ll need to set up a service that can update your database or provide a file to check locally.
-
Perhaps we can be given the option to change the timeout length?
Is there a PHP edit we can do to change it? Maybe the FOG web GUI can pull this value from a file, instead of it being hard-coded into the PHP ?
-
[quote=“Wayne Workman, post: 43326, member: 28155”]Perhaps we can be given the option to change the timeout length?
Is there a PHP edit we can do to change it? Maybe the FOG web GUI can pull this value from a file, instead of it being hard-coded into the PHP ?[/quote]
You can change the socket timeout, but it is still going to eat performance with more users. Like I said before, it’s easy to cause a DoS by requesting that Ajax call repeatedly. This could also happen if you have three people on the website at the same time. Any time a user visits the login or views other pages that leverage fetchURL() like the configuration settings page, there is going to be a server freeze.
Considering that the only usage of fetchURL is for getting latest version or server count data, that could be handled outside of the web server. I’ll even write a service script to help with this.
-
[quote=“cspence, post: 43329, member: 28749”]You can change the socket timeout, but it is still going to eat performance with more users. Like I said before, it’s easy to cause a DoS by requesting that Ajax call repeatedly. This could also happen if you have three people on the website at the same time. Any time a user visits the login or views other pages that leverage fetchURL() like the configuration settings page, there is going to be a server freeze.
Considering that the only usage of fetchURL is for getting latest version or server count data, that could be handled outside of the web server. I’ll even write a service script to help with this.[/quote]
On a serious note, how likely is a DoS to happen from within your own network? Nobody is going to build a FOG server inside a de-militarized zone, with a public IP address…
FOG is for internal use.
-
[quote=“Wayne Workman, post: 43330, member: 28155”]On a serious note, how likely is a DoS to happen from within your own network? Nobody is going to build a FOG server inside a de-militarized zone, with a public IP address…
FOG is for internal use. ;-)[/quote]
Unless you’re running an air-gapped hacking proving ground for Information Assurance (Cyber Security) students at a University where you need to deploy fresh machine images in short amounts of time. Then it’s definitely a threat.
In that case, FOG is not for internal use. It’s for anyone who needs a copy of a machine image deployed.
-
Forgive my ignorance I suppose, but how, exactly, is fread a blocking function? I’ve explicitly set the file descriptor to be done in a non-blocking mode. This means, if there’s data to return, it returns it, if not it returns none at the time it’s being read.
-
No problem. I’ll just paste a bit of the documentation on that function here:
[COLOR=#0000ff][B]fread()[/B] reads up to length bytes from the file pointer referenced by handle. Reading stops as soon as one of the following conditions is met: [/COLOR]
[LIST]
[][COLOR=#0000ff] length bytes have been read [/COLOR]
[][COLOR=#0000ff] EOF (end of file) is reached [/COLOR]
[][COLOR=#0000ff] a packet becomes available or the [URL=‘http://php.net/manual/en/function.socket-set-timeout.php’][COLOR=#0000ff] socket timeout[/COLOR][/URL] occurs (for network streams) [/COLOR]
[][COLOR=#0000ff] if the stream is read buffered and it does not represent a plain file, at most one read of up to a number of bytes equal to the chunk size (usually 8192) is made; depending on the previously buffered data, the size of the returned data may be larger than the chunk size. [/COLOR]
[/LIST]
So let’s walk through the bullets:- If the file descriptor is waiting for input, it will definitely not reach the length.
- There is no EOF because the socket is still waiting for input.
- The socket will ultimately fail out because it can’t contact the server. In the meantime, the command still waits.
- I’m not exactly sure how this behaves with a network socket, but it is still waiting for buffered data to arrive.
One more thing to think about when trying to handle this process: If you were able to make fread() non-blocking (or multi-threaded), what affect does that have? Instant failure to grab any input. Additionally, since the function getting the data is not reading from readily available source (like a database entry), it has to wait or fail. This is why you need to have something like a service that does this outside of the server. The service can update the database or a file that the webserver could check. Instead of waiting for something that is not there at the instant it checks, it will always be there or not be there (EOF).
The fopen() function can be non-blocking because it can be redirected somewhere like a log or database entry. But be careful, this doesn’t mean you should redirect this to a database each time there is a call to fetchURL(). You will get data before it is overwritten by the latest fetchURL() call. In other words, stuff will wack out. If a lock is placed on that database entry while trying to write to it, you’ll have another source of I/O blocking. This is yet another reason to just set up a service that will only update periodically and not every time you visit pages.
If I made a mistake somewhere, feel free to point it out.
-
I’ll step through my knowledge, though I’m sure I am more likely wrong, but according to PHP’s own documentation:
[url]http://php.net/manual/en/function.stream-set-blocking.php[/url]
The first few lines:
[quote]mode
If mode is 0, the given stream will be switched to non-blocking mode, and if 1, it will be switched to blocking mode. This affects calls like fgets() and fread() that read from the stream. In non-blocking mode an fgets() call will always return right away while in blocking mode it will wait for data to become available on the stream.[/quote]The “mode” is to set the stream that’s being read directly into non-blocking.
Add to this that we’re using an AJAX call to even perform the fetchURL phase, and now a completely non-blocking stream that simply checks for data, this means it should be entirely nonblocking. For further test, I’ve even added the ajax call a specific async: true, call to ensure that if for whatever reason it can’t make it to the local ajax caller, it should return false. The ONLY think I believe could be potentially causing the “block” at this point would be the foreach loop at this point.
So maybe I’m thinking wrong, but when the only documentation I have to work off of states that something is non-blocking, I haven’t much choice but to entrust the official sources. I’m more on the line that the fread isn’t our blocking point there, but rather the for each loop itself.
I can see if removing the loop is better, but still don’t know EXACTLY what’s causing the block itself. It just seems odd that now that (from my perspective) we’re using ajax and an explicitly set non-blocking stream, that it’s blocking at all.[/quote]
-
It’s definitely the server-side though. You can lock up the server by just visiting the URL the Ajax call uses. After all, client-side scripts should not lock up the server.
-
Alright,
So I’m attempting more things. It is only using the stream to read the contents of the data at any given time. From my testing, the hwinfo (which also uses the same fetchURL command) doesn’t block the page anymore which was occurring with the fread. While potentially not perfect, I don’t know what else to try. The whole idea of the current version and number of sites is simply to give the normal fog user an idea of what is in use. Hopefully this new approach may work better.
Thank you,
-
Before you try to code around an issue we don’t quite understand (I would hate to see you waste time on the wrong chunk of code.), I have some more test results.
I tried commenting out the fetchURL() function contents, but it still froze up on the ajax call. So that was unexpected from what I know so far from analyzing the code. So I made sure that the apache server had been restarted and that you guys didn’t do something crazy with PHP (I had to make sure you weren’t running something like HipHop or some other compiled PHP project).
Next I installed Wireshark to see how it was potentially doing on the network. FOG was sending DNS requests like crazy to my internal DNS (which of course is also isolated in my network). The requests would fail out naturally.
By this point, I realized that this was a good time to follow my advice from earlier. I added 0.0.0.0 fogproject.org to my /etc/hosts. Instant page loads!
This leaves us with two+ questions:
[LIST=1]
[]Where in execution outside of fetchURL() does the server attempt to resolve fogproject.org when http(s)://foginstallation.tld/fog/management/index.php?node=client&sub=loginInfo?
[]Are those other queries being run non-blocking? (probably not)
[*]If everything is running non-blocking (if, remember), does that mean that the way PHP does DNS is still blocking? (Unlikely from what I would expect so far.)
[/LIST] -
There’s only two calls to fogproject.org during the login phase. They’re both, now, and even before, using the fetchURL method.
