better web performance?

[Gabor]

I don’t completely understand, how fog works, so if it’s stupid I am sorry

As I understand now the https server works as a user interface, and also used when booting machines and communicate with fog clients and how knows for what else. I always felt the web interface laggy, and now I think it is because of the “KeepAlive Off” in the apache config, which I think is needed because of the pxe boot (or whatever).

I wanted to reach the fog web ui with company signed cert. Thanks @Sebastian-Roth for trying to help me,
but it was too complicated, and I messed it up.

So I decided that I will use my cert on tcp 444 and proxying the requests to the original site locally or a same virtualhost what fog generated but on different port (81) without ssl (why use ssl locally?) and also removed the KeepAlive Off setting. Unfortunately this doesn’t work because the program doesn’t handle the different port number and as I see some requests are “missrouted”. But if I use the original 443 instead of 81, it works (not tested too much yet), and I still notice that the site response time is better (on 444 than on 443).

So if it possible I think it worth considering to separate the web and this way gain some speed on the ui, and also would easier to install custom certs.

What do you think?

[george1421]

I guess I have a few comments here.

1. Where are you seeing a slow web interface?
2. What version of FOG are you using?
3. How many computers in your environment have the FOG client installed?
4. What is your FOG client check in period (time interval)?

There are some things that can be done post install to help improve performance quite a bit.

Now this is just a suspicion, but with apache why not just create a new virtual site on http port 443 and use your enterprise certs here? PXE booting and fog client will continue to use port 80 which will not have ssl turned on. If your goal is to use your company signed certs for just the web interface then do that. You may have to play with the default (index.php/index.html) so if people land without a path it redirects to the https interface. I would think both the fog client and pxe booting would know the fully defined path to the files they want.

Personally I’ve wanted the developers to move the fog client target port to something other than 80 so the current ssl certs and methodology can continue for the fog client without changing much code. Performance wise its still the same since its the fog server as a whole that has the impact on performance.

[Gabor]

@george1421 said in better web performance?:

I guess I have a few comments here.

1. Where are you seeing a slow web interface?

Not slow but rather lagging its about 1-2 tenth of a second, which is enough to feel that is not responding well.

2. What version of FOG are you using?

1.5.8

3. How many computers in your environment have the FOG client installed?

about a hundred, but i feel the lag even if they all turned off.

4. What is your FOG client check in period (time interval)?

I don’t know, default.

There are some things that can be done post install to help improve performance quite a bit.

Now this is just a suspicion, but with apache why not just create a new virtual site on http port 443 and use your enterprise certs here? PXE booting and fog client will continue to use port 80 which will not have ssl turned on.

I don’t know about pxe booting, but my fog clients uses https, and I think they should as a security reason.

If your goal is to use your company signed certs for just the web interface then do that. You may have to play with the default (index.php/index.html) so if people land without a path it redirects to the https interface. I would think both the fog client and pxe booting would know the fully defined path to the files they want.

As I mentioned I have tried to change the default cert, but it is not so easy. And my point here is that with this proxy I ended up an overall better situation, where I even got rid off the lag. Maybe this doesn’t annoys others as me, but I am really happy that it’s gone. And it would be even better if I could turn off ssl and enable Keepalive behind the proxy. And I think some modification would be enogh on port 80 virtualhost, but as I don’t understand the mechanisms I am afraid to toch that settings.

Personally I’ve wanted the developers to move the fog client target port to something other than 80 so the current ssl certs and methodology can continue for the fog client without changing much code. Performance wise its still the same since its the fog server as a whole that has the impact on performance.

[george1421]

@Gabor 100 is not that busy of a fog server. But when your clients are online, on the fog server console. Go into top and sort by processor consumption P and tell me what the top process is and its percentage of CPU.

Is your fog server a VM or a physical box? If its a VM how many vCPU do you have allocated and how much ram (top will also tell you free ram)

The default check in time is 5 minutes for the fog clients. If you had 500 clients then I would suggest increasing it to 10 minutes. But with 100 clients 5 minutes is OK.

I don’t know about pxe booting, but my fog clients uses https, and I think they should as a security reason.

By default FOG only installs http server not https. The clients kind of use ssl inside http protocol. The data is encrypted with the fog ssl certificate but the traffic flows across http. That’s why I suggested leaving this bit alone and just enable https on apache where you can then use your company ssl certificate without touching the fog client stuff.

[Sebastian Roth]

@Gabor It’s great to see you try to dig into this yourself and play with settings to see if it makes a difference. From my experience proxying and using separate ports makes these things even more complicated and prone to errors. So I am wondering if you want to take one step back, re-install FOG plain as default and then play with the “KeepAlive” setting.

Not slow but rather lagging its about 1-2 tenth of a second, which is enough to feel that is not responding well.

While I have not looked into this myself yet I’d imagine this is due to database queries in the background. The web UI is not optimized similar to a lot of websites that use caching techniques or anything. You’d need to re-invent the whole FOG web UI to make this go away I reckon! But maybe I am wrong.

I just did a quick test, switched on KeepAlive in a test VM and tried PXE booting. It does work for PXE boot and I can see it re-using TCP connection using netstat command. Though I am not sure what other consequences this change might have. We do internal curl calls within FOG, fog-client communication, replication between nodes and so on. This would need to be tested thoroughly before we’d add this as default.

To find out when this was added and why I started digging through github commit messages and the forums. I found the changes but can’t tell you why it was initially added set to “On” and then switched to “Off” just a day later. Though I found no discussion in the forums about keep alive around that time. No idea why this was added or changed. @Tom-Elliott might remember, though he was pretty active back then and literally pushed hundreds of changes and fixes in those months in 2015/16.

[Gabor]

@george1421 if the default is to not use https, I think I turned it on. I didn’t know that clients use encryption over http.
The server by the way is an Intel i3 540@3GHz (4core) with 8G RAM. (but I can change it later if it’s needed)

[Gabor]

@Sebastian-Roth I am glad to see that you took the time to investigate this setting. I am definitely a newbie here. FOG has so many thing I don’t understand yet, but I am trying. It’s a greet project which already saved me a lot of work, so I am trying to help make it even better, and I am really glad to see your positive attitude. Devs often feels attacked and decline every idea arrogantly.
I think I am going to follow your advice and install a new FOG server from the beginning.

From my experience proxying and using separate ports makes these things even more complicated and prone to errors.

Of course proxying just a workaround now, and I think I complicated it unnecessarily. I try to rethink it and do it again without proxy. I keep you updated.

[Gabor]

@Sebastian-Roth Well proxying was just overcomplicated. I could just copy the 443 virtualhost as 444, remove the KeepAlive and change the SSL cert.
Why I prefer this solution?

• after an update I just have to check wheter the virtualhost config changed, and copy the chages, and I don’t need to touch the FOG code.
• I can left the KeepAlive on it’s place, maybe it is important.
• have the little better response time

However if it is safe to remove KeepAlive, then running just a script after upgrade is not a big deal.

[george1421]

@Gabor said in better web performance?:

if the default is to not use https, I think I turned it on. I didn’t know that clients use encryption over http.

So if you run into a road block on your current path, what I’m recommending is that you think about resetting the FOG server back to default settings (your existing fog client installs will need to be address to fix the certificates) then do what you suggested in your post "I could just copy the 443 virtualhost as 444"but instead of creating a virtual host just setup https on your fog server. There is really 2 sides to this equation. First you have all of the fog inner workings of pxe booting and fog clients. Leave them on port 80. Second use your enterprise ssl certificates on the https port. (actually I’m going to be doing something similar in about a week to our fog server because more browsers are complaining about http sites.)

The fog clients use the certificate of the fog install to communicate over http. If you reinstall your fog server new certificates are created. The fog clients out in the wild will not be able to communicate with the fog server if this happens. On a fresh fog client install the fog client reaches out to the fog server and downloads the certificate, so in a way the fog client tattoos itself to the FOG server. The intent is to keep rogue fog servers from doing bad things to your target computers via the FOG client communications.

The server by the way is an Intel i3 540@3GHz (4core) with 8G RAM. (but I can change it later if it’s needed)

So in regards to performance: your CPU is sufficient. In my home lab I have fog (still 1.4.4) running on a raspberry PI. So the CPU performance is not really a factor. For disclosure I don’t have any fog clients hitting the Raspberry PI either. What disk subsystem does your FOG server have in it? Is it a single spinning disk, SSD or a raid array? As for the network a single 1GbE interface is sufficient for 100 clients. You can saturate a 1GbE link with 3 simultaneous unicast imaging, just be aware. Adding additional network adapter and setting up a LAG group will help here but only under heavy network traffic. I think I would focus on disk subsystem and the top processes in top to start to understand why the server appears slow.

[Gabor]

@george1421 I think I will give it a try. Really fog clients and pxe boot is secure even on http? I will try it in a test environment with a new vm server , then I just have to figure out how to reinstall the FOG clients to connect them to the new server. What do you think is it possible with a snippet? So the clients basically reinstall themselves?

If you reinstall your fog server new certificates are created

What if I inject my own key,crt,ca files at this point of the installation and enable https as in the first install?

The “server” by the way just a desktop computer I picked up from a storeroom of old stuff when I heard about FOG, and wanted to try it out. Running debian on a 1TB sata software RAID1, and as I monitored with iostat it is performing well.

[george1421]

@Gabor said in better web performance?:

I think I will give it a try. Really fog clients and pxe boot is secure even on http?

As I said before the fog clients communicate using the ssl certificate over http. So the communications is encrypted and protected from remote command injection. You have to think about what pxe booting is really doing. pxe booting involves moving bzImage and init.xz between the FOG Server and target computer. There is nothing in bzImage/init.xz (FOS Linux) that is not publicly available to download. There is no PII or really any data that you need to protect.

I will try it in a test environment with a new vm server , then I just have to figure out how to reinstall the FOG clients to connect them to the new server.

I don’t use the fog client in my environment, but if you were to look into the fog client directory on the target computer, I bet you would find the certificate it downloads from the FOG server. One might think you could swap that certificate with one from the new fog server and live happily.

What if I inject my own key,crt,ca files at this point of the installation and enable https as in the first install?

I haven’t looked at the installer scirpt, but I bet it calls openssl to create the certificates. If one would short circuit that code and slip his/her enterprise certificates in its place, would the fog installer know any difference?

The “server” by the way just a desktop computer I picked up from a storeroom of old stuff when I heard about FOG, and wanted to try it out. Running debian on a 1TB sata software RAID1, and as I monitored with iostat it is performing well.

I was kind of figuring your disk subsystem might be the slowness in your environment. If you were to rebuild your fog server OS again on a 128GB SSD and then before installing FOG mount that 1TB software raid onto /images then install FOG that would help with your performance. All of the images will be kept on the slower 1TB array with the OS on the SSD. That single disk spindle will hurt you if you try to deploy 2 images at the same time, but for a single deployment it will work just OK.

[Sebastian Roth]

@Gabor said in better web performance?:

Really fog clients and pxe boot is secure even on http?

As George already said, fog-client communication is encrypted and save but PXE boot over HTTP is not. So unless you use join multicast sessions or other PXE menu things that ask FOG credentials you don’t need to worry.

I will try it in a test environment with a new vm server , then I just have to figure out how to reinstall the FOG clients to connect them to the new server.

I don’t use the fog client in my environment, but if you were to look into the fog client directory on the target computer, I bet you would find the certificate it downloads from the FOG server. One might think you could swap that certificate with one from the new fog server and live happily.

Not that easy, sorry. The cert is being installed into the Windows certificate store. So you’d need to find a way to remove and install certs from that using some kind of automated way. Doing it manually for 100 clients is a nightmare. There have been a few people in the forums trying to automate this but I think we never got any detailed feedback on how they did it, see 1, 2, 3. I have too many other things on my list and never really looked into this myself.

What if I inject my own key,crt,ca files at this point of the installation and enable https as in the first install?

I haven’t looked at the installer scirpt, but I bet it calls openssl to create the certificates. If one would short circuit that code and slip his/her enterprise certificates in its place, would the fog installer know any difference?

While it might work for PXE boot this would surely break the fog-client communication. As described in the wiki the current fog-client code checks the common name of the certificate to be FOG Server CA. I am fairly sure not too many CAs will allow you to generate a sub CA using that CN (common name). I know this is not good practice but it’s not me who came up with this initially and I have thought about different ways to get away from this. Sure we could remove that restriction and allow for any certificate to be used by the fog-client. I am open to discuss this with all of you. Though on the other hand I am questioning the secure encrypted communication over plain HTTP altogether with HTTPS becoming state of the art more and more. But moving FOG and the fog-client to be HTTPS only and remove the internal encryption is a major project which I don’t have the time to right now.

[Gabor]

@Sebastian-Roth I went trough the topics you linked, and considering all the trouble just to reach the webUI trough 443 instead of 444 which I already have, I decided I am fine with 444

As described in the wiki the current fog-client code checks the common name of the certificate to be FOG Server CA […] Sure we could remove that restriction and allow for any certificate to be used by the fog-client.

Absolutely as the FOG client has a configured server address any cert which is considered secure (matching ip or domain, and signed with a known issuer) should be sufficient.

By the way I noticed that FOG client installs two CAs. Why it is needed?

[george1421]

@Sebastian-Roth said in better web performance?:

@Gabor said in better web performance?:
Not that easy, sorry. The cert is being installed into the Windows certificate store.

Just for completeness, the windows certutil.exe program can be used to automate adding and removing certificates from the local cert store. You can deploy the command as a snapin (with the new certificates) or with your favorite application deployment tool. I suppose you could deploy the certificates with an AD GPP too, but I haven’t looked into that.

[Sebastian Roth]

@Gabor said in better web performance?:

By the way I noticed that FOG client installs two CAs. Why it is needed?

The other one is for code signing.

[Tom Elliott]

The FOG Server CA was used as a way for us to verify the information being received from the server is indeed from the server it’s requesting it from. When the client gets installed, it downloads the CA from the server you configure. When it does this, the certificate is Pinned to the client. This way the client can trust the information from the server is indeed from that server.

When the client checks in, it uses the certificate to encrypt its information and sends the encryption key (reset every 30 minutes) it is looking for. This way the fog server can trust the initial authentication attempt from the client, and then you have 2 way trust, essentially.

So data being sent from the client or to the client is always encrypted.

Client -> Server (CA Cert initial authorization) -> Client uses this encryption to pass the FOG Server the key it randomly created to encrypt each message.
Server -> Client (using key sent above) 1 way trust
Client -> Server (using its encryption key) 1 way trust

Each cycle the client must reauthorize at least:
Client -> Server (using ca cert and its encryption key)
Server -> Client (using the encryption key)
Both sides now trust the other until the 30 minute timeout at least.

If at any point, the authorization process fails, no information is stored on the server and the client will keep trying to reauthorize. No trust and the server cannot do anything with the client data.

If, say, you authorize successfully, and then reboot the client into another operating system with a client as well, the authorization will then fail and remove the ability to access for the client even on the “working OS”. (Invalid Security Token) You can fix this by resetting encryption data, but it can also be a good indicator of attack.

This all works regardless of the http/https debate going on here. All the data passed between client and server is encrypted in some way. The key in use for regular connections is never passed in clear text. The data is never passed in clear text.

[george1421]

@george1421 said in better web performance?:

… "I could just copy the 443 virtualhost as 444"but instead of creating a virtual host just setup https on your fog server. There is really 2 sides to this equation. First you have all of the fog inner workings of pxe booting and fog clients. Leave them on port 80. Second use your enterprise ssl certificates on the https port. (actually I’m going to be doing something similar in about a week to our fog server because more browsers are complaining about http sites.)

I’ve started outlining a basic (hack) configuration here to add ssl support to the FOG sever without much pain or messing with PXE or FOG Clients. https://forums.fogproject.org/topic/14464/adding-https-support-to-fog-server-with-centos-7

So far its working.