FOG 1.5.5 SLOW OPEN STORAGES NODES

[Sebastian Roth]

@wanderson This issue is not specific to 1.5.5 but more 1.5.x and has to do with master node trying to contact the storage nodes when you click the listing. I am working on this part and hopefully have a solution to it soon.

[wanderson]

@Sebastian-Roth

True, I had tried to migrate to version 1.5.0 and I got this problem I gave up, then tried again in version 1.5.4 and the problem still persisted, now I saw that version 1.5.5 was out and the problem is still happening, but I am happy that you understand that there is this problem for those who use many nodes, I hope that I can solve and I know that you will get … thanks for the help.

[Sebastian Roth]

@wanderson The issue as I understand it so far is not that much about many storage nodes but about the nodes taking a long time (timeout) when trying to talk to each other. Please do me a favor and open the FOG configuration page http://ip.of.fog.srv/fog/management/index.php?node=about and post a screenshot of that here. What I hope to see is if all the storage nodes show proper version info. Mind you this will take a long time as well, and could possibly timeout in the end (Gateway error).

[george1421]

@Sebastian-Roth said in FOG 1.5.5 SLOW OPEN STORAGES NODES:

…and could possibly timeout in the end (Gateway error).

If we get this error there is a apache config file change we can make to extend the timeout to what ever length we need.

ref: https://forums.fogproject.org/topic/12573/intermittent-error-504/10

[Sebastian Roth]

@george1421 Nice idea but I’d better make our code to not wait for such a long time!

@wanderson I was now able to replicate the issue and still hope that we can fix this in the code of the web UI. As a quick fix you need to check all your storage node configs (/var/www/html/fog/lib/fog/config.class.php). I am fairly sure at least one of your storage nodes has the wrong database host (or user/pass) set which is causing the very long delay. They should all point to your main FOG server.

[wanderson]

@Sebastian-Roth But the 35 nodes using version 1.4.0 do not give this problem, only in version 1.5.X

[wanderson]

@Sebastian-Roth

Watch ----> https://youtu.be/EmL0gVIMbEI

[wanderson]

@george1421

Watch ----> https://youtu.be/EmL0gVIMbEI

[Sebastian Roth]

@wanderson Thanks for the video again. You are right, 1.4.x seems fast on the mass storage node setup. Possibly a different mysql driver was used which is handling timeouts differently and did a more appropriate asynchronous check on the storage nodes. I am still digging through the code. Fairly sure I can come up with a fix in the next couple of days.

[Sebastian Roth]

@wanderson Would you please test the latest dev-branch version? You should see an improvement if only upgrading the master node already. But to get to experience the full scope of changes I made you’d need to upgrade all storage nodes as well!

[wanderson]

@Sebastian-Roth

Okay, I’ll test it, and then put the result here.

[wanderson]

@Sebastian-Roth said in FOG 1.5.5 SLOW OPEN STORAGES NODES:

@wanderson Would you please test the latest dev-branch version? You should see an improvement if only upgrading the master node already. But to get to experience the full scope of changes I made you’d need to upgrade all storage nodes as well!

I tried to install via git, but I could not, where I get the .zip of the branch version??

[Sebastian Roth]

@wanderson said in FOG 1.5.5 SLOW OPEN STORAGES NODES:

I tried to install via git, but I could not, where I get the .zip of the branch version??

Errors you got when trying git?

git clone https://github.com/FOGProject/fogproject
cd fogproject/
git checkout dev-branch
cd bin/
./installfog.sh

But sure you can download the ZIP archive from github as well: https://github.com/FOGProject/fogproject/archive/dev-branch.zip

[wanderson]

@Sebastian-Roth said in FOG 1.5.5 SLOW OPEN STORAGES NODES:

@wanderson Would you please test the latest dev-branch version? You should see an improvement if only upgrading the master node already. But to get to experience the full scope of changes I made you’d need to upgrade all storage nodes as well!

Watch video new test: -------------> https://youtu.be/r4T6oDEhQhQ

[Tom Elliott]

@wanderson That’s much faster than in the past. This is because we’re also testing if the machines are actually available and turned on. We do want to know if a machine is online and accessible right?

[wanderson]

@Tom-Elliott said in FOG 1.5.5 SLOW OPEN STORAGES NODES:

@wanderson That’s much faster than in the past. This is because we’re also testing if the machines are actually available and turned on. We do want to know if a machine is online and accessible right?

would it be this ------> https://youtu.be/I5Mr8dJIMpo

[Sebastian Roth]

@wanderson Compared to your first video it improved immensely as you couldn’t even load the pages with 1.5.4 and now we improved to only 5 second delay when loading the storage node list or configuration page - for checking 35 storage nodes to be online!!

As I already said, if you update all your storage nodes to dev-branch it would be even faster!

@wanderson said:

would it be this ------> https://youtu.be/I5Mr8dJIMpo

Yes, one of your storage nodes seems a bit slower than the others. Because we test the online status on nodes in FOG 1.5.x it takes a little bit of time to load.

I would suggest you update the slowest storage nodes to dev-branch as well (best if you upgrade all of them!) and see if that helps.

Sure we could improve page loading speed even more if we do the online status check using AJAX.

[wanderson]

@Sebastian-Roth

afraid to update everyone and stay slow … it takes a lot of work to update everyone

[Tom Elliott]

@wanderson That’s why we’re asking you to update in the first place.

WHen upgrading servers, it’s best to keep all servers/nodes at the same level (though certainly not a 100% requirement sometimes.)

We’re asking you to update because you reported a problem and we’ve been (mostly @Sebastian-Roth) working to improve and fix the slow UI. We won’t know if all is doing well until we know all machines are at the same level.

This is like a catch 22 now.

Either upgrade all of your servers and stay slow (where we can work towards improving and correcting as needed),

OR

Stay in the state you are where you already are slow.

We’re simply trying to help.

[wanderson]

@Tom-Elliott

Okay, I’ll update and post the result here.