FOG 1.5.5 SLOW OPEN STORAGES NODES

[george1421]

Did you upgrade all of your storage nodes to FOG 1.5.5 first then upgrade your master node?

[wanderson]

not the nodes are in version 1.40

[Tom Elliott]

@wanderson Then it would be highly recommended to upgrade all nodes to 1.5.5

Yes it will take some time, and it may not even (entirely) fix the “slow” issue you’re seeing, but it will certainly help us ascertain more information on what we can find out.

Sometimes we update and refactor the code between versions. While attempts to maintain Backward compatibility is made, sometimes the simplest solution is to just upgrade.

[wanderson]

Plus it only gets slow with the 35, when I leave only 5 gets fast for example, I do not believe upgrading the nodes will make it work, the funny one that in the 1.40 fog works fast with the 35 nodes

[george1421]

@wanderson said in FOG 1.5.5 SLOW OPEN STORAGES NODES:

not the nodes are in version 1.40

This is going to be a problem if they are all not on the same release, especially with the replication code changes made in 1.5.5. In your current configuration the master node will continue to replicate all images to all storage nodes even if the images haven’t changed. This WILL be a bandwidth issue for your environment.

Edit: ref-> https://news.fogproject.org/fog-1-5-5-officially-released/

[wanderson]

@george1421 esse é um ambiente de homologação, não há imagens nem snapins no servidor.

[george1421]

@wanderson said in FOG 1.5.5 SLOW OPEN STORAGES NODES:

esse é um ambiente de homologação, não há imagens nem snapins no servidor.

OK but its still recommended to be on the same release for all nodes in the storage group… but wait. Why do you have storage nodes if you don’t have images to replicate?

[wanderson]

@george1421

only the 1.5.5 server is ratified, the main server is in production in version 1.4.0 and uses the 35 storage nodes in version 1.4.0 and is working perfectly.

I just imported the FOG version 1.4.0 server database into the FOG server 1.5.5 environment to see how it would behave with the database import and whether it would maintain communication with the storage nodes .

[george1421]

@wanderson Ok I understand a bit better how you have things setup. As long as the 1.5.5 sever doesn’t have any images to push then you can run mixed.

I can tell you fog 1.5.x is very much different than fog 1.4.x as windows 10 is very much different than windows 7. While both 1.4.x and 1.5.x is FOG, they are different in how they manage things internally. There is a bigger demand on system resources in FOG 1.5.x over 1.4.x. Look at the top command to see if you need to add more vCPU or memory to your 1.5.5 fog server.

[wanderson]

@george1421

[Sebastian Roth]

@wanderson This issue is not specific to 1.5.5 but more 1.5.x and has to do with master node trying to contact the storage nodes when you click the listing. I am working on this part and hopefully have a solution to it soon.

[wanderson]

@Sebastian-Roth

True, I had tried to migrate to version 1.5.0 and I got this problem I gave up, then tried again in version 1.5.4 and the problem still persisted, now I saw that version 1.5.5 was out and the problem is still happening, but I am happy that you understand that there is this problem for those who use many nodes, I hope that I can solve and I know that you will get … thanks for the help.

[Sebastian Roth]

@wanderson The issue as I understand it so far is not that much about many storage nodes but about the nodes taking a long time (timeout) when trying to talk to each other. Please do me a favor and open the FOG configuration page http://ip.of.fog.srv/fog/management/index.php?node=about and post a screenshot of that here. What I hope to see is if all the storage nodes show proper version info. Mind you this will take a long time as well, and could possibly timeout in the end (Gateway error).

[george1421]

@Sebastian-Roth said in FOG 1.5.5 SLOW OPEN STORAGES NODES:

…and could possibly timeout in the end (Gateway error).

If we get this error there is a apache config file change we can make to extend the timeout to what ever length we need.

ref: https://forums.fogproject.org/topic/12573/intermittent-error-504/10

[Sebastian Roth]

@george1421 Nice idea but I’d better make our code to not wait for such a long time!

@wanderson I was now able to replicate the issue and still hope that we can fix this in the code of the web UI. As a quick fix you need to check all your storage node configs (/var/www/html/fog/lib/fog/config.class.php). I am fairly sure at least one of your storage nodes has the wrong database host (or user/pass) set which is causing the very long delay. They should all point to your main FOG server.

[wanderson]

@Sebastian-Roth But the 35 nodes using version 1.4.0 do not give this problem, only in version 1.5.X

[wanderson]

@Sebastian-Roth

Watch ----> https://youtu.be/EmL0gVIMbEI

[wanderson]

@george1421

Watch ----> https://youtu.be/EmL0gVIMbEI

[Sebastian Roth]

@wanderson Thanks for the video again. You are right, 1.4.x seems fast on the mass storage node setup. Possibly a different mysql driver was used which is handling timeouts differently and did a more appropriate asynchronous check on the storage nodes. I am still digging through the code. Fairly sure I can come up with a fix in the next couple of days.

[Sebastian Roth]

@wanderson Would you please test the latest dev-branch version? You should see an improvement if only upgrading the master node already. But to get to experience the full scope of changes I made you’d need to upgrade all storage nodes as well!