voison

voison

@wayne-workman Good call on the MySQL issue. The database on the Master Node would not allow a login for ‘fogstorage’ from anything except localhost. I fixed the database issues for both storage nodes, testing all three machines abilities to access the MySQL server with success. After this I looked over the .fogsettings file on both storage nodes and configured them accordingly, then ran the installer again. All was successful and nothing failed. I attempted another batch of images, and nothing changed. I then went to see what was happening with the replicator, just out of curiosity, and it seemed that the Master Node was trying to replicate something, but neither of the storage nodes seemed to recognize anything. While writing this, they have both since come back with “disabled replication” messages:

[01-30-18 11:20:31 pm] * Starting ImageReplicator Service
[01-30-18 11:20:31 pm] * Checking for new items every 600 seconds
[01-30-18 11:20:31 pm] * Starting service loop
[01-30-18 11:20:31 pm] * | This is not the master node
[01-30-18 11:30:31 pm] * | This is not the master node
[01-30-18 11:55:59 pm] * * Image replication is globally disabled

For a very short brief moment both of the servers were showing up on both the home page in the bandwidth graph as well as on the web gui > Fog Configuration > Kernel Versions; however they have now disappeared from both. I’m not sure if that helps with the problem but I figured I would report it.

Do I need to re-run the installer on the master node as well?

voison

Thank you wayne, that is good to know about the replication scripts.

Sorry I disappeared for a bit, I had to go out of town for a little while. Unfortunately now that I have returned, there has been no change in the status of this issue. I have confirmed that the replicator ran perfectly fine, and the /images directory on all three servers are identical. What would you all suggest for me to try next? At this point I am starting to consider exporting all of the images to a hard drive and starting over from scratch.

The server is still running well, even while supporting as many unicast streams as it does. However I would like to try to solve this issue. I have gone back over this evening and checked on all of the passwords, thoroughly eliminating that as a possible culprit.

voison

@wayne-workman Good call on the MySQL issue. The database on the Master Node would not allow a login for ‘fogstorage’ from anything except localhost. I fixed the database issues for both storage nodes, testing all three machines abilities to access the MySQL server with success. After this I looked over the .fogsettings file on both storage nodes and configured them accordingly, then ran the installer again. All was successful and nothing failed. I attempted another batch of images, and nothing changed. I then went to see what was happening with the replicator, just out of curiosity, and it seemed that the Master Node was trying to replicate something, but neither of the storage nodes seemed to recognize anything. While writing this, they have both since come back with “disabled replication” messages:

[01-30-18 11:20:31 pm] * Starting ImageReplicator Service
[01-30-18 11:20:31 pm] * Checking for new items every 600 seconds
[01-30-18 11:20:31 pm] * Starting service loop
[01-30-18 11:20:31 pm] * | This is not the master node
[01-30-18 11:30:31 pm] * | This is not the master node
[01-30-18 11:55:59 pm] * * Image replication is globally disabled

For a very short brief moment both of the servers were showing up on both the home page in the bandwidth graph as well as on the web gui > Fog Configuration > Kernel Versions; however they have now disappeared from both. I’m not sure if that helps with the problem but I figured I would report it.

Do I need to re-run the installer on the master node as well?

voison

Well, an interesting find. No the passwords were not correct. I checked both of the nodes (/opt/fog/.fogsettings) and their passwords did differ from the the passwords on the web gui. I tried updating the web gui however it has not fixed the problem, all unicast images are still being pulled from the Master Node. I do remember watching the replicator log for a while when I first created the nodes. So, that being said, I made a little comparison of all three /images/ directors on each server. The master node does have more that the other two, but these are just old images that we deleted out of FOG but not off of the disk. Everything in the directories on the storage servers are the same as what is found in the web gui. I would say that the replicator did run at some point, even with the differing passwords. Is there any kind of manual system refresh I need to do after updating the passwords to associate the storage nodes to the master node?

voison

Thank you both for all the information. I have considered link aggregation but have not worked on implementing such a solution yet. I have not tried a multicast session yet but have just started reading more on it. In terms of imaging speeds, before I made the nodes I was usually getting about 2.9 to 5.3 GB/min according to Partclone, depending on the age of the hard drives in the machines that were being imaged. This was still while imaging 5-7 machines at least. When only imaging 2-4, the speeds were often higher. Each of the servers have 1TB SSDs, so they are pretty quick when it comes to their own disk speeds. I’ll look at the link saturations tomorrow when I get in the office and follow up here, thank you for the helpful links! Here is the Storage Node webpage from the GUI:

Here are the screen shots of the specific details per node. I played around with replication speeds but that did not make a difference.
-Master Node-

-Storage Node 1-

-Storage Node 2-

voison

Hi everyone, first off as this is my first post on this forum I want to thank the wonderful FOG team for the outstanding product they have made and all of the great documentation on it.

On to the good stuff:
I’m facing an issue with the use of FOG Storage Nodes in my setup. I have three servers, all of identical hardware and specifications, running Centos 7 and FOG v1.4.4. I utilize FOG unicast imaging for deploying batches of computers to ship to clients with pre-made images; however the quantity I was imaging at a time were taking a tole on imaging speeds and throughput, thus slowing down imaging times substantially. Using the documentation on the Wiki and the youtube video suggest there as well, I (believe I) successfully created two storage nodes on the same network to help the Master Node load balance. The replicator script did run and the images appear to have all been copied over to the correct directories on the new servers.

All of this being said, every machine is still taking its image from the Master Node, and thus the imaging speed problem persists. All images image from 10.0.0.1, and never from the storage nodes (IPs Below). I even set the max clients on each storage node under the storage management tab to 5 instead of the intended 10 to see if that would force the use of the other machines, but it did not change anything. imaging between 15 and 30 machines at once, they all still pulled from the master node (10.0.0.1).

I would greatly appreciate any help or advice the community could give me! Thank you all in advance! Below is a list of information on each of the servers, as well as a network map showing the general setup of the servers, imaging stations, and switch configuration. If there are any other questions I can answer or config files that would assist in fixing the issue, please let me know and I will answer/upload them as soon as possible. Thank you!

voison

@voison

Best posts made by voison

Latest posts made by voison