NODES - how are images shared between the master and the node?

[andy.king]

Background <Optional reading>
I am thinking of deploying some nodes across the business, we have 4 sites all interconnected.
A few site have different images to other sites.
The main site has a fog master server at present.
The other offices are connected via leased lines so obvious choice would be a node in each site.

Issue:
The only issue i have is I can’t find any documentation regarding how the master and the nodes exchange image data.
Does it sync back and forth ?
If so how and when ?
Is this automatic ? or Manual ?

If its automatic
Is it done on a schedule ?
Can you control the schedule ? the bandwidth ?

You can appreciate Obviously i cant have a 40GiB image floating across leased lines from one site to the next during the day.

So any help would be gratefully received.

[Wayne Workman]

I made this a while back - has some great info in it but the video is not totally complete. I need to re-record it and make it as comprehensive as possible. The text in the article is very brief, but is pretty complete.
https://wiki.fogproject.org/wiki/index.php?title=Location_Plugin

[andy.king]

Thank you,
A Very useful explanation and Video.
I have our fog master set up pretty much as you describe in the video so I at least I know I’m on the right tracks.

So what I’ve got from you and please correct me if I’m wrong is
*with master to node image replication, I can control bandwidth of image replication and how often it checks for new images from the master?

• if I upload an image it is always uploaded directly to the master node ? if so can the bandwidth of this be controlled ?
  (we have a few sites that use different images so having these available across the whole business would be ideal)

**I didn’t think fog 1.3.0 was available yet ? I’m still running 1.2.0

[Wayne Workman]

@andy.king said in NODES - how are images shared between the master and the node?:

*with master to node image replication, I can control bandwidth of image replication and how often it checks for new images from the master?

Yes.

• if I upload an image it is always uploaded directly to the master node ? if so can the bandwidth of this be controlled ?

Yes, and no the bandwidth of uploads cannot be controlled. However, uploads are not bandwidth intensive at all. During an upload, there are small peaks on the bandwidth graph every 3 to 10 or 15 seconds, depending on processor and HDD performance, and the peak usually doesn’t hit the network cieling. This is due to the image being compressed on the target host. A chunk is read, compressed, and then shot to the network… rinse, repeat.

**I didn’t think fog 1.3.0 was available yet ? I’m still running 1.2.0

It’s not available yet, but it’s close. I’ve been writing articles from the perspective of 1.3.0 because I want documentation ready when it releases.

[andy.king]

HI Wayne the only thing I am worried about is the master is in head office and the images may be uploaded during the day over leased lines.
Hence my worry over bandwidth. As you understand the network celling is significantly lower on these lines then over the lan.

[Wayne Workman]

@andy.king While the rules I’ve already stated are absolutely true - FOG maintains a great deal of flexibility.

Have a storage group per location, and put one storage node in each group, and make it the master node.

this way, each site can upload to their local master node - as long as the image they are uploading to is assigned to the storage group for that location.

images can also be shared across groups, from master to master - however - this replication is a one-way highway. For example, say we have group A and group B. Each group has one node and it’s the master of it’s respective group. Say you make an image on B and upload. Then say you share that image with group A. The image will replicate to the A group’s master (and then from that master to any other nodes in group A).

But - if you update that image and it uploads to group A, replication will just delete that image and then re-replicate from group B because group B was the original.

These complications and features would only be of use to you if you even wanted to replicate images. If you don’t care about replication, don’t even worry about it. Also - if your WAN link is that slow, maybe you shouldn’t even use replication with fog, maybe you should just sneaker-net the images, or mail them.