Dynamic FOG Replicator transfer rates

[george1421]

Note: this code has not been completely tested. As it stands right now its more of a proof of concept than a functioning system

I have a desire to have different replication rates between storage nodes based on time of day. During the day time I want to cap transfer rates between the FOG Master node and Storage nodes to 1Mb/s and then during the night time transfer as fast as possible (based on the WAN link speed).

So hacking about for a while I came up with this solution. Its a bit complex but simple at the same time. The first step is to update the bitrate values in the FOG database and then restart the FOG replicator service.

For my project I only need 2 different bit rates day and night. You might expand this concept to any number of transfer rates per day. I might suggest against having a more than 2 transfer periods, but that decision is based on speculation.

For this hack I needed to create 2 bash scripts. One for daytime rates and one for nighttime rates.

1. Lets start by making a place to put our bash scripts. For this example I’m placing my files in a new directory in the /opt/fog directory.
   sudo mkdir /opt/fog/cron
2. Next we need to create our daytime bash script
   sudo touch /opt/fog/cron/replicator.daytime
sudo chmod 755 /opt/fog/cron/replicator.daytime
sudo vi /opt/fog/cron/replicator.daytime
3. In the replicator.daytime we’ll paste the following code

#!/bin/bash

. /opt/fog/.fogsettings

## Update the array with the storage node [name]=value value pairs ##
declare -A StorageNodes=( [ATLNode]=1000 [NYCNode]=1000 [LANode]=600 )
## don't change anything below this line ##

# Loop through all nodes in the associative array
for snode in "${!StorageNodes[@]}"
do 
  sql="mysql -u ${snmysqluser} --password='${snmysqlpass}' -e 'UPDATE nfsGroupMembers SET ngmBandwidthLimit=${StorageNodes[$snode]} WHERE ngmMemberName like \"$snode\" ' fog";
  eval $sql;
done

service FOGImageReplicator restart;

The notable bits in this is the line declare -A StorageNodes=( [ATLNode]=1000 NYCNode]=1000 [LANode]=600 ) This is a key value pair that will be used to update the FOG database with the new bit rates. You can add as many or as few storage nodes to this key value array as necessary. Just ensure you keep the proper formatting, capitalization and spacing. The key value must match exactly the name of the storage node from the FOG management GUI. The value must be an integer value. Again watch your spacing adding or removing storage nodes. There may be better ways to go about this, but “it works for me” YMMV.
4) Next we need to create our nighttime bash script
sudo touch /opt/fog/cron/replicator.nighttime
sudo chmod 755 /opt/fog/cron/replicator.nighttime
sudo vi /opt/fog/cron/replicator.nighttime
5) In the replicator.nighttime we’ll paste the following code

#!/bin/bash

. /opt/fog/.fogsettings

## Update the array with the storage node [name]=value value pairs ##
## in our declarations below we'll set the value to 0 (no restrictions on speed)
declare -A StorageNodes=( [ATLNode]=0 [NYCNode]=0 [LANode]=0 )
## don't change anything below this line ##

# Loop through all nodes in the associative array
for snode in "${!StorageNodes[@]}"
do 
  sql="mysql -u ${snmysqluser} --password='${snmysqlpass}' -e 'UPDATE nfsGroupMembers SET ngmBandwidthLimit=${StorageNodes[$snode]} WHERE ngmMemberName like \"$snode\" ' fog";
  eval $sql;
done

service FOGImageReplicator restart;

6. We should probably test our bash scripts to ensure we didn’t have a type-o in the script.
7. Key in the following from the command line:
   sudo /opt/fog/cron/replicator.nighttime
sudo /opt/fog/cron/replicator.daytime
8. The last bit we need to do is create an entry for the cron service to run the appropriate bash script at the right time of the day.
9. We need to edit the crontab files using this command:
   sudo crontab -e
10. Add this command for the night time schedule (I want to start at 5:30pm)

30 17 * * * /opt/fog/cron/replicator.nighttime

9. Add this command for the day time schedule (I want to start at 5:00am)

0 5 * * * /opt/fog/cron/replicator.daytime

9. Exit out of the crontab editor
10. Restart cron with latest data:
    sudo service crond restart

To help you understand the cryptic contab format here is the field position structure.

0 5 * * * /opt/fog/cron/replicator.daytime

MIN HOUR DOM MON DOW CMD

Format definitions and allowed value:
MIN     Minute field    0 to 59
HOUR    Hour field      0 to 23
DOM     Day of Month    1-31
MON     Month field     1-12
DOW     Day Of Week     0-6
CMD     Command to be executed.

Note: the star ( * ) is a position holder that matches 'any' 

This concludes the setup required to change the replication rates based on time of day. Is this an ideal solution, no. Will it work, yes. Are there some caveats in the design, yes. So in short it worked for me.

[Wayne Workman]

I like it but there are issues with how the image replicator works with this approach.

Any image that isn’t completed when the replicator gets restarted, the image transfer too gets restarted. Tom made this change due to a bug that was discovered (re-discovered by me) in how lftp mirrors changes in files. It didn’t do it right - hashing the source and destination image files and comparing the hashes showed they were not the same. The solution was - whenever it was determined that replication needed to happen, delete the destination file first.

If this issue with lftp could be checked on, and if that’s solved and the FOG replicator code can be tweaked, this stuff you’ve wrote here would be that much better.

But - for the purposes of rate limiting, what you posted would do it.

[george1421]

@Wayne-Workman said in Dynamic FOG Replicator transfer rates:

Any image that isn’t completed when the replicator gets restarted, the image transfer too gets restarted.

I is why I think that rsync would have been a better tool for image replication than ftp. Then it could possibly limit the number of NAS devices supported, because not all NAS devices have rsync as an option. But that is a topic of another thread.

If you look at the above concept from a purest standpoint. At 5am I want daytime transfer rate rules applied, I’ll have to live with the results of having to send one of the partitions (hopefully just a partition) that was in transit over again. I guess that is an OK trade off. I absolutely want to ensure that full WAN bandwidth is available as the workforce starts at 6am.

[Wayne Workman]

@george1421 as long as people understand that, then yeah this is great.