UNSOLVED Image replication between nodes keeps restarting

  • I’ve got two servers running 1.4.0.
    The master storage node is located in Denmark and the other storage node is located in Norway.
    The replication works, but every time it has replicated an image, it just deletes the image on the server in Norway and starts over once again. It has been doing this for about a week, any ideas?

  • @jgallo I’d put everything on the same particularly as it addresses replication directly. The ftp error however is not because of the password you add when installing a new node. That password is what’s created for the node automatically. Sometimes, however, it’s possible it creates unusable characters.

  • @tom-elliott

    So today I got a chance to start adding even more storage nodes. The FOG Server is on the working branch as well as the storage node. Once I’m finished installing the storage node, I go to the console and make changes like enable graph and change the storage node name. I also place a value in the Replication Bandwidth because it won’t let me use 0. So when I tail the fogreplicator.log file, it states a

    *Type: 2, File: /var/www/fog/lib/fog/fogftp.class.php, Line: 463, Message: ftp_login(): Login incorrect., Host:, Username: fog

    Now when I ran the installer, I did type the storage node password from FOG correctly. I know this because the IP shows up under storage. This is awkward because my first two storage nodes worked but were on RC9. Should I keep the storage nodes at RC9 or put the storage nodes on working branch?

  • @jgallo Stay on the working branch until the next RC.

  • @tom-elliott

    Added another storage node and replication seems to operate normally. I still have the linux sleeptimer at 10800 seconds so I think we are good for now. My question is do I stay on the working branch or do I go back to RC9? I’m not too familiar when changes are made to RC’s and working branches.

  • @tom-elliott

    I uploaded an image and tried to manually start the replication. I did a tail on the fogreplicator.log file and it seemed like it wasn’t working. So I went to lunch and came back. Looks like it eventually worked. I see the image replicated to my only storage node right now. There are some other weird things occurring but I will create separate post as it has to do with my custom fog.man.reg being copied. Thank you for your help.

  • @tom-elliott

    I don’t mind at all. I will install working branch once this image that is currently replicating is complete. Once I install working branch, I will upload an image and check if replication i working.

  • Mind installing working branch and see if it is working for you now? I looked over the available code and apparently I made one mistake on a variable for the “domain” element being checked. This variable was not defined anywhere. So i have now setup what should be accurate and just need something to test off of.

    git checkout working
    git pull
    cd bin
    ./installfog.sh -y

  • @tom-elliott

    Thank you Tom. I went ahead and made the changes. Worked as soon as a reboot occurred.

  • @jgallo specifically lines 533 - 549 of /var/www/fog/lib/service/fogservice.class.php

  • @jgallo the only reason I can think rc9 is broken is the available checks in place. However this would lead me to think it’s a firewall issue. You c an fix this by commenting the new lines in the file. That is the only thing that would be causing the issue from what I can see.

  • @wayne-workman

    I increased it to 10800 which is 3 hours just for the hell of it. Seems to work fine on 1.5.0 RC7 unless replication takes longer than that LOL but issue I ran into was when I try to go and upload an image I get some weird DHCP issues. So I went ahead and went to RC9 and then replication does’t work. I came across a post regarding same issue and suggestion was to go back to RC7. So I’m in a bind and not sure what to do. I’m going to start from scratch and add each node and tail the fogreplicator.log file and see if the same errors occur. I’m getting a (storage node) server does not appear to be online. This only occurs in RC9 and as soon as I go back to RC7 replication works fine.

  • @jgallo If the image replicator is deleting the files if it’s not completed by the next iteration - this is a bug. I thought this was fixed completely somewhere between 1.2 and 1.3… I guess the issue has cropped up again. So yeah, just to see if this is the case, set your IMAGEREPSLEEPTIME to something huge and see if it’s fixed. If that fixes it, then that’s bug confirmed.

  • @wayne-workman

    I know this is a really old post and I apologize if I should have started a new topic. I’m experiencing the same symptoms after only accidentally discovering the replication keeps deleting my images from the nodes. By increasing the value of the FOG Linux Service Sleep Times - IMAGEREPSLEEPTIME, will this stop the image from being deleted?

    Basically would I need to set the value to a point beyond the time it takes FOG to finish replicating? If the value is less than the time it takes FOG to finish replicating, would this explain why the images keeps being deleted and replication restarting?

  • @Wayne-Workman Thank you so much! It worked, it sync the images and don’t delete them over and over again.

  • @AndersHoeg said in Image replication between nodes keeps restarting:

    Any clue where I can see/change how often it tries to replicate an image?

    FOG Web Interface -> FOG Configuration -> FOG Settings -> FOG Linux Service Sleep Times -> IMAGEREPSLEEPTIME

    FOG Replication logs are also viewable in the web interface:
    FOG Web Interface -> FOG Configuration -> Log Viewer -> Image Replicator

  • Hmmm maybe it is trying to replicate to often and that’s the issue.
    Any clue where I can see/change how often it tries to replicate an image?

  • Moderator

    This is an interesting issue.

    The fog replicator compares the md5sum of a certain number of bytes of each file before it decides if they are different. If the md5sum key is not the same then it makes the decision that the storage node is out of sync and starts replicating the image again.

    The fog replicator logs are in /opt/fog/logs does the master replication log or the replication log for the norway server give you an idea why it just keeps replicating?