master node sending data to storage node.

Sebastian Roth

@mwolfe60 Hope you have both nodes still on 1.5.4! Please make sure you read and understand the release notes for 1.5.5 - https://news.fogproject.org/fog-1-5-5-officially-released/

IMPORTANT NOTICE:
Nodes being on different versions (1.5.4 vs. 1.5.5) will replicate images over and over again as some of the hashing code needed to be changed. Therefore we advise you to update all nodes in one go! Please make sure you stop replication on the master first systemctl stop FOGImageReplicator; systemctl stop FOGSnapinReplicator, then update the storage node(s) and then update master node as a last step.

Now to your points:

I’m not sure where the image replicator logs are located to review.

Take a look at /opt/fog/log/fogreplicator.log on your FOG server.

I should also mention that today we attempted to update the server 1.5.5 and the update failed.

We need more details to be able to help. At what stage exactly did it fail? Please post the exact output you get. Either as text or take a picture and post here.

… in the past with auto updates filling up the root file system.

Have you made sure there is sufficient space on both nodes?? Run df -h and post output here.

mwolfe60

@Sebastian-Roth results from df -h

df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 799M 8.8M 790M 2% /run
/dev/mapper/FOGMTNA–vg-root 786G 570G 181G 76% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda1 472M 160M 289M 36% /boot
tmpfs 799M 0 799M 0% /run/user/1000

snippet of the error updating

Skipping package: php7.1-mysql…(Already Installed)
Skipping package: php-gettext…(Already Installed)
Installing package: sysv-rc-conf…Failed! (Will try later)
Skipping package: tar…(Already Installed)
Skipping package: tftpd-hpa…(Already Installed)
Skipping package: tftp-hpa…(Already Installed)
Skipping package: unzip…(Already Installed)
Skipping package: vsftpd…(Already Installed)
Skipping package: wget…(Already Installed)
Skipping package: xinetd…(Already Installed)
Skipping package: zlib1g…(Already Installed)
Updating packages as needed…OK
Installing now everything is updated…Failed!

I think I want to move this fog server onto a new server. This server was not setup by me and has some things that are not idea. The images directory is not its own mount point.

Sebastian Roth

@mwolfe60 Ok, disk space does not seem to be your issue here. It’s something about the package manager not being able to update packages.

Please run apt-get update ; apt-get upgrade and I reckon you’ll run into an issue there. Please post the full output you get here. We’ll probably be able to help.

I think I want to move this fog server onto a new server. This server was not setup by me and has some things that are not idea. The images directory is not its own mount point.

Well that’s up to you. Quiet often it’s not too bad of an idea to start from scratch but make sure you have a backup copy of all your images read our wiki article: https://wiki.fogproject.org/wiki/index.php?title=Migrate_FOG

Sebastian Roth

@mwolfe60 By the way, what about your initial request? Did you take a look at the replication log? Please post here so we can help on this issue.

mwolfe60

@Sebastian-Roth putty.log

Apt-get update ran fine.

I attached the output from apt-get upgrade

When I looked at the Replicator logs it was trying to copy images over. But this was after we updated the storage node to 1.5.5 so we don’t know what it was sending prior to that.

I have read the wiki on migrating when i attempted do the migration. The web interface would not load after running through the SSL migration part. Prior to that it would load just fine.

Thanks for the help. I’m new to fog and rusty on linux.

Sebastian Roth

@mwolfe60 Seems like something corrupted your mysql package. But there is a solution: https://askubuntu.com/questions/643251/having-trouble-installing-and-removing-mysql-in-ubuntu

Simply create the file /etc/mysql/my.cnf.fallback with the following content: https://www.apt-browse.org/browse/ubuntu/xenial-security/main/all/mysql-common/5.7.25-0ubuntu0.16.04.2/file/etc/mysql/my.cnf.fallback

After that again run apt-get upgrade

mwolfe60

@Sebastian-Roth
I successfully ran apt-get upgrade after creating the my.cnf.fallback file and updated fog to 1.5.5. Unfortunately the data started replicating to the storage node. It looked like its was replicating all the images across our site to site link to the storage node. I let this run for the evening and checked the for replication log and found that it was all synced.

This shows that all files are synced but it is still transmitting about 100Mb/s of data.

Sebastian Roth

@mwolfe60 Are you sure ALL nodes are updated to 1.5.5??

Do you use snapins? We have a bug in 1.5.5 where snapin replication is not working properly and will re-transfer over and over. This is fixed and will be in the next release coming very soon!

mwolfe60

@Sebastian-Roth The only snapin ( I’m assuming plugins are what they are called in the web console) is location which I removed. It still looks like its replicating images to the storage node.

Sebastian Roth

@mwolfe60 Snapins is not Plugins!

mwolfe60

Then we do not use snapins. I let the replication run all day Saturday and it never stopped. I stopped before it fill the storage node completely.

Tom Elliott

@mwolfe60 Did you update all the storage nodes in your environment to be at 1.5.5? If not, It’d be highly recommended that you do. We’re aware of a bug in replication for Snapins, but this does not sound like to be your case. There is a release note explaining the reasoning for needing all nodes to be at the same version.

mwolfe60

Yes all nodes are running 1.5.5

mwolfe60

are there any logs that I could provide that would help figure out what is going on here. I’m going to need the server as I have about 70 computers to image in the coming months. I would like to get this one fixed or stand up a new server whichever is going to be the easiest solution.

Sebastian Roth

@mwolfe60 said:

are there any logs that I could provide that would help figure out what is going on here.

Yes, simply post the full fogreplication.log! If you fear to share this with the whole world you can upload to a private share and send me a link or password protect it and send me the PW. Whatever you like.

mwolfe60

@Sebastian-Roth I sent the link in a chat.

Sebastian Roth

@mwolfe60 Thanks for the log. Unfortunately there is only one loop of the replicator in the log and therefore we don’t really see what it’s doing further on. Need more of the log!

What I see in the log so far is that only one single file is not matching (file size different) and it tries to re-transfer that. My guess is that this will take a fair amount of time to transfer as it is roughly 45 GB in size. What speed is the link between your two nodes??

As well, can you post the contents of /opt/fog/log/fogreplicator.Automation_TB_CF54.transfer.Greenville.log or send me a link again?

mwolfe60

we have 100Mb/s dedicated link between the 2 sites. I’ll send you the links tomorrow when I get back in the office.

mwolfe60

@Sebastian-Roth I was looking at that log and there are some images that do not need to be replicated to the other node. They are not needed at that site as the systems are not located there. can we not replicate those images and delete them from the storage node or is all or nothing ?

Sebastian Roth

@mwolfe60 said in master node sending data to storage node.:

can we not replicate those images and delete them from the storage node or is all or nothing ?

Open the image settings in the web UI and un-check the option Replicate…

I had a look a the new logs you send me and I still can’t see there being an issue. Possibly you just restart the fogreplicaiton service too early before it finishes transferring the huge file? From my point of view the replication algorithm seems to work. Here is an example:

...
[04-30-19 11:48:01 am]  * Found Image to transfer to 1 node
[04-30-19 11:48:01 am]  | Image Name: Automation_TB_CF54
...
[04-30-19 11:48:10 am]   # Automation_TB_CF54: File size mismatch - d1p2.img: 48334490621 != 1202149362
[04-30-19 11:48:10 am]   # Automation_TB_CF54: Deleting remote file d1p2.img
[04-30-19 11:48:10 am]  * Starting Sync Actions
...
[04-30-19 11:48:10 am]  | Started sync for Image Automation_TB_CF54 - Resource id #1105
...
[04-30-19 12:06:13 pm]  | Image Name: Automation_TB_CF54
[04-30-19 12:06:13 pm] | Replication already running with PID: 2651
...

So it finds that d1p2.img is not the same size on the storage node, deletes it and starts replicating it again. This is at 11:48. Then it checks all the other images and on the next loop (at 12:06) when it comes back to that image it tells us there the sync of that file is still going on and goes ahead.

Doing some quick maths here. Transferring a 45 GB file over a 100 Mbit/s link will take at least 1 hour 07 minutes and 35 seconds. And that calculation is most certainly still a fair way off. I’d expect it to take one and a half to three hours depending on the other traffic on that link. So you just need to be patient!

If the link collapses somewhere on between it needs to start over again. I know this sounds stupid but implementing a continuing sync that can pick up on a broken file is way more advanced. Not something we can implement on short notice.

master node sending data to storage node.

147

12.1k

17.3k

155.3k