master node sending data to storage node.
-
Our fog master node started sending large amounts of data to one of the storage node. This started 2 days ago. No changes have been made to either of the nodes. What could of lead to this behavior? I can get more info but right now I’m not at work.
Thanks
-
What version of fog are you using?
Do you know what kind of data is it sending?
Have you reviewed the FOG image replicator logs to see if the replicator is stuck on something?
Is your storage node a full fog server or is it a nas device? -
We are currently using Fog 1.5.4
no we do not know what data
I’m not sure where the image replicator logs are located to review.
the storage node is a full fog server and not a NAS device.I should also mention that today we attempted to update the server 1.5.5 and the update failed.
It is running on an Ubuntu Linux server that had problems in the past with auto updates filling up the root file system. -
@mwolfe60 Hope you have both nodes still on 1.5.4! Please make sure you read and understand the release notes for 1.5.5 - https://news.fogproject.org/fog-1-5-5-officially-released/
IMPORTANT NOTICE:
Nodes being on different versions (1.5.4 vs. 1.5.5) will replicate images over and over again as some of the hashing code needed to be changed. Therefore we advise you to update all nodes in one go! Please make sure you stop replication on the master firstsystemctl stop FOGImageReplicator; systemctl stop FOGSnapinReplicator, then update the storage node(s) and then update master node as a last step.Now to your points:
I’m not sure where the image replicator logs are located to review.
Take a look at
/opt/fog/log/fogreplicator.logon your FOG server.I should also mention that today we attempted to update the server 1.5.5 and the update failed.
We need more details to be able to help. At what stage exactly did it fail? Please post the exact output you get. Either as text or take a picture and post here.
… in the past with auto updates filling up the root file system.
Have you made sure there is sufficient space on both nodes?? Run
df -hand post output here. -
@Sebastian-Roth results from df -h
df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 799M 8.8M 790M 2% /run
/dev/mapper/FOGMTNA–vg-root 786G 570G 181G 76% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda1 472M 160M 289M 36% /boot
tmpfs 799M 0 799M 0% /run/user/1000snippet of the error updating
- Skipping package: php7.1-mysql…(Already Installed)
- Skipping package: php-gettext…(Already Installed)
- Installing package: sysv-rc-conf…Failed! (Will try later)
- Skipping package: tar…(Already Installed)
- Skipping package: tftpd-hpa…(Already Installed)
- Skipping package: tftp-hpa…(Already Installed)
- Skipping package: unzip…(Already Installed)
- Skipping package: vsftpd…(Already Installed)
- Skipping package: wget…(Already Installed)
- Skipping package: xinetd…(Already Installed)
- Skipping package: zlib1g…(Already Installed)
- Updating packages as needed…OK
- Installing now everything is updated…Failed!
I think I want to move this fog server onto a new server. This server was not setup by me and has some things that are not idea. The images directory is not its own mount point.
-
@mwolfe60 Ok, disk space does not seem to be your issue here. It’s something about the package manager not being able to update packages.
Please run
apt-get update ; apt-get upgradeand I reckon you’ll run into an issue there. Please post the full output you get here. We’ll probably be able to help.I think I want to move this fog server onto a new server. This server was not setup by me and has some things that are not idea. The images directory is not its own mount point.
Well that’s up to you. Quiet often it’s not too bad of an idea to start from scratch but make sure you have a backup copy of all your images read our wiki article: https://wiki.fogproject.org/wiki/index.php?title=Migrate_FOG
-
@mwolfe60 By the way, what about your initial request? Did you take a look at the replication log? Please post here so we can help on this issue.
-
Apt-get update ran fine.
I attached the output from apt-get upgrade
When I looked at the Replicator logs it was trying to copy images over. But this was after we updated the storage node to 1.5.5 so we don’t know what it was sending prior to that.
I have read the wiki on migrating when i attempted do the migration. The web interface would not load after running through the SSL migration part. Prior to that it would load just fine.
Thanks for the help. I’m new to fog and rusty on linux.
-
@mwolfe60 Seems like something corrupted your mysql package. But there is a solution: https://askubuntu.com/questions/643251/having-trouble-installing-and-removing-mysql-in-ubuntu
Simply create the file
/etc/mysql/my.cnf.fallbackwith the following content: https://www.apt-browse.org/browse/ubuntu/xenial-security/main/all/mysql-common/5.7.25-0ubuntu0.16.04.2/file/etc/mysql/my.cnf.fallbackAfter that again run
apt-get upgrade -
@Sebastian-Roth
I successfully ran apt-get upgrade after creating the my.cnf.fallback file and updated fog to 1.5.5. Unfortunately the data started replicating to the storage node. It looked like its was replicating all the images across our site to site link to the storage node. I let this run for the evening and checked the for replication log and found that it was all synced.This shows that all files are synced but it is still transmitting about 100Mb/s of data.
-
@mwolfe60 Are you sure ALL nodes are updated to 1.5.5??
Do you use snapins? We have a bug in 1.5.5 where snapin replication is not working properly and will re-transfer over and over. This is fixed and will be in the next release coming very soon!
-
@Sebastian-Roth The only snapin ( I’m assuming plugins are what they are called in the web console) is location which I removed. It still looks like its replicating images to the storage node.
-
@mwolfe60 Snapins is not Plugins!
-
Then we do not use snapins. I let the replication run all day Saturday and it never stopped. I stopped before it fill the storage node completely.
-
@mwolfe60 Did you update all the storage nodes in your environment to be at 1.5.5? If not, It’d be highly recommended that you do. We’re aware of a bug in replication for Snapins, but this does not sound like to be your case. There is a release note explaining the reasoning for needing all nodes to be at the same version.
-
Yes all nodes are running 1.5.5
-
are there any logs that I could provide that would help figure out what is going on here. I’m going to need the server as I have about 70 computers to image in the coming months. I would like to get this one fixed or stand up a new server whichever is going to be the easiest solution.
-
@mwolfe60 said:
are there any logs that I could provide that would help figure out what is going on here.
Yes, simply post the full fogreplication.log! If you fear to share this with the whole world you can upload to a private share and send me a link or password protect it and send me the PW. Whatever you like.
-
@Sebastian-Roth I sent the link in a chat.
-
@mwolfe60 Thanks for the log. Unfortunately there is only one loop of the replicator in the log and therefore we don’t really see what it’s doing further on. Need more of the log!
What I see in the log so far is that only one single file is not matching (file size different) and it tries to re-transfer that. My guess is that this will take a fair amount of time to transfer as it is roughly 45 GB in size. What speed is the link between your two nodes??
As well, can you post the contents of
/opt/fog/log/fogreplicator.Automation_TB_CF54.transfer.Greenville.logor send me a link again?
