Migrate images from a storage to another.
-
I have a fog server which was installed as default with image stored in a local mount (from an iSCSI link).
I wanted to move the images from the server to a storage node.So I built a storage node, unmounted images LUN from the server and mounted it on the storage node.
I wanted to move the images from the storage node to another using the WebUI and it does not seem to be possible.
So I exported image infos for a specific image. tried to reimport it, but storage info remain the same.
So I delete it again and recreated the image manually.
Using exactly the same parameters but the storage node. (choosing the new one)unfortunately this did not worked either always getting this error :
Could not mount images folder (/bin/fog.download)On another hand I can create new image and deploy them without any issue. (they are stored on the new storage)
I tried the plugin location the enforce the use of the new storage node on my test PC but it did not helped.
Any idea of what could be my problem? -
@processor May I ask why you want to move the images at all? What is wrong with having the iSCSI LUN to your main FOG Server??
-
HI Sebastian,
No worries, right now the fog server is an Ubuntu 16.04 VM build in an ESXI 5.5
the VM has 2 vNIC. One on our education network and another one storage network.
Each vNIC are supported by 2 physical Ethernet NIC teamed and configured to use ip hashing to load balance traffic.The Ubuntu server is configured to use multipath iSCSI to access the NAS LUNs and multicast is massively used to install our classes.
We have to install between 200 and 300 machines per week. At start we have no severe issue but with programmed tasks starting the webui is dramatically slowing down. With 2 or 3 tasks running we can start seeing slowdowns.
Using nload and top to monitor where could be the issue, we could only see very high network load but no issue with memory or CPU.
We deploy at speed between 5GB/m and 27GB/m depending on the receiving hardware. No other resources that the FOG server are showing slowdowns.
So we thought the mutliple multicast sessions were cannibalizing the server bandwidth and having a server for the management and a storage node for the deploys would be a good idea.
Sorry for the long explanation. Are we wrong?
Then we are multi-site, right now only our main site is testing fog and we already have several TB of images.
At the end, sync via wan link will be OK but for the first load we will have to send a disk with all images on it. that’s the other reason we would like to test import.
Proc
-
Ok, seems to work finally.
The error was because I deleted the test image thinking it will only remove infos from database whereas it has trully deleted the image in the storage node, even if it appeared in default() storage in image description.Now, new image will be stored in the storage associate with the location of the captured machine despite the selected storage during image creation.
It’s quite confusing because this mean we could think that images are stored in the default storage whereas they are not and inversely.
Is it a bug or is this is a normal behaviour?
Does the storage setting in image creation has an influence or everything depends of location of the computer which deploy or capture? -
I’m having difficulties following this, maybe because its late in the day for me.
Let me restate the issue with my words to see if I understand it.
You image between 200 and 300 systems per week. Your actual images are stored on a NAS LUN on the storage network and not local to the FOG server. When you get several unicast streams running the access to the webui becomes very slow. Looking at the server stats the CPU and MEMORY numbers are OK, but network utilization is very high.
Do I understand your issue correctly?
Now I have a few comments.
- I have proven through testing that you can flood a 1GbE link with 3 unicast streams running at the same time. This isn’t a FOG issue, but a physical issue being able to move that much data down the wire. So if you are running more than 3 unicast streams at a time I can see where the web ui may appear to be slow. Even with LACP bonding on the NICs you are only increasing your feed rate by a little, I would guess you might flood at 5 unicast streams or less. It really depends on the IP hashing used to select the link. You may have one LAG link flooded and the other lightly used. Its hard to say.
- I would surely use mutlicast imaging in this case. With 200 machines, that means you would have to do 25/hr in an 8hr day. (its still possible to do with unicast at least by the numbers). Multicast imaging would free up a lot of system resources over unicast imaging. On the FOG servers site, but also on the SAN resources as well as networking.
- If you didn’t or couldn’t use multicast imaging then you could add additional FOG storage nodes to share the load. But understand the load or restriction point is where the data meets the network infrastructure. So lets say you have a virtual fog server and its running on vm host A. If you add a second storage node and it is also running on vm host A, your bottle neck will between the vm host A nic adapter and your network switch. So adding more fog storage nodes to vm host A will not help. You would need to add the fog storage node to vm host b to get any benefit, but then your SAN is taking the impact of having multiple data streams.
If you add a fog storage server to your existing fog server, the images will automatically replicate from the FOG Master node to all slave nodes in the storage group. You don’t/should’t need to mess with any backend storage LUNs. If the storage nodes are in the same storage group when the mast node fills up to max clients then the next storage node will be used. If you have storage nodes at different locations then you will want to use the fog location plugin to assign a storage node and clients to a location so the clients will use the local fog server.
-
Hi,
My skills in english are fair but that’s all, that’s may be why my message was not clear enough.
To resume :
-We had a NAS LUN linked to the legacy FOG Server
-We rarely do unicast deploy, 95% are multicast.
-Running 3 or more multicasts start causing slowdowns on the WebUI.
-Sometimes we have up to 10 simultaneous multicasts. Very have rarely more, even
with 23 classrooms on the main site.As the nic used to access the WebUI is the same as the one that flood the images through the network we thought that we could mitigate this phenomenon using storage node.
We are note stuck to a model or another, we are still searching for the best compromise.
Now we have/had a second issue it’s image export/Import. We plan to use replication at the end but for the first load which represent several TB sync through a wan link is not possible.
It seems this issue is now solved, the remaining “problem” is only cosmetic, but may be I did something wrong, it was the first time I moved images from storage to another and tried the export/import.
