FOG 1.2.0 and Ubuntu Server 12.04.5 LTS - Workload distribution on storage nodes

[ChristianUOS]

Hello together,

because Multicast still doesn’t work within our infrastructure, tested:

• Fedora 21/FOG SVN2922 as mentioned in [url]http://www.fogproject.org/wiki/index.php/FOG_(r2922)_Configuration_on_Fedora_21_Workstation_inside_Windows_Server_2012_Hyper-V_using_ProxyDHCP[/url]
  Result: Multicast Imaging doesn’t start at all
• Ubuntu Server 12.04 LTS and FOG 1.2.0 as mentioned in [url]http://www.fogproject.org/wiki/index.php/Ubuntu_12.04[/url]
  –> without UDPCast because it refers to Version 0.32 and not to 1.2.0
  –> without activating root
  –> without echo “greeter-show-manual-login=true” >> /etc/lightdm/lightdm.conf because of running the Ubuntu Server Version
  Result: Multicast hangs always on the last partition in “Starting to restore image (-) to device (/dev/sda4)” and if I’m deleting the last one “Starting to restore image (-) to device (/dev/sda3)”
  - seems to be the same problem like posted by snoopsean: [url]http://fogproject.org/forum/threads/fog-1-1-0-multicast-sits-at-starting-to-restore-image-to-device-dev-sda1.10782/[/url]
  => Conclusion: Ok, Multicast seems to be a really big and open issue.

I’ve now installed a fog server 1.2.0 on Ubuntu Server 12.04.5 LTS and 4 storage nodes for improving the unicast speed. In addition to running the installscript - because of some errors - I’ve done the following steps:

• chown -R fog /images -> on the fog server, because before Images couldn’t be moved from /images/dev to /images/ and upload tasks didn’t complete (seems to be done by local fog user)
• removed the sql bind address 127.0.0.1 in /etc/mysql/my.cnf --> before no connection could be established by the storage nodes
• chown -R fog /images -> on the storage nodes, because the graph on the dashboard couldn’t get any information from the storage nodes
• sudo service restart FOGImageReplicator restart --> Before he wasn’t running and Images were not be synced

Now, the Image was successfully synchronised to the 4 storage nodes and the log seems to be ok:

[03-17-15 12:32:21 pm] * SubProcess ->

[03-17-15 12:32:21 pm] * SubProcess -> Complete

[03-17-15 12:42:21 pm] * Starting Image Replication.

[03-17-15 12:42:21 pm] * We are group ID: #1

[03-17-15 12:42:21 pm] * We have node ID: #1

[03-17-15 12:42:21 pm] * Found: 4 other member(s).

[03-17-15 12:42:21 pm]

[03-17-15 12:42:21 pm] * My root: /images

[03-17-15 12:42:21 pm] * Starting Sync.

[03-17-15 12:42:21 pm] * Syncing: fognode1

[03-17-15 12:42:21 pm] * SubProcess -> Mirroring directory `20142015WS001’

[03-17-15 12:42:21 pm] * SubProcess -> Mirroring directory `postdownloadscripts’

[03-17-15 12:42:21 pm] * SubProcess ->

[03-17-15 12:42:21 pm] * SubProcess -> Complete

[03-17-15 12:42:21 pm] * Syncing: fognode2

[03-17-15 12:42:21 pm] * SubProcess -> Mirroring directory `20142015WS001’

[03-17-15 12:42:21 pm] * SubProcess -> Mirroring directory `postdownloadscripts’

[03-17-15 12:42:21 pm] * SubProcess ->

[03-17-15 12:42:21 pm] * SubProcess -> Complete

[03-17-15 12:42:21 pm] * Syncing: fognode3

[03-17-15 12:42:21 pm] * SubProcess -> Mirroring directory `20142015WS001’

[03-17-15 12:42:21 pm] * SubProcess -> Mirroring directory `postdownloadscripts’

[03-17-15 12:42:21 pm] * SubProcess ->

[03-17-15 12:42:21 pm] * SubProcess -> Complete

[03-17-15 12:42:21 pm] * Syncing: fognode4

[03-17-15 12:42:21 pm] * SubProcess -> Mirroring directory `20142015WS001’

[03-17-15 12:42:21 pm] * SubProcess -> Mirroring directory `postdownloadscripts’

[03-17-15 12:42:21 pm] * SubProcess ->

[03-17-15 12:42:21 pm] * SubProcess -> Complete

Unfortunately, the unicast speed were not improved as expected. After the first slow deployment, I decided to streamline the passwords of the local fog user on all servers and inserted them in storage management section of the web GUI as mentioned in the forum.

But the next deployment of the 64 pcs wasn’t as fast as expected. The image has got a server size of about 34 GB. I also wondered why the CPU usage of fognode1 and fognode2 was at 0% whereas it was at 20%-30% for the fog server. So I took at the RX rates:

fog - eth0 RX 2.97 TiB - max clients 12
fognode1 - eth0 RX 9.73 GiB - max clients 12
fognode2 - eth0 RX 31.46 MiB - max clients 12
fognode3 - eth0 RX 66.94 GiB - max clients 14
fognode4 - eth0 RX 2.97 TiB - max clients 14

It seems that fognode1 and fognode2 never were used for deployment. fognode3 also doesn’t have the workload of 14 expected clients. Because I didn’t found any helpful further information inside the wiki, I’d like to know:

• Why are the nodes 1-3 not beeing used? What are the requirements (running services, file permissions, etc.)?
• Are there any troubleshooting possibilities?
• Which user is used to access the images?
• How is the workload distributed to the nodes?
• Is the Ubuntu documentation in the wiki still valid for the latest staple version 1.2.0/valid for Ubuntu server? I doubt because of my mentioned additional steps above and the out-dated udpcast-step according to version 0.32

Thanks for any additional information!

Kind regards
Christian

[Wayne Workman]

I can probably help with the multicasting aspects on Fedora FOG configuration, but for the storage nodes, someone else will need to hop in for that.

I’m actually the person that wrote: [url]http://www.fogproject.org/wiki/index.php/FOG_(r2922)_Configuration_on_Fedora_21_Workstation_inside_Windows_Server_2012_Hyper-V_using_ProxyDHCP[/url]
Tom (the senior developer) and I worked together so he could integrate official Fedora 21 support for FOG.

Those steps are literally, step by step, every single thing I did to set FOG up in my environment, I’ve tested those steps probably 3 or 4 different times with various development revisions of FOG, perfecting those steps. To guarantee they work for you, you’ll have to follow, step by step, those instructions. Every step is important. You can deviate with some stuff, but for the majority, it needs to be the same.

Do you still have the FOG Fedora 21 machine set up? Did you run any of the troubleshooting commands towards the bottom of the instructions? What’s your compression settings for the images set to?

Also, a 34GB image size on the server is radically large. I can tell you that a 16GB image size on the server will deploy via unicast to 1 host on a 1Gbps network in about 8 minutes.

[Junkhacker]

[quote=“Wayne Workman, post: 43990, member: 28155”]
Also, a 34GB image size on the server is radically large. I can tell you that a 16GB image size on the server will deploy via unicast to 1 host on a 1Gbps network in about 8 minutes.[/quote]

34GB is pretty big, but I’ve certainly got larger images that are deployed here. some nearly double that size.

[ChristianUOS]

Thanks Wyne Workman for your quick reply. I’ll try to setup a new machine with fedora 21 and the svn-Version 2922. Meanwhile I’ve updated to 3121 to see if it works. But I’m bounded to Windows Server 2012 as virtualization host. Because I’m the next few days out of office, I can not try this before Tuesday. Or should it also work with the SVN-Version 3121?

If someone have got addition information on how to troubleshoot the load balancing of storage nodes to improve unicast performance I would be very thankful.

[Wayne Workman]

[quote=“ChristianUOS, post: 44003, member: 28875”]Thanks Wyne Workman for your quick reply. I’ll try to setup a new machine with fedora 21 and the svn-Version 2922. Meanwhile I’ve updated to 3121 to see if it works. But I’m bounded to Windows Server 2012 as virtualization host. Because I’m the next few days out of office, I can not try this before Tuesday. Or should it also work with the SVN-Version 3121?

If someone have got addition information on how to troubleshoot the load balancing of storage nodes to improve unicast performance I would be very thankful.[/quote]

You could follow the Fedora 21 instructions but instead of using 2922, you could use 3121.

Please follow the instructions, even for installing Fedora 21 itself. There’s a custom configuration regarding the /images directory for it.