You are correct, I found the issue imaging across vlans. Stupid oversight on my part, not all my interfaces had the same pim mode. So for anyone else having issues imaging across vlans pay special attention that your pim modes are the same on the interfaces in question.
https://wiki.fogproject.org/wiki/index.php?title=Cisco_Multi_Cast
I’ve wiped out my 1.3.5-RC-11 installation and have installed the latest stable release 1.3.4 and it experiences the same issue. I mentioned I changed the IP address, I was using fog on an isolated vlan, just fog server, a 24port Netgear 10/100/1000 switch and my test machines. The new connection the fog server is on one VLAN and the machines are on another. I’m beginning to think this is the issue.
I’m configured like this:
https://wiki.fogproject.org/wiki/index.php?title=Cisco_Multi_Cast
Here is a fresh attempt:
[03-06-17 7:30:08 pm] | Task (30) Multi-Cast Task is new!
[03-06-17 7:30:08 pm] | Task (30) Multi-Cast Task has been cleaned.
[03-06-17 7:30:08 pm] | Task (30) /images/base_W5xx_Win764_AUG2016 image file found.
[03-06-17 7:30:08 pm] | Task (30) Multi-Cast Task 1 client found.
[03-06-17 7:30:08 pm] | Task (30) Multi-Cast Task sending on base port: 63158.
[03-06-17 7:30:08 pm] | Command: /usr/local/sbin/udp-sender --interface eth1 --min-receivers 1 --max-wait 600 --portbase 63158 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/base_W5xx_Win764_AUG2016/d1p1.img;
[03-06-17 7:30:09 pm] | Task (30) Multi-Cast Task has started!
[03-06-17 7:30:19 pm] | Task (30) Multi-Cast Task is already running with pid: 20332.
[03-06-17 7:30:29 pm] | Task (30) Multi-Cast Task is already running with pid: 20332.
[03-06-17 7:30:39 pm] | Task (30) Multi-Cast Task is already running with pid: 20332.
And it’s running:
ps -ef | grep 20332
root 20332 11450 0 13:30 ? 00:00:00 sh -c /usr/local/sbin/udp-sender --interface eth1 --min-receivers 1 --max-wait 600 --portbase 63158 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/base_W5xx_Win764_AUG2016/d1p1.img;
root 20333 20332 0 13:30 ? 00:00:00 /usr/local/sbin/udp-sender --interface eth1 --min-receivers 1 --max-wait 600 --portbase 63158 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/base_W5xx_Win764_AUG2016/d1p1.img
root 20459 17105 0 13:31 pts/0 00:00:00 grep --color=auto 20332
When I boot the host, it just says
Partclone v0.2.89 http://partclone.org
Starting to restore image (-) to device (/dev/md126p1)
The task in the Active Task has a spinner that says “In Progress” and the Active Multicast task says “Queued” and it never starts actually imaging.
FOG Multicast log says:
[03-06-17 5:35:48 pm] | Task (29) Multi-Cast Task is new!
[03-06-17 5:35:48 pm] | Task (29) Multi-Cast Task has been cleaned.
[03-06-17 5:35:48 pm] | Task (29) /images/base_W5xx_Win764_AUG2016 image file found.
[03-06-17 5:35:48 pm] | Task (29) Multi-Cast Task 1 client found.
[03-06-17 5:35:48 pm] | Task (29) Multi-Cast Task sending on base port: 58318.
[03-06-17 5:35:48 pm] | Command: /usr/local/sbin/udp-sender --interface eth1 --min-receivers 1 --max-wait 600 --portbase 58318 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/base_W5xx_Win764_
AUG2016/d1p1.img;
[03-06-17 5:35:48 pm] | Task (29) Multi-Cast Task has started!
[03-06-17 5:35:58 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:36:08 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:36:18 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:36:28 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:36:38 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:36:48 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:36:58 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:37:08 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:37:18 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:37:28 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:37:38 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:37:48 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:37:58 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:38:08 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:38:18 pm] | Task (29) Multi-Cast Task is already running with pid: 12190.
[03-06-17 5:38:28 pm] | Cleaning 1 task as they have been removed
[03-06-17 5:38:28 pm] | Task (29) Multi-Cast Task is being cleaned.
[03-06-17 5:38:28 pm] | Task (29) Multi-Cast Task has been cancelled.
Eth1 is the correct interface for the connection.
And here I cancel it because it’s not started… The Image was made with FOG1.3.1 if it matters…
I reconfigured the FOG servers IP address and NIC and moved it’s network connection and broke multicast.
I changed the interface setting in FOG_UDPCAST_INTERFACE, FOG_NFS_ETH_MONITOR and in the Storage Manager.
I’ve also verified all the settings here show the correct ip:
https://wiki.fogproject.org/wiki/index.php/Change_FOG_Server_IP_Address
Unicast works just fine, but Multicast fails. It starts partclone and just hangs there, not doing anything. Going through the steps here:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshooting_a_multicast
I try the first test:
sudo udp-sender --file /opt/fog/.fogsettings --log /opt/fog/log/multicast.log --ttl 1 --nopointopoint
Then “udp-receiver” on the client. Doesn’t work, the server doesn’t see the client connection attempt. It doesn’t work even if the client is directly connected to the server by a crossover cable.
udp-receiver --mcast-rdv-address <fog server ip address>
If I specify the mcast-rdv-address the test works. This works with the test machine directly connected to the fog server by crossover cable or with the test machine connected to the LAN on all my vlans. So it seems my network connection is fine.
I’m not finding any settings are incorrect, any pointers?
I haven’t updated in a few weeks, but checking the change log I didn’t see any changes related to groups, I’ll update later today and confirm if the issue is still present.
Adding a host to a group, the new host doesn’t inherit group settings. Is this intended?
ie: I have a group called P50, which has custom Kernel Arguments and Primary Disk settings. I register a new Lenovo P50, I add it to the group. The new P50 has no settings for Kernel Arguments or Primary Disk. I must edit and re-save the group for the new host to receive those settings.
Another unrelated bug, when I edit a group the “Primary Disk” setting is always empty. If I enter something and save the changes, it is saved to the group members, but the setting remains blank if I reopen the group for edit.
@Tom-Elliott said in dmraid and mdadm?:
@rbaldwin while we await your suggestions/ideas/etc… Anyway we could have you try the --metadata flag during auto assembly? While I will likely add the postinitload feature anyway, I’d still be interested in seeing if we can get automated detection working.
Since mdadm --examine --scan shows the metadata for the array as metadata=imsm, I used that in my attempt.
madadm --assemble --scan --metadata=imsm
Same results as before, the array isn’t found.
@Tom-Elliott said in dmraid and mdadm?:
@rbaldwin I’ll likely still add the postinitload scripts though my implementation will likely be a little different.
For example, the S99fog.sh script should not deal with mounting anything (that should be handled in the fog system for ease. Also postinitscripts should probably exist in the dev folder because you might want to upload, not always download – just my suggestion).
I understand. My implementation is a dirty hack.
@Tom-Elliott said in dmraid and mdadm?:
@rbaldwin while we await your suggestions/ideas/etc… Anyway we could have you try the --metadata flag during auto assembly? While I will likely add the postinitload feature anyway, I’d still be interested in seeing if we can get automated detection working.
I’ll have a look at that parameter and get back with you.
@Tom-Elliott
Here is how I added a postinitscript to my running FOG.
Using information from here I modified my init image (https://wiki.fogproject.org/wiki/index.php?title=Modifying_the_Init_Image)
In the S99FOG.sh I added at line 18:
if [ ! -z "$postinitscript" ]; then
mkdir /images
mount -o nolock,proto=tcp,rsize=32768,intr,noatime $storage /images
/images/postinitscripts/$postinitscript
umount /images
fi
I on the server I created a folder /images/dev/postinitscripts and a symbolic link to it in /images so that /images/postiniscripts is a valid path booting in both capture and deploy mode.
mkdir /images/dev/postinitscripts
ln -s /images/dev/postinitscripts /images/postinitscripts
Then I created the /images/postinitscripts/P50.sh which is a bash script which executes my mdadm build command to build my RAID array.
#!/bin/sh
#
mdadm --build /dev/md0 --raid-devices=2 --chunk=16 --level=0 /dev/sda /dev/sdb
Then on my P50 host I added the postinitscript=P50.sh to the “Host Kernel Arguments”
This way when the host boots in capture or deploy mode, S99FOG.sh checks to see if $postinitscript is set, if it is it mounts the images path and executes the specified script. I only have the one storage node, which is my FOG server. So I’m not sure if my method would work on a multinode configuration.
With this configuration I was able to boot and capture an image of my RAID configured Lenovo P50s and deploy that image to another identically configured Lenovo P50. Because the script can be specified per host, it’s customizable for other host which might need some postinit scripting.
I agree with this summary. I would just like to add I have modified my copy to execute a specified postinit script. I’ve done this by “kernel arguments”, I have added two and modified S99FOG to look for them and run the specified script. This way my Lenovo P50 hosts can run the commands to manually build the RAID array postinit and prior to imaging.
I would share my changes here, but I’m unsure the rules. Would that be welcome here?
@Tom-Elliott said in dmraid and mdadm?:
Looking a bit more, I’ve added three hopeful bits for solving this.
First, the mdadm --auto-detect should be by itself as mdadm is then asking the kernel to activate the arrays. (Source: https://linux.die.net/man/8/mdadm)
So --assemble --scan is not “compatible” with the --auto-detect argument (
mdadm --auto-detect --assemble --scanvsmdadm --auto-detect).I’ve updated the code base in the init’s for handling this to try these three things:
mdadm --auto-detect mdadm --assemble --scan mdadm --incremental --run --scanIf you jump on the dev-branch, it should update the inits for this new stuff. If these new changes still don’t work, please see if there’s a way we can do an auto assembly on the iMSM raid. If we cannot auto-assemble, it is extremely difficult (currently) to make a working method for you. I suppose I could add a “postinitload” scripts thing, similar to postdownload scripts. I’d really prefer seeing if the current code could handle this with these modifications or an auto assembly model over implicit command to generate the array.
I booted in debug mode and ran the following.
mdadm --auto-detect
mdadm --assemble --scan
mdadm --incremental --run --scan
/proc/mdstats shows 0 devices.
mdadm --detail /dev/md0
Shows inactive with 0 devices.
A postinitload script would be great, if I could specify that per host. As I have many Lenovo W530/W520s with RAID that work just fine with the existing FOG RAID Configurations. It’s only the brand new EUFI Bios’ed Lenovo P50s that are having this issue.
Is there perhaps a newer version of mdadm tools? Maybe these new Intel fake RAID controllers aren’t supported by the older version?
@Tom-Elliott said in dmraid and mdadm?:
Specifically, I’m hoping that you could run the
mdadm --assemble --scanwith achunksize specified in the command line. If this successfully builds your imsm array, then I can adjust our arguments to accept the “chunk” size.Perhaps:
mdadm --auto-detect --assemble --scanOf course I don’t have an array I could test this with, sorry.
–chunk is an invalid command on the --assemble command as is --auto-detect
Any pointers on this, I just need to be able to specify these commands to FOG when these hosts boot up for imaging?
When Attempting to send inventory… it repeatly attempts to send the inventory then finally fails an error is recorded in the Apache error.log
[Tue Jan 03 15:56:24.811397 2017] [php7:error] [pid 16365] [client 10.67.55.12:56504] PHP Fatal error: Uncaught Error: Cannot access self:: when no class scope is active in /var/www/html/fog/service/inventory.php:25\nStack trace:\n#0 {main}\n thrown in /var/www/html/fog/service/inventory.php on line 25
Error is seen when registering the host as well as when imaging the host, registration and imaging is successful, only inventory data is missing.
Host task progress is not shown, only an error is shown in the Apache error.log:
[Tue Jan 03 16:01:23.528751 2017] [php7:error] [pid 18616] [client 10.67.55.12:56700] PHP Fatal error: Uncaught Error: Cannot access self:: when no class scope is active in /var/www/html/fog/service/progress.php:24\nStack trace:\n#0 {main}\n thrown in /var/www/html/fog/service/progress.php on line 24
The task still runs, imaging is successful only the progress on the web interface isn’t shown.
I was able to build the array with the mdadm command shown here:
https://goo.gl/photos/7gtJf1hHVi4VAjj18
The build command seems to be the key here, as It works with and without the chunk command.
I haven’t mounted it yet to see if it’s readable, but this is progress from what I’ve seen so far. Not sure what to do now, since this works great in Debug mode, but wont be done automatically on fresh reboots to debug, capture or deploy.
Sorry just noticed you said mdadm --assemble --scan
“no arrays found in configure file or automatically”
It’s in my last reply:
https://goo.gl/photos/Hq3CN682Hc3jshq1A
The raid array has a Windows 7 64bit installation on it and is defined as ~1TB
Raid configuration in Bios:
https://goo.gl/photos/iLZveDYK1vJ6Tijt6
mdadm --examine on /dev/sda
https://goo.gl/photos/9DTovJpVxsrLPApA9
mdadm --examine on /dev/sdb
https://goo.gl/photos/BY8ynhEzmsFNHtES6
mdadm --examine --scan output:
https://goo.gl/photos/Hq3CN682Hc3jshq1A
As you can see mdadm can see the volume but for some reason doesn’t automatically assemble it. I don’t know eneough about how this works to say why. Just looking at dmraid because I’ve seen it work on the Ubuntu 16.04 live cd. Something I can’t say for mdadm so for on the P50. mdadm works fine on the Lenovo W530.
fdisk -l output:
https://goo.gl/photos/jMvtUtfyjsoyPZBH6
I haven’t been able to make mdadm assemble the disk.
For example:
mdadm --assemble /dev/md0 /dev/sda /dev/sdb
Complains that disk /dev/sda and /dev/sdb do not contain a superblock.