dmraid and mdadm?

[rbaldwin]

@Tom-Elliott

It’s in my last reply:
https://goo.gl/photos/Hq3CN682Hc3jshq1A

[rbaldwin]

@Tom-Elliott

Sorry just noticed you said mdadm --assemble --scan

“no arrays found in configure file or automatically”

[Tom Elliott]

http://unix.stackexchange.com/questions/273819/how-do-i-rebuild-create-assemble-an-imsm-raid-0-array-from-disk-images-instead

It appears you might need to specify a chunk size? I don’t know what that chunksize will be, but if you can test the ideas on the above link, and figure out, I can work to add a “chunk” size argument for the raid if all works.

[rbaldwin]

@Tom-Elliott

I was able to build the array with the mdadm command shown here:
https://goo.gl/photos/7gtJf1hHVi4VAjj18

The build command seems to be the key here, as It works with and without the chunk command.
I haven’t mounted it yet to see if it’s readable, but this is progress from what I’ve seen so far. Not sure what to do now, since this works great in Debug mode, but wont be done automatically on fresh reboots to debug, capture or deploy.

[rbaldwin]

Any pointers on this, I just need to be able to specify these commands to FOG when these hosts boot up for imaging?

[Tom Elliott]

@rbaldwin The difference is your commands are implicit (with or without the chunk commands). The methods that are in use when you run just a “normal” runner, the code is greatly different.

Maybe it’s my unknowing how to do so. If you’d like, you can play with the code a bit (in a debug mode maybe?)

The information FOG currently uses (and works for the few other cases we have of people using RAID layout) is located here:
https://github.com/FOGProject/fogproject/blob/dev-branch/src/buildroot/package/fog/scripts/etc/init.d/S99fog#L17

[Tom Elliott]

Specifically, I’m hoping that you could run the mdadm --assemble --scan with a chunk size specified in the command line. If this successfully builds your imsm array, then I can adjust our arguments to accept the “chunk” size.

Perhaps:

mdadm --auto-detect --assemble --scan

Of course I don’t have an array I could test this with, sorry.

[Tom Elliott]

I’m working on thoughts so if my messages seem to drift around it’s probably just thinking a lot.

[Tom Elliott]

Looking a bit more, I’ve added three hopeful bits for solving this.

First, the mdadm --auto-detect should be by itself as mdadm is then asking the kernel to activate the arrays. (Source: https://linux.die.net/man/8/mdadm)

So --assemble --scan is not “compatible” with the --auto-detect argument (mdadm --auto-detect --assemble --scan vs mdadm --auto-detect).

I’ve updated the code base in the init’s for handling this to try these three things:

mdadm --auto-detect
mdadm --assemble --scan
mdadm --incremental --run --scan

If you jump on the dev-branch, it should update the inits for this new stuff. If these new changes still don’t work, please see if there’s a way we can do an auto assembly on the iMSM raid. If we cannot auto-assemble, it is extremely difficult (currently) to make a working method for you. I suppose I could add a “postinitload” scripts thing, similar to postdownload scripts. I’d really prefer seeing if the current code could handle this with these modifications or an auto assembly model over implicit command to generate the array.

[rbaldwin]

@Tom-Elliott said in dmraid and mdadm?:

Specifically, I’m hoping that you could run the mdadm --assemble --scan with a chunk size specified in the command line. If this successfully builds your imsm array, then I can adjust our arguments to accept the “chunk” size.

Perhaps:

mdadm --auto-detect --assemble --scan

Of course I don’t have an array I could test this with, sorry.

–chunk is an invalid command on the --assemble command as is --auto-detect

[rbaldwin]

@Tom-Elliott said in dmraid and mdadm?:

Looking a bit more, I’ve added three hopeful bits for solving this.

First, the mdadm --auto-detect should be by itself as mdadm is then asking the kernel to activate the arrays. (Source: https://linux.die.net/man/8/mdadm)

So --assemble --scan is not “compatible” with the --auto-detect argument (mdadm --auto-detect --assemble --scan vs mdadm --auto-detect).

I’ve updated the code base in the init’s for handling this to try these three things:

mdadm --auto-detect
mdadm --assemble --scan
mdadm --incremental --run --scan

If you jump on the dev-branch, it should update the inits for this new stuff. If these new changes still don’t work, please see if there’s a way we can do an auto assembly on the iMSM raid. If we cannot auto-assemble, it is extremely difficult (currently) to make a working method for you. I suppose I could add a “postinitload” scripts thing, similar to postdownload scripts. I’d really prefer seeing if the current code could handle this with these modifications or an auto assembly model over implicit command to generate the array.

I booted in debug mode and ran the following.
mdadm --auto-detect
mdadm --assemble --scan
mdadm --incremental --run --scan

/proc/mdstats shows 0 devices.

mdadm --detail /dev/md0
Shows inactive with 0 devices.

A postinitload script would be great, if I could specify that per host. As I have many Lenovo W530/W520s with RAID that work just fine with the existing FOG RAID Configurations. It’s only the brand new EUFI Bios’ed Lenovo P50s that are having this issue.

Is there perhaps a newer version of mdadm tools? Maybe these new Intel fake RAID controllers aren’t supported by the older version?

[Tom Elliott]

@rbaldwin But they are supported, they’re just to supporting auto assembly. You showed that they did get detected if you structured the call.

[Tom Elliott]

Maybe this could help?

mdadm --assemble --scan --metadata=imsm
mdadm --incremental --run --scan --metadata=imsm

I am hoping to find something.

[Wayne Workman]

Can we please summarize this thread in 1 or 2 posts please? It’s now 24 posts long and getting unwieldy. I want to help - probably as others do - but I’ve just read through 30 other posts and it’s now 1:30 AM.

Please someone summarize this thread.

[Tom Elliott]

@Wayne-Workman I guess I don’t understand what you mean “summarize”.

Essentially (if this is what you mean), @rbaldwin has a RAID setup that he’d like to capture/deploy images from/to. The common RAID setup is using a metadata of 0.90 (which I suspect is causing the inability to assemble the array currently).

This thread is trying to get the RAID array in use to work with FOG. I’m trying to get it to figure out how to do this without necessarily having to know the drives and layouts just to perform the tasking.

Capture is really the only thing that should need this except we capture partition data based on MBR/GPT layout on the disk. This will need to be rethought so images can be deployed to any disk regardless of if it is a RAID array. – This is for later though (more a todo than will be done right now).

So far, however, we have not had any luck so many of the runs on here is just a collaborative effort to see if we can get it working automatically.

All we’re trying to do right now is figure out a means to get this somewhat automated. If we cannot I might add a postinitload feature that will allow users to perform tasks after the init is loaded but before fog start performing its own tasks.

[rbaldwin]

@Tom-Elliott

I agree with this summary. I would just like to add I have modified my copy to execute a specified postinit script. I’ve done this by “kernel arguments”, I have added two and modified S99FOG to look for them and run the specified script. This way my Lenovo P50 hosts can run the commands to manually build the RAID array postinit and prior to imaging.

I would share my changes here, but I’m unsure the rules. Would that be welcome here?

[Tom Elliott]

@rbaldwin Any help is always welcome. I’m not perfect.

[Tom Elliott]

@rbaldwin while we await your suggestions/ideas/etc… Anyway we could have you try the --metadata flag during auto assembly? While I will likely add the postinitload feature anyway, I’d still be interested in seeing if we can get automated detection working.

[rbaldwin]

@Tom-Elliott
Here is how I added a postinitscript to my running FOG.

Using information from here I modified my init image (https://wiki.fogproject.org/wiki/index.php?title=Modifying_the_Init_Image)

In the S99FOG.sh I added at line 18:

if [ ! -z "$postinitscript" ]; then
    mkdir /images
    mount -o nolock,proto=tcp,rsize=32768,intr,noatime $storage /images
    /images/postinitscripts/$postinitscript
   umount /images
fi

I on the server I created a folder /images/dev/postinitscripts and a symbolic link to it in /images so that /images/postiniscripts is a valid path booting in both capture and deploy mode.

mkdir /images/dev/postinitscripts
ln -s /images/dev/postinitscripts /images/postinitscripts

Then I created the /images/postinitscripts/P50.sh which is a bash script which executes my mdadm build command to build my RAID array.

#!/bin/sh
#
mdadm --build /dev/md0 --raid-devices=2 --chunk=16 --level=0 /dev/sda /dev/sdb

Then on my P50 host I added the postinitscript=P50.sh to the “Host Kernel Arguments”

This way when the host boots in capture or deploy mode, S99FOG.sh checks to see if $postinitscript is set, if it is it mounts the images path and executes the specified script. I only have the one storage node, which is my FOG server. So I’m not sure if my method would work on a multinode configuration.

With this configuration I was able to boot and capture an image of my RAID configured Lenovo P50s and deploy that image to another identically configured Lenovo P50. Because the script can be specified per host, it’s customizable for other host which might need some postinit scripting.

[rbaldwin]

@Tom-Elliott said in dmraid and mdadm?:

@rbaldwin while we await your suggestions/ideas/etc… Anyway we could have you try the --metadata flag during auto assembly? While I will likely add the postinitload feature anyway, I’d still be interested in seeing if we can get automated detection working.

I’ll have a look at that parameter and get back with you.