Intel Raid0 Image Capture


  • Senior Developer

    I updated the source for this to allow for if ($2 ~ /part/ || $2 ~ /md/).

    Hopefully this will work for you needs.



  • @george1421 Have you been able to capture an image from the machine?

    Do you think it’d be possible to capture a Single Disk Resizable image from these fake raid machines?



  • @Sebastian-Roth said in Intel Raid0 Image Capture:

    @jpmartin George’s mdadm -D /dev/md126 seems to be very handy and informative. Give that a try!

    No image file found that would match the partitions to be restored
    args passed /dev/md126 /images/WIN7ENTX64 all
    

    Guess that’s just a matter of tuning the init scripts to make this work. Will have a look tomorrow. Marking this unread… ;-)

    Here you go:

    mdadm -D /dev/md126.txt


  • Moderator

    Updating the trouble code
    lsblk -I 3,8,9,179,259 -lpno KNAME,TYPE /dev/md126 | awk '{if ($2 ~ /part/) print $1}'

    to
    lsblk -I 3,8,9,179,259 -lpno KNAME,TYPE /dev/md126 | awk '{if ($2 ~ /md/) print $1}'

    Allowed the system to image in debug mode. I doubt it will run because I don’t have the raid drivers in my pushed image, but imaging did complete fully without error.


  • Moderator

    @Tom-Elliott getPartitons from download. I’ll have to look at the code again

    [edit] here is the call chain
    fog.download:restorePartition->funcs.sh:runPartprobe->funcs.sh:performRestore->funcs.sh:getValidRestorePartitions->funcs.sh:getPartitions


  • Senior Developer

    @george1421 what was the function the error came from?


  • Moderator

    With the last bit of brain power I have left today I think I narrowed it down to the following (I don’t know what I’m looking at only that it doesn’t work"

    In /usr/share/fog/lib/funcs.sh at function: getPartitions()

    There is a call to lsblk that should be returning something, but its returning and empty string instead.

    This is the command
    lsblk -I 3,8,9,179,259 -lpno KNAME,TYPE /dev/md126 | awk '{if ($2 ~ /part/) print $1}'

    If I shorten the command to just the lsblk without the awk I get

    # lsblk -I 3,8,9,179,259 -lpno KNAME,TYPE /dev/md126
    /dev/md126   raid0
    /dev/md126p1 md
    /dev/md126p2 md
    

    If I run the full command on my fog server using /dev/sda I get

    # lsblk -I 3,8,9,179,259 -lpno KNAME,TYPE /dev/sda | awk '{if ($2 ~ /part/) print $1}'
    /dev/sda1
    /dev/sda2
    

    Running lsblk alone on FOS

    lsblk /dev/md126
    NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
    md126       9:126  0 149.1G  0 raid0
    |-md126p1 259:4    0   499M  0 md
    `-md126p2 259:5    0 148.6G  0 md
    

    Running lsblk alone on FOG

    # lsblk
    NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    fd0               2:0    1    4K  0 disk
    sda               8:0    0   20G  0 disk
    ├─sda1            8:1    0  500M  0 part /boot
    └─sda2            8:2    0 19.5G  0 part
      ├─centos-root 253:0    0 17.5G  0 lvm  /
      └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
    sr0              11:0    1    4G  0 rom
    

    Hopefully the devs can make heads or tails of why the lsblk command is not returning the expected value.

    [edit] Heck, now that I spell it out I see the issue. The awk regular expression is only looking for part in the type column. awk '{if ($2 ~ /part/) print $1}' But the mdraid its type md not part !!!


  • Moderator

    I left the FOS kernel running. Now that I’m home I’m able to remote into FOS to continue debugging.

    Looking at what FOS has done so far, I can see that it did create the partitions

    Disk /dev/md126: 149.1 GiB, 160058834944 bytes, 312614912 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 131072 bytes / 262144 bytes
    Disklabel type: dos
    Disk identifier: 0x9bfaf142
    
    Device       Boot   Start       End   Sectors   Size Id Type
    /dev/md126p1 *       2048   1023999   1021952   499M  7 HPFS/NTFS/exFAT
    /dev/md126p2      1024000 312614911 311590912 148.6G  7 HPFS/NTFS/exFAT
    

    I’m going to keep walking down the fog.download script to see where its falling down.

    Here is the exact error:

     * Attempting to expand/fill partitions..............Done
     * Press [Enter] key to continue
    
     * Seems like you are trying to restore to an empty disk. Be aware this will most probably cause trouble.
    
       +--------------------------------+
       |  Attempting to download image  |
       +--------------------------------+
       |         Using Partclone        |
       +--------------------------------+
    ##############################################################################
    #                                                                            #
    #                         An error has been detected!                        #
    #                                                                            #
    ##############################################################################
    No image file(s) found that would match the partition(s) to be restored (performRestore)
       Args Passed: /dev/md126 /images/WIN7ENTSP1X6401 all
    

    Its dieing somewhere in this section in funcs.sh in the performRestore() sub.

        local disk_number=1
        local part_number=0
        local restoreparts=""
        local mainuuidfilename=""
        [[ $imgType =~ [Nn] ]] && local tmpebrfilename=""
        for disk in $disks; do
            mainuuidfilename=""
            mainUUIDFileName "$imagePath" "$disk_number"
            getValidRestorePartitions "$disk" "$disk_number" "$imagePath" "$restoreparts"
            [[ -z $restoreparts ]] && handleError "No image file(s) found that would match the partition(s) to be restored (${FUNCNAME[0]})\n   Args Passed: $*"
            for restorepart in $restoreparts; do
    

  • Senior Developer

    @george1421 Great stuff man! Just wondering that you had the same md127 : inactive ... Possibly that’s just the way it is with those fake RAID controllers?

    @jpmartin George’s mdadm -D /dev/md126 seems to be very handy and informative. Give that a try!

    No image file found that would match the partitions to be restored
    args passed /dev/md126 /images/WIN7ENTX64 all
    

    Guess that’s just a matter of tuning the init scripts to make this work. Will have a look tomorrow. Marking this unread… ;-)


  • Moderator

    @jpmartin OK here is what I did.

    1. Installed 2 test hard drives in the 780
    2. Changed the 780 disk mode to Raid
    3. Booted the system and went into the Raid setup <Ctrl-I>
    4. Created a new Raid0 array (Name Volume0)
    5. I used my FOS usb in the debug grub entry I added “mdraid=true”
    6. Booted off the FOS USB stick and it hung after loading the inits

    I remember that Tom upgraded the kernel after I created my FOS USB so I downloaded the latest firmware and inits to the fos-usb stick. This kernel was 4.6.2 (not sure what was before)
    7. I again booted the FOS engine from the USB stick.
    8. I inspected /proc/mdstat and received the following

    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
    md126 : active raid0 sda[1] sdb[0]
          156307456 blocks super external:/md127/0 128k chunks
    
    md127 : inactive sdb[1](S) sda[0](S)
          4856 blocks super external:imsm
    
    unused devices: <none>
    
    1. Hey! the kernel saw the fake raid!
    2. Now lets see if mdadm understands it mdadm -D /dev/md126
    /dev/md126:
          Container : /dev/md/imsm0, member 0
         Raid Level : raid0
         Array Size : 156307456 (149.07 GiB 160.06 GB)
       Raid Devices : 2
      Total Devices : 2
    
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
    
         Chunk Size : 128K
    
    
               UUID : d0202e14:9f7ca368:a44d0646:e4480443
        Number   Major   Minor   RaidDevice State
           1       8        0        0      active sync   /dev/sda
           0       8       16        1      active sync   /dev/sdb
    
    1. Nice!
    2. I wondered if it was real or a mirage 'fdisk /dev/md126`
    Welcome to fdisk (util-linux 2.27.1).
    Changes will remain in memory only, until you decide to write them.
    Be careful before using the write command.
    
    Device does not contain a recognized partition table.
    Created a new DOS disklabel with disk identifier 0x7f7c460d.
    
    1. I was able to create a partion on the disk
    Disk /dev/md126: 149.1 GiB, 160058834944 bytes, 312614912 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 131072 bytes / 262144 bytes
    Disklabel type: dos
    Disk identifier: 0x7f7c460d
    
    Device       Boot Start       End   Sectors   Size Id Type
    /dev/md126p1       2048 312614911 312612864 149.1G 83 Linux
    
    1. I wonder if I can format it? mkfs -t ext4 /dev/md126p1
    mke2fs 1.42.13 (17-May-2015)
    Creating filesystem with 39076608 4k blocks and 9773056 inodes
    Filesystem UUID: ba6c9de0-0032-4fbc-82c3-054bbb92df20
    Superblock backups stored on blocks:
            32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
            4096000, 7962624, 11239424, 20480000, 23887872
    
    Allocating group tables: done
    Writing inode tables: done
    Creating journal (32768 blocks): done
    Writing superblocks and filesystem accounting information: done
    
    1. Well can I mount it? mount -t ext4 /dev/md126p1 /mnt
    # df
    Filesystem     1K-blocks  Used Available Use% Mounted on
    /dev/root          95538 82836      7897  92% /
    /dev/md126p1   153721892 60864 145829324   1% /mnt
    
    1. Success!!

    So now to turn to FOG

    I ran out of time to play for today.

    But I updated the host definition with
    Host Kernel Arguments: mdraid=true
    Host Primary Disk: /dev/md126

    The image deploy threw an error.

    Seems like you are trying to restore to an empty disk. Be aware that this most probably will cause trouble

    No image file found that would match the partitions to be restored
    args passed /dev/md126 /images/WIN7ENTX64 all

    Conclusion: The array is there and I can mount it. We just need to identify why this error is being thrown.

    I also saw a device /dev/md/Volume0_0 which seemed to match the name of the array I created in the Intel firmware.



  • @george1421 We’re on Raid0, if that makes a difference.


  • Moderator

    @jpmartin My work hours just ended so I have time to play at work now. The dell optiplex 780s have this built in fake raid controller. I’ll let you know in a little bit if I’m able to get the array to be seen by the FOS engine.



  • @Sebastian-Roth said in Intel Raid0 Image Capture:>

    The tools we use in the inits are not the ones you usually have on your normal linux desktop or server as this is a buildroot “toolchain”. The options are mostly similar to all the conventional linux tools but some are different. lsblk is definitely one having quite different options. While you are in debug mode run lsblk --help to see it’s options.

    Great you found my old post and tried all the commands. While it’s still true that I don’t have a system to test I guess we can take this one step further if you keep on posting valuable information.

    • mdadm examine output looks ok to me.
    • mdstat not great as md127 is inactive. Maybe this helps?
    • possibly this is just an issue because we don’t have /etc/mdadm.conf - maybe? Take a look at that file on your system before running the debug upload task. Maybe put a copy on your FOG server (/images/dev), boot in FOG debug and copy from FOG server to the client. Then see if you can assemble the array properly?!

    These are Windows 7 systems, so I don’t have a mdadm.conf to go off of.

    I went to that link and started down the list, but it got over my head pretty quickly.

    These are the results of mdadm -examine of sda and sdb.

    mdadm_examine_sdb.txt
    mdadm_examine_sda.txt


  • Senior Developer

    @jpmartin said:

    I’m not very experienced with linux (specifically .sh scripting), but the portion of the functions.sh script that looks for partitions doesn’t make much sense to me, especially this function:

    The tools we use in the inits are not the ones you usually have on your normal linux desktop or server as this is a buildroot “toolchain”. The options are mostly similar to all the conventional linux tools but some are different. lsblk is definitely one having quite different options. While you are in debug mode run lsblk --help to see it’s options.

    Great you found my old post and tried all the commands. While it’s still true that I don’t have a system to test I guess we can take this one step further if you keep on posting valuable information.

    • mdadm examine output looks ok to me.
    • mdstat not great as md127 is inactive. Maybe this helps?
    • possibly this is just an issue because we don’t have /etc/mdadm.conf - maybe? Take a look at that file on your system before running the debug upload task. Maybe put a copy on your FOG server (/images/dev), boot in FOG debug and copy from FOG server to the client. Then see if you can assemble the array properly?!


  • @Tom-Elliott said in Intel Raid0 Image Capture:

    if you can reboot a few times and you have the mdraid=‘true’ argument set on this host, can you verify the name ‘/dev/md126’ is always the same?

    If it is, you will need to have ‘/dev/md126’ added to the host’s primary device as well.

    I didn’t specify in my earlier posts, but I believe /dev/md126 was consistent. I set Host primary disk to /dev/md126 and tested a couple different times.

    As Multiple Partition - Single Disk: It could find the disk (array), but not the partitions.
    As Multiple Partition - All Disks: It was unable to find the disk (array).

    @george1421 said in Intel Raid0 Image Capture:

    @jpmartin It would be interesting to know the output of

    mdadm –D /dev/md126

    Will get that output for you shortly.

    I’m going to run a bunch of different tests and dump the outputs to .txt files. Let me know if there is a command that you need me to run in addition to the one just above, fdisk -l, and from this post:

    @Sebastian-Roth said in Intel RAID:

    I am sorry but I feel pretty lost with this as I don’t have such a machine here to test!

    The only thing I can offer is to go through this again from the start step by step. Just an offer. It’s up to you if you want to.

    Let’s start with the client you are getting the image from. Configure a complete new image for that client in FOG and make it Multiple partition image - single disk (not resizeable). The run a debug session for that client, boot it up and wait till you get to the command shell. Then run the following commands and post the full output here in the forums or upload the text files. Replace x.x.x.x with the IP address of your FOG server.

    mkdir -p /mnt
    mount -t nfs -o rw,nolock x.x.x.x:/images/dev /mnt
    mdadm --examine /dev/sd? > /mnt/mdadm_examine.txt
    cat /proc/mdstat > /mnt/mdstat.txt
    mdadm --assemble --scan --verbose > /mnt/mdadm_assemble.txt
    ls -al /dev/md* > /mnt/md_devices.txt
    umount /mnt
    

    I just typed and copied those commands without testing. There might be typos.


  • Moderator

    @jpmartin It would be interesting to know the output of

    mdadm –D /dev/md126


  • Senior Developer

    if you can reboot a few times and you have the mdraid=‘true’ argument set on this host, can you verify the name ‘/dev/md126’ is always the same?

    If it is, you will need to have ‘/dev/md126’ added to the host’s primary device as well.



  • @george1421 said in Intel Raid0 Image Capture:

    I can say that dealing with the hybrid software / hardware raid (fake raid) is kind of a pita.

    Yes it is.

    What you say when you reference /sda and /sdb is that you are talking to the individual disks of the array not the array. This is true. While I haven’t attempted to mess with this is the logical array referenced with /dev/md126 from above? I might expect the array itself to be something like /dev/md/{array0} or something similar.

    /dev/md126 is what as best I can tell is the Raid array. mdstat.txt in OP is the output of:

    cat /proc/mdstat
    

    while in debug - capture mode.

    I was hoping that if I switched to the multiple disks that it would look for the individual disks and image each of them instead of looking for the Raid array. But, it seems as though partclone is looking for the total number of blocks for the entire array and is trying to read/write all of them from/to sda1.


  • Moderator

    I can say that dealing with the hybrid software / hardware raid (fake raid) is kind of a pita.

    What you say when you reference /sda and /sdb is that you are talking to the individual disks of the array not the array. This is true. While I haven’t attempted to mess with this is the logical array referenced with /dev/md126 from above? I might expect the array itself to be something like /dev/md/{array0} or something similar.


Log in to reply
 

273
Online

7.1k
Users

14.3k
Topics

135.1k
Posts