Posts made by david.burgess

@george1421 said in FOS fails to capture image from md RAID host:

since you referenced my tutorial I assumed you are using the intel raid.

I referenced your Intel RAID tutorial because I found another forum post on mdadm where you linked to it and stated that it was similar in principle. I am actually working on capturing an image from a Linux host with md RAID, and deploying the same image to other hosts with identical hardware.

By way of update, I removed the large sda device (a hardware RAID) and the USB stick, so my disk layout was thus (from the running host OS):

# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda         8:0    0 139.8G  0 disk  
├─sda1      8:1    0   300M  0 part  
├─sda2      8:2    0    30G  0 part  
│ └─md0     9:0    0    30G  0 raid1 /
└─sda3      8:3    0     2G  0 part  
  └─md127   9:127  0     2G  0 raid1 
sdb         8:16   0 139.8G  0 disk  
├─sdb1      8:17   0   300M  0 part  /boot/efi
├─sdb2      8:18   0    30G  0 part  
│ └─md0     9:0    0    30G  0 raid1 /
└─sdb3      8:19   0     2G  0 part  
  └─md127   9:127  0     2G  0 raid1

I then configured the following in FOG, with the noted outcomes:

Host Primary Disk: /dev/md126 (equivalent to md0 on live system above)
Image Type: Multiple Partition Image - All Disks (3)
Host Kernel Arguments: mdraid=true
Outcome: fails to find disks

Host Primary Disk: /dev/md126 (equivalent to md0 on live system above)
Image Type: Multiple Partition Image - Single Disk (2)
Host Kernel Arguments: mdraid=true
Outcome: fails to read partition table

Host Primary Disk: /dev/sdb
Image Type: Multiple Partition Image - Single Disk (2)
Host Kernel Arguments: [not noted. I forget]
Outcome: fails to read partition table

Host Primary Disk: /dev/sda
Image Type: Multiple Partition Image - All Disks (3)
Host Kernel Arguments: [none]
Outcome: capture succeeds. Deployment on new system succeeds. System boots and partition table on sda looks correct. md127 is lost. Partition table on sdb is not correct:

# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda       8:0    0 139.8G  0 disk  
├─sda1    8:1    0   300M  0 part  
├─sda2    8:2    0    30G  0 part  
│ └─md0   9:0    0    30G  0 raid1 /
└─sda3    8:3    0     2G  0 part  
sdb       8:16   0 139.8G  0 disk  
├─sdb1    8:17   0     1G  0 part  
└─sdb2    8:18   0 138.8G  0 part

I’m not sure if the deployment procedure touched sdb or not. I suspect not. I will try again and watch closer or record it. I suppose I can rebuild my RAIDs from here, but I would of course prefer to have FOG automate that process to the extent possible.

Edit: added FOG settings detail.

@george1421

If I use the whole device for md-RAID, then partition the md device, this doesn’t prevent the OS from seeing the underlying physical devices. It will still see sdb, sdc and md126. I’m not familiar with Intel RAID on Linux, but it sounds from your description like the result is similar to what I’m seeing. In any case, I don’t know of any way to present the md-RAID device to Linux and hide the member physical devices.

At this point I have removed sda and disabled USB for now until I can get somebody to remove the USB drive from the system. The unfortunate side effect is that I have no keyboard input in FOS with USB disabled (over IPMI, and presumably on the physical console as well). So I will retry the capture job in non-debug mode and see what happens.

db

@george1421

I can see how mirroring two disks and then partitioning the RAID device would be simpler, but we have ruled it out as an option because we want to overprovision the SSDs.

We can remove the USB drive and probably the large unpartitioned drive (sda) as well, just leaving us with sdb, sdc, md126 and md127, but it sounds like we may still not have success. I think I will try it anyway just to be thorough.

db

@nadachowski said in FOG and different types of hardware:

FOG goes into a loop.

Please be as specific as possible. Can you provide screenshots or pictures?

Ubuntu 18.04.6 LTS
Web Server 10.13.6.7

I’m trying to capture an image from a host that boots UEFI from two hard disks in RAID 1. The way this works is that both disks have an EFI partition that is kept in sync by the OS. The computer boots from either of these disks and then mdadm assembles the root partition from the second partition on these two disks.

I followed the handy guide here and determined that FOS sees my root partition as /dev/md126. On FOG’s host management screen I set the Host Kernel Arguments to mdraid=true and Host Primary Disk to /dev/md126. I have tried setting the image type to each of the first three values.

In every case, attempts to capture the image have resulted in an error in reading the disk or partition table. I hope the output from one such attempt will prove helpful in identifying the source of my troubles and a possible solution:

[Tue Jun 14 root@fogclient ~]# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda         8:0    0 116.4T  0 disk  
sdb         8:16   0 139.8G  0 disk  
|-sdb1      8:17   0   300M  0 part  
|-sdb2      8:18   0    30G  0 part  
| `-md126   9:126  0    30G  0 raid1 
`-sdb3      8:19   0     2G  0 part  
  `-md127   9:127  0     2G  0 raid1 
sdc         8:32   0 139.8G  0 disk  
|-sdc1      8:33   0   300M  0 part  
|-sdc2      8:34   0    30G  0 part  
| `-md126   9:126  0    30G  0 raid1 
`-sdc3      8:35   0     2G  0 part  
  `-md127   9:127  0     2G  0 raid1 
sdd         8:48   1  14.3G  0 disk  
`-sdd1      8:49   1  14.3G  0 part  

[Tue Jun 14 root@fogclient ~]# mdadm -D /dev/md126
/dev/md126:
           Version : 1.2
     Creation Time : Wed May 25 20:32:17 2022
        Raid Level : raid1
        Array Size : 31439872 (29.98 GiB 32.19 GB)
     Used Dev Size : 31439872 (29.98 GiB 32.19 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Tue Jun 14 15:26:24 2022
             State : clean 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : ubuntu-server:0
              UUID : 7f76bb36:09c97d07:2528dfc0:ade215db
            Events : 222

    Number   Major   Minor   RaidDevice State
       0       8       34        0      active sync   /dev/sdc2
       1       8       18        1      active sync   /dev/sdb2

   ==================================
   ===        ====    =====      ====
   ===  =========  ==  ===   ==   ===
   ===  ========  ====  ==  ====  ===
   ===  ========  ====  ==  =========
   ===      ====  ====  ==  =========
   ===  ========  ====  ==  ===   ===
   ===  ========  ====  ==  ====  ===
   ===  =========  ==  ===   ==   ===
   ===  ==========    =====      ====
   ==================================
   ===== Free Opensource Ghost ======
   ==================================
   ============ Credits =============
   = https://fogproject.org/Credits =
   ==================================
   == Released under GPL Version 3 ==
   ==================================
   Version: 1.5.9
   Init Version: 20200906
 * Press [Enter] key to continue

 * Verifying network interface configuration.........Done
 * Press [Enter] key to continue

 * Checking Operating System.........................Linux
 * Checking CPU Cores................................40
 * Send method.......................................NFS
 * Attempting to check in............................
Done
 * Press [Enter] key to continue
 * Mounting File System..............................Done
 * Press [Enter] key to continue

 * Checking Mounted File System......................Done
 * Press [Enter] key to continue

 * Checking img variable is set......................Done
 * Press [Enter] key to continue

 * Preparing to send image file to server
 * Preparing backup location.........................Done
 * Press [Enter] key to continue

 * Setting permission on /images/0cc47abc024c........Done
 * Press [Enter] key to continue

 * Removing any pre-existing files...................Done
 * Press [Enter] key to continue

 * Using Image: veeam_barracuda_2022-06-14
 * Looking for Hard Disks............................Failed
 * Press [Enter] key to continue

##############################################################################
#                                                                            #
#                         An error has been detected!                        #
#                                                                            #
##############################################################################
Init Version: 20200906
Could not find any disks (/bin/fog.upload)
   Args Passed: 

Kernel variables and settings:
bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 web=http://10.13.6.7/fog/ consoleblank=0 rootfstype=ext4 mdraid=true nvme_core.default_ps_max_latency_us=0 mac=0c:c4:7a:bc:02:4c ftp=10.13.6.7 storage=10.13.6.7:/images/dev/ storageip=10.13.6.7 osid=50 irqpoll hostname=vcc-ldc-vs-11 chkdsk=0 img=veeam_barracuda_2022-06-14 imgType=mpa imgPartitionType=all imgid=2 imgFormat=5 PIGZ_COMP=-6 fdrive=/dev/md126 hostearly=1 pct=5 ignorepg=1 isdebug=yes type=up mdraid=true
 * Press [Enter] key to continue

Note also that /dev/sda is an empty filesystem and doesn’t need to be imaged. /dev/sdd is a USB stick and also unnecessary to the image. I don’t know if there’s a way to tell FOG to skip these devices.

I didn’t find a solution, but went with the workaround of designating a host primary disk, then creating a debug deploy task. Once booted, I was able to check nvme order. If correct, I ran ‘fog’ to deploy. If incorrect, I rebooted the host and tried again. Time consuming, but the outcome was correct.

FOG Web Server 10.15.9.4
Ubuntu 18.04.6 LTS

I’m deploying to a small number of hosts with identical hardware. These hosts have 3 nvme drives, which are detected as nvme0n1, nvme1n1 and nvme2n1. One of these is ~500GB and the other two are ~3TB:

Disk /dev/nvme0n1: 2.93 TiB, 3200631791616 bytes, 6251233968 sectors
Disk /dev/nvme1n1: 2.93 TiB, 3200631791616 bytes, 6251233968 sectors
Disk /dev/nvme2n1: 476.96 GiB, 512110190592 bytes, 1000215216 sectors

My objective is to deploy my image to the small nvme drive. The challenge is that each time I reboot, these drives are detected in random order, so designating the group or host primary disk does not deploy the image to the correct drive in all cases.

Is it possible to change the ‘hd’ variable on the fly, such as during a debug deploy task? Or is there a better approach? I could ask remote hands to disconnect the larger drives during deploy, but a software-based intervention is preferred.

@sebastian-roth said in Image capture: reading partition tables failed:

@david-burgess You might want to point to a specific Primary Disk within the host settings, e.g. /dev/sda

Thank you. This was the fix.

@george1421

This is helpful, thanks.

I have read a few tutorials about the post-download script, and maybe I’m just too tired to get it, but I am unclear on what path or binaries are available to the script.

To put my problem in context, I need to wipe the nvme drive in the client after the image is deployed. The OS (Ubuntu) will be installed on a SATA drive, so we’re just wiping the secondary drive and the only nvme device in the host.

So I’ve added these lines to fog.postdownload:

drive=`nvme list|awk -F'[/ ]' '/dev/ {print $3}'`
nvme format $drive

What’s not clear to me is the environment this will run in, and thus where nvme-cli needs to be installed. The host’s OS? FOG?

And here’s the output from a host that has not yet been imaged. I will image it and post back with the result.

[Fri Apr 22 root@fogclient ~]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 446.6G  0 disk 
nvme0n1     259:0    0   3.7T  0 disk 
|-nvme0n1p1 259:1    0   190M  0 part 
`-nvme0n1p2 259:2    0 372.5G  0 part 

[Fri Apr 22 root@fogclient ~]# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4c53] (rev 01)
00:01.0 PCI bridge [0604]: Intel Corporation Device [8086:4c01] (rev 01)
00:06.0 PCI bridge [0604]: Intel Corporation Device [8086:4c09] (rev 01)
00:14.0 USB controller [0c03]: Intel Corporation Device [8086:43ed] (rev 11)
00:14.2 RAM memory [0500]: Intel Corporation Device [8086:43ef] (rev 11)
00:16.0 Communication controller [0780]: Intel Corporation Device [8086:43e0] (rev 11)
00:16.4 Communication controller [0780]: Intel Corporation Device [8086:43e4] (rev 11)
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:43d2] (rev 11)
00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:43c0] (rev 11)
00:1b.6 PCI bridge [0604]: Intel Corporation Device [8086:43c6] (rev 11)
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:43b8] (rev 11)
00:1c.1 PCI bridge [0604]: Intel Corporation Device [8086:43b9] (rev 11)
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:438d] (rev 11)
00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:43a3] (rev 11)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:43a4] (rev 11)
01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation Device [1987:5018] (rev 01)
02:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3416 [1000:0015] (rev 01)
05:00.0 PCI bridge [0604]: PLDA Device [1556:be00] (rev 02)
06:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller [102b:0536] (rev 04)
07:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
07:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]

[Fri Apr 22 root@fogclient ~]# grep -i -e "firmware" /var/log/messages
Apr 22 14:37:57 fogclient user.info kernel: megaraid_sas 0000:02:00.0: firmware supports msix	: (128)
Apr 22 14:37:57 fogclient user.info kernel: megaraid_sas 0000:02:00.0: Current firmware supports maximum commands: 1517	 LDIO threshold: 0
Apr 22 14:37:57 fogclient user.info kernel: megaraid_sas 0000:02:00.0: Configured max firmware commands: 1516
Apr 22 14:37:57 fogclient user.info kernel: megaraid_sas 0000:02:00.0: firmware type	: Legacy(64 VD) firmware
Apr 22 14:37:57 fogclient user.info kernel: megaraid_sas 0000:02:00.0: firmware crash dump	: no

Thanks for the guidance. I have created and run a new capture task in debug mode on the source host. I will create a separate post for a target host. Here is the output of the requested commands:

[Thu Apr 21 root@fogclient ~]# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda       8:0    0 447.1G  0 disk 
|-sda1    8:1    0   190M  0 part 
`-sda2    8:2    0 372.5G  0 part 
nvme0n1 259:0    0   3.7T  0 disk 

[Thu Apr 21 root@fogclient ~]# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4c53] (rev 01)
00:01.0 PCI bridge [0604]: Intel Corporation Device [8086:4c01] (rev 01)
00:01.1 PCI bridge [0604]: Intel Corporation Device [8086:4c05] (rev 01)
00:14.0 USB controller [0c03]: Intel Corporation Device [8086:43ed] (rev 11)
00:14.2 RAM memory [0500]: Intel Corporation Device [8086:43ef] (rev 11)
00:16.0 Communication controller [0780]: Intel Corporation Device [8086:43e0] (rev 11)
00:16.4 Communication controller [0780]: Intel Corporation Device [8086:43e4] (rev 11)
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:43d2] (rev 11)
00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:43c0] (rev 11)
00:1b.6 PCI bridge [0604]: Intel Corporation Device [8086:43c6] (rev 11)
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:43b8] (rev 11)
00:1c.1 PCI bridge [0604]: Intel Corporation Device [8086:43b9] (rev 11)
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:438d] (rev 11)
00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:43a3] (rev 11)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:43a4] (rev 11)
01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation Device [1987:5018] (rev 01)
02:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3416 [1000:0015] (rev 01)
05:00.0 PCI bridge [0604]: PLDA Device [1556:be00] (rev 02)
06:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller [102b:0536] (rev 04)
07:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
07:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]

[Thu Apr 21 root@fogclient ~]# grep -i -e "firmware" /var/log/messages 
Apr 21 22:44:43 fogclient user.info kernel: megaraid_sas 0000:02:00.0: firmware supports msix	: (128)
Apr 21 22:44:43 fogclient user.info kernel: megaraid_sas 0000:02:00.0: Current firmware supports maximum commands: 1517	 LDIO threshold: 0
Apr 21 22:44:43 fogclient user.info kernel: megaraid_sas 0000:02:00.0: Configured max firmware commands: 1516
Apr 21 22:44:43 fogclient user.info kernel: megaraid_sas 0000:02:00.0: firmware type	: Legacy(64 VD) firmware
Apr 21 22:44:43 fogclient user.info kernel: megaraid_sas 0000:02:00.0: firmware crash dump	: no

I guess I spoke too soon. The image was pulled from a host that had the OS installed on a SATA SSD. The host also has an nvme drive that was unconfigured, but when the image is written to any host, including that from which the image was pulled, it is installed on the nvme drive. I will have to investigate further, but I suspect that FOG is not recognising the SATA drive on the PERC controller and is installing on the nvme drive as it is the only drive seen.

Any know workarounds for properly detecting and writing to the SATA drive? I find it odd that FOG can pull the image from the drive but apparently not write to it.

Well, wouldn’t you know it. I selected “Multiple Partition Image - All Disks (Not Resizable)” and it worked.

Fog server: Ubuntu 18.04.6
Client hardware: Dell PowerEdge R250
Client OS: Ubuntu 18.04.6

The client registered to the server fine, but while trying to capture an image I get this error (image below):

Reading Partition Tables...............Failed

For image partition type, I have tried Single Disk - Resizable, Multiple Partition Image - All Disks (Not Resizable), and Raw Image (Sector By Sector, DD, Slow). All three failed with exactly the same message. I did not try Multiple Partition Image - Single Disk (Not Resizable) as I did not believe it would succeed where the others had failed.

Previous to this, I was able to successfully capture an image from a virtual machine on which I had made a nearly identical install of Ubuntu. The only material difference I can think of is real versus virtual hardware.

My searches found results indicating that FOG doesn’t play well with RAID setups, however this PERC disk is a dingle SATA SSD configured in non-RAID mode. Could it be that FOG doesn’t play well with the PERC H345 controller in this server? I tried searching for results including ‘perc’ but came up with nothing relevant.

Any recommendations? If I can’t get this hardware configured in a way to work with FOG I will have to reluctantly find another solution, as I have 40 of these hosts to image.