Error Restoring GPT Partition Tables



  • (moving this to another thread so others might find it easier)

    I’m running an error stating that Restoring GPT Partition Tables failed (see pic).

    IMG_20190725_143623 (1).jpg

    This error is particularly annoying because it happens intermittently, not every time you try to image to a particular machine. This is on an HP Z2 G4 workstation with two M.2 drives installed. I have successfully imaged to these computers a few times already.

    Initial Reponse:

    We were able to track down an issue that sounds very similar last week. Find the details here: https://forums.fogproject.org/topic/13440/fog-1-5-6-auto-resize-is-unpredictable

    To try the fixed binaries download 64 Bit and 32 Bit and put those in /var/www/html/fog/service/ipxe/ dir on your FOG server. Probably good to rename the original ones instead of overwriting.


  • Developer

    @tlehrian Can you please run the following commands in a debug session on one of your machines:

    basename $(readlink /sys/block/nvme0n1/device)
    basename $(readlink /sys/block/nvme0n1/device/device)
    basename $(readlink /sys/block/nvme1n1/device)
    basename $(readlink /sys/block/nvme1n1/device/device)
    

    I borrowed that from this script. As well referenced in this topic.

    I am wondering if the so called BDF (bus/device/function) notation is consistent across reboots.

    Edit: Quite possibly the BDF will also change. Reading the section on “PCI Bus enumeration” in this wikipedia article I can imagine that it’s just the nature of those kind of devices.

    Edit 2: Hmmmmmmm: https://superuser.com/questions/488833/do-pci-pcie-buses-and-devices-always-enumerate-in-the-same-order (not sure if this applies to PCIe NVMe devices at all)


  • Developer

    @george1421 said in Error Restoring GPT Partition Tables:

    Its not something that only impacts FOG, but all distros of Linux.

    As @Quazz said, distros don’t have that issue because they mostly use UUIDs to identify partitions. Once the identifier is set and configured in your grub.conf/fstab there is no issue finding the right one again. But we can’t do that as we need to identify the whole disk, one that we possibly have never seen before (fresh machine).

    So I kind of understand why this topic is not being discussed in the Linux world too much. But I am still wondering why this is the case for NVMe drives and if there is a way to query the controller itself. Within FOG we can do a lot of things. We can even implement our very own low level tool in C to query that information for us.


  • Moderator

    @tlehrian Ok so this IS something that the Linux kernel developers are going to have to address. Its not something that only impacts FOG, but all distros of Linux.



  • @Quazz I was curious about the LiveCD thing, so I tried it. I can confirm that it DOES reorder the drives on successive LiveCD boots. lsblk switched the drive order after one successive boot.

    In fact I think I ran into this when first installing the OSes on these machines, and thought it was weird when it happened, but didn’t think much of it at the time.


  • Moderator

    @george1421 If you actually install Linux, it will generally mount it using the UUID, which should always be the same.

    As for the BIOS, the PCI path doesn’t change for NVME drives, just which one initalizes first can be volatile (and thus the block naming scheme in Linux gets messed up). BIOS looks for the boot file on the PCI path afaik.

    However, I’d be interested in seeing if this issue occurs on successive Ubuntu/Centos LiveCD boots


  • Moderator

    @tlehrian Thank you for taking the time to help us debug this. Without the hardware the developers rely on good quality feedback from you guys.

    Ok so we are down to firmware on the computer and firmware on the drives them selves (if that’s a thing).

    Something else I’m thinking about (just out loud at the moment) does a commercial linux distribution like ubuntu or centos do the same thing? If it does then it should fail to boot every second time or so if the drives are swapping position. If it boots correctly every time how do it do that? Just thinking about a computer when it cold boots it looks to the bios to find which drive to boot. The reference for uefi is disk/path/file. It doesn’t (at the point of cold boot) know anything about uuid so how does it find the boot drive? (assuming that it works every time)


  • Developer

    @tlehrian said in Error Restoring GPT Partition Tables:

    The initial issue was that we were receiving the error restoring GPT partition tables sometimes on these particular machines. We have two NVME M.2 drives on these machines, …

    Right yeah. I just totally lost the string here. When reading some of the messages I had the impression that there was another poblem causing this. No worries.



  • @george1421 I placed one of these machines in legacy mode to see if the drives would exhibit the same behavior and ran the tests you prescribed earlier. Indeed, after two reboots, they did switch, so we can probably rule BIOS type out. Here are the outputs:

    State 1:

    
    NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    nvme0n1     259:0    0 238.5G  0 disk
    nvme1n1     259:1    0   477G  0 disk
    |-nvme1n1p1 259:2    0   499M  0 part
    |-nvme1n1p2 259:3    0   100M  0 part
    |-nvme1n1p3 259:4    0    16M  0 part
    |-nvme1n1p4 259:5    0 341.2G  0 part
    `-nvme1n1p5 259:6    0 135.1G  0 part
    
    > nvme list
    
    Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
    ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
    /dev/nvme0n1     S499NX0M113634       SAMSUNG MZVLB256HAHQ-000H2               1           2.95  GB / 256.06  GB    512   B +  0 B   EXD71HAQ
    /dev/nvme1n1     S498NA0M403426       SAMSUNG MZVLB512HAJQ-000H2               1         149.49  GB / 512.11  GB    512   B +  0 B   EXA71HAQ
    
    > nvme id-ctrl /dev/nvme0n1 -H
    
    NVME Identify Controller:
    vid     : 0x144d
    ssvid   : 0x144d
    sn      : S499NX0M113634
    mn      : SAMSUNG MZVLB256HAHQ-000H2
    fr      : EXD71HAQ
    rab     : 2
    ieee    : 002538
    cmic    : 0
      [2:2] : 0     PCI
      [1:1] : 0     Single Controller
      [0:0] : 0     Single Port
    
    mdts    : 9
    cntlid  : 4
    ver     : 10200
    rtd3r   : 186a0
    rtd3e   : 7a1200
    oaes    : 0
      [8:8] : 0     Namespace Attribute Changed Event Not Supported
    
    oacs    : 0x17
     [15:4] : 0x1   Reserved
      [3:3] : 0     NS Management and Attachment Not Supported
      [2:2] : 0x1   FW Commit and Download Supported
      [1:1] : 0x1   Format NVM Supported
      [0:0] : 0x1   Sec. Send and Receive Supported
    
    acl     : 7
    aerl    : 7
    frmw    : 0x16
      [4:4] : 0x1   Firmware Activate Without Reset Supported
      [3:1] : 0x3   Number of Firmware Slots
      [0:0] : 0     Firmware Slot 1 Read/Write
    
    lpa     : 0x3
      [1:1] : 0x1   Command Effects Log Page Supported
      [0:0] : 0x1   SMART/Health Log Page per NS Supported
    
    elpe    : 255
    npss    : 4
    avscc   : 0x1
      [0:0] : 0x1   Admin Vendor Specific Commands uses NVMe Format
    
    apsta   : 0x1
      [0:0] : 0x1   Autonomous Power State Transitions Supported
    
    wctemp  : 354
    cctemp  : 355
    mtfa    : 50
    hmpre   : 0
    hmmin   : 0
    tnvmcap : 256060514304
    unvmcap : 0
    rpmbs   : 0
     [31:24]: 0     Access Size
     [23:16]: 0     Total Size
      [5:3] : 0     Authentication Method
      [2:0] : 0     Number of RPMB Units
    
    sqes    : 0x66
      [7:4] : 0x6   Max SQ Entry Size (64)
      [3:0] : 0x6   Min SQ Entry Size (64)
    
    cqes    : 0x44
      [7:4] : 0x4   Max CQ Entry Size (16)
      [3:0] : 0x4   Min CQ Entry Size (16)
    
    nn      : 1
    oncs    : 0x1f
      [5:5] : 0     Reservations Not Supported
      [4:4] : 0x1   Save and Select Supported
      [3:3] : 0x1   Write Zeroes Supported
      [2:2] : 0x1   Data Set Management Supported
      [1:1] : 0x1   Write Uncorrectable Supported
      [0:0] : 0x1   Compare Supported
    
    fuses   : 0
      [0:0] : 0     Fused Compare and Write Not Supported
    
    fna     : 0
      [2:2] : 0     Crypto Erase Not Supported as part of Secure Erase
      [1:1] : 0     Crypto Erase Applies to Single Namespace(s)
      [0:0] : 0     Format Applies to Single Namespace(s)
    
    vwc     : 0x1
      [0:0] : 0x1   Volatile Write Cache Present
    
    awun    : 1023
    awupf   : 0
    nvscc   : 1
      [0:0] : 0x1   NVM Vendor Specific Commands uses NVMe Format
    
    acwu    : 0
    sgls    : 0
      [0:0] : 0     Scatter-Gather Lists Not Supported
    
    subnqn  :
    ps    0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
              rwt:0 rwl:0 idle_power:- active_power:-
    ps    1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
              rwt:1 rwl:1 idle_power:- active_power:-
    ps    2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
              rwt:2 rwl:2 idle_power:- active_power:-
    ps    3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
              rwt:3 rwl:3 idle_power:- active_power:-
    ps    4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
              rwt:4 rwl:4 idle_power:- active_power:-
    
    nvme id-ctrl /dev/nvme1n1 -H
    NVME Identify Controller:
    vid     : 0x144d
    ssvid   : 0x144d
    sn      : S498NA0M403426
    mn      : SAMSUNG MZVLB512HAJQ-000H2
    fr      : EXA71HAQ
    rab     : 2
    ieee    : 002538
    cmic    : 0
      [2:2] : 0     PCI
      [1:1] : 0     Single Controller
      [0:0] : 0     Single Port
    
    mdts    : 9
    cntlid  : 4
    ver     : 10200
    rtd3r   : 186a0
    rtd3e   : 7a1200
    oaes    : 0
      [8:8] : 0     Namespace Attribute Changed Event Not Supported
    
    oacs    : 0x17
     [15:4] : 0x1   Reserved
      [3:3] : 0     NS Management and Attachment Not Supported
      [2:2] : 0x1   FW Commit and Download Supported
      [1:1] : 0x1   Format NVM Supported
      [0:0] : 0x1   Sec. Send and Receive Supported
    
    acl     : 7
    aerl    : 7
    frmw    : 0x16
      [4:4] : 0x1   Firmware Activate Without Reset Supported
      [3:1] : 0x3   Number of Firmware Slots
      [0:0] : 0     Firmware Slot 1 Read/Write
    
    lpa     : 0x3
      [1:1] : 0x1   Command Effects Log Page Supported
      [0:0] : 0x1   SMART/Health Log Page per NS Supported
    
    elpe    : 255
    npss    : 4
    avscc   : 0x1
      [0:0] : 0x1   Admin Vendor Specific Commands uses NVMe Format
    
    apsta   : 0x1
      [0:0] : 0x1   Autonomous Power State Transitions Supported
    
    wctemp  : 354
    cctemp  : 355
    mtfa    : 50
    hmpre   : 0
    hmmin   : 0
    tnvmcap : 512110190592
    unvmcap : 0
    rpmbs   : 0
     [31:24]: 0     Access Size
     [23:16]: 0     Total Size
      [5:3] : 0     Authentication Method
      [2:0] : 0     Number of RPMB Units
    
    sqes    : 0x66
      [7:4] : 0x6   Max SQ Entry Size (64)
      [3:0] : 0x6   Min SQ Entry Size (64)
    
    cqes    : 0x44
      [7:4] : 0x4   Max CQ Entry Size (16)
      [3:0] : 0x4   Min CQ Entry Size (16)
    
    nn      : 1
    oncs    : 0x1f
      [5:5] : 0     Reservations Not Supported
      [4:4] : 0x1   Save and Select Supported
      [3:3] : 0x1   Write Zeroes Supported
      [2:2] : 0x1   Data Set Management Supported
      [1:1] : 0x1   Write Uncorrectable Supported
      [0:0] : 0x1   Compare Supported
    
    fuses   : 0
      [0:0] : 0     Fused Compare and Write Not Supported
    
    fna     : 0
      [2:2] : 0     Crypto Erase Not Supported as part of Secure Erase
      [1:1] : 0     Crypto Erase Applies to Single Namespace(s)
      [0:0] : 0     Format Applies to Single Namespace(s)
    
    vwc     : 0x1
      [0:0] : 0x1   Volatile Write Cache Present
    
    awun    : 1023
    awupf   : 0
    nvscc   : 1
      [0:0] : 0x1   NVM Vendor Specific Commands uses NVMe Format
    
    acwu    : 0
    sgls    : 0
      [0:0] : 0     Scatter-Gather Lists Not Supported
    
    subnqn  :
    ps    0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
              rwt:0 rwl:0 idle_power:- active_power:-
    ps    1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
              rwt:1 rwl:1 idle_power:- active_power:-
    ps    2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
              rwt:2 rwl:2 idle_power:- active_power:-
    ps    3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
              rwt:3 rwl:3 idle_power:- active_power:-
    ps    4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
              rwt:4 rwl:4 idle_power:- active_power:-
    

    After one reboot, the lsblk command reversed the order of the listing, but still had the drives identified correctly:

    
    NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    nvme1n1     259:0    0   477G  0 disk
    |-nvme1n1p1 259:2    0   499M  0 part
    |-nvme1n1p2 259:3    0   100M  0 part
    |-nvme1n1p3 259:4    0    16M  0 part
    |-nvme1n1p4 259:5    0 341.2G  0 part
    `-nvme1n1p5 259:6    0 135.1G  0 part
    nvme0n1     259:1    0 238.5G  0 disk
    

    (I did not run the nvme commands in this state)

    After one more reboot, the drives switched:

    
    NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    nvme0n1     259:0    0   477G  0 disk
    |-nvme0n1p1 259:2    0   499M  0 part
    |-nvme0n1p2 259:3    0   100M  0 part
    |-nvme0n1p3 259:4    0    16M  0 part
    |-nvme0n1p4 259:5    0 341.2G  0 part
    `-nvme0n1p5 259:6    0 135.1G  0 part
    nvme1n1     259:1    0 238.5G  0 disk
    
    Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
    ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
    /dev/nvme0n1     S498NA0M403426       SAMSUNG MZVLB512HAJQ-000H2               1         149.49  GB / 512.11  GB    512   B +  0 B   EXA71HAQ
    /dev/nvme1n1     S499NX0M113634       SAMSUNG MZVLB256HAHQ-000H2               1           2.95  GB / 256.06  GB    512   B +  0 B   EXD71HAQ
    
    > nvme id-ctrl /dev/nvme0n1 -H
    
    NVME Identify Controller:
    vid     : 0x144d
    ssvid   : 0x144d
    sn      : S498NA0M403426
    mn      : SAMSUNG MZVLB512HAJQ-000H2
    fr      : EXA71HAQ
    rab     : 2
    ieee    : 002538
    cmic    : 0
      [2:2] : 0     PCI
      [1:1] : 0     Single Controller
      [0:0] : 0     Single Port
    
    mdts    : 9
    cntlid  : 4
    ver     : 10200
    rtd3r   : 186a0
    rtd3e   : 7a1200
    oaes    : 0
      [8:8] : 0     Namespace Attribute Changed Event Not Supported
    
    oacs    : 0x17
     [15:4] : 0x1   Reserved
      [3:3] : 0     NS Management and Attachment Not Supported
      [2:2] : 0x1   FW Commit and Download Supported
      [1:1] : 0x1   Format NVM Supported
      [0:0] : 0x1   Sec. Send and Receive Supported
    
    acl     : 7
    aerl    : 7
    frmw    : 0x16
      [4:4] : 0x1   Firmware Activate Without Reset Supported
      [3:1] : 0x3   Number of Firmware Slots
      [0:0] : 0     Firmware Slot 1 Read/Write
    
    lpa     : 0x3
      [1:1] : 0x1   Command Effects Log Page Supported
      [0:0] : 0x1   SMART/Health Log Page per NS Supported
    
    elpe    : 255
    npss    : 4
    avscc   : 0x1
      [0:0] : 0x1   Admin Vendor Specific Commands uses NVMe Format
    
    apsta   : 0x1
      [0:0] : 0x1   Autonomous Power State Transitions Supported
    
    wctemp  : 354
    cctemp  : 355
    mtfa    : 50
    hmpre   : 0
    hmmin   : 0
    tnvmcap : 512110190592
    unvmcap : 0
    rpmbs   : 0
     [31:24]: 0     Access Size
     [23:16]: 0     Total Size
      [5:3] : 0     Authentication Method
      [2:0] : 0     Number of RPMB Units
    
    sqes    : 0x66
      [7:4] : 0x6   Max SQ Entry Size (64)
      [3:0] : 0x6   Min SQ Entry Size (64)
    
    cqes    : 0x44
      [7:4] : 0x4   Max CQ Entry Size (16)
      [3:0] : 0x4   Min CQ Entry Size (16)
    
    nn      : 1
    oncs    : 0x1f
      [5:5] : 0     Reservations Not Supported
      [4:4] : 0x1   Save and Select Supported
      [3:3] : 0x1   Write Zeroes Supported
      [2:2] : 0x1   Data Set Management Supported
      [1:1] : 0x1   Write Uncorrectable Supported
      [0:0] : 0x1   Compare Supported
    
    fuses   : 0
      [0:0] : 0     Fused Compare and Write Not Supported
    
    fna     : 0
      [2:2] : 0     Crypto Erase Not Supported as part of Secure Erase
      [1:1] : 0     Crypto Erase Applies to Single Namespace(s)
      [0:0] : 0     Format Applies to Single Namespace(s)
    
    vwc     : 0x1
      [0:0] : 0x1   Volatile Write Cache Present
    
    awun    : 1023
    awupf   : 0
    nvscc   : 1
      [0:0] : 0x1   NVM Vendor Specific Commands uses NVMe Format
    
    acwu    : 0
    sgls    : 0
      [0:0] : 0     Scatter-Gather Lists Not Supported
    
    subnqn  :
    ps    0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
              rwt:0 rwl:0 idle_power:- active_power:-
    ps    1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
              rwt:1 rwl:1 idle_power:- active_power:-
    ps    2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
              rwt:2 rwl:2 idle_power:- active_power:-
    ps    3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
              rwt:3 rwl:3 idle_power:- active_power:-
    ps    4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
              rwt:4 rwl:4 idle_power:- active_power:-
    
    > nvme id-ctrl /dev/nvme1n1 -H
    
    NVME Identify Controller:
    vid     : 0x144d
    ssvid   : 0x144d
    sn      : S499NX0M113634
    mn      : SAMSUNG MZVLB256HAHQ-000H2
    fr      : EXD71HAQ
    rab     : 2
    ieee    : 002538
    cmic    : 0
      [2:2] : 0     PCI
      [1:1] : 0     Single Controller
      [0:0] : 0     Single Port
    
    mdts    : 9
    cntlid  : 4
    ver     : 10200
    rtd3r   : 186a0
    rtd3e   : 7a1200
    oaes    : 0
      [8:8] : 0     Namespace Attribute Changed Event Not Supported
    
    oacs    : 0x17
     [15:4] : 0x1   Reserved
      [3:3] : 0     NS Management and Attachment Not Supported
      [2:2] : 0x1   FW Commit and Download Supported
      [1:1] : 0x1   Format NVM Supported
      [0:0] : 0x1   Sec. Send and Receive Supported
    
    acl     : 7
    aerl    : 7
    frmw    : 0x16
      [4:4] : 0x1   Firmware Activate Without Reset Supported
      [3:1] : 0x3   Number of Firmware Slots
      [0:0] : 0     Firmware Slot 1 Read/Write
    
    lpa     : 0x3
      [1:1] : 0x1   Command Effects Log Page Supported
      [0:0] : 0x1   SMART/Health Log Page per NS Supported
    
    elpe    : 255
    npss    : 4
    avscc   : 0x1
      [0:0] : 0x1   Admin Vendor Specific Commands uses NVMe Format
    
    apsta   : 0x1
      [0:0] : 0x1   Autonomous Power State Transitions Supported
    
    wctemp  : 354
    cctemp  : 355
    mtfa    : 50
    hmpre   : 0
    hmmin   : 0
    tnvmcap : 256060514304
    unvmcap : 0
    rpmbs   : 0
     [31:24]: 0     Access Size
     [23:16]: 0     Total Size
      [5:3] : 0     Authentication Method
      [2:0] : 0     Number of RPMB Units
    
    sqes    : 0x66
      [7:4] : 0x6   Max SQ Entry Size (64)
      [3:0] : 0x6   Min SQ Entry Size (64)
    
    cqes    : 0x44
      [7:4] : 0x4   Max CQ Entry Size (16)
      [3:0] : 0x4   Min CQ Entry Size (16)
    
    nn      : 1
    oncs    : 0x1f
      [5:5] : 0     Reservations Not Supported
      [4:4] : 0x1   Save and Select Supported
      [3:3] : 0x1   Write Zeroes Supported
      [2:2] : 0x1   Data Set Management Supported
      [1:1] : 0x1   Write Uncorrectable Supported
      [0:0] : 0x1   Compare Supported
    
    fuses   : 0
      [0:0] : 0     Fused Compare and Write Not Supported
    
    fna     : 0
      [2:2] : 0     Crypto Erase Not Supported as part of Secure Erase
      [1:1] : 0     Crypto Erase Applies to Single Namespace(s)
      [0:0] : 0     Format Applies to Single Namespace(s)
    
    vwc     : 0x1
      [0:0] : 0x1   Volatile Write Cache Present
    
    awun    : 1023
    awupf   : 0
    nvscc   : 1
      [0:0] : 0x1   NVM Vendor Specific Commands uses NVMe Format
    
    acwu    : 0
    sgls    : 0
      [0:0] : 0     Scatter-Gather Lists Not Supported
    
    subnqn  :
    ps    0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
              rwt:0 rwl:0 idle_power:- active_power:-
    ps    1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
              rwt:1 rwl:1 idle_power:- active_power:-
    ps    2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
              rwt:2 rwl:2 idle_power:- active_power:-
    ps    3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
              rwt:3 rwl:3 idle_power:- active_power:-
    ps    4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
              rwt:4 rwl:4 idle_power:- active_power:-
    


  • @george1421 No worries. Just making sure there wasn’t something you knew that I didn’t :)


  • Moderator

    @tlehrian said in Error Restoring GPT Partition Tables:

    Are either of those companies related to Lenovo in some way?

    Sorry I’m working on too many threads at the moment. I got the threads mixed up with similar issues.



  • @george1421 Sure. I understand about narrowing down the possibilities. I can switch one of these to legacy and see what happens with the drives. To be clear, these are HP workstations with Samsung NVME M.2 drives. Are either of those companies related to Lenovo in some way?



  • @Sebastian-Roth Thanks for asking!

    The initial issue was that we were receiving the error restoring GPT partition tables sometimes on these particular machines. We have two NVME M.2 drives on these machines, that we now know are becoming available seemingly randomly based upon a race condition as to which one becomes available to the OS first. Our drives happen to be different sizes (256GB/512GB), and we want the larger drive to be the primary drive with the OS installed, and the smaller drive to be an extra storage drive. I have set up an image as Multiple Partition - All Disks (not resizeable) to deploy to these machines. Also, these machines are booting UEFI, without secure boot.

    If the correct drive wins the race, the image goes through swimmingly. If it doesn’t, I get the error that started this thread as the drive it’s trying to restore the partition tables to is not large enough to hold the image.

    So, long story short, no. But that’s OK because now I know what’s causing the issue, and although it’s a bit annoying, I can work around it knowing that it’s not a larger hardware issue. And I’m happy to help troubleshoot to try to get to a resolution for it, as it would benefit me greatly as this setup composes about 25% of my lab deployments at present.


  • Developer

    @tlehrian Is the inital issue solved at all? I seem to have lost track of that part of the discussion. ;-)


  • Moderator

    @tlehrian What I’m trying to rule out is if its a Lenovo thing that is causing the drives to switch, I don’t think it is, but I wanted to rule out hardware (or stale firmware) as the culprit.

    As for the question about bios (legacy) mode. I wanted to see with the same physical hardware, is the device swapping location related to the uefi firmware to see if it does the same thing in bios mode.

    Right now I’m looking at this as I don’t know where the issue is. So I’m trying to rule out where the problem isn’t first to get the number of possibilities down.



  • @george1421 I forgot I still had one of these up and running. It looks like the firmware for the MOBO is not at latest version, and the BIOS is also not at latest version. Not sure about NVME firmware. I think this is the 981 which is an OEM version of the Samsung 970, but not sure if the same firmware applies.



  • @george1421 Ok. This is on an HP Z2 G4 workstation.

    1. and

    2. I’m not sure about the firmware, but will check and let you know which firmware was installed. This will probably be tomorrow as there is a scheduled power outage for our campus tomorrow morning and we’ve shut these machines down for the day in preparation.

    3. it took 2 reboots to switch, and did not require a hard reboot.

    4. The BIOS does have legacy mode…I have not tried this in legacy mode, mainly as we are moving our dual-boot setup using GRUB2 to EFI on boot. I’ve never booted this machine in legacy mode. Secure boot is disabled.


  • Moderator

    @george1421 After reading up a bit more, it seems the conclusion is that it’s not so much about the disks themselves, but rather that the assignment is arbitrary if they’re connected to different controllers, which in the case of NVME will always be the case and in the case of SATA (on most modern consumer grade hardware at least) will not be the case.


  • Moderator

    @tlehrian Ok I have a few more tasks for you, well kind of.

    1. Does this lenovo computer have the latest firmware updates installed
    2. Does this lenovo computer’s nvme drives have the latest firmware installed? (you should be able to get disk firmware for this model of computer from the lenovo site)
    3. How many times of rebooting did it take to get the drive order to flip? Did you have to do a hard boot to get them to flip or was just a reboot enough?
    4. Does this laptop have legacy (bios) mode? If it does, do these drives change order in bios mode?

    So right now its not clear in my head if its a hardware, firmware (bios), or linux kernel error.


  • Moderator

    @george1421 Created an issue over at github

    https://github.com/FOGProject/fos/issues/27

    This one will need some consideration I think, not really straightforward


Log in to reply
 

424
Online

6.1k
Users

13.4k
Topics

126.3k
Posts