Error Restoring GPT Partition Tables
-
@george1421 I placed one of these machines in legacy mode to see if the drives would exhibit the same behavior and ran the tests you prescribed earlier. Indeed, after two reboots, they did switch, so we can probably rule BIOS type out. Here are the outputs:
State 1:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 238.5G 0 disk nvme1n1 259:1 0 477G 0 disk |-nvme1n1p1 259:2 0 499M 0 part |-nvme1n1p2 259:3 0 100M 0 part |-nvme1n1p3 259:4 0 16M 0 part |-nvme1n1p4 259:5 0 341.2G 0 part `-nvme1n1p5 259:6 0 135.1G 0 part > nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 S499NX0M113634 SAMSUNG MZVLB256HAHQ-000H2 1 2.95 GB / 256.06 GB 512 B + 0 B EXD71HAQ /dev/nvme1n1 S498NA0M403426 SAMSUNG MZVLB512HAJQ-000H2 1 149.49 GB / 512.11 GB 512 B + 0 B EXA71HAQ > nvme id-ctrl /dev/nvme0n1 -H NVME Identify Controller: vid : 0x144d ssvid : 0x144d sn : S499NX0M113634 mn : SAMSUNG MZVLB256HAHQ-000H2 fr : EXD71HAQ rab : 2 ieee : 002538 cmic : 0 [2:2] : 0 PCI [1:1] : 0 Single Controller [0:0] : 0 Single Port mdts : 9 cntlid : 4 ver : 10200 rtd3r : 186a0 rtd3e : 7a1200 oaes : 0 [8:8] : 0 Namespace Attribute Changed Event Not Supported oacs : 0x17 [15:4] : 0x1 Reserved [3:3] : 0 NS Management and Attachment Not Supported [2:2] : 0x1 FW Commit and Download Supported [1:1] : 0x1 Format NVM Supported [0:0] : 0x1 Sec. Send and Receive Supported acl : 7 aerl : 7 frmw : 0x16 [4:4] : 0x1 Firmware Activate Without Reset Supported [3:1] : 0x3 Number of Firmware Slots [0:0] : 0 Firmware Slot 1 Read/Write lpa : 0x3 [1:1] : 0x1 Command Effects Log Page Supported [0:0] : 0x1 SMART/Health Log Page per NS Supported elpe : 255 npss : 4 avscc : 0x1 [0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format apsta : 0x1 [0:0] : 0x1 Autonomous Power State Transitions Supported wctemp : 354 cctemp : 355 mtfa : 50 hmpre : 0 hmmin : 0 tnvmcap : 256060514304 unvmcap : 0 rpmbs : 0 [31:24]: 0 Access Size [23:16]: 0 Total Size [5:3] : 0 Authentication Method [2:0] : 0 Number of RPMB Units sqes : 0x66 [7:4] : 0x6 Max SQ Entry Size (64) [3:0] : 0x6 Min SQ Entry Size (64) cqes : 0x44 [7:4] : 0x4 Max CQ Entry Size (16) [3:0] : 0x4 Min CQ Entry Size (16) nn : 1 oncs : 0x1f [5:5] : 0 Reservations Not Supported [4:4] : 0x1 Save and Select Supported [3:3] : 0x1 Write Zeroes Supported [2:2] : 0x1 Data Set Management Supported [1:1] : 0x1 Write Uncorrectable Supported [0:0] : 0x1 Compare Supported fuses : 0 [0:0] : 0 Fused Compare and Write Not Supported fna : 0 [2:2] : 0 Crypto Erase Not Supported as part of Secure Erase [1:1] : 0 Crypto Erase Applies to Single Namespace(s) [0:0] : 0 Format Applies to Single Namespace(s) vwc : 0x1 [0:0] : 0x1 Volatile Write Cache Present awun : 1023 awupf : 0 nvscc : 1 [0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format acwu : 0 sgls : 0 [0:0] : 0 Scatter-Gather Lists Not Supported subnqn : ps 0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:- active_power:- ps 1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1 rwt:1 rwl:1 idle_power:- active_power:- ps 2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2 rwt:2 rwl:2 idle_power:- active_power:- ps 3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3 rwt:3 rwl:3 idle_power:- active_power:- ps 4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4 rwt:4 rwl:4 idle_power:- active_power:- nvme id-ctrl /dev/nvme1n1 -H NVME Identify Controller: vid : 0x144d ssvid : 0x144d sn : S498NA0M403426 mn : SAMSUNG MZVLB512HAJQ-000H2 fr : EXA71HAQ rab : 2 ieee : 002538 cmic : 0 [2:2] : 0 PCI [1:1] : 0 Single Controller [0:0] : 0 Single Port mdts : 9 cntlid : 4 ver : 10200 rtd3r : 186a0 rtd3e : 7a1200 oaes : 0 [8:8] : 0 Namespace Attribute Changed Event Not Supported oacs : 0x17 [15:4] : 0x1 Reserved [3:3] : 0 NS Management and Attachment Not Supported [2:2] : 0x1 FW Commit and Download Supported [1:1] : 0x1 Format NVM Supported [0:0] : 0x1 Sec. Send and Receive Supported acl : 7 aerl : 7 frmw : 0x16 [4:4] : 0x1 Firmware Activate Without Reset Supported [3:1] : 0x3 Number of Firmware Slots [0:0] : 0 Firmware Slot 1 Read/Write lpa : 0x3 [1:1] : 0x1 Command Effects Log Page Supported [0:0] : 0x1 SMART/Health Log Page per NS Supported elpe : 255 npss : 4 avscc : 0x1 [0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format apsta : 0x1 [0:0] : 0x1 Autonomous Power State Transitions Supported wctemp : 354 cctemp : 355 mtfa : 50 hmpre : 0 hmmin : 0 tnvmcap : 512110190592 unvmcap : 0 rpmbs : 0 [31:24]: 0 Access Size [23:16]: 0 Total Size [5:3] : 0 Authentication Method [2:0] : 0 Number of RPMB Units sqes : 0x66 [7:4] : 0x6 Max SQ Entry Size (64) [3:0] : 0x6 Min SQ Entry Size (64) cqes : 0x44 [7:4] : 0x4 Max CQ Entry Size (16) [3:0] : 0x4 Min CQ Entry Size (16) nn : 1 oncs : 0x1f [5:5] : 0 Reservations Not Supported [4:4] : 0x1 Save and Select Supported [3:3] : 0x1 Write Zeroes Supported [2:2] : 0x1 Data Set Management Supported [1:1] : 0x1 Write Uncorrectable Supported [0:0] : 0x1 Compare Supported fuses : 0 [0:0] : 0 Fused Compare and Write Not Supported fna : 0 [2:2] : 0 Crypto Erase Not Supported as part of Secure Erase [1:1] : 0 Crypto Erase Applies to Single Namespace(s) [0:0] : 0 Format Applies to Single Namespace(s) vwc : 0x1 [0:0] : 0x1 Volatile Write Cache Present awun : 1023 awupf : 0 nvscc : 1 [0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format acwu : 0 sgls : 0 [0:0] : 0 Scatter-Gather Lists Not Supported subnqn : ps 0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:- active_power:- ps 1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1 rwt:1 rwl:1 idle_power:- active_power:- ps 2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2 rwt:2 rwl:2 idle_power:- active_power:- ps 3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3 rwt:3 rwl:3 idle_power:- active_power:- ps 4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4 rwt:4 rwl:4 idle_power:- active_power:-After one reboot, the
lsblkcommand reversed the order of the listing, but still had the drives identified correctly:NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme1n1 259:0 0 477G 0 disk |-nvme1n1p1 259:2 0 499M 0 part |-nvme1n1p2 259:3 0 100M 0 part |-nvme1n1p3 259:4 0 16M 0 part |-nvme1n1p4 259:5 0 341.2G 0 part `-nvme1n1p5 259:6 0 135.1G 0 part nvme0n1 259:1 0 238.5G 0 disk(I did not run the nvme commands in this state)
After one more reboot, the drives switched:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 477G 0 disk |-nvme0n1p1 259:2 0 499M 0 part |-nvme0n1p2 259:3 0 100M 0 part |-nvme0n1p3 259:4 0 16M 0 part |-nvme0n1p4 259:5 0 341.2G 0 part `-nvme0n1p5 259:6 0 135.1G 0 part nvme1n1 259:1 0 238.5G 0 disk Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 S498NA0M403426 SAMSUNG MZVLB512HAJQ-000H2 1 149.49 GB / 512.11 GB 512 B + 0 B EXA71HAQ /dev/nvme1n1 S499NX0M113634 SAMSUNG MZVLB256HAHQ-000H2 1 2.95 GB / 256.06 GB 512 B + 0 B EXD71HAQ > nvme id-ctrl /dev/nvme0n1 -H NVME Identify Controller: vid : 0x144d ssvid : 0x144d sn : S498NA0M403426 mn : SAMSUNG MZVLB512HAJQ-000H2 fr : EXA71HAQ rab : 2 ieee : 002538 cmic : 0 [2:2] : 0 PCI [1:1] : 0 Single Controller [0:0] : 0 Single Port mdts : 9 cntlid : 4 ver : 10200 rtd3r : 186a0 rtd3e : 7a1200 oaes : 0 [8:8] : 0 Namespace Attribute Changed Event Not Supported oacs : 0x17 [15:4] : 0x1 Reserved [3:3] : 0 NS Management and Attachment Not Supported [2:2] : 0x1 FW Commit and Download Supported [1:1] : 0x1 Format NVM Supported [0:0] : 0x1 Sec. Send and Receive Supported acl : 7 aerl : 7 frmw : 0x16 [4:4] : 0x1 Firmware Activate Without Reset Supported [3:1] : 0x3 Number of Firmware Slots [0:0] : 0 Firmware Slot 1 Read/Write lpa : 0x3 [1:1] : 0x1 Command Effects Log Page Supported [0:0] : 0x1 SMART/Health Log Page per NS Supported elpe : 255 npss : 4 avscc : 0x1 [0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format apsta : 0x1 [0:0] : 0x1 Autonomous Power State Transitions Supported wctemp : 354 cctemp : 355 mtfa : 50 hmpre : 0 hmmin : 0 tnvmcap : 512110190592 unvmcap : 0 rpmbs : 0 [31:24]: 0 Access Size [23:16]: 0 Total Size [5:3] : 0 Authentication Method [2:0] : 0 Number of RPMB Units sqes : 0x66 [7:4] : 0x6 Max SQ Entry Size (64) [3:0] : 0x6 Min SQ Entry Size (64) cqes : 0x44 [7:4] : 0x4 Max CQ Entry Size (16) [3:0] : 0x4 Min CQ Entry Size (16) nn : 1 oncs : 0x1f [5:5] : 0 Reservations Not Supported [4:4] : 0x1 Save and Select Supported [3:3] : 0x1 Write Zeroes Supported [2:2] : 0x1 Data Set Management Supported [1:1] : 0x1 Write Uncorrectable Supported [0:0] : 0x1 Compare Supported fuses : 0 [0:0] : 0 Fused Compare and Write Not Supported fna : 0 [2:2] : 0 Crypto Erase Not Supported as part of Secure Erase [1:1] : 0 Crypto Erase Applies to Single Namespace(s) [0:0] : 0 Format Applies to Single Namespace(s) vwc : 0x1 [0:0] : 0x1 Volatile Write Cache Present awun : 1023 awupf : 0 nvscc : 1 [0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format acwu : 0 sgls : 0 [0:0] : 0 Scatter-Gather Lists Not Supported subnqn : ps 0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:- active_power:- ps 1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1 rwt:1 rwl:1 idle_power:- active_power:- ps 2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2 rwt:2 rwl:2 idle_power:- active_power:- ps 3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3 rwt:3 rwl:3 idle_power:- active_power:- ps 4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4 rwt:4 rwl:4 idle_power:- active_power:- > nvme id-ctrl /dev/nvme1n1 -H NVME Identify Controller: vid : 0x144d ssvid : 0x144d sn : S499NX0M113634 mn : SAMSUNG MZVLB256HAHQ-000H2 fr : EXD71HAQ rab : 2 ieee : 002538 cmic : 0 [2:2] : 0 PCI [1:1] : 0 Single Controller [0:0] : 0 Single Port mdts : 9 cntlid : 4 ver : 10200 rtd3r : 186a0 rtd3e : 7a1200 oaes : 0 [8:8] : 0 Namespace Attribute Changed Event Not Supported oacs : 0x17 [15:4] : 0x1 Reserved [3:3] : 0 NS Management and Attachment Not Supported [2:2] : 0x1 FW Commit and Download Supported [1:1] : 0x1 Format NVM Supported [0:0] : 0x1 Sec. Send and Receive Supported acl : 7 aerl : 7 frmw : 0x16 [4:4] : 0x1 Firmware Activate Without Reset Supported [3:1] : 0x3 Number of Firmware Slots [0:0] : 0 Firmware Slot 1 Read/Write lpa : 0x3 [1:1] : 0x1 Command Effects Log Page Supported [0:0] : 0x1 SMART/Health Log Page per NS Supported elpe : 255 npss : 4 avscc : 0x1 [0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format apsta : 0x1 [0:0] : 0x1 Autonomous Power State Transitions Supported wctemp : 354 cctemp : 355 mtfa : 50 hmpre : 0 hmmin : 0 tnvmcap : 256060514304 unvmcap : 0 rpmbs : 0 [31:24]: 0 Access Size [23:16]: 0 Total Size [5:3] : 0 Authentication Method [2:0] : 0 Number of RPMB Units sqes : 0x66 [7:4] : 0x6 Max SQ Entry Size (64) [3:0] : 0x6 Min SQ Entry Size (64) cqes : 0x44 [7:4] : 0x4 Max CQ Entry Size (16) [3:0] : 0x4 Min CQ Entry Size (16) nn : 1 oncs : 0x1f [5:5] : 0 Reservations Not Supported [4:4] : 0x1 Save and Select Supported [3:3] : 0x1 Write Zeroes Supported [2:2] : 0x1 Data Set Management Supported [1:1] : 0x1 Write Uncorrectable Supported [0:0] : 0x1 Compare Supported fuses : 0 [0:0] : 0 Fused Compare and Write Not Supported fna : 0 [2:2] : 0 Crypto Erase Not Supported as part of Secure Erase [1:1] : 0 Crypto Erase Applies to Single Namespace(s) [0:0] : 0 Format Applies to Single Namespace(s) vwc : 0x1 [0:0] : 0x1 Volatile Write Cache Present awun : 1023 awupf : 0 nvscc : 1 [0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format acwu : 0 sgls : 0 [0:0] : 0 Scatter-Gather Lists Not Supported subnqn : ps 0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:- active_power:- ps 1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1 rwt:1 rwl:1 idle_power:- active_power:- ps 2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2 rwt:2 rwl:2 idle_power:- active_power:- ps 3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3 rwt:3 rwl:3 idle_power:- active_power:- ps 4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4 rwt:4 rwl:4 idle_power:- active_power:- -
@tlehrian said in Error Restoring GPT Partition Tables:
The initial issue was that we were receiving the error restoring GPT partition tables sometimes on these particular machines. We have two NVME M.2 drives on these machines, …
Right yeah. I just totally lost the string here. When reading some of the messages I had the impression that there was another poblem causing this. No worries.
-
@tlehrian Thank you for taking the time to help us debug this. Without the hardware the developers rely on good quality feedback from you guys.
Ok so we are down to firmware on the computer and firmware on the drives them selves (if that’s a thing).
Something else I’m thinking about (just out loud at the moment) does a commercial linux distribution like ubuntu or centos do the same thing? If it does then it should fail to boot every second time or so if the drives are swapping position. If it boots correctly every time how do it do that? Just thinking about a computer when it cold boots it looks to the bios to find which drive to boot. The reference for uefi is disk/path/file. It doesn’t (at the point of cold boot) know anything about uuid so how does it find the boot drive? (assuming that it works every time)
-
@george1421 If you actually install Linux, it will generally mount it using the UUID, which should always be the same.
As for the BIOS, the PCI path doesn’t change for NVME drives, just which one initalizes first can be volatile (and thus the block naming scheme in Linux gets messed up). BIOS looks for the boot file on the PCI path afaik.
However, I’d be interested in seeing if this issue occurs on successive Ubuntu/Centos LiveCD boots
-
@Quazz I was curious about the LiveCD thing, so I tried it. I can confirm that it DOES reorder the drives on successive LiveCD boots.
lsblkswitched the drive order after one successive boot.In fact I think I ran into this when first installing the OSes on these machines, and thought it was weird when it happened, but didn’t think much of it at the time.
-
@tlehrian Ok so this IS something that the Linux kernel developers are going to have to address. Its not something that only impacts FOG, but all distros of Linux.
-
@george1421 said in Error Restoring GPT Partition Tables:
Its not something that only impacts FOG, but all distros of Linux.
As @Quazz said, distros don’t have that issue because they mostly use UUIDs to identify partitions. Once the identifier is set and configured in your grub.conf/fstab there is no issue finding the right one again. But we can’t do that as we need to identify the whole disk, one that we possibly have never seen before (fresh machine).
So I kind of understand why this topic is not being discussed in the Linux world too much. But I am still wondering why this is the case for NVMe drives and if there is a way to query the controller itself. Within FOG we can do a lot of things. We can even implement our very own low level tool in C to query that information for us.
-
@tlehrian Can you please run the following commands in a debug session on one of your machines:
basename $(readlink /sys/block/nvme0n1/device) basename $(readlink /sys/block/nvme0n1/device/device) basename $(readlink /sys/block/nvme1n1/device) basename $(readlink /sys/block/nvme1n1/device/device)I borrowed that from this script. As well referenced in this topic.
I am wondering if the so called BDF (bus/device/function) notation is consistent across reboots.
Edit: Quite possibly the BDF will also change. Reading the section on “PCI Bus enumeration” in this wikipedia article I can imagine that it’s just the nature of those kind of devices.
Edit 2: Hmmmmmmm: https://superuser.com/questions/488833/do-pci-pcie-buses-and-devices-always-enumerate-in-the-same-order (not sure if this applies to PCIe NVMe devices at all)
-
@Sebastian-Roth I’m sorry it took me so long to get back to this, but I was able to run the testing this morning. It does appear that the device names stay consistent, even though the drives may change order in
lsblk. Here are my results:State 1
> lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme1n1 259:0 0 477G 0 disk |-nvme1n1p1 259:2 0 499M 0 part |-nvme1n1p2 259:3 0 100M 0 part |-nvme1n1p3 259:4 0 16M 0 part |-nvme1n1p4 259:5 0 341.2G 0 part `-nvme1n1p5 259:6 0 135.1G 0 part nvme0n1 259:1 0 238.5G 0 disk > basename $(readlink /sys/block/nvme0n1/device) nvme0 > basename $(readlink /sys/block/nvme0n1/device/device) 0000:02:00.0 > basename $(readlink /sys/block/nvme1n1/device) nvme1 > basename $(readlink /sys/block/nvme1n1/device/device) 0000:03:00.0State 2
> lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 477G 0 disk |-nvme0n1p1 259:2 0 499M 0 part |-nvme0n1p2 259:3 0 100M 0 part |-nvme0n1p3 259:4 0 16M 0 part |-nvme0n1p4 259:5 0 341.2G 0 part `-nvme0n1p5 259:6 0 135.1G 0 part nvme1n1 259:1 0 238.5G 0 disk > basename $(readlink /sys/block/nvme0n1/device) nvme1 > basename $(readlink /sys/block/nvme0n1/device/device) 0000:03:00.0 > basename $(readlink /sys/block/nvme1n1/device) nvme0 > basename $(readlink /sys/block/nvme1n1/device/device) 0000:02:00.0Note how the
deviceanddevice/devicenames remain unchanged per disk. -
I am busy getting ready for our Fall semester to start next week, BUT…
I will keep a debug task set up for one of our machines in order to do more testing for you all if need be. I’m glad to be part of the solution here, and I promise not to take as long to respond back to a testing request next time…
-
@tlehrian Just wanted to let you know that I still have this on my list but just don’t get to it.
-
I’m just checking back on this (rather old) topic to see if any headway had been made. It seems the last tests I ran indicated that the
basenamecommands did indicate the device names remained consistent even when the OS switches them. I won’t need to image these again for a while, but would be nice to see if a fix is available.Thanks!
Tim -
@tlehrian Great you are bringing this topic up again. Even though I try to keep track of all open topics I still miss one or the other sometimes.
Which version of the FOS inits do you currently use?
-
@Sebastian-Roth Looks like we’re still on 4.19.48. I have not needed to image labs since August, and don’t expect to need to do any large-scale imaging before Spring, but could certainly update and test with a newer kernel if need be.
Tim
-
@tlehrian It’s not the kernel version I am after but the version of the init file. Unfortunately there is not as easy a method to find which version you have. We added a versioning that is printed on the console when an error happens but this hasn’t been added long ago.
I may ask you do download the latest inits (from our jenkins repo) and see if you still run into the same issue. I have worked on that part a bit a couple of days ago.
-
Oh, OK. I guess I misread that. I’ll check out the new inits and see what happens, but probably won’t hit this again hard until January. Thanks for the response!
-
@tlehrian It would be really great if you could test this fairly soon so we have a little bit of time in case it needs further work. The earlier we know the better! We can’t guarantee a quick fix some time in January when you really need it.
