Posts made by tlehrian

@Sebastian-Roth Thanks for asking!

The initial issue was that we were receiving the error restoring GPT partition tables sometimes on these particular machines. We have two NVME M.2 drives on these machines, that we now know are becoming available seemingly randomly based upon a race condition as to which one becomes available to the OS first. Our drives happen to be different sizes (256GB/512GB), and we want the larger drive to be the primary drive with the OS installed, and the smaller drive to be an extra storage drive. I have set up an image as Multiple Partition - All Disks (not resizeable) to deploy to these machines. Also, these machines are booting UEFI, without secure boot.

If the correct drive wins the race, the image goes through swimmingly. If it doesn’t, I get the error that started this thread as the drive it’s trying to restore the partition tables to is not large enough to hold the image.

So, long story short, no. But that’s OK because now I know what’s causing the issue, and although it’s a bit annoying, I can work around it knowing that it’s not a larger hardware issue. And I’m happy to help troubleshoot to try to get to a resolution for it, as it would benefit me greatly as this setup composes about 25% of my lab deployments at present.

@george1421 I forgot I still had one of these up and running. It looks like the firmware for the MOBO is not at latest version, and the BIOS is also not at latest version. Not sure about NVME firmware. I think this is the 981 which is an OEM version of the Samsung 970, but not sure if the same firmware applies.

@george1421 Ok. This is on an HP Z2 G4 workstation.

1. and

2. I’m not sure about the firmware, but will check and let you know which firmware was installed. This will probably be tomorrow as there is a scheduled power outage for our campus tomorrow morning and we’ve shut these machines down for the day in preparation.

3. it took 2 reboots to switch, and did not require a hard reboot.

4. The BIOS does have legacy mode…I have not tried this in legacy mode, mainly as we are moving our dual-boot setup using GRUB2 to EFI on boot. I’ve never booted this machine in legacy mode. Secure boot is disabled.

@george1421 Cool. Glad I could help. Let me know if/when you need any other testing done.

@george1421 And from State 2 (reversed):

> nvme list

Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     S498NA0M403426       SAMSUNG MZVLB512HAJQ-000H2               1         149.49  GB / 512.11  GB    512   B +  0 B   EXA71HAQ
/dev/nvme1n1     S499NX0M113634       SAMSUNG MZVLB256HAHQ-000H2               1           2.95  GB / 256.06  GB    512   B +  0 B   EXD71HAQ

> nvme id-ctrl /dev/nvme0n1 -H

NVME Identify Controller:
vid     : 0x144d
ssvid   : 0x144d
sn      : S498NA0M403426
mn      : SAMSUNG MZVLB512HAJQ-000H2
fr      : EXA71HAQ
rab     : 2
ieee    : 002538
cmic    : 0
  [2:2] : 0     PCI
  [1:1] : 0     Single Controller
  [0:0] : 0     Single Port

mdts    : 9
cntlid  : 4
ver     : 10200
rtd3r   : 186a0
rtd3e   : 7a1200
oaes    : 0
  [8:8] : 0     Namespace Attribute Changed Event Not Supported

oacs    : 0x17
 [15:4] : 0x1   Reserved
  [3:3] : 0     NS Management and Attachment Not Supported
  [2:2] : 0x1   FW Commit and Download Supported
  [1:1] : 0x1   Format NVM Supported
  [0:0] : 0x1   Sec. Send and Receive Supported

acl     : 7
aerl    : 7
frmw    : 0x16
  [4:4] : 0x1   Firmware Activate Without Reset Supported
  [3:1] : 0x3   Number of Firmware Slots
  [0:0] : 0     Firmware Slot 1 Read/Write

lpa     : 0x3
  [1:1] : 0x1   Command Effects Log Page Supported
  [0:0] : 0x1   SMART/Health Log Page per NS Supported

elpe    : 255
npss    : 4
avscc   : 0x1
  [0:0] : 0x1   Admin Vendor Specific Commands uses NVMe Format

apsta   : 0x1
  [0:0] : 0x1   Autonomous Power State Transitions Supported

wctemp  : 354
cctemp  : 355
mtfa    : 50
hmpre   : 0
hmmin   : 0
tnvmcap : 512110190592
unvmcap : 0
rpmbs   : 0
 [31:24]: 0     Access Size
 [23:16]: 0     Total Size
  [5:3] : 0     Authentication Method
  [2:0] : 0     Number of RPMB Units

sqes    : 0x66
  [7:4] : 0x6   Max SQ Entry Size (64)
  [3:0] : 0x6   Min SQ Entry Size (64)

cqes    : 0x44
  [7:4] : 0x4   Max CQ Entry Size (16)
  [3:0] : 0x4   Min CQ Entry Size (16)

nn      : 1
oncs    : 0x1f
  [5:5] : 0     Reservations Not Supported
  [4:4] : 0x1   Save and Select Supported
  [3:3] : 0x1   Write Zeroes Supported
  [2:2] : 0x1   Data Set Management Supported
  [1:1] : 0x1   Write Uncorrectable Supported
  [0:0] : 0x1   Compare Supported

fuses   : 0
  [0:0] : 0     Fused Compare and Write Not Supported

fna     : 0
  [2:2] : 0     Crypto Erase Not Supported as part of Secure Erase
  [1:1] : 0     Crypto Erase Applies to Single Namespace(s)
  [0:0] : 0     Format Applies to Single Namespace(s)

vwc     : 0x1
  [0:0] : 0x1   Volatile Write Cache Present

awun    : 1023
awupf   : 0
nvscc   : 1
  [0:0] : 0x1   NVM Vendor Specific Commands uses NVMe Format

acwu    : 0
sgls    : 0
  [0:0] : 0     Scatter-Gather Lists Not Supported

subnqn  :
ps    0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-
ps    1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
          rwt:1 rwl:1 idle_power:- active_power:-
ps    2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
          rwt:2 rwl:2 idle_power:- active_power:-
ps    3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
          rwt:3 rwl:3 idle_power:- active_power:-
ps    4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
          rwt:4 rwl:4 idle_power:- active_power:-

> nvme id-ctrl /dev/nvme1n1 -H

NVME Identify Controller:
vid     : 0x144d
ssvid   : 0x144d
sn      : S499NX0M113634
mn      : SAMSUNG MZVLB256HAHQ-000H2
fr      : EXD71HAQ
rab     : 2
ieee    : 002538
cmic    : 0
  [2:2] : 0     PCI
  [1:1] : 0     Single Controller
  [0:0] : 0     Single Port

mdts    : 9
cntlid  : 4
ver     : 10200
rtd3r   : 186a0
rtd3e   : 7a1200
oaes    : 0
  [8:8] : 0     Namespace Attribute Changed Event Not Supported

oacs    : 0x17
 [15:4] : 0x1   Reserved
  [3:3] : 0     NS Management and Attachment Not Supported
  [2:2] : 0x1   FW Commit and Download Supported
  [1:1] : 0x1   Format NVM Supported
  [0:0] : 0x1   Sec. Send and Receive Supported

acl     : 7
aerl    : 7
frmw    : 0x16
  [4:4] : 0x1   Firmware Activate Without Reset Supported
  [3:1] : 0x3   Number of Firmware Slots
  [0:0] : 0     Firmware Slot 1 Read/Write

lpa     : 0x3
  [1:1] : 0x1   Command Effects Log Page Supported
  [0:0] : 0x1   SMART/Health Log Page per NS Supported

elpe    : 255
npss    : 4
avscc   : 0x1
  [0:0] : 0x1   Admin Vendor Specific Commands uses NVMe Format

apsta   : 0x1
  [0:0] : 0x1   Autonomous Power State Transitions Supported

wctemp  : 354
cctemp  : 355
mtfa    : 50
hmpre   : 0
hmmin   : 0
tnvmcap : 256060514304
unvmcap : 0
rpmbs   : 0
 [31:24]: 0     Access Size
 [23:16]: 0     Total Size
  [5:3] : 0     Authentication Method
  [2:0] : 0     Number of RPMB Units

sqes    : 0x66
  [7:4] : 0x6   Max SQ Entry Size (64)
  [3:0] : 0x6   Min SQ Entry Size (64)

cqes    : 0x44
  [7:4] : 0x4   Max CQ Entry Size (16)
  [3:0] : 0x4   Min CQ Entry Size (16)

nn      : 1
oncs    : 0x1f
  [5:5] : 0     Reservations Not Supported
  [4:4] : 0x1   Save and Select Supported
  [3:3] : 0x1   Write Zeroes Supported
  [2:2] : 0x1   Data Set Management Supported
  [1:1] : 0x1   Write Uncorrectable Supported
  [0:0] : 0x1   Compare Supported

fuses   : 0
  [0:0] : 0     Fused Compare and Write Not Supported

fna     : 0
  [2:2] : 0     Crypto Erase Not Supported as part of Secure Erase
  [1:1] : 0     Crypto Erase Applies to Single Namespace(s)
  [0:0] : 0     Format Applies to Single Namespace(s)

vwc     : 0x1
  [0:0] : 0x1   Volatile Write Cache Present

awun    : 1023
awupf   : 0
nvscc   : 1
  [0:0] : 0x1   NVM Vendor Specific Commands uses NVMe Format

acwu    : 0
sgls    : 0
  [0:0] : 0     Scatter-Gather Lists Not Supported

subnqn  :
ps    0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-
ps    1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
          rwt:1 rwl:1 idle_power:- active_power:-
ps    2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
          rwt:2 rwl:2 idle_power:- active_power:-
ps    3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
          rwt:3 rwl:3 idle_power:- active_power:-
ps    4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
          rwt:4 rwl:4 idle_power:- active_power:-

@george1421 Here is the information from State 1:

State 1:
> nvme list

Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     S499NX0M113634       SAMSUNG MZVLB256HAHQ-000H2               1           2.95  GB / 256.06  GB    512   B +  0 B   EXD71HAQ
/dev/nvme1n1     S498NA0M403426       SAMSUNG MZVLB512HAJQ-000H2               1         149.49  GB / 512.11  GB    512   B +  0 B   EXA71HAQ


> nvme id-ctrl /dev/nvme0n1 -H

NVME Identify Controller:
vid     : 0x144d
ssvid   : 0x144d
sn      : S499NX0M113634
mn      : SAMSUNG MZVLB256HAHQ-000H2
fr      : EXD71HAQ
rab     : 2
ieee    : 002538
cmic    : 0
  [2:2] : 0     PCI
  [1:1] : 0     Single Controller
  [0:0] : 0     Single Port

mdts    : 9
cntlid  : 4
ver     : 10200
rtd3r   : 186a0
rtd3e   : 7a1200
oaes    : 0
  [8:8] : 0     Namespace Attribute Changed Event Not Supported

oacs    : 0x17
 [15:4] : 0x1   Reserved
  [3:3] : 0     NS Management and Attachment Not Supported
  [2:2] : 0x1   FW Commit and Download Supported
  [1:1] : 0x1   Format NVM Supported
  [0:0] : 0x1   Sec. Send and Receive Supported

acl     : 7
aerl    : 7
frmw    : 0x16
  [4:4] : 0x1   Firmware Activate Without Reset Supported
  [3:1] : 0x3   Number of Firmware Slots
  [0:0] : 0     Firmware Slot 1 Read/Write

lpa     : 0x3
  [1:1] : 0x1   Command Effects Log Page Supported
  [0:0] : 0x1   SMART/Health Log Page per NS Supported

elpe    : 255
npss    : 4
avscc   : 0x1
  [0:0] : 0x1   Admin Vendor Specific Commands uses NVMe Format

apsta   : 0x1
  [0:0] : 0x1   Autonomous Power State Transitions Supported

wctemp  : 354
cctemp  : 355
mtfa    : 50
hmpre   : 0
hmmin   : 0
tnvmcap : 256060514304
unvmcap : 0
rpmbs   : 0
 [31:24]: 0     Access Size
 [23:16]: 0     Total Size
  [5:3] : 0     Authentication Method
  [2:0] : 0     Number of RPMB Units

sqes    : 0x66
  [7:4] : 0x6   Max SQ Entry Size (64)
  [3:0] : 0x6   Min SQ Entry Size (64)

cqes    : 0x44
  [7:4] : 0x4   Max CQ Entry Size (16)
  [3:0] : 0x4   Min CQ Entry Size (16)

nn      : 1
oncs    : 0x1f
  [5:5] : 0     Reservations Not Supported
  [4:4] : 0x1   Save and Select Supported
  [3:3] : 0x1   Write Zeroes Supported
  [2:2] : 0x1   Data Set Management Supported
  [1:1] : 0x1   Write Uncorrectable Supported
  [0:0] : 0x1   Compare Supported

fuses   : 0
  [0:0] : 0     Fused Compare and Write Not Supported

fna     : 0
  [2:2] : 0     Crypto Erase Not Supported as part of Secure Erase
  [1:1] : 0     Crypto Erase Applies to Single Namespace(s)
  [0:0] : 0     Format Applies to Single Namespace(s)

vwc     : 0x1
  [0:0] : 0x1   Volatile Write Cache Present

awun    : 1023
awupf   : 0
nvscc   : 1
  [0:0] : 0x1   NVM Vendor Specific Commands uses NVMe Format

acwu    : 0
sgls    : 0
  [0:0] : 0     Scatter-Gather Lists Not Supported

subnqn  :
ps    0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-
ps    1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
          rwt:1 rwl:1 idle_power:- active_power:-
ps    2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
          rwt:2 rwl:2 idle_power:- active_power:-
ps    3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
          rwt:3 rwl:3 idle_power:- active_power:-
ps    4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
          rwt:4 rwl:4 idle_power:- active_power:-

> nvme id-ctrl /dev/nvme1n1 -H

NVME Identify Controller:
vid     : 0x144d
ssvid   : 0x144d
sn      : S498NA0M403426
mn      : SAMSUNG MZVLB512HAJQ-000H2
fr      : EXA71HAQ
rab     : 2
ieee    : 002538
cmic    : 0
  [2:2] : 0     PCI
  [1:1] : 0     Single Controller
  [0:0] : 0     Single Port

mdts    : 9
cntlid  : 4
ver     : 10200
rtd3r   : 186a0
rtd3e   : 7a1200
oaes    : 0
  [8:8] : 0     Namespace Attribute Changed Event Not Supported

oacs    : 0x17
 [15:4] : 0x1   Reserved
  [3:3] : 0     NS Management and Attachment Not Supported
  [2:2] : 0x1   FW Commit and Download Supported
  [1:1] : 0x1   Format NVM Supported
  [0:0] : 0x1   Sec. Send and Receive Supported

acl     : 7
aerl    : 7
frmw    : 0x16
  [4:4] : 0x1   Firmware Activate Without Reset Supported
  [3:1] : 0x3   Number of Firmware Slots
  [0:0] : 0     Firmware Slot 1 Read/Write

lpa     : 0x3
  [1:1] : 0x1   Command Effects Log Page Supported
  [0:0] : 0x1   SMART/Health Log Page per NS Supported

elpe    : 255
npss    : 4
avscc   : 0x1
  [0:0] : 0x1   Admin Vendor Specific Commands uses NVMe Format

apsta   : 0x1
  [0:0] : 0x1   Autonomous Power State Transitions Supported

wctemp  : 354
cctemp  : 355
mtfa    : 50
hmpre   : 0
hmmin   : 0
tnvmcap : 512110190592
unvmcap : 0
rpmbs   : 0
 [31:24]: 0     Access Size
 [23:16]: 0     Total Size
  [5:3] : 0     Authentication Method
  [2:0] : 0     Number of RPMB Units

sqes    : 0x66
  [7:4] : 0x6   Max SQ Entry Size (64)
  [3:0] : 0x6   Min SQ Entry Size (64)

cqes    : 0x44
  [7:4] : 0x4   Max CQ Entry Size (16)
  [3:0] : 0x4   Min CQ Entry Size (16)

nn      : 1
oncs    : 0x1f
  [5:5] : 0     Reservations Not Supported
  [4:4] : 0x1   Save and Select Supported
  [3:3] : 0x1   Write Zeroes Supported
  [2:2] : 0x1   Data Set Management Supported
  [1:1] : 0x1   Write Uncorrectable Supported
  [0:0] : 0x1   Compare Supported

fuses   : 0
  [0:0] : 0     Fused Compare and Write Not Supported

fna     : 0
  [2:2] : 0     Crypto Erase Not Supported as part of Secure Erase
  [1:1] : 0     Crypto Erase Applies to Single Namespace(s)
  [0:0] : 0     Format Applies to Single Namespace(s)

vwc     : 0x1
  [0:0] : 0x1   Volatile Write Cache Present

awun    : 1023
awupf   : 0
nvscc   : 1
  [0:0] : 0x1   NVM Vendor Specific Commands uses NVMe Format

acwu    : 0
sgls    : 0
  [0:0] : 0     Scatter-Gather Lists Not Supported

subnqn  :
ps    0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-
ps    1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
          rwt:1 rwl:1 idle_power:- active_power:-
ps    2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
          rwt:2 rwl:2 idle_power:- active_power:-
ps    3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
          rwt:3 rwl:3 idle_power:- active_power:-
ps    4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
          rwt:4 rwl:4 idle_power:- active_power:-

@george1421 Ok. I should have a chance to do this later today. I’m hopeful this leads to a fix for the issue.

@george1421 Sure, I’d be happy to help where I can. After reading through the thread @Quazz shared, I guess I’m lucky that we’re trying to use the large NVME drive as our OS drive…

@george1421 Possibly. Unfortunately I have a small window of opportunity here where I could help test before we need to have all of our systems with two M2 drives up and running for our Fall semester (starting Aug 27). At that point, I’d no longer have any systems to play with.

@Quazz Hmmmm… very interesting. I guess I’m OK that it seems to be a random thing. I can keeping booting until it finally, randomly, picks the right drive to be the primary. At least it doesn’t seem to be a larger HW issue. I’ll be looking for a solution to this issue if one does eventually pop up. Thanks for the quick reply.

@Quazz Do you know of any solution to the issue?

@Sebastian-Roth The interesting thing is, the process seems to work every once in a while. I’m suspecting maybe the drives are being switched around and the process is trying to use the 256GB drive as the primary OS drive, when it won’t be large enough. Is there some sort of random or race condition that might cause that? I see an option in the host setup for “Host Primary Disk” - would that alleviate this issue?

@Sebastian-Roth Still throwing the same error (original error…restoring the GPT partition tables, not the kernel panic).

FYI the image is set up as a Multiple partition image - all disks (Not resizeable). The computer contains two M.2 drives: 1 512GB with several partitions for boot/OS (dual boot Linux/Windows), and another 256GB with two paritions (one ext4 and another ntfs). The OS on the image is set to Linux-50.

@Sebastian-Roth Perfect. Thanks for the quick reply. Will do that.

After downloading the init files linked to previously, I get a kernel panic when attempting to image:

The ACPI error has been cropping up before, but has never impeded an image. The kernel panic is new to these init files. Any ideas?

Thanks
Tim

I have downloaded the init files and will be trying another deploy in this lab today.

(moving this to another thread so others might find it easier)

I’m running an error stating that Restoring GPT Partition Tables failed (see pic).

This error is particularly annoying because it happens intermittently, not every time you try to image to a particular machine. This is on an HP Z2 G4 workstation with two M.2 drives installed. I have successfully imaged to these computers a few times already.

Initial Reponse:

We were able to track down an issue that sounds very similar last week. Find the details here: https://forums.fogproject.org/topic/13440/fog-1-5-6-auto-resize-is-unpredictable

To try the fixed binaries download 64 Bit and 32 Bit and put those in /var/www/html/fog/service/ipxe/ dir on your FOG server. Probably good to rename the original ones instead of overwriting.

@Sebastian-Roth So attached is the multicast.log file after restarting the Multicast service on the server. multicastlog.txt

I’m running into another error that Restoring GPT Partition Tables failed (see pic).

This error is particularly annoying because it happens intermittently, not every time you try to image to a particular machine. This is on an HP Z2 G4 workstation with two M.2 drives installed. I have successfully imaged to these computers a few times already.

@Sebastian-Roth I have had opportunity to start another multicast via Groups, and it did not work. This time, in the multicast.log, there are entries saying “This is not the master node.” What’s going on? I have made no configuration changes to the fog server.

@Sebastian-Roth I just wanted to follow up with this from last week. I subsequently tried running a multicast session set up via the Groups (with another group), and it worked. Twice. I’m not sure what caused the hiccup on the one last week. I’m willing to let this go as an anomaly for now, but will follow up on this if I run into the issue again.