@george1421 I replaced RAM with new, and now I get a BSOD: CLOCK_WATCHDOG_TIMEOUT which indicates it might be the processor.
This computer has been a thorn in our side - we have replaced the motherboard, and so it’s possible the processor didn’t survive the move. (or it went bad with the previous mobo).
I’m satisfied that this is not a FOG issue, but a hardware issue, so I think we can call this issue closed. Thanks for the help!
Tim
@george1421 So, I did update to BIOS version 01.08.08, but that still resulted in a Windows install BSOD:
IRQ_NOT_LESS_THAN_OR_EQUAL
followed by a reboot and another BSOD:
KMODE_EXCEPTION_NOT_HANDLED
Both of these point to either bad memory or device drivers. I’ll keep nosing around there.
@george1421 I thought I’d see if I could just install Windows on this computer without deploying an image to it. It gave me a BSOD. I did end up seeing the BIOS version was old, so I updated it to 01.08.08. Windows install BSOD again.
Memory and NVME hardware tests all come back OK. I have already tried swapping memory.
A successful deploy of this image on another computer indicates that it is not an issue with the image itself or with our server.
Any ideas what the cryptic messaging is saying? Which hardware caused this?
Thanks
Tim
I’m in the process of trying to deploy the same image on another computer with known good hardware to try to rule out any issues with the image itself.
I’m trying to image a computer that we fixed (replaced several components), but halfway through the imaging process I get a kernel panic, like the following:
Closeup of text:
I’m not sure whether the cause is hardware or on the fog server side.
Imaging HP Z2 G4
Fog version 1.5.10
bzImage and bzImage32 are both on version 6.1.63 (I originally tried the stock 5.15.93 but upgraded in case that was the issue).
Any ideas?
@george1421 Thanks again for this. I’ll tuck this away for later if I need it.
@george1421 Thanks so much! So, in the interim I decided to update FOG to 1.5.10 and now it works 
I supposed that upgrade also updated the iPXE file?
We have some newer HP Z2 G9 computers that are failing at DHCP in PXE. They get the correct IP address at first, then inside iPXE they fail (time out). It takes a long time to get the first address before iPXE. I’ve talked to our IT Dept and they insist it’s not a spanning tree protocol issue (they use MSTP). We have other computers that do not have this issue. Any ideas?
Screen shot of what’s happening:
I recently ran across this same issue, but it was not due to the fast spanning tree protocols. Our IT department added 802.1x authentication on the wired ports of all our switches, but the time out to fail back to MAC authentication was just longer than the 27 seconds @george1421 referenced in his post. We had a handful of hosts work, but a large majority failed. Our IT department had to shorten the 802.1X authentication timeout in order to make this work for us.
Oh, OK. I guess I misread that. I’ll check out the new inits and see what happens, but probably won’t hit this again hard until January. Thanks for the response!
@Sebastian-Roth Looks like we’re still on 4.19.48. I have not needed to image labs since August, and don’t expect to need to do any large-scale imaging before Spring, but could certainly update and test with a newer kernel if need be.
Tim
I’m just checking back on this (rather old) topic to see if any headway had been made. It seems the last tests I ran indicated that the basename commands did indicate the device names remained consistent even when the OS switches them. I won’t need to image these again for a while, but would be nice to see if a fix is available.
Thanks!
Tim
I am busy getting ready for our Fall semester to start next week, BUT…
I will keep a debug task set up for one of our machines in order to do more testing for you all if need be. I’m glad to be part of the solution here, and I promise not to take as long to respond back to a testing request next time…
@Sebastian-Roth I’m sorry it took me so long to get back to this, but I was able to run the testing this morning. It does appear that the device names stay consistent, even though the drives may change order in lsblk. Here are my results:
State 1
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1 259:0 0 477G 0 disk
|-nvme1n1p1 259:2 0 499M 0 part
|-nvme1n1p2 259:3 0 100M 0 part
|-nvme1n1p3 259:4 0 16M 0 part
|-nvme1n1p4 259:5 0 341.2G 0 part
`-nvme1n1p5 259:6 0 135.1G 0 part
nvme0n1 259:1 0 238.5G 0 disk
> basename $(readlink /sys/block/nvme0n1/device)
nvme0
> basename $(readlink /sys/block/nvme0n1/device/device)
0000:02:00.0
> basename $(readlink /sys/block/nvme1n1/device)
nvme1
> basename $(readlink /sys/block/nvme1n1/device/device)
0000:03:00.0
State 2
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 477G 0 disk
|-nvme0n1p1 259:2 0 499M 0 part
|-nvme0n1p2 259:3 0 100M 0 part
|-nvme0n1p3 259:4 0 16M 0 part
|-nvme0n1p4 259:5 0 341.2G 0 part
`-nvme0n1p5 259:6 0 135.1G 0 part
nvme1n1 259:1 0 238.5G 0 disk
> basename $(readlink /sys/block/nvme0n1/device)
nvme1
> basename $(readlink /sys/block/nvme0n1/device/device)
0000:03:00.0
> basename $(readlink /sys/block/nvme1n1/device)
nvme0
> basename $(readlink /sys/block/nvme1n1/device/device)
0000:02:00.0
Note how the device and device/device names remain unchanged per disk.
@Quazz I was curious about the LiveCD thing, so I tried it. I can confirm that it DOES reorder the drives on successive LiveCD boots. lsblk switched the drive order after one successive boot.
In fact I think I ran into this when first installing the OSes on these machines, and thought it was weird when it happened, but didn’t think much of it at the time.
@george1421 I placed one of these machines in legacy mode to see if the drives would exhibit the same behavior and ran the tests you prescribed earlier. Indeed, after two reboots, they did switch, so we can probably rule BIOS type out. Here are the outputs:
State 1:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 238.5G 0 disk
nvme1n1 259:1 0 477G 0 disk
|-nvme1n1p1 259:2 0 499M 0 part
|-nvme1n1p2 259:3 0 100M 0 part
|-nvme1n1p3 259:4 0 16M 0 part
|-nvme1n1p4 259:5 0 341.2G 0 part
`-nvme1n1p5 259:6 0 135.1G 0 part
> nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 S499NX0M113634 SAMSUNG MZVLB256HAHQ-000H2 1 2.95 GB / 256.06 GB 512 B + 0 B EXD71HAQ
/dev/nvme1n1 S498NA0M403426 SAMSUNG MZVLB512HAJQ-000H2 1 149.49 GB / 512.11 GB 512 B + 0 B EXA71HAQ
> nvme id-ctrl /dev/nvme0n1 -H
NVME Identify Controller:
vid : 0x144d
ssvid : 0x144d
sn : S499NX0M113634
mn : SAMSUNG MZVLB256HAHQ-000H2
fr : EXD71HAQ
rab : 2
ieee : 002538
cmic : 0
[2:2] : 0 PCI
[1:1] : 0 Single Controller
[0:0] : 0 Single Port
mdts : 9
cntlid : 4
ver : 10200
rtd3r : 186a0
rtd3e : 7a1200
oaes : 0
[8:8] : 0 Namespace Attribute Changed Event Not Supported
oacs : 0x17
[15:4] : 0x1 Reserved
[3:3] : 0 NS Management and Attachment Not Supported
[2:2] : 0x1 FW Commit and Download Supported
[1:1] : 0x1 Format NVM Supported
[0:0] : 0x1 Sec. Send and Receive Supported
acl : 7
aerl : 7
frmw : 0x16
[4:4] : 0x1 Firmware Activate Without Reset Supported
[3:1] : 0x3 Number of Firmware Slots
[0:0] : 0 Firmware Slot 1 Read/Write
lpa : 0x3
[1:1] : 0x1 Command Effects Log Page Supported
[0:0] : 0x1 SMART/Health Log Page per NS Supported
elpe : 255
npss : 4
avscc : 0x1
[0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format
apsta : 0x1
[0:0] : 0x1 Autonomous Power State Transitions Supported
wctemp : 354
cctemp : 355
mtfa : 50
hmpre : 0
hmmin : 0
tnvmcap : 256060514304
unvmcap : 0
rpmbs : 0
[31:24]: 0 Access Size
[23:16]: 0 Total Size
[5:3] : 0 Authentication Method
[2:0] : 0 Number of RPMB Units
sqes : 0x66
[7:4] : 0x6 Max SQ Entry Size (64)
[3:0] : 0x6 Min SQ Entry Size (64)
cqes : 0x44
[7:4] : 0x4 Max CQ Entry Size (16)
[3:0] : 0x4 Min CQ Entry Size (16)
nn : 1
oncs : 0x1f
[5:5] : 0 Reservations Not Supported
[4:4] : 0x1 Save and Select Supported
[3:3] : 0x1 Write Zeroes Supported
[2:2] : 0x1 Data Set Management Supported
[1:1] : 0x1 Write Uncorrectable Supported
[0:0] : 0x1 Compare Supported
fuses : 0
[0:0] : 0 Fused Compare and Write Not Supported
fna : 0
[2:2] : 0 Crypto Erase Not Supported as part of Secure Erase
[1:1] : 0 Crypto Erase Applies to Single Namespace(s)
[0:0] : 0 Format Applies to Single Namespace(s)
vwc : 0x1
[0:0] : 0x1 Volatile Write Cache Present
awun : 1023
awupf : 0
nvscc : 1
[0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format
acwu : 0
sgls : 0
[0:0] : 0 Scatter-Gather Lists Not Supported
subnqn :
ps 0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
ps 1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
rwt:1 rwl:1 idle_power:- active_power:-
ps 2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
rwt:2 rwl:2 idle_power:- active_power:-
ps 3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
rwt:3 rwl:3 idle_power:- active_power:-
ps 4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
rwt:4 rwl:4 idle_power:- active_power:-
nvme id-ctrl /dev/nvme1n1 -H
NVME Identify Controller:
vid : 0x144d
ssvid : 0x144d
sn : S498NA0M403426
mn : SAMSUNG MZVLB512HAJQ-000H2
fr : EXA71HAQ
rab : 2
ieee : 002538
cmic : 0
[2:2] : 0 PCI
[1:1] : 0 Single Controller
[0:0] : 0 Single Port
mdts : 9
cntlid : 4
ver : 10200
rtd3r : 186a0
rtd3e : 7a1200
oaes : 0
[8:8] : 0 Namespace Attribute Changed Event Not Supported
oacs : 0x17
[15:4] : 0x1 Reserved
[3:3] : 0 NS Management and Attachment Not Supported
[2:2] : 0x1 FW Commit and Download Supported
[1:1] : 0x1 Format NVM Supported
[0:0] : 0x1 Sec. Send and Receive Supported
acl : 7
aerl : 7
frmw : 0x16
[4:4] : 0x1 Firmware Activate Without Reset Supported
[3:1] : 0x3 Number of Firmware Slots
[0:0] : 0 Firmware Slot 1 Read/Write
lpa : 0x3
[1:1] : 0x1 Command Effects Log Page Supported
[0:0] : 0x1 SMART/Health Log Page per NS Supported
elpe : 255
npss : 4
avscc : 0x1
[0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format
apsta : 0x1
[0:0] : 0x1 Autonomous Power State Transitions Supported
wctemp : 354
cctemp : 355
mtfa : 50
hmpre : 0
hmmin : 0
tnvmcap : 512110190592
unvmcap : 0
rpmbs : 0
[31:24]: 0 Access Size
[23:16]: 0 Total Size
[5:3] : 0 Authentication Method
[2:0] : 0 Number of RPMB Units
sqes : 0x66
[7:4] : 0x6 Max SQ Entry Size (64)
[3:0] : 0x6 Min SQ Entry Size (64)
cqes : 0x44
[7:4] : 0x4 Max CQ Entry Size (16)
[3:0] : 0x4 Min CQ Entry Size (16)
nn : 1
oncs : 0x1f
[5:5] : 0 Reservations Not Supported
[4:4] : 0x1 Save and Select Supported
[3:3] : 0x1 Write Zeroes Supported
[2:2] : 0x1 Data Set Management Supported
[1:1] : 0x1 Write Uncorrectable Supported
[0:0] : 0x1 Compare Supported
fuses : 0
[0:0] : 0 Fused Compare and Write Not Supported
fna : 0
[2:2] : 0 Crypto Erase Not Supported as part of Secure Erase
[1:1] : 0 Crypto Erase Applies to Single Namespace(s)
[0:0] : 0 Format Applies to Single Namespace(s)
vwc : 0x1
[0:0] : 0x1 Volatile Write Cache Present
awun : 1023
awupf : 0
nvscc : 1
[0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format
acwu : 0
sgls : 0
[0:0] : 0 Scatter-Gather Lists Not Supported
subnqn :
ps 0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
ps 1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
rwt:1 rwl:1 idle_power:- active_power:-
ps 2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
rwt:2 rwl:2 idle_power:- active_power:-
ps 3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
rwt:3 rwl:3 idle_power:- active_power:-
ps 4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
rwt:4 rwl:4 idle_power:- active_power:-
After one reboot, the lsblk command reversed the order of the listing, but still had the drives identified correctly:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1 259:0 0 477G 0 disk
|-nvme1n1p1 259:2 0 499M 0 part
|-nvme1n1p2 259:3 0 100M 0 part
|-nvme1n1p3 259:4 0 16M 0 part
|-nvme1n1p4 259:5 0 341.2G 0 part
`-nvme1n1p5 259:6 0 135.1G 0 part
nvme0n1 259:1 0 238.5G 0 disk
(I did not run the nvme commands in this state)
After one more reboot, the drives switched:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 477G 0 disk
|-nvme0n1p1 259:2 0 499M 0 part
|-nvme0n1p2 259:3 0 100M 0 part
|-nvme0n1p3 259:4 0 16M 0 part
|-nvme0n1p4 259:5 0 341.2G 0 part
`-nvme0n1p5 259:6 0 135.1G 0 part
nvme1n1 259:1 0 238.5G 0 disk
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 S498NA0M403426 SAMSUNG MZVLB512HAJQ-000H2 1 149.49 GB / 512.11 GB 512 B + 0 B EXA71HAQ
/dev/nvme1n1 S499NX0M113634 SAMSUNG MZVLB256HAHQ-000H2 1 2.95 GB / 256.06 GB 512 B + 0 B EXD71HAQ
> nvme id-ctrl /dev/nvme0n1 -H
NVME Identify Controller:
vid : 0x144d
ssvid : 0x144d
sn : S498NA0M403426
mn : SAMSUNG MZVLB512HAJQ-000H2
fr : EXA71HAQ
rab : 2
ieee : 002538
cmic : 0
[2:2] : 0 PCI
[1:1] : 0 Single Controller
[0:0] : 0 Single Port
mdts : 9
cntlid : 4
ver : 10200
rtd3r : 186a0
rtd3e : 7a1200
oaes : 0
[8:8] : 0 Namespace Attribute Changed Event Not Supported
oacs : 0x17
[15:4] : 0x1 Reserved
[3:3] : 0 NS Management and Attachment Not Supported
[2:2] : 0x1 FW Commit and Download Supported
[1:1] : 0x1 Format NVM Supported
[0:0] : 0x1 Sec. Send and Receive Supported
acl : 7
aerl : 7
frmw : 0x16
[4:4] : 0x1 Firmware Activate Without Reset Supported
[3:1] : 0x3 Number of Firmware Slots
[0:0] : 0 Firmware Slot 1 Read/Write
lpa : 0x3
[1:1] : 0x1 Command Effects Log Page Supported
[0:0] : 0x1 SMART/Health Log Page per NS Supported
elpe : 255
npss : 4
avscc : 0x1
[0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format
apsta : 0x1
[0:0] : 0x1 Autonomous Power State Transitions Supported
wctemp : 354
cctemp : 355
mtfa : 50
hmpre : 0
hmmin : 0
tnvmcap : 512110190592
unvmcap : 0
rpmbs : 0
[31:24]: 0 Access Size
[23:16]: 0 Total Size
[5:3] : 0 Authentication Method
[2:0] : 0 Number of RPMB Units
sqes : 0x66
[7:4] : 0x6 Max SQ Entry Size (64)
[3:0] : 0x6 Min SQ Entry Size (64)
cqes : 0x44
[7:4] : 0x4 Max CQ Entry Size (16)
[3:0] : 0x4 Min CQ Entry Size (16)
nn : 1
oncs : 0x1f
[5:5] : 0 Reservations Not Supported
[4:4] : 0x1 Save and Select Supported
[3:3] : 0x1 Write Zeroes Supported
[2:2] : 0x1 Data Set Management Supported
[1:1] : 0x1 Write Uncorrectable Supported
[0:0] : 0x1 Compare Supported
fuses : 0
[0:0] : 0 Fused Compare and Write Not Supported
fna : 0
[2:2] : 0 Crypto Erase Not Supported as part of Secure Erase
[1:1] : 0 Crypto Erase Applies to Single Namespace(s)
[0:0] : 0 Format Applies to Single Namespace(s)
vwc : 0x1
[0:0] : 0x1 Volatile Write Cache Present
awun : 1023
awupf : 0
nvscc : 1
[0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format
acwu : 0
sgls : 0
[0:0] : 0 Scatter-Gather Lists Not Supported
subnqn :
ps 0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
ps 1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
rwt:1 rwl:1 idle_power:- active_power:-
ps 2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
rwt:2 rwl:2 idle_power:- active_power:-
ps 3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
rwt:3 rwl:3 idle_power:- active_power:-
ps 4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
rwt:4 rwl:4 idle_power:- active_power:-
> nvme id-ctrl /dev/nvme1n1 -H
NVME Identify Controller:
vid : 0x144d
ssvid : 0x144d
sn : S499NX0M113634
mn : SAMSUNG MZVLB256HAHQ-000H2
fr : EXD71HAQ
rab : 2
ieee : 002538
cmic : 0
[2:2] : 0 PCI
[1:1] : 0 Single Controller
[0:0] : 0 Single Port
mdts : 9
cntlid : 4
ver : 10200
rtd3r : 186a0
rtd3e : 7a1200
oaes : 0
[8:8] : 0 Namespace Attribute Changed Event Not Supported
oacs : 0x17
[15:4] : 0x1 Reserved
[3:3] : 0 NS Management and Attachment Not Supported
[2:2] : 0x1 FW Commit and Download Supported
[1:1] : 0x1 Format NVM Supported
[0:0] : 0x1 Sec. Send and Receive Supported
acl : 7
aerl : 7
frmw : 0x16
[4:4] : 0x1 Firmware Activate Without Reset Supported
[3:1] : 0x3 Number of Firmware Slots
[0:0] : 0 Firmware Slot 1 Read/Write
lpa : 0x3
[1:1] : 0x1 Command Effects Log Page Supported
[0:0] : 0x1 SMART/Health Log Page per NS Supported
elpe : 255
npss : 4
avscc : 0x1
[0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format
apsta : 0x1
[0:0] : 0x1 Autonomous Power State Transitions Supported
wctemp : 354
cctemp : 355
mtfa : 50
hmpre : 0
hmmin : 0
tnvmcap : 256060514304
unvmcap : 0
rpmbs : 0
[31:24]: 0 Access Size
[23:16]: 0 Total Size
[5:3] : 0 Authentication Method
[2:0] : 0 Number of RPMB Units
sqes : 0x66
[7:4] : 0x6 Max SQ Entry Size (64)
[3:0] : 0x6 Min SQ Entry Size (64)
cqes : 0x44
[7:4] : 0x4 Max CQ Entry Size (16)
[3:0] : 0x4 Min CQ Entry Size (16)
nn : 1
oncs : 0x1f
[5:5] : 0 Reservations Not Supported
[4:4] : 0x1 Save and Select Supported
[3:3] : 0x1 Write Zeroes Supported
[2:2] : 0x1 Data Set Management Supported
[1:1] : 0x1 Write Uncorrectable Supported
[0:0] : 0x1 Compare Supported
fuses : 0
[0:0] : 0 Fused Compare and Write Not Supported
fna : 0
[2:2] : 0 Crypto Erase Not Supported as part of Secure Erase
[1:1] : 0 Crypto Erase Applies to Single Namespace(s)
[0:0] : 0 Format Applies to Single Namespace(s)
vwc : 0x1
[0:0] : 0x1 Volatile Write Cache Present
awun : 1023
awupf : 0
nvscc : 1
[0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format
acwu : 0
sgls : 0
[0:0] : 0 Scatter-Gather Lists Not Supported
subnqn :
ps 0 : mp:7.02W operational enlat:0 exlat:0 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
ps 1 : mp:6.30W operational enlat:0 exlat:0 rrt:1 rrl:1
rwt:1 rwl:1 idle_power:- active_power:-
ps 2 : mp:3.50W operational enlat:0 exlat:0 rrt:2 rrl:2
rwt:2 rwl:2 idle_power:- active_power:-
ps 3 : mp:0.0760W non-operational enlat:210 exlat:1200 rrt:3 rrl:3
rwt:3 rwl:3 idle_power:- active_power:-
ps 4 : mp:0.0050W non-operational enlat:2000 exlat:8000 rrt:4 rrl:4
rwt:4 rwl:4 idle_power:- active_power:-
@george1421 No worries. Just making sure there wasn’t something you knew that I didn’t
@george1421 Sure. I understand about narrowing down the possibilities. I can switch one of these to legacy and see what happens with the drives. To be clear, these are HP workstations with Samsung NVME M.2 drives. Are either of those companies related to Lenovo in some way?