Identical NVMe drives

[mrp]

Hi!

We have a lab of 20 or so identical PCs, all with 2 identical NVMe drives and 1 SATA drive. We’ve been using FOG since years without major issues, but this is our first setup with two identical NVMe drives. We are experiencing the issue with FOG mixing up the NVMe drives both with single disk and multi disk images. I’ve read the discussion in NVME madness, based on which there is currently no solution for this with identical NVMe drives. Are there any plans or hope for a solution?

In NVME madness it was mentioned that the disk identifiers could be used to identify the right disk, but they cannot be used for capturing since it would cause issues when deploying the image to another PC. Have you thought of implementing the usage of disk identifiers for the deployment? I’m thinking of the possibility to specify the order of drives in the Host Primary Disk field by listing all disk identifiers in the right order. I’m not worried about the drives being captured in the wrong order, I can sort that out after capturing.

[Sebastian Roth]

@mrp Added to the official FOS inits now. Can’t wait any longer for you to test.

[Sebastian Roth]

@mrp I just did a fresh build using the newer version of buildroot which we now use officially as well. Please download the init.xz file here for your next tests: https://github.com/FOGProject/fos/releases/testing

[Sebastian Roth]

@mrp Thanks for bringing this topic up again. I lost track of this as well. Just found some time to bring it all back to mind. We’ll surely merge this into the official FOG code sooner or later. Right now we are preparing for a next release soon I am a bit worried about adding the changes I made back in November now in a rush to an official release. We plan to put out another release this year and this would be perfect to be included then.

We might even add the full lsblk code mentioned by George to make WWN work as well.

Please let me know when you get to capture and deploy your NMVe PCs.

[mrp]

Hi, sorry for bumping this thread, I myself got carried away with other tasks.

As I read the test results, the serial should work for NVME drives on all tested systems. Does this solution have a chance to be merged into the next FOG release?

In the meantime I’m alright with using the modified init file provided by @Sebastian-Roth.

There may also be a chance that I need to capture and deploy our nvme PCs in the near future, in that case I will post the results here.

[Sebastian Roth]

@george1421 Nice, thanks heaps for your testing and the research! Will see if I can add this. Interesting you didn’t even get the serial in the mSATA…

[george1421]

@sebastian-roth said in Identical NVMe drives:

feedback on the output of lsblk -pdo NAME,SERIAL,WWN in a debug session

Testing shows that the wwn value is not displayed/present on
Dell 7450 (msata)
Dell 7400 nvme
Dell 9420 nvme
Dell 5420 nvme

Except for the msata drive the serial number was present.

Patch to add real lsblk to buildroot: https://patchwork.ozlabs.org/project/buildroot/patch/20160225171433.7e200dfc@free-electrons.com/

(I didn’t look into it more than to see what needed to be updated in package/util-linux/util-linux.mk)

[Sebastian Roth]

@george1421 Thanks! My guess is that the WWN will be empty on any hardware. Though it would be good to test on three or so different systems. Yes, could be a minified version of lsblk that we have in buildroot, not sure. Though SERIAL seems to work.

[george1421]

@sebastian-roth said in Identical NVMe drives:

Anyone keen to test and give some feedback

Sorry I forgot about this thread. I’ll give you the results tomorrow. You just need a yes or no on the WWN or do you want me to test with multiple hardware to see if its hardware specific?

I think lsblk is part of busybox. I don’t know if buildroot has the full app or not. I haven’t look as of now.

[Sebastian Roth]

@testers Anyone keen to test and give some feedback on the output of lsblk -pdo NAME,SERIAL,WWN in a debug session? No need to take a picture and post that here. Just let me know of WWN is shown in the output or empty.

[Sebastian Roth]

@george1421 said in Identical NVMe drives:

On any computer with an NVMe drive or only one that has more than one nvme drive??

Really on any machine you have available, be IT single or multi disk, HDD or NVMe…

My guess is that our FOS lsblk has this kind of bug where it cannot read the WWN.

[mrp]

@sebastian-roth Sorry for the delay, I have some busy days behind me. I ran the debug task and lsblk on an Intel NUC with a single NVME drive. On the NUC, the results were the same, the WWN field was empty, while the name and serial were ok.

[george1421]

@sebastian-roth On any computer with an NVMe drive or only one that has more than one nvme drive??

[Sebastian Roth]

@testers @moderators Can some people please test running lsblk -pdo NAME,SERIAL,WWN in a debug session (doesn’t matter if it’s capture or deploy). No need to take a picture and post that here. Just let me know of WWN is shown in the output.

[Sebastian Roth]

@mrp Ah well I see. Our FOS lsblk command is not able to retrieve the WWN information. Too bad. Not sure if there is anything we can do about it. Good you can use the serial numbers for now.

Would be interesting to see if this is not working in general or only might be an issue on your computers.

[mrp]

@sebastian-roth No problem, here it is:

[Sebastian Roth]

@mrp Thanks again for testing. Looks better now but still not perfect. I wonder why it doesn’t find the WWN at all. Can you please run the following commands in a debug command prompt and post output here:

lsblk -pdno WWN /dev/sda
lsblk -pdno WWN /dev/nvme0n1
lsblk -pdno WWN /dev/nvme1n1

[mrp]

@sebastian-roth I tested the same debug tasks with the new init. It seems that the WWN cannot be retrieved by the init, but with the serial it seems to work now.

Find another updated init binary on github that might possibly solve the issue

serial(nvme1),serial(nvme2),device-name(sata) = eui.0025385601500953,eui.0025385601500954,/dev/sda

wwn(nvme1),wwn(nvme2),device-name(sata) = S4EVNG0N600905D,S4EVNG0N600906P,/dev/sda

wwn(nvme1),serial(nvme2),device-name(sata) = S4EVNG0N600905D,eui.0025385601500954,/dev/sda

[Sebastian Roth]

@mrp Thanks for the pictures, great documentation of the test and your patience!!

Find another updated init binary on github that might possibly solve the issue but if not it will definitely give us further insight on why it doesn’t work with serial and WWN yet.

[mrp]

@sebastian-roth Sorry for the delay, today I had some time to test this.

Please download the updated init, place it on your FOG server, schedule a debug deploy task and boot the host up.

I tested the new init with debug tasks with the following host primary field values:

serial(nvme1),serial(nvme2),device-name(sata) = eui.0025385601500953,eui.0025385601500954,/dev/sda

wwn(nvme1),wwn(nvme2),device-name(sata) = S4EVNG0N600905D,S4EVNG0N600906P,/dev/sda

wwn(nvme1),serial(nvme2),device-name(sata) = S4EVNG0N600905D,eui.0025385601500954,/dev/sda

Just for double checking, I also checked lsblk -pdno NAME,SERIAL,WWN again: