No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.

[ITsecWalrus]

Re: Dell Latitude 5510 No network interface found! Kernel might not have the correct driver!

Also Re: Lenovo ThinkPad P15 & T15 No network interface found! Kernel might not have the correct driver!

Hello!

I have been able to image lenovo machines ranging from thinkcentres and thinkpads. I have also been able to image Thinkpads ranging from T480s to P14s.

The only issue machine I am having is a T14 Gen 2. I have done everything from updating kernels through FOG, kernels through Linux, BIOS/UEFI updates for the machines themselves. I have tried everything in the linked pages.

One thing that I am seeing that could be the issue is that when I do the commands:

cd /var/www/fog/service/ipxe/
file bzImage

It shows the following output.

bzImage: Linux kernel x86 boot executable bzImage, version 4.19.145 (sebastian@Tollana) #1 SMP Sun Sep 13 05:35:01 CDT 2020, RO-rootFS, swap_dev 0x8, Normal VGA

It shows the same thing for the bzImage32 file. I know it should be updated to the 5.10.71, and I thought I took the steps to do that. Can you please highlight the steps to update it. Thank you

[ITsecWalrus]

I have also tried using a different managed switch between the machine and the box.

I have made sure that the patch cable is good and functional, even trying several different ones to make sure that it is not the patch cable.

I should notate the the CPU is an Intel Core i5
the architecture is x65

If any more information is needed please let me know. Thank you!

[ITsecWalrus]

Ref: Kernel is missing the correct driver

So I did the steps in this to manually download the Kernel updates and that fixed the Kernel version in the bzimage file.

That said, now when I go to deploy the image to the T14 Gen 2, it goes incredibly slow. Use to, the rate would be about 10 gb/min (give or take), now it is going at 11-20 mb/min and says that it will take about 56 hours to finish.

I checked the server’s nic to make sure it is set to 10gbs/second at full duplex. Ensured that the port it is connected to is allowing the same speeds. Is there anything that I am missing?

I don’t have a different machine to try and see if it works. Let me know if you have any questions. Thanks!

[george1421]

@itsecwalrus We are seeing on the HP Gen8 models that they are imaging super slow too. In the HP bios there is a setting that says configure storage controlller for VMD. On the HPs if you turn that feature off, imaging returns to normal. I’m not saying that is the same on lenovo, but its something to look in the firmware for. In the HP case its not the network that is running slow, but the writing to the storage controller. The devs don’t have either lenovo or hp on thier campuses, and I only have Dell so we may need to dig into what is causing this issue if the problem expands beyond HP.

If your system doesn’t have something that references VMD then we can debug this deeper if you want.

[ITsecWalrus]

@george1421 Sorry I haven’t responded. I left work literally minutes before you commented.

I checked the BIOS and UEFI setup pages and could not find a setting referring to VMD. I tried looking on the internet to see if I maybe missing something and couldn’t find anything on that setting for thinkpads.

Please advise on next steps when you can. Thanks!

[george1421]

@itsecwalrus Well to debug this a bit more we will need a sample computer where we can overwrite the hard drive with stuff. But first as silly as it sounds, lets plug a usb memory device/flash drive into the computer then try to image again. Let see if that changes the speed issue. If it doesn’t don’t let it run to completion there is no need.

If not then what we will do is boot into debug mode, where we need to run a few commands. For debug mode configure another deploy task, but before you hit the schedule task button, tick the debug checkbox, then schedule the task. Now pxe boot the computer. After a few screens of text that you need to clear with the enter key, you will be dropped to the fos linux command prompt.

Key i the following commands and post the results.
lspci -nn | grep -i net That will list out the hardware IDs of the network adapters built into the computer.
ip a s gives us info on the network adapters
lsblk gives us info on the drives

This next one I can’t remember if its syslog or messages file. I should know this but after too much holiday cheers I can seem to remember.
grep -i firmware /var/log/syslog see if that returns an error messages related to missing firmware. If its not syslog then its messages where the system log file is. Lets see what that info gives us. Stay in debug mode so we can do additional testing.

One thing I didn’t see is what version of FOG are you using. You downloaded the latest kernel but what’s version of FOG?

Also since this IS an lenovo, please make sure the firmware is up to date. We’ve seen some pretty flaky firmware in the past. Also just for completeness, what cpu is installed intel or amd?

[ITsecWalrus]

@george1421

I will try both of those and report back.

Currently we have 1.5.9 installed for FOG. At least that it* what is says. Screenshot provided:

I also suspected that the firmware was not up to date, so I checked and the firmware version was from 10-08-2021. I updated the firmware to the latest version and the issue still persisted. I imaged a Lenovo ThinkCentre and yesterday we could image a P14 just fine so it seems to be just the T14 Gen 2.

Currently a Intel core i5 is installed. Standby for debug info. Thanks!

[ITsecWalrus]

@george1421 said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:
Okay so imaging with two different USBs did not change anything. still was imaging at 11-20 mb/min.

This is what happened with the debugging task:
Command:

lspci -nn | grep -i net

Results:

00:1f.6 Ethernet controller [0200]: Intel Corporation Device [8086:15fc] (rev 20)
09:00.0 Network controller [0280]: Intel Corporation Device [8086:2725] (rev 1a)

Command:

ip a s

Results:

1: lo: <LOOPBACK,UP ,LOWER _UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever

2: emp0s31f6: (BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 84:a9:38:7a:3f:e6 brd ff:ff:ff:ff:ff:ff
net xxx.xxx.xxx.xxx brdxxx.xxx.xxx.xxx scope global empOs31f6
valid_lft forever preferred_lft forever

Command:

lsblk

Results:

NAME MAT:MIN RM SIZE RO TYPE MOUNTPOINT
nume0n1 259:0 0 4776 0 disk
|-nume0n1p1 259:1 0 100M 0 part
|-nume0n1p2 259:2 0 16M 0 part
|-nume0n1p3 259:3 0 476.3G 0 part
`-nume0n1p4 259:4 0 508M 0 part

and lastly Command:

grep -i firmware /var/log/messages
//when the path was /var/log/syslog, is said that there was no such file or directory.

Results:

Dec 28 12:54:37 fogclient user.notice kernel: ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
Dec 28 12:54:37 fogclient user.info kernel: psmouse serio2: trackpoint: Elan TrackPoint firmuare: 0x12, buttons: 3/3

Not sure what to take from that but that is what happened with those commands

[george1421]

@itsecwalrus said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:

8086:15fc ==“Ethernet Connection (13) I219-V”

First thank you for the data collection here it saved us a few turns of Q&A.

OK for the nic above that was first introduced into the linux kernel at version 5.5.

for the syslog, ok thank you for the path, you found the right log file. This shows there is no missing linux firmware to make things work. This was one concern that either the nic or disk controller needs something that linux isn’t providing.

From the version of FOG you don’t have the latest, but you have the newest LTS supported version. There was some windows 20H1 fixes in the dev channel that hasn’t been pushed to the 1.5.10 GA release yet. That partition 4 being marked as non movable by MS will give 1.5.9 an issue on resizing that 476GB partition if you try to deploy it to a smaller disk than your source disk. But that isn’t your problem here.

This next bit we will try to debug if its a network issue or disk issue. We will use that 476GB partition on the disk to write a file to it and time the writes.

from the fos linux command prompt key in:

mkdir /ntfs
mount /dev/nume0n1p3 /ntfs
ls /ntfs

Hopefully all of those commands execute without issue. You should see the contents on the 3rd partition. I don’t care what the answer is as long as if you can see the files on that partition. If you can’t or get an error during the mount, let me know and we will correct the issue.

[ITsecWalrus]

@george1421 said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:

mkdir /ntfs
mount /dev/nume0n1p3 /ntfs
ls /ntfs

I tried those commands and this is what it resulted in:

mkdir /ntfs

//no output

Command:

mount /dev/nvme0n1p3 /ntfs
//I mistyped earlier. nvme is the correct spelling. Apologies

Result:

/dev/nvme0n1p3: Can't open blockdev
/dev/nvme0n1p3: Can't open blockdev
mount: mounting /dev/nvme0n1p3 on /ntfs failed: device or resource busy

Command:

ls /ntfs

Results:

'$Recycle Bin' 'Document and Setting' DumpStack.log.tmp Intel PerfLogs  'Program Files' 'Program Files x86' ProgramData Recovery 'System Volume Information' Users Windows hiberfil.sys pagefile.sys swapfile.sys

not sure if /dev/nvme0n1p3 is supposed to me there.

[george1421]

@itsecwalrus Thank you for sticking with me here. Some of this I’m shooting from the hip, the rest just guessing (no really I know the direction we need to move it).

So from what it looks like the partition is created but probably doesn’t have a format on it. So lets put a format on it.

mkfs -t ext4 /dev/nvme0n1p3 

What this will do is format that partition as the linux EXT4 format. Windows will not be able to read this, but in the end once we get some benchmark numbers you will reimage that computer anyway.

Once it takes the format then again try to connect it to the /ntfs directory. This time lets use a bit more advanced mount command.

ntfs-3g -o force,rw  /dev/nvme0n1p3  /ntfs

Hopefully that command after we placed a format on the partition it will complete successfully. So if you are from the windows world, we just connected that partition on the nvme drive to the /ntfs directory. We can use the linux command touch /ntfs/bob.txt to create a zero byte file on that partition 3 on the nvme disk. You should also be able to see that partition connected to fos linux by using the df -h command. The whole point of this is to connect the disk to fos linux because the next command we are going to run we are going to create a 1GB file on that partition to see how log it takes to create it.

A few years ago I created this benchmarking post: https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast We will use a few commands here to see what part of FOG imaging is slow. You have already mentioned that you can image other models at normal speed, so there has to be something localized to this specific model.

So from that post lets use this command.

dd if=/dev/zero of=/ntfs/test1.img bs=1G count=1 oflag=direct

That command should run to completion. Here is the results from a rotating sata HDD from that benchmark post.

[root@localhost ~]# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 13.9599 s, 76.9 MB/s

For an nvme drive I would expect 1000MB/s plus. On my main linux computer with a samsung evo nvme I have 3500MB/s sequential write speed. Just for reference for a sata ssd I would expect to see about 500MB/s sequential write. What we are testing here is to see how fast your hard drive can intake an image file.

The speed you see on the partclone screen is a composite speed of the entire data path, fog server send, network throughput, target system expand image, target system write to disk. We are testing the last bit. Just remember we need to multiple the number that comes out of dd X 60 to get a theoretical maximum speed as compared to the partclone screen since that is measured in GB/min or MB/min and not by seconds.

[ITsecWalrus]

@george1421

I should thank you for sticking with ME. So… thank you

May have gotten ahead of myself and accidentally messed it up but this is what resulted of:

ntfs-3g -o force,rw /dev/nvme0n1p3 /ntfs

I have resorted to pictures. I got tired of copy pasting lol

[george1421]

@itsecwalrus pictures ok, I will give you a quick debug sidebar.

<sidebar>
On fos linux you can do remote debugging. Issue a ip a s command to get the IP address of fos linux target computer. Then reset root’s password with this command passwd set it to something simple like hello No worries it will be reset when fos linux reboots. Now from a windows computer use putty or linux use ssh and remote into this target computer using the IP address you collected and with root and the password you just set. It makes copy and pasting easier from a remote computer a bit easier. </sidebar>

OK it almost appears that the partition is currently mounted. That’s good and bad, but we can fix. If you issue the command df -h does it show that partition 3 is mounted on /ntfs or /images ?

For example from my laptop issuing that command this is what I see

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.8G     0  3.8G   0% /dev
tmpfs           787M  1.7M  785M   1% /run
/dev/sda2       234G   40G  183G  18% /
tmpfs           3.9G   38M  3.9G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1       511M  5.3M  506M   2% /boot/efi
tmpfs           787M  120K  787M   1% /run/user/1000

In my case the /dev/sda2 is connected to the root /

[george1421]

@george1421 If it is mounted then issue either umount /ntfs or umount /images to disconnect that partition. Then use the mkfs command to reformat that partition 3 and then remount it to /ntfs directory.

So I can start working on the next part, what is the FOG Server host OS? We will need to get the iperf3 command loaded there. On FOS Linux key in which iperf3 to confirm the program is in fos linux. I’m pretty sure it was added a few years ago for debugging. With iperf we will test the bandwidth between the target computer and the FOG server. We’ll do that after we are satisfied with the local disk performance.

[ITsecWalrus]

@george1421

What I see from that command is below:

df -h

Result:

Filesystem Size Used Avail Use% Mounted on
/dev/root 248M 97M 139M 42% /
/dev/nvme0n1p3 477G 26G 452G 6% /ntfs

So It looks like it was mounted so I did that:

umount /ntfs

Result:

//no output

Command:

mkfs -t ext4 /dev/nvme0n1p3 

Result:

nke2fs 1.45.6 (20-Mar-2020)
/dev/numeOn1p3 contains a ntfs file system
Proceed anyway? (y.N) y
Discarding device blocks: done
Creating filesysten with 124866880 4k blocks and 31219712 inodes
Filesysten UUID: 5652bad-814c-4a2d-811a-fd5fb50a6dc4
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Now I did that and then tried to mount by the command suggested earlier:

ntfs-3g -o force,rw  /dev/nvme0n1p3  /ntfs

and that resulted in:

NTFS signature is missing.
Failed to mount 'dev/nvme0n1p3': Invalid argument
The device '/dev/nvme0n1p3' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?

Did I miss a step? I thought That I didn’t but I believe I did. was I supposed to also make /ntfs an EXT4 filesystem? Please advise when you can. Thanks for your help so far.

As far as your other questions, FOS shows iperf3 being stored in /usr/bin/iperf3. The fog server host OS is currently Ubuntu 20.10, screenshot provided below:

I am aware that there are newer ubuntu releases available, but I have been advised not to upgrade to them. Could that be the issue we are having? Thanks again!

[george1421]

@itsecwalrus Ok I tried to take the lazy way and its just cost us a bunch of time.

Do you know how to run fdisk? Lets run these commands:

fidsk /dev/nvme0n1

Use the d command to delete all of the current partitions on that disk. What’s wrong with partition 3 is the partition type is ntfs but I had you reformat it as ext4. Just use the d command and delete all of the current partitions on disk. Then use w to write the changes to disk, and then finally create a new partitions with n then p for primary, select partition 1, and the defaults to the rest of the values. w write the values to disk then e exit fidsk. Key in sync twice.

Now format with mkfs -t ext4 /dev/nvme0n1p1

Edit: And mount with ntfs-3g -o force,rw /dev/nvme0n1p1 /ntfs if that mount command fails then use the standard mount mount /dev/nvme0n1p1 /ntfs If this doesn’t work I’ll mock up the configuration you have in the test lab. I feel I need to turn these steps into a debugging tutorial. You are not the first to have a slow <something> in fog.

[Sebastian Roth]

@george1421 You keep mixing up ext4 formating and ntfs mounting. Won’t work!

[ITsecWalrus]

@george1421 said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:

fidsk /dev/nvme0n1

Okay sorry for the delay in response. I had to leave work yesterday. Here is the results of the last sesh.

It seems that all the fdisk commands worked, I was able to delete all the partitions and then create 1.

When doing the command:

fidsk /dev/nvme0n1

Results:

nke2fs 1.45.6 (20-Mar-2020)
Discarding device blocks: done
Creating filesysten with 124866880 4k blocks and 31219712 inodes
Filesysten UUID: 5652bad-814c-4a2d-811a-fd5fb50a6dc4
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Now after that I noticed that you crossed out the other command for mounting so I assumed you wanted me to use the command:

mount /dev/nvme0n1p1 /ntfs

Results:

//there was no output so I assumed it mounted since there wasn't an error

With that I went back to earlier replies and did this command:

touch /ntfs/bob.txt

Result:

//no output

After that I went to see if the partition is connected:

df -h

Result:

Size Used Avail Use% Mounted on
/dev/root 248M 97M 139M 42% /
/dev/nvme0n1p1 477G 26G 452G 6% /ntfs

I assumed that is what we wanted to see, so I continued to this command:

dd if=/dev/zero of=/ntfs/test1.img bs=1G count=1 oflag=direct

Result:

1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0GiB) copied, 0.546232 s, 2.0 GB/s

Interestingly fast. I was assuming it wouldn’t be. Does this provide any insight?

[george1421]

@itsecwalrus well it did show us something. The disk subsystem isn’t the problem. Those numbers are really good for local disk performance. I know it took us a lot of time to get here, but at least we know the issue isn’t with the nvme drive access (kind of what i was thinking was wrong with the new hardware. We’ve seen this in the past).

The next point we should focus on is network performance. We’ll use iperf3 for that. We’ll install iperf3 on the server and set it up in server mode and then from the target computer have it connect to the fog server and send a sample file and record the timing.

Since you have ubuntu then you will want to do a sudo apt-get install iperf3 to install iperf3 on your fog server. On a side note, hopefully you did not enable the ubuntu firewall because we will use a non standard port for sending data.

[george1421]

@itsecwalrus once you have iperf3 installed we need to go to the fog server and turn on the iperf service in server mode.

sudo iperf3 -s

Now go to the target computer and run this command
iperf3 -c <fog_server_ip>

This will give you an output similar to this. https://forums.fogproject.org/post/98230

Understand this bit is ONLY testing network throughput