No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.

ITsecWalrus

I will try both of those and report back.

Currently we have 1.5.9 installed for FOG. At least that it* what is says. Screenshot provided:

I also suspected that the firmware was not up to date, so I checked and the firmware version was from 10-08-2021. I updated the firmware to the latest version and the issue still persisted. I imaged a Lenovo ThinkCentre and yesterday we could image a P14 just fine so it seems to be just the T14 Gen 2.

Currently a Intel core i5 is installed. Standby for debug info. Thanks!

ITsecWalrus

@george1421 said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:
Okay so imaging with two different USBs did not change anything. still was imaging at 11-20 mb/min.

This is what happened with the debugging task:
Command:

lspci -nn | grep -i net

Results:

00:1f.6 Ethernet controller [0200]: Intel Corporation Device [8086:15fc] (rev 20)
09:00.0 Network controller [0280]: Intel Corporation Device [8086:2725] (rev 1a)

Command:

ip a s

Results:

1: lo: <LOOPBACK,UP ,LOWER _UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever

2: emp0s31f6: (BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 84:a9:38:7a:3f:e6 brd ff:ff:ff:ff:ff:ff
net xxx.xxx.xxx.xxx brdxxx.xxx.xxx.xxx scope global empOs31f6
valid_lft forever preferred_lft forever

Command:

lsblk

Results:

NAME MAT:MIN RM SIZE RO TYPE MOUNTPOINT
nume0n1 259:0 0 4776 0 disk
|-nume0n1p1 259:1 0 100M 0 part
|-nume0n1p2 259:2 0 16M 0 part
|-nume0n1p3 259:3 0 476.3G 0 part
`-nume0n1p4 259:4 0 508M 0 part

and lastly Command:

grep -i firmware /var/log/messages
//when the path was /var/log/syslog, is said that there was no such file or directory.

Results:

Dec 28 12:54:37 fogclient user.notice kernel: ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
Dec 28 12:54:37 fogclient user.info kernel: psmouse serio2: trackpoint: Elan TrackPoint firmuare: 0x12, buttons: 3/3

Not sure what to take from that but that is what happened with those commands

george1421

@itsecwalrus said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:

8086:15fc ==“Ethernet Connection (13) I219-V”

First thank you for the data collection here it saved us a few turns of Q&A.

OK for the nic above that was first introduced into the linux kernel at version 5.5.

for the syslog, ok thank you for the path, you found the right log file. This shows there is no missing linux firmware to make things work. This was one concern that either the nic or disk controller needs something that linux isn’t providing.

From the version of FOG you don’t have the latest, but you have the newest LTS supported version. There was some windows 20H1 fixes in the dev channel that hasn’t been pushed to the 1.5.10 GA release yet. That partition 4 being marked as non movable by MS will give 1.5.9 an issue on resizing that 476GB partition if you try to deploy it to a smaller disk than your source disk. But that isn’t your problem here.

This next bit we will try to debug if its a network issue or disk issue. We will use that 476GB partition on the disk to write a file to it and time the writes.

from the fos linux command prompt key in:

mkdir /ntfs
mount /dev/nume0n1p3 /ntfs
ls /ntfs

Hopefully all of those commands execute without issue. You should see the contents on the 3rd partition. I don’t care what the answer is as long as if you can see the files on that partition. If you can’t or get an error during the mount, let me know and we will correct the issue.

ITsecWalrus

@george1421 said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:

mkdir /ntfs
mount /dev/nume0n1p3 /ntfs
ls /ntfs

I tried those commands and this is what it resulted in:

mkdir /ntfs

//no output

Command:

mount /dev/nvme0n1p3 /ntfs
//I mistyped earlier. nvme is the correct spelling. Apologies

Result:

/dev/nvme0n1p3: Can't open blockdev
/dev/nvme0n1p3: Can't open blockdev
mount: mounting /dev/nvme0n1p3 on /ntfs failed: device or resource busy

Command:

ls /ntfs

Results:

'$Recycle Bin' 'Document and Setting' DumpStack.log.tmp Intel PerfLogs  'Program Files' 'Program Files x86' ProgramData Recovery 'System Volume Information' Users Windows hiberfil.sys pagefile.sys swapfile.sys

not sure if /dev/nvme0n1p3 is supposed to me there.

george1421

@itsecwalrus Thank you for sticking with me here. Some of this I’m shooting from the hip, the rest just guessing (no really I know the direction we need to move it).

So from what it looks like the partition is created but probably doesn’t have a format on it. So lets put a format on it.

mkfs -t ext4 /dev/nvme0n1p3

What this will do is format that partition as the linux EXT4 format. Windows will not be able to read this, but in the end once we get some benchmark numbers you will reimage that computer anyway.

Once it takes the format then again try to connect it to the /ntfs directory. This time lets use a bit more advanced mount command.

ntfs-3g -o force,rw  /dev/nvme0n1p3  /ntfs

Hopefully that command after we placed a format on the partition it will complete successfully. So if you are from the windows world, we just connected that partition on the nvme drive to the /ntfs directory. We can use the linux command touch /ntfs/bob.txt to create a zero byte file on that partition 3 on the nvme disk. You should also be able to see that partition connected to fos linux by using the df -h command. The whole point of this is to connect the disk to fos linux because the next command we are going to run we are going to create a 1GB file on that partition to see how log it takes to create it.

A few years ago I created this benchmarking post: https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast We will use a few commands here to see what part of FOG imaging is slow. You have already mentioned that you can image other models at normal speed, so there has to be something localized to this specific model.

So from that post lets use this command.

dd if=/dev/zero of=/ntfs/test1.img bs=1G count=1 oflag=direct

That command should run to completion. Here is the results from a rotating sata HDD from that benchmark post.

[root@localhost ~]# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 13.9599 s, 76.9 MB/s

For an nvme drive I would expect 1000MB/s plus. On my main linux computer with a samsung evo nvme I have 3500MB/s sequential write speed. Just for reference for a sata ssd I would expect to see about 500MB/s sequential write. What we are testing here is to see how fast your hard drive can intake an image file.

The speed you see on the partclone screen is a composite speed of the entire data path, fog server send, network throughput, target system expand image, target system write to disk. We are testing the last bit. Just remember we need to multiple the number that comes out of dd X 60 to get a theoretical maximum speed as compared to the partclone screen since that is measured in GB/min or MB/min and not by seconds.

ITsecWalrus

@george1421

I should thank you for sticking with ME. So… thank you

May have gotten ahead of myself and accidentally messed it up but this is what resulted of:

ntfs-3g -o force,rw /dev/nvme0n1p3 /ntfs

I have resorted to pictures. I got tired of copy pasting lol

george1421

@itsecwalrus pictures ok, I will give you a quick debug sidebar.

<sidebar>
On fos linux you can do remote debugging. Issue a ip a s command to get the IP address of fos linux target computer. Then reset root’s password with this command passwd set it to something simple like hello No worries it will be reset when fos linux reboots. Now from a windows computer use putty or linux use ssh and remote into this target computer using the IP address you collected and with root and the password you just set. It makes copy and pasting easier from a remote computer a bit easier. </sidebar>

OK it almost appears that the partition is currently mounted. That’s good and bad, but we can fix. If you issue the command df -h does it show that partition 3 is mounted on /ntfs or /images ?

For example from my laptop issuing that command this is what I see

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.8G     0  3.8G   0% /dev
tmpfs           787M  1.7M  785M   1% /run
/dev/sda2       234G   40G  183G  18% /
tmpfs           3.9G   38M  3.9G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1       511M  5.3M  506M   2% /boot/efi
tmpfs           787M  120K  787M   1% /run/user/1000

In my case the /dev/sda2 is connected to the root /

george1421

@george1421 If it is mounted then issue either umount /ntfs or umount /images to disconnect that partition. Then use the mkfs command to reformat that partition 3 and then remount it to /ntfs directory.

So I can start working on the next part, what is the FOG Server host OS? We will need to get the iperf3 command loaded there. On FOS Linux key in which iperf3 to confirm the program is in fos linux. I’m pretty sure it was added a few years ago for debugging. With iperf we will test the bandwidth between the target computer and the FOG server. We’ll do that after we are satisfied with the local disk performance.

ITsecWalrus

@george1421

What I see from that command is below:

df -h

Result:

Filesystem Size Used Avail Use% Mounted on
/dev/root 248M 97M 139M 42% /
/dev/nvme0n1p3 477G 26G 452G 6% /ntfs

So It looks like it was mounted so I did that:

umount /ntfs

Result:

//no output

Command:

mkfs -t ext4 /dev/nvme0n1p3

Result:

nke2fs 1.45.6 (20-Mar-2020)
/dev/numeOn1p3 contains a ntfs file system
Proceed anyway? (y.N) y
Discarding device blocks: done
Creating filesysten with 124866880 4k blocks and 31219712 inodes
Filesysten UUID: 5652bad-814c-4a2d-811a-fd5fb50a6dc4
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Now I did that and then tried to mount by the command suggested earlier:

ntfs-3g -o force,rw  /dev/nvme0n1p3  /ntfs

and that resulted in:

NTFS signature is missing.
Failed to mount 'dev/nvme0n1p3': Invalid argument
The device '/dev/nvme0n1p3' doesn't seem to have a valid NTFS.
Maybe the wrong device is used? Or the whole disk instead of a
partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?

Did I miss a step? I thought That I didn’t but I believe I did. was I supposed to also make /ntfs an EXT4 filesystem? Please advise when you can. Thanks for your help so far.

As far as your other questions, FOS shows iperf3 being stored in /usr/bin/iperf3. The fog server host OS is currently Ubuntu 20.10, screenshot provided below:

I am aware that there are newer ubuntu releases available, but I have been advised not to upgrade to them. Could that be the issue we are having? Thanks again!

george1421

@itsecwalrus Ok I tried to take the lazy way and its just cost us a bunch of time.

Do you know how to run fdisk? Lets run these commands:

fidsk /dev/nvme0n1

Use the d command to delete all of the current partitions on that disk. What’s wrong with partition 3 is the partition type is ntfs but I had you reformat it as ext4. Just use the d command and delete all of the current partitions on disk. Then use w to write the changes to disk, and then finally create a new partitions with n then p for primary, select partition 1, and the defaults to the rest of the values. w write the values to disk then e exit fidsk. Key in sync twice.

Now format with mkfs -t ext4 /dev/nvme0n1p1

Edit: ~~And mount with ntfs-3g -o force,rw /dev/nvme0n1p1 /ntfs if that mount command fails then~~ use the standard mount mount /dev/nvme0n1p1 /ntfs If this doesn’t work I’ll mock up the configuration you have in the test lab. I feel I need to turn these steps into a debugging tutorial. You are not the first to have a slow <something> in fog.

Sebastian Roth

@george1421 You keep mixing up ext4 formating and ntfs mounting. Won’t work!

ITsecWalrus

@george1421 said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:

fidsk /dev/nvme0n1

Okay sorry for the delay in response. I had to leave work yesterday. Here is the results of the last sesh.

It seems that all the fdisk commands worked, I was able to delete all the partitions and then create 1.

When doing the command:

fidsk /dev/nvme0n1

Results:

nke2fs 1.45.6 (20-Mar-2020)
Discarding device blocks: done
Creating filesysten with 124866880 4k blocks and 31219712 inodes
Filesysten UUID: 5652bad-814c-4a2d-811a-fd5fb50a6dc4
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Now after that I noticed that you crossed out the other command for mounting so I assumed you wanted me to use the command:

mount /dev/nvme0n1p1 /ntfs

Results:

//there was no output so I assumed it mounted since there wasn't an error

With that I went back to earlier replies and did this command:

touch /ntfs/bob.txt

Result:

//no output

After that I went to see if the partition is connected:

df -h

Result:

Size Used Avail Use% Mounted on
/dev/root 248M 97M 139M 42% /
/dev/nvme0n1p1 477G 26G 452G 6% /ntfs

I assumed that is what we wanted to see, so I continued to this command:

dd if=/dev/zero of=/ntfs/test1.img bs=1G count=1 oflag=direct

Result:

1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0GiB) copied, 0.546232 s, 2.0 GB/s

Interestingly fast. I was assuming it wouldn’t be. Does this provide any insight?

george1421

@itsecwalrus well it did show us something. The disk subsystem isn’t the problem. Those numbers are really good for local disk performance. I know it took us a lot of time to get here, but at least we know the issue isn’t with the nvme drive access (kind of what i was thinking was wrong with the new hardware. We’ve seen this in the past).

The next point we should focus on is network performance. We’ll use iperf3 for that. We’ll install iperf3 on the server and set it up in server mode and then from the target computer have it connect to the fog server and send a sample file and record the timing.

Since you have ubuntu then you will want to do a sudo apt-get install iperf3 to install iperf3 on your fog server. On a side note, hopefully you did not enable the ubuntu firewall because we will use a non standard port for sending data.

george1421

@itsecwalrus once you have iperf3 installed we need to go to the fog server and turn on the iperf service in server mode.

sudo iperf3 -s

Now go to the target computer and run this command
iperf3 -c <fog_server_ip>

This will give you an output similar to this. https://forums.fogproject.org/post/98230

Understand this bit is ONLY testing network throughput

ITsecWalrus

@george1421

That is also what I assumed it would be. I am surprised it isn’t that causing the issue.

I installed iperf3 on the server without any errors.
With the command:

sudo iperf3 -s

Resulted in:

-----------------------------------------------------------
Server listening on xxxx
-----------------------------------------------------------

I went to the target machine and ran:

iperf3 -c <fog_server_ip>

Target machine results:

Accepted connection from xxx.xxx.xxx.xxx, port xxxx
[  5] local xxx.xxx.xxx.xxx port xxxx connected to xxx.xxx.xxx.xxx port xxxx
[ ID] Interval           Transfer     Bitrate	Retr	Cwnd
[  5]   0.00-1.00   sec   112 MBytes   918 Mbits/sec 	0	413 KBytes
[  5]   1.00-2.00   sec   110 MBytes   930 Mbits/sec 	0	413 KBytes
[  5]   2.00-3.00   sec   111 MBytes   929 Mbits/sec 	0	413 KBytes
[  5]   3.00-4.00   sec   111 MBytes   929 Mbits/sec 	0	413 KBytes
[  5]   4.00-5.00   sec   111 MBytes   931 Mbits/sec 	0	413 KBytes
[  5]   5.00-6.00   sec   111 MBytes   935 Mbits/sec 	0	413 KBytes
[  5]   6.00-7.00   sec   111 MBytes   935 Mbits/sec 	0	413 KBytes
[  5]   7.00-8.00   sec   111 MBytes   934 Mbits/sec 	0	413 KBytes
[  5]   8.00-9.00   sec   111 MBytes   935 Mbits/sec 	0	413 KBytes
[  5]   9.00-10.00  sec   112 MBytes   934 Mbits/sec 	0	413 KBytes
[  5]  10.00-10.00  sec  1.12 MBytes   898 Mbits/sec 	0	413 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.09 GBytes   933 Mbits/sec                  sender
[  5]   0.00-10.01  sec  1.08 GBytes   931 Mbits/sec                  receiver

Results on Server:

Accepted connection from xxx.xxx.xxx.xxx, port xxxx
[  5] local xxx.xxx.xxx.xxx port xxxx connected to xxx.xxx.xxx.xxx port xxxx
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   109 MBytes   918 Mbits/sec
[  5]   1.00-2.00   sec   111 MBytes   930 Mbits/sec
[  5]   2.00-3.00   sec   111 MBytes   929 Mbits/sec
[  5]   3.00-4.00   sec   111 MBytes   929 Mbits/sec
[  5]   4.00-5.00   sec   111 MBytes   931 Mbits/sec
[  5]   5.00-6.00   sec   111 MBytes   935 Mbits/sec
[  5]   6.00-7.00   sec   111 MBytes   935 Mbits/sec
[  5]   7.00-8.00   sec   111 MBytes   934 Mbits/sec
[  5]   8.00-9.00   sec   111 MBytes   935 Mbits/sec
[  5]   9.00-10.00  sec   111 MBytes   934 Mbits/sec
[  5]  10.00-10.01  sec  1.05 MBytes   898 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.08 GBytes   931 Mbits/sec                  receiver

george1421

@itsecwalrus Yes these settings are exactly what I would expect on a very healthy FOG deployment.

So you are sure that this same computer from this same network jack it was moving at 10-22MB/minute ?

I’m a bit hesitant to put you through the NFS tests because the other two were the likely trouble spots.

I want to keep this configuration on this target system, but it would be interesting to know what version of partclone your version of FOG is using. This number would be visible on the blue partclone screen. It would be a number like 0.3.13 . Maybe you could find out what version using a different computer. Its not vitally important to know now, but the underlying subsystems seem to be working as it should. The next tests are NFS file transfer and then decompression and partclone.

Does this target computer have at least 4GB of ram?

This is the same exact image as you are sending to other lenovo computers, just this model is having the speed issue consuming your standard image?

ITsecWalrus

@george1421

So you are sure that this same computer from this same network jack it was moving at 10-22MB/minute ?

Yes I have not moved it from the last time I tried to image it. We have a few others of the same model we attempted to image from different drops in different rooms. All of them had the same result.

Does this target computer have at least 4GB of ram?

Yes, I believe it has 8.

This is the same exact image as you are sending to other lenovo computers, just this model is having the speed issue consuming your standard image?

Yes we use the same image for all of our Lenovo machines because it has never given us problems to do so.

I want to keep this configuration on this target system, but it would be interesting to know what version of partclone your version of FOG is using. This number would be visible on the blue partclone screen. It would be a number like 0.3.13 . Maybe you could find out what version using a different computer. Its not vitally important to know now, but the underlying subsystems seem to be working as it should. The next tests are NFS file transfer and then decompression and partclone.

It looks like the version of Partclone being used is Partclone v0.3.13 like you said.

ITsecWalrus

@george1421 Any update?

ITsecWalrus

@george1421 Hello!

I wanted to check in on this post to see if there was an update.

george1421

@itsecwalrus I started working on this next bit here: https://forums.fogproject.org/post/146681

I still need to work out a process for testing partclone performance. I think if I grab an existing image file I can send it to the local disk. All of these steps are to help up identify where its slow. From your testing we know its not the local disk (what I initially suspected) and not the network (suspected less since other models work OK). I doubt its NFS performance but we need to test for that. It could be partclone and something recently incompatible.

In the past we’ve also seen issue with the NVMe controller putting the nvme drive in a low power state during imaging. That should have been addressed in fog 1.5.9 released. But we might look into that too. But the drive speed tests didn’t indicate that was an issue.

No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.

262

12.0k

17.3k

155.2k