No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.
-
@itsecwalrus We are seeing on the HP Gen8 models that they are imaging super slow too. In the HP bios there is a setting that says configure storage controlller for VMD. On the HPs if you turn that feature off, imaging returns to normal. I’m not saying that is the same on lenovo, but its something to look in the firmware for. In the HP case its not the network that is running slow, but the writing to the storage controller. The devs don’t have either lenovo or hp on thier campuses, and I only have Dell so we may need to dig into what is causing this issue if the problem expands beyond HP.
If your system doesn’t have something that references VMD then we can debug this deeper if you want.
-
@george1421 Sorry I haven’t responded. I left work literally minutes before you commented.
I checked the BIOS and UEFI setup pages and could not find a setting referring to VMD. I tried looking on the internet to see if I maybe missing something and couldn’t find anything on that setting for thinkpads.
Please advise on next steps when you can. Thanks!
-
@itsecwalrus Well to debug this a bit more we will need a sample computer where we can overwrite the hard drive with stuff. But first as silly as it sounds, lets plug a usb memory device/flash drive into the computer then try to image again. Let see if that changes the speed issue. If it doesn’t don’t let it run to completion there is no need.
If not then what we will do is boot into debug mode, where we need to run a few commands. For debug mode configure another deploy task, but before you hit the schedule task button, tick the debug checkbox, then schedule the task. Now pxe boot the computer. After a few screens of text that you need to clear with the enter key, you will be dropped to the fos linux command prompt.
Key i the following commands and post the results.
lspci -nn | grep -i netThat will list out the hardware IDs of the network adapters built into the computer.
ip a sgives us info on the network adapters
lsblkgives us info on the drivesThis next one I can’t remember if its syslog or messages file. I should know this but after too much holiday cheers I can seem to remember.
grep -i firmware /var/log/syslogsee if that returns an error messages related to missing firmware. If its not syslog then its messages where the system log file is. Lets see what that info gives us. Stay in debug mode so we can do additional testing.One thing I didn’t see is what version of FOG are you using. You downloaded the latest kernel but what’s version of FOG?
Also since this IS an lenovo, please make sure the firmware is up to date. We’ve seen some pretty flaky firmware in the past. Also just for completeness, what cpu is installed intel or amd?
-
I will try both of those and report back.
Currently we have 1.5.9 installed for FOG. At least that it* what is says. Screenshot provided:
I also suspected that the firmware was not up to date, so I checked and the firmware version was from 10-08-2021. I updated the firmware to the latest version and the issue still persisted. I imaged a Lenovo ThinkCentre and yesterday we could image a P14 just fine so it seems to be just the T14 Gen 2.
Currently a Intel core i5 is installed. Standby for debug info. Thanks!
-
@george1421 said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:
Okay so imaging with two different USBs did not change anything. still was imaging at 11-20 mb/min.This is what happened with the debugging task:
Command:lspci -nn | grep -i net
Results:
00:1f.6 Ethernet controller [0200]: Intel Corporation Device [8086:15fc] (rev 20)
09:00.0 Network controller [0280]: Intel Corporation Device [8086:2725] (rev 1a)Command:
ip a s
Results:
1: lo: <LOOPBACK,UP ,LOWER _UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever2: emp0s31f6: (BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 84:a9:38:7a:3f:e6 brd ff:ff:ff:ff:ff:ff
net xxx.xxx.xxx.xxx brdxxx.xxx.xxx.xxx scope global empOs31f6
valid_lft forever preferred_lft foreverCommand:
lsblk
Results:
NAME MAT:MIN RM SIZE RO TYPE MOUNTPOINT
nume0n1 259:0 0 4776 0 disk
|-nume0n1p1 259:1 0 100M 0 part
|-nume0n1p2 259:2 0 16M 0 part
|-nume0n1p3 259:3 0 476.3G 0 part
`-nume0n1p4 259:4 0 508M 0 partand lastly Command:
grep -i firmware /var/log/messages
//when the path was /var/log/syslog, is said that there was no such file or directory.Results:
Dec 28 12:54:37 fogclient user.notice kernel: ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
Dec 28 12:54:37 fogclient user.info kernel: psmouse serio2: trackpoint: Elan TrackPoint firmuare: 0x12, buttons: 3/3Not sure what to take from that but that is what happened with those commands
-
@itsecwalrus said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:
8086:15fc ==“Ethernet Connection (13) I219-V”
First thank you for the data collection here it saved us a few turns of Q&A.
OK for the nic above that was first introduced into the linux kernel at version 5.5.
for the syslog, ok thank you for the path, you found the right log file. This shows there is no missing linux firmware to make things work. This was one concern that either the nic or disk controller needs something that linux isn’t providing.
From the version of FOG you don’t have the latest, but you have the newest LTS supported version. There was some windows 20H1 fixes in the dev channel that hasn’t been pushed to the 1.5.10 GA release yet. That partition 4 being marked as non movable by MS will give 1.5.9 an issue on resizing that 476GB partition if you try to deploy it to a smaller disk than your source disk. But that isn’t your problem here.
This next bit we will try to debug if its a network issue or disk issue. We will use that 476GB partition on the disk to write a file to it and time the writes.
from the fos linux command prompt key in:
mkdir /ntfs mount /dev/nume0n1p3 /ntfs ls /ntfsHopefully all of those commands execute without issue. You should see the contents on the 3rd partition. I don’t care what the answer is as long as if you can see the files on that partition. If you can’t or get an error during the mount, let me know and we will correct the issue.
-
@george1421 said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:
mkdir /ntfs
mount /dev/nume0n1p3 /ntfs
ls /ntfsI tried those commands and this is what it resulted in:
mkdir /ntfs
//no outputCommand:
mount /dev/nvme0n1p3 /ntfs
//I mistyped earlier. nvme is the correct spelling. ApologiesResult:
/dev/nvme0n1p3: Can't open blockdev /dev/nvme0n1p3: Can't open blockdev mount: mounting /dev/nvme0n1p3 on /ntfs failed: device or resource busyCommand:
ls /ntfs
Results:
'$Recycle Bin' 'Document and Setting' DumpStack.log.tmp Intel PerfLogs 'Program Files' 'Program Files x86' ProgramData Recovery 'System Volume Information' Users Windows hiberfil.sys pagefile.sys swapfile.sysnot sure if /dev/nvme0n1p3 is supposed to me there.
-
@itsecwalrus Thank you for sticking with me here. Some of this I’m shooting from the hip, the rest just guessing (no really I know the direction we need to move it).
So from what it looks like the partition is created but probably doesn’t have a format on it. So lets put a format on it.
mkfs -t ext4 /dev/nvme0n1p3What this will do is format that partition as the linux EXT4 format. Windows will not be able to read this, but in the end once we get some benchmark numbers you will reimage that computer anyway.
Once it takes the format then again try to connect it to the
/ntfsdirectory. This time lets use a bit more advanced mount command.ntfs-3g -o force,rw /dev/nvme0n1p3 /ntfsHopefully that command after we placed a format on the partition it will complete successfully. So if you are from the windows world, we just connected that partition on the nvme drive to the /ntfs directory. We can use the linux command
touch /ntfs/bob.txtto create a zero byte file on that partition 3 on the nvme disk. You should also be able to see that partition connected to fos linux by using thedf -hcommand. The whole point of this is to connect the disk to fos linux because the next command we are going to run we are going to create a 1GB file on that partition to see how log it takes to create it.A few years ago I created this benchmarking post: https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast We will use a few commands here to see what part of FOG imaging is slow. You have already mentioned that you can image other models at normal speed, so there has to be something localized to this specific model.
So from that post lets use this command.
dd if=/dev/zero of=/ntfs/test1.img bs=1G count=1 oflag=directThat command should run to completion. Here is the results from a rotating sata HDD from that benchmark post.
[root@localhost ~]# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=direct 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.9599 s, 76.9 MB/sFor an nvme drive I would expect 1000MB/s plus. On my main linux computer with a samsung evo nvme I have 3500MB/s sequential write speed. Just for reference for a sata ssd I would expect to see about 500MB/s sequential write. What we are testing here is to see how fast your hard drive can intake an image file.
The speed you see on the partclone screen is a composite speed of the entire data path, fog server send, network throughput, target system expand image, target system write to disk. We are testing the last bit. Just remember we need to multiple the number that comes out of
dd X 60to get a theoretical maximum speed as compared to the partclone screen since that is measured in GB/min or MB/min and not by seconds. -
I should thank you for sticking with ME. So… thank you
May have gotten ahead of myself and accidentally messed it up but this is what resulted of:
ntfs-3g -o force,rw /dev/nvme0n1p3 /ntfs
I have resorted to pictures. I got tired of copy pasting lol
-
@itsecwalrus pictures ok, I will give you a quick debug sidebar.
<sidebar>
On fos linux you can do remote debugging. Issue aip a scommand to get the IP address of fos linux target computer. Then reset root’s password with this commandpasswdset it to something simple like hello No worries it will be reset when fos linux reboots. Now from a windows computer use putty or linux use ssh and remote into this target computer using the IP address you collected and with root and the password you just set. It makes copy and pasting easier from a remote computer a bit easier. </sidebar>OK it almost appears that the partition is currently mounted. That’s good and bad, but we can fix. If you issue the command
df -hdoes it show that partition 3 is mounted on /ntfs or /images ?For example from my laptop issuing that command this is what I see
df -h Filesystem Size Used Avail Use% Mounted on udev 3.8G 0 3.8G 0% /dev tmpfs 787M 1.7M 785M 1% /run /dev/sda2 234G 40G 183G 18% / tmpfs 3.9G 38M 3.9G 1% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/sda1 511M 5.3M 506M 2% /boot/efi tmpfs 787M 120K 787M 1% /run/user/1000In my case the /dev/sda2 is connected to the root /
Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
-
@george1421 If it is mounted then issue either
umount /ntfsorumount /imagesto disconnect that partition. Then use the mkfs command to reformat that partition 3 and then remount it to/ntfsdirectory.So I can start working on the next part, what is the FOG Server host OS? We will need to get the iperf3 command loaded there. On FOS Linux key in
which iperf3to confirm the program is in fos linux. I’m pretty sure it was added a few years ago for debugging. With iperf we will test the bandwidth between the target computer and the FOG server. We’ll do that after we are satisfied with the local disk performance. -
What I see from that command is below:
df -hResult:
Filesystem Size Used Avail Use% Mounted on /dev/root 248M 97M 139M 42% / /dev/nvme0n1p3 477G 26G 452G 6% /ntfsSo It looks like it was mounted so I did that:
umount /ntfsResult:
//no outputCommand:
mkfs -t ext4 /dev/nvme0n1p3Result:
nke2fs 1.45.6 (20-Mar-2020) /dev/numeOn1p3 contains a ntfs file system Proceed anyway? (y.N) y Discarding device blocks: done Creating filesysten with 124866880 4k blocks and 31219712 inodes Filesysten UUID: 5652bad-814c-4a2d-811a-fd5fb50a6dc4 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000 Allocating group tables: done Writing inode tables: done Creating journal (262144 blocks): done Writing superblocks and filesystem accounting information: doneNow I did that and then tried to mount by the command suggested earlier:
ntfs-3g -o force,rw /dev/nvme0n1p3 /ntfsand that resulted in:
NTFS signature is missing. Failed to mount 'dev/nvme0n1p3': Invalid argument The device '/dev/nvme0n1p3' doesn't seem to have a valid NTFS. Maybe the wrong device is used? Or the whole disk instead of a partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?Did I miss a step? I thought That I didn’t but I believe I did. was I supposed to also make /ntfs an EXT4 filesystem? Please advise when you can. Thanks for your help so far.
As far as your other questions, FOS shows iperf3 being stored in /usr/bin/iperf3. The fog server host OS is currently Ubuntu 20.10, screenshot provided below:
I am aware that there are newer ubuntu releases available, but I have been advised not to upgrade to them. Could that be the issue we are having? Thanks again!
-
@itsecwalrus Ok I tried to take the lazy way and its just cost us a bunch of time.
Do you know how to run fdisk? Lets run these commands:
fidsk /dev/nvme0n1Use the
dcommand to delete all of the current partitions on that disk. What’s wrong with partition 3 is the partition type is ntfs but I had you reformat it as ext4. Just use thedcommand and delete all of the current partitions on disk. Then usewto write the changes to disk, and then finally create a new partitions withnthenpfor primary, select partition1, and the defaults to the rest of the values.wwrite the values to disk theneexit fidsk. Key insynctwice.Now format with
mkfs -t ext4 /dev/nvme0n1p1Edit:
And mount with ntfs-3g -o force,rw /dev/nvme0n1p1 /ntfs if that mount command fails thenuse the standard mountmount /dev/nvme0n1p1 /ntfsIf this doesn’t work I’ll mock up the configuration you have in the test lab. I feel I need to turn these steps into a debugging tutorial. You are not the first to have a slow <something> in fog. -
@george1421 You keep mixing up ext4 formating and ntfs mounting. Won’t work!
-
@george1421 said in No network interface found! Kernel might not have the correct driver! Lenovo T14 Gen 2.:
fidsk /dev/nvme0n1
Okay sorry for the delay in response. I had to leave work yesterday. Here is the results of the last sesh.
It seems that all the fdisk commands worked, I was able to delete all the partitions and then create 1.
When doing the command:
fidsk /dev/nvme0n1Results:
nke2fs 1.45.6 (20-Mar-2020) Discarding device blocks: done Creating filesysten with 124866880 4k blocks and 31219712 inodes Filesysten UUID: 5652bad-814c-4a2d-811a-fd5fb50a6dc4 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000 Allocating group tables: done Writing inode tables: done Creating journal (262144 blocks): done Writing superblocks and filesystem accounting information: doneNow after that I noticed that you crossed out the other command for mounting so I assumed you wanted me to use the command:
mount /dev/nvme0n1p1 /ntfsResults:
//there was no output so I assumed it mounted since there wasn't an errorWith that I went back to earlier replies and did this command:
touch /ntfs/bob.txtResult:
//no outputAfter that I went to see if the partition is connected:
df -hResult:
Size Used Avail Use% Mounted on /dev/root 248M 97M 139M 42% / /dev/nvme0n1p1 477G 26G 452G 6% /ntfsI assumed that is what we wanted to see, so I continued to this command:
dd if=/dev/zero of=/ntfs/test1.img bs=1G count=1 oflag=directResult:
1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0GiB) copied, 0.546232 s, 2.0 GB/sInterestingly fast. I was assuming it wouldn’t be. Does this provide any insight?
-
@itsecwalrus well it did show us something. The disk subsystem isn’t the problem. Those numbers are really good for local disk performance. I know it took us a lot of time to get here, but at least we know the issue isn’t with the nvme drive access (kind of what i was thinking was wrong with the new hardware. We’ve seen this in the past).
The next point we should focus on is network performance. We’ll use iperf3 for that. We’ll install iperf3 on the server and set it up in server mode and then from the target computer have it connect to the fog server and send a sample file and record the timing.
Since you have ubuntu then you will want to do a
sudo apt-get install iperf3to install iperf3 on your fog server. On a side note, hopefully you did not enable the ubuntu firewall because we will use a non standard port for sending data. -
@itsecwalrus once you have iperf3 installed we need to go to the fog server and turn on the iperf service in server mode.
sudo iperf3 -sNow go to the target computer and run this command
iperf3 -c <fog_server_ip>This will give you an output similar to this. https://forums.fogproject.org/post/98230
Understand this bit is ONLY testing network throughput
-
That is also what I assumed it would be. I am surprised it isn’t that causing the issue.
I installed iperf3 on the server without any errors.
With the command:sudo iperf3 -sResulted in:
----------------------------------------------------------- Server listening on xxxx -----------------------------------------------------------I went to the target machine and ran:
iperf3 -c <fog_server_ip>Target machine results:
Accepted connection from xxx.xxx.xxx.xxx, port xxxx [ 5] local xxx.xxx.xxx.xxx port xxxx connected to xxx.xxx.xxx.xxx port xxxx [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 112 MBytes 918 Mbits/sec 0 413 KBytes [ 5] 1.00-2.00 sec 110 MBytes 930 Mbits/sec 0 413 KBytes [ 5] 2.00-3.00 sec 111 MBytes 929 Mbits/sec 0 413 KBytes [ 5] 3.00-4.00 sec 111 MBytes 929 Mbits/sec 0 413 KBytes [ 5] 4.00-5.00 sec 111 MBytes 931 Mbits/sec 0 413 KBytes [ 5] 5.00-6.00 sec 111 MBytes 935 Mbits/sec 0 413 KBytes [ 5] 6.00-7.00 sec 111 MBytes 935 Mbits/sec 0 413 KBytes [ 5] 7.00-8.00 sec 111 MBytes 934 Mbits/sec 0 413 KBytes [ 5] 8.00-9.00 sec 111 MBytes 935 Mbits/sec 0 413 KBytes [ 5] 9.00-10.00 sec 112 MBytes 934 Mbits/sec 0 413 KBytes [ 5] 10.00-10.00 sec 1.12 MBytes 898 Mbits/sec 0 413 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 1.09 GBytes 933 Mbits/sec sender [ 5] 0.00-10.01 sec 1.08 GBytes 931 Mbits/sec receiverResults on Server:
Accepted connection from xxx.xxx.xxx.xxx, port xxxx [ 5] local xxx.xxx.xxx.xxx port xxxx connected to xxx.xxx.xxx.xxx port xxxx [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 109 MBytes 918 Mbits/sec [ 5] 1.00-2.00 sec 111 MBytes 930 Mbits/sec [ 5] 2.00-3.00 sec 111 MBytes 929 Mbits/sec [ 5] 3.00-4.00 sec 111 MBytes 929 Mbits/sec [ 5] 4.00-5.00 sec 111 MBytes 931 Mbits/sec [ 5] 5.00-6.00 sec 111 MBytes 935 Mbits/sec [ 5] 6.00-7.00 sec 111 MBytes 935 Mbits/sec [ 5] 7.00-8.00 sec 111 MBytes 934 Mbits/sec [ 5] 8.00-9.00 sec 111 MBytes 935 Mbits/sec [ 5] 9.00-10.00 sec 111 MBytes 934 Mbits/sec [ 5] 10.00-10.01 sec 1.05 MBytes 898 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 1.08 GBytes 931 Mbits/sec receiver -
@itsecwalrus Yes these settings are exactly what I would expect on a very healthy FOG deployment.
So you are sure that this same computer from this same network jack it was moving at 10-22MB/minute ?
I’m a bit hesitant to put you through the NFS tests because the other two were the likely trouble spots.
I want to keep this configuration on this target system, but it would be interesting to know what version of partclone your version of FOG is using. This number would be visible on the blue partclone screen. It would be a number like 0.3.13 . Maybe you could find out what version using a different computer. Its not vitally important to know now, but the underlying subsystems seem to be working as it should. The next tests are NFS file transfer and then decompression and partclone.
Does this target computer have at least 4GB of ram?
This is the same exact image as you are sending to other lenovo computers, just this model is having the speed issue consuming your standard image?
-
So you are sure that this same computer from this same network jack it was moving at 10-22MB/minute ?
Yes I have not moved it from the last time I tried to image it. We have a few others of the same model we attempted to image from different drops in different rooms. All of them had the same result.
Does this target computer have at least 4GB of ram?
Yes, I believe it has 8.
This is the same exact image as you are sending to other lenovo computers, just this model is having the speed issue consuming your standard image?
Yes we use the same image for all of our Lenovo machines because it has never given us problems to do so.
I want to keep this configuration on this target system, but it would be interesting to know what version of partclone your version of FOG is using. This number would be visible on the blue partclone screen. It would be a number like 0.3.13 . Maybe you could find out what version using a different computer. Its not vitally important to know now, but the underlying subsystems seem to be working as it should. The next tests are NFS file transfer and then decompression and partclone.
It looks like the version of Partclone being used is Partclone v0.3.13 like you said.
