Crash due to timeout in tg3 kernel module: tg3_stop_block timed out, ofs=4c00, enable_bit=2
@sebastian-roth Hello Sebastian,
I believe it’s a regression. I saw the old messages on the 2.6 kernels, this bug has happened before, maybe a few times already. Most probably the module worked for similar devices, but for this specific combination of new device plus old switch, it breaks. If I understood it correctly, there is a watchdog which keeps running, expecting something that never happens (such as a reply/control flow message/packet).
Anyway, I agree that solid evidence is always better than the best guess. I will schedule a debug test as you mentioned. Hopefully, will have a result in the next few days (things are a bit tricky in here, this week). By the way, I can add some printk messages inside the module, in case you want to see something specific into the sequence of function calls or about the variables (e.g. state of the device, etc.).
@Paulo-Guedes I have looked through the code and search the web a fair bit now and I have a feeling that we might be mislead by this old 2012 to 2013 (kernel 2.6) posts. Not saying this is totally different but I am not sure if it’s a regression or maybe something new.
The messages “Host status block”, “NAPI info” and partly also “tg3_stop_block timed out” don’t help us much right now as those are all just messages that tell us that the NIC has been reset. But so far we have no idea why this happens. Can you schedule a debug task and when you hit the error break out using keys Ctrl+c? Now get an usb key and run:
mount /dev/sdb1 /mnt dmesg > /mnt/tg3_dmesg.txt umout /mnt
Possibly the sdb1 could be sdc1 or what on your system. Just see what you have. Please post the full text message here (in a code block) or upload that file somewhere and post a link.
@sebastian-roth Hello, you described it very well: it was a really difficult time for us to manually clone all our labs. That’s something I don’t wish for anyone.
Anyway, I should have time to check again on this issue next week, I hope. Will try with the latest fog + latest kernel, to see if the issue is still happening. I already checked some posts and it seems that the problem was not yet fixed. If anyone has more information on the problem, please let me know. I will report back when I reinstall, run and test it.
@Paulo-Guedes Oh man, this sounds very unfortunate that you need to sail this with one broken arm, crippled eyes and let’s hope there is no storm coming up on the last leg of the turn. I really hope and keep my fingers crossed that you can get this done in time. After that I am more than happy to get into finding and fixing this with you. Maybe I can even get a piece of hardware myself to test.
So yes, finish that ugly job and let me know when you have time again. Wish you all the best!
well, yes and no. I have a lot of information on this, but I’m still working to bring up my labs. Please allow me a few days to work this out. I have a few hundred students and teachers that are eagerly expecting our labs to be ready, so it’s a big issue for us.
Currently I am working with a very small team, cloning about 100+ machines, one by one. Yes, you’ve read it right: it’s currently impossible to multicast images with this bug and our infrastructure (gigabit ethernet mixed with 10/100 switches).
We setup three fog servers and are using them with crossover cables. I’ve got also two external hard drives (USB 3.0), and hacked my way out through them by using a few shell scripts and a lot of tinkering.
We are able to clone about five machines in parallel with this scheme. However, the cloning process is very very unstable. About 30% to 50% of the cloning operations are failing (roughly).
From these, only about half is due to the tg3 problem and is related to fog. Yes, that’s right: with a pair of distinct machines and a single crossover cable across them, the “tg3 timeout” issue is still happening. Both machines (in each pair) have gigabit cards, but they are different. The bug is way less frequent, and we managed to finish many cloning operations successfully. But it’s still hapening.
This means the 10/100 switch makes the bug more reproducible, but it’s not the root cause. It still happens, even without any 10/100 network interface in the middle.
The other half of the failures are due to crashes and freezes from a couple of live memory sticks running ubuntu and pumping about 200GB over USB3.0 (about 45 min to 1h to finish).
I could not dig deeper into this since we need to finish the work. Hope to have it done by next friday, maybe before of that.
About iommu=soft, I also tried it a few times, without any success. Both in a 64 and 32bit kernel, and also with the latest “vanilla + firmware repo” kernel. I also tried many other things, such as noapic, nolapic, both of them, turning off autonegotiation, raising the log level to look for more messages and the like. Oh, and I also updated the HP BIOS firmware, turned on traffic shaping and tried other things (isolated and combined).
Nothing solved the problem. It’s clearly a regression somewhere between the HW, the firmware and the kernel driver.
With all the respect to Broadcom, this is something that they should have catched in a reasonably easy way. Since they gives explicit support for the kernel module, a goot testbench should have exposed the problem. Most probably their test setup has only gigabit cards, otherwise the bug would be exposed more easily.
I really believe that a testbench with Fog, a set of machines (with many distinct cards) and a set of images (with many distinct sizes, partition layouts and the like) would be great to catch this kind of thing. Oh my, I would love to help them setup something like that…
Aham. Well, let me see how things are moving. Will get back to you in a few days.
Thank you all for your support,
@Paulo-Guedes Are there any news on this? Did you get Gigabit on both ends? Reading the other post I am fairly sure this would work for you as well.
Did you try the kernel parameter suggested in the Ubuntu bug report? Edit the host’s settings in the web UI and add
iommu=softas kernel parameter. Then try again.
Other than that I don’t have any other ideas as of right now.
Hello, here is the output for lspci.
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1576
00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Carrizo (rev e4)
00:01.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Kabini HDMI/DP Audio
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 157b
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 157c
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 157c
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 157b
00:08.0 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1578
00:09.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 157d
00:09.2 Audio device: Advanced Micro Devices, Inc. [AMD] Device 157a
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 20)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 49)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 49)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 4a)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1570
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1571
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1572
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1573
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1574
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1575
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5762 Gigabit Ethernet PCIe (rev 10)
02:00.0 Network controller: Intel Corporation Wireless 3165 (rev 81)
The interesting line is the following:
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5762 Gigabit Ethernet PCIe (rev 10)
This is a Broadcom BCM5762 device.
Unfortunately, I still don’t know how to workaround this issue. I have systematically tried all sorts of kernel parameters, ethtool parameters and other things with no luck. And yes, I tried to turn off autoneg, with no luck.
In the meantime, I gathered a lot of information. It’s a bit messy, so I’ll have to organize it.
I don’t have gigabit in both ends, other than a couple of machines. That is making my cloning process a real pain, and is also the main reason I’ve not answered before :(.
I also downloaded and rebuilt the latest kernel (linux-4.12.10) based on your .config files and instructions in here.
This still does not solved the issue. And yes, I’m sure my kernel is running because I added a few messages in Portuguese to make sure it was going up, instead of the previous one.
Currently I am trying more things, including tinkering with this module myself. It seems that there is something wrong with ACPI. But tg3 is a quite complex module (at least for me). Looks more like a ton of modules merged together, with dozens of special cases, switches and paths to accommodate a large family of devices. Ouch!
Will try to better organize my ideas in order to share the details with you.
Talk to you soon. Thank you for helping me out with this crazy bug!
By the way, there are others looking at it right now. Check out this:
@Paulo-Guedes Again I think you are doing great on testing things and trying to figure this out. Using a crossover cable and different cables to verify that it must be a kernel issue is a great step. From what I see you’ve done the best to make sure we are not on the wrong path with this assumption.
So I went ahead and tried to find out one of the very important detail that you have missed out so far. What NIC exactly this is?! The PCI ID of it I mean. Luckily searching for this piece of information I stumbled upon postings in our forums that tell me that others had this issue in April already (missed that as I have been out of business for a couple of weeks back then). Read through this: https://forums.fogproject.org/topic/9976/hp-elitedesk-705-g2-mini
The issue seems to be the auto negotiation. Make sure you have Gigabit on both ends of the client connection (switch and client or FOG server and client if crossover) and you should be imaging fine!
And read this to see what happens.
Hmm, good finding. Possibly we need to add a firmware blob for this NIC to get around the issue. But please try the above forced Gigabit workaround first.