@andrew-asph
Hello, this thread has a patch that solved the bug.
https://www.mail-archive.com/netdev@vger.kernel.org/msg189347.html
The patch is here:
https://www.mail-archive.com/netdev@vger.kernel.org/msg189923/0001-tg3-Add-clock-override-support-for-5762.patch
More details in here:
https://forums.fogproject.org/topic/10731/crash-due-to-timeout-in-tg3-kernel-module-tg3_stop_block-timed-out-ofs-4c00-enable_bit-2?loggedin=true
@sebastian-roth
As far as I can tell, the patch for tg3 was not inside the release candidates for the current kernel. I’ve tested 4.15-RC8 and it was not working. Then RC9 was released (no idea about it). Two days ago a brand new stable version was released. Will try it and see what happens.
https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.15.tar.xz
I just checked the changelog and it mentions nothing related to tg3, tigon, timeout or broadcom. I would bet this patch is not in here yet. Here it is.
https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.14.15
I will try to run more tests today. One with a 4.13.3 without the patch, to see if it breaks (and hence, the patch is the real fix). And another with 4.15 (with and without patch), to see if it is fixed and, in case it’s not, if the patch applies cleandly and works. Meanwhile, yesterday I wrote in another thread (with the same bug), asking people from there to double check our findings. Maybe they can take a look too, and see what happens.
Hello, just updating.
No answer so far from Broadcom. Tom, adding the patch would be good.
Added a link to this discussion in another thread. I think it’s the same problem.
Maybe they can also report on the problem.
https://forums.fogproject.org/topic/9976/hp-elitedesk-705-g2-mini
Mentioned the patch and test results in another forum. Hope this helps the patch to enter the main kernel faster.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1447664
Hello, just updating.
No answer so far from Broadcom. Tom, adding the patch would be good.
Added a link to this discussion in another thread. I think it’s the same problem.
Maybe they can also report on the problem.
https://forums.fogproject.org/topic/9976/hp-elitedesk-705-g2-mini
Mentioned the patch and test results in another forum. Hope this helps the patch to enter the main kernel faster.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1447664
@andrew-asph
Hello, this thread has a patch that solved the bug.
https://www.mail-archive.com/netdev@vger.kernel.org/msg189347.html
The patch is here:
https://www.mail-archive.com/netdev@vger.kernel.org/msg189923/0001-tg3-Add-clock-override-support-for-5762.patch
More details in here:
https://forums.fogproject.org/topic/10731/crash-due-to-timeout-in-tg3-kernel-module-tg3_stop_block-timed-out-ofs-4c00-enable_bit-2?loggedin=true
@sebastian-roth
Hello Sebastian, all,
Stable kernels 4.13.3 and 4.15 crash without the patch. Patch is not merged yet in the main branch.
Stable kernels 4.13.3 and 4.15 work great with the patch: no timeouts on tg3. Fast transfers on gigabit links and 10/100 links.
Wrote to the patch author as Sebastian suggested, with my results and asking when it will be merged. Waiting for his answers. Patch has a slight offset for 4.15 (2 lines, probably new comments or code) but works anyway. Will keep you updated on this.
Deploy for single machines (in parallel without multicast) is finally checked. Tested overnight with a bunch of machines and it’s ok.
If you wish, I can upload the patched 4.15 kernel tomorrow, just in case someone wants to use it.
Multicast deploy for groups of machines is working too, but much slower (about 10x) than my 10/100 network could transfer. Same network, same machines, no cable touched, nothing reset and… the deploy already starts at a slow speed (between 100 and 200 MB/min). Just reporting. Will start reading about it, to try to understand the problem. If anyone can point me on the right direction, please answer this message.
@sebastian-roth
As far as I can tell, the patch for tg3 was not inside the release candidates for the current kernel. I’ve tested 4.15-RC8 and it was not working. Then RC9 was released (no idea about it). Two days ago a brand new stable version was released. Will try it and see what happens.
https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.15.tar.xz
I just checked the changelog and it mentions nothing related to tg3, tigon, timeout or broadcom. I would bet this patch is not in here yet. Here it is.
https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.14.15
I will try to run more tests today. One with a 4.13.3 without the patch, to see if it breaks (and hence, the patch is the real fix). And another with 4.15 (with and without patch), to see if it is fixed and, in case it’s not, if the patch applies cleandly and works. Meanwhile, yesterday I wrote in another thread (with the same bug), asking people from there to double check our findings. Maybe they can take a look too, and see what happens.
@andrew-asph
Hello, today I just managed to make it work, based on directions provided by the fog team. I will double check it tomorrow, but the details (and the patch) are all described on the other thread (please see my last message in this thread).
By the way, if any of you could repeat what I did in order to check that my findings also work for you, it would provide valuable evidence that the issue was nailed. Can you please try the fix and report the results? Any result will help, since a failure can show that there’s something else that I missed.
Looking forward for your testing.
Regards,
Paulo
@tom-elliott
Hello Tom, I have added a few dmesg logs in the messages below. I think it’s not related to the firmwares, since the kernel builds ok, but the module crashes.
Hello all, it’s a real pleasure to finally say that IT WORKED!!! Wow, it finally worked! I almost can’t believe it. Thank you so much for all your help.
Aham. The solution was found by Sebastian (thanks Sebastian!!!). Here I just describe the process.
The message thread that contains the solution and a patch. It describes precisely the failure scenario: The same NIC, boot over the network, then a 10/100 switch, then the way the tg3 kernel module breaks with a timeout.
https://www.mail-archive.com/netdev@vger.kernel.org/msg189347.html
The kernel version: 4.13.3
https://www.kernel.org/pub/linux/kernel/v4.x/
https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.13.3.tar.xz
Basically I followed the instructions to rebuild a static image.
Download the kernel and the patch; extract the kernel, apply the patch. Build an image (mine was a 64 bit one).
https://wiki.fogproject.org/wiki/index.php?title=Build_TomElliott_Kernel
Install the build inside fog, then try to image something over ethernet with the regular procedure: using pxe to boot.
Without a patch, the deploy will fail with a timeout crash inside tg3. Now it should work flawlessly.
If you wish to just
If you wish, I’ve built a 64-bit image, ready to be used inside fog. Here it is.
https://goo.gl/n1qBES
Regards,
Paulo
p.s.: I really hope nothing has changed inside the firmware repository, and the fix is not due to a new firmware. Maybe it’s worth trying the same kernel with the same firmware repository, but without the patch (to see if it breaks). Anyway, it works, and this is what matters:)
Short version:
Now the long version:
@sebastian-roth
Hello Sebastian,
the bug is still hapenning. Let me explain.
I have tried what you suggested first about modifying the patch for Dell machines. Tried it with a few printk comments, to prove it was being correctly built. The patch “as is” fails the if condition (as expected), most probably because my machines are not Dell. It prints my message that proves the module is being compiled, but not the one stating that the “if body” runs.
Then I commented the “if condition” as you suggested, in order to force the body to always execute. It runs (added a printk to prove that), but the problem still happens.
I could not yet try your last suggestion (next link).
https://www.mail-archive.com/netdev@vger.kernel.org/msg189347.html
But it states that “Booting from a harddrive works fine”, which is very encouraging. It also describes precisely the scenario I see in here, with the bios code loading the kernel and ramdisk file correctly over ethernet, and with the network breaking after the boot process.
Well, I will prepare a new kernel today and will run two new tests tomorow.
Will return to you as soon as I have more information.
By the way, you’re right: “Nasty stuff and really hard to find and fix”. By the way, I stumbled upon the NIC development datasheet and it’s quite large (600+ pages: ouch!), so I gave up this route.
Thanks!
@george1421
Hello George,
Next monday I will test with “lspci -nn”.
Actually the issue only hapens in slow speeds (100). With a gigabit link it never happens.
Good to know about the usb boot FOS image. May I create it as described in the following link, or it’s something else? Anyway I just created a bootx64.efi as it describes. Will try it tomorrow morning, as soon as I get at work.
https://wiki.fogproject.org/wiki/index.php?title=USB_Bootable_Media
About the fog usb boot drive and the USB network adapter, if it works, it will be a great solution. Currently, we really don’t care much about the link speed, since we have more than 100 machines to clone. We can let them work overnight and that would be just fine, even if it takes two days.
@george1421
Hello George,
The HW id: this is a Broadcom card. This came from lspci. Is that what you’re asking?
01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5762 Gigabit Ethernet PCIe (rev 10)
I have tried that. Forcing the port to 100mb does not change anything when it’s connected to a 10/100 switch: the bug still happens. Actually, the bug ALWAYS happens when connected to a slower switch (10/100). When I connect to a gigabit port (crossover cable or gigabit switch), the communication flows perfectly. It’s clearly something time-related.
We tried to do that, but I actually… don’t know how. I mean, I can use a tethered cell phone with an “ethernet over USB” inside Ubuntu, quite easily. I can write udev rules and the like. However, I never had to do that inside busybox. The kernel is clearly recognizing the device, but I don’t know how to proceed in order to set a (virtual) network interface. Today I tried with “mdev -s”, but could not properly search for a tutorial to learn how to finish it. If you can point me on the right direction, it would be great.
Sebastian, I’ve also tried the kernel patch without success. With and without the “if condition” (and with printk messages). Later I will elaborate on that a bit further.
Unfortunately we’re still stuck.
Thank you all for your ideas,
Paulo
@andrew-asph
Hello, I’m having the very same issue. We’re trying to work around this since last semester, and the best we could get was to create a set of scripts in order to use fog files for MANUAL cloning, without network (ouch!). Basically we managed to:
Grab an image using a single ethernet cable (crossover), with gigabit board on both ends and NO SWITCH in between. One of the machines was NOT the HP (hence, it most probably has another kind of NIC).
Copy fog files to an external, usb hard disk. Adjust scripts to fix paths.
Boot a live ubuntu from a memory stick (already prepared with the necessary tools inside).
Run the scripts, in order to setup the gpt partitioning and the disk deploy
My machines are “HP EliteDesk 705 G3 MINI” (G3 instead of G2), but most probably the motherboard is the same. Or at least close enough to have the same issue. The process has to be repeated for every single machine (yeah, it’s a pain). Here it takes about 50 minutes each, when everything works fine (over USB 3).
Maybe you would like to take a look at the issue I created. It has a lot of information, including kernel logs and such. Just updated it today. Here it is. Maybe when a fix (or workaround) is found in the future, you can use it too.
Regards,
Paulo
Hello, sorry for taking long to answer this. Too much (time-sensitive) things at work. Well, I managed to find out a few new details on this bug. It seems that this AMD architecture is still not yet well supported.
This message mentions a few changes on the tigon3, including a workaround that is specific for my network card. I tested it, but it’s not working.
https://lkml.org/lkml/2017/12/31/125
<…>
Siva Reddy Kallam (3):
tg3: Update copyright
tg3: Add workaround to restrict 5762 MRRS to 2048
tg3: Enable PHY reset in MTU change path for 5720
<…>
According to this thread, the fix still does not solve the issue. Last post: 2018-01-16.
It’s the patch for tg3, aimed to my specific ethernet card (5762).
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1447664
Meanwhile, I have downloaded and rebuilt the latest linux release candidate, which has this patch for the tg3 module.
The 4.15-rc8 is available here:
https://git.kernel.org/torvalds/t/linux-4.15-rc8.tar.gz
The bzimage file was created as a static TomElliot 64 bit image.
https://wiki.fogproject.org/wiki/index.php?title=Build_TomElliott_Kernel
Unfortunately, my tests with this kernel showed no improvements on the timeout issue. The problem still happens. I tried a few kernel parameters, without success. This is a vanilla (+TomElliot config) kernel. Not tainted, although it has the firmware repository inside.
However, I finally got kernel logs. You can check them in the links below.
log_01_acpi_off.txt
https://pastebin.com/FGQNiLqk
log_02_maxcpus_1.txt
https://pastebin.com/2eEJnA3Z
log_03_nmi_watchdog_off.txt
https://pastebin.com/Su44AqiX
log_04_nmi_watchdog_off.txt
https://pastebin.com/4ja0UZ0c
log_05_noapic_nolapic.txt
https://pastebin.com/fZNJbME5
The kernel parameters were used as follows. Some were inspired by the logs (tsc), some just to… see what happens.
debug loglevel=7
debug loglevel=7 acpi=off
debug loglevel=7 acpi=off tsc=unstable
debug loglevel=7 acpi=off tsc=unstable maxcpus=1
debug loglevel=7 acpi=off tsc=unstable maxcpus=1 nmi_watchdog=0
debug loglevel=7 acpi=off tsc=unstable maxcpus=1 nmi_watchdog=0 noapic nolapic
Sometimes it’s difficult to get logs as the machine hangs right after the network stops working.
Here is the mrrs patch for tg3, related to the 5762 hw version. My test has this applied, but still does not fix the problem.
https://github.com/torvalds/linux/commit/4419bb1cedcda0272e1dc410345c5a1d1da0e367#diff-ee9b0abeec638cc316efd5b30e0e01e8
Any ideas? Would you like logs with other parameters? Is there anything I can do to provide further information? lsusb? lspci? lscpu? anything?
Regards,
Paulo
p.s.: by the way, I also spotted network issues on a live Ubuntu image (17.10.1), both on wired (tg3) and wireless (iwlwifi) network cards.