Capturing at Gigabit speeds however only Unicasting at 50Mbps
-
Hi all,
I am new to FOG and am currently trying to get it setup within my workplace. If there are any troubleshooting requirements or logs you need to see, please let me know / spell it out fairly plainly.
I have currently setup my FOG server on a 1GBe network, with plans to upgrade to fibre 10GBe. My server is running on a Fedora 41 machine and I am mainly imaging Windows 11. When I am capturing images I can see that the speeds are reaching 1GBe speeds however whenever I try to deploy, using unicast, each unicast is stuck at 50Mbps.
When I say 50Mbps, I can unicast 1 machine at 50Mbps (takes around 2 hours), or I can Unicast 3 images / machines at 50Mbps (150Mbps over the switch) which still takes about 2 hours. My IT brethren in the workplace have asked me not to use multicast.
I am mostly imaging Dell Laptops with secure boot and Bitlocker turned off, so this shouldnt be the limitation.I have used iperf3 to test the connection between the windows machines and the linux server, and everything is running at close to 950Mbps. I have also run multiple speedtests on both the server and clients and all are as expected (about 500 up and down). The client machines are using i9’s and i7’s, so I don’t think it is decompression slowing things down. I have also tried to image no compression on the image and again I am limited at 50Mbps, as it was not compressed it took close to 4 hours.
This feels like a unicast limit, as if either A) FOG limits each unicast connection or B) hardware in between (such as the NIC or Ubiquiti switch) are limiting ftp / udp on the network.
Any thoughts?
Deployment
Capture Task
Server Setup
And some evidence that the server and connection can transmit at Gigabit
-
Hi all,
I believe I have figured out this issue, however it is unideal. I seems that between Kernel 5.15.68 and Kernel 5.15.93 there must have been some update to Realtek Driver R8169. As such, this driver now has degraded performance on my Dell G15 5510 machines.
I have also tested a machine with a intel NIC, which runs the e1000e driver in the FOG-OS environment. It works beautifully.
For other issues with similar issues out there, I would suggest for you to consider these 2 steps to resolve this issue:
- You can purchase PXE enabled USB to Ethernet adapters. Each brand of laptops will have their specified version that it allows use of (Dell’s is here: https://www.dell.com/en-au/shop/dell-adapter-usb-30-to-ethernet-pxe-boot/apd/492-11726/pc-accessories). I have tested this and it works beautifully, however would get costly above a certain amount.
- You can also roll you own Linux Kernel with your own selected drivers (ie. using R8168 instead of R8169). I also did this and it is fairly simple, but… in my case the time-cost benefit was not up to par.
Also just to put it out there, FOG debug mode is very very useful and can be found in the advanced tasks menu:
Let me know if there are any follow up questions out there.
-
Hi all,
New information. When I load the client into FOS debug mode and run iperf3, the client device can only recieve around 50Mbps but can send at 1GBe.
Thoughts:
- Given this only happens in FOS, I dare say one of the drivers are not behaving.
- The machines I am imaging are Dell Laptops G15 5510. These have Realtek ethernet controllers. I have read these are troublesome, is there any work arounds.
- I have tried updating the kernels to the latest dev kernels via the FOG Web UI, but this does not seem to help.
I would appreciate any help here.
-
Hi all,
I believe I have figured out this issue, however it is unideal. I seems that between Kernel 5.15.68 and Kernel 5.15.93 there must have been some update to Realtek Driver R8169. As such, this driver now has degraded performance on my Dell G15 5510 machines.
I have also tested a machine with a intel NIC, which runs the e1000e driver in the FOG-OS environment. It works beautifully.
For other issues with similar issues out there, I would suggest for you to consider these 2 steps to resolve this issue:
- You can purchase PXE enabled USB to Ethernet adapters. Each brand of laptops will have their specified version that it allows use of (Dell’s is here: https://www.dell.com/en-au/shop/dell-adapter-usb-30-to-ethernet-pxe-boot/apd/492-11726/pc-accessories). I have tested this and it works beautifully, however would get costly above a certain amount.
- You can also roll you own Linux Kernel with your own selected drivers (ie. using R8168 instead of R8169). I also did this and it is fairly simple, but… in my case the time-cost benefit was not up to par.
Also just to put it out there, FOG debug mode is very very useful and can be found in the advanced tasks menu:
Let me know if there are any follow up questions out there.
-
[[undefined-ago, C Clebboii, ]]
-
@Clebboii So, you could use the 5.15.67 and earlier versions (whatever they actually may be) and all will be fine?
-
@Clebboii I just checked through the commits between kernels 5.15.34 and 5.15.98. I didn’t see any changes that would cause this issue.
I have a theory of what it could be, so can you try this kernel I added to my forked repo. Let us know if the networking issue still persists.
-
@rodluz I have tried this kernel in debug mode and below is the results. Still unsuccessful:
I tested again on 5.15.68 released October 9, 2022, and back to working. Same experience on earlier kernels:
Same machine, same network connection, same FOG Server version. It is just FOS version from what I can tell.
-
@Tom-Elliott Yes correct, if its not to do with R8169, then I am unsure what is going on.
-
@Clebboii Sorry for the delay on this.
Others are having the same issue and George has something good to try. Please try this out and let us know.@george1421 said in FOG Very Slow to Deploy Image - Lenovo Neo 50Q Gen 4:
@olivier-bonnici If you have time to test, I have an idea what maybe causing this. I just saw a video on hacking a chromebox (seemingly unrelated), and they had a similar issue with downloads being unnaturally slow but uploads were OK.
It came down to the energy efficient ethernet built into the realtek nic.
What I want you to do this this:
- schedule another deployment to this hardware, before you hit the schedule task button tick the debug checkbox then schedule the task.
- PXE boot the target computer, it should go right into imaging, well it should drop you to a linux command prompt after several screens of text.
- At the linux command prompt, we need to find the name of the ethernet adapter device. Issue the following command
ip a s. Look in the list for the network device that has an IP address. Write down that name for the next stel. - Use the linux tool ethtool to disable ‘eee’ on the network adapter, in this case the ethernet adapter name is eno0 yours might be different, fix accordingly
ethtool --set-eee eno0 eee off - On success of setting green ethernet to off lets start the imaging process. Key in
fogand press enter. This will place you in single step mode. You will need to press enter at each breakpoint in the code but you will be able to see the partclone screen to see if the speed has improved.
I don’t know specifically if this is your issue but its worth a shot. If it does work we can do a few things to make this a forever change, but first we need to find out if it works or not.
-
@rodluz Unfortunately this changes nothing. I received the “eee unchanged” message, which I assume the Realtek Environmental Ethernet policies are already off.
Any other thoughts?
-
Greetings.
Same issues here with a LENOVO 50S, realteck chipset.
I’va speed of 9 Gbit/min on capture and 0.6 Gbits/min on deploy.
Debian Fog Server, version 1.5.10.1660, latest and beta kernel : same issue.
Do a bunch of test with an another hard disk, new kernel, same issue.I tested with 5.15.68 kernel and my download speed go from 0.6 Gib/min to 4.5 Gbib/min !
