FOG Very Slow to Deploy Image - Lenovo Neo 50Q Gen 4

[Quintin Giesbrecht]

Good morning! We have several Lenovo Neo 50Q Gen 4 machines that we just got. I used one to create a new image - it captured a 40GB image in about 5 minutes. When I send that same image out to any of these new machines it takes 30 minutes. If I send that image to any of our existing machines, it takes 5 minutes or so. I am thinking it must be a NIC driver issue, or some kind of write issue to the SSD or something like that.

Anyone have any ideas? I did look through the forum, but couldn’t find anything applicable.

Thanks!

Q

[Tom Elliott]

@Quintin-Giesbrecht We have seen this in the past, though I’ve kind of been out of the loop for some time.

I believe this is specific to the kernel and the way it deals with paging and block size.

While this is a very old image now, This was effectively the problem.

Basically, what I found back here was that the page_size value being less than the disk size was causing issues by defaulting to anything less than the 4k sector. So if page_size - 1 was less than 4096, default the page size to 4096 to maintain 4k structures.

Now we may need to revisit this, but i’m not so certain. Even this method could be wrong.

[Tom Elliott]

@Quintin-Giesbrecht basically I think this is something in the kernel, but what it is we don’t have enough information yet. It’s possible a newer kernel might help.

Then again it could be something like:
https://forums.fogproject.org/topic/13620/very-slow-cloning-speed-on-specific-model/105?_=1729271500055

[george1421]

@Quintin-Giesbrecht said in FOG Very Slow to Deploy Image - Lenovo Neo 50Q Gen 4:

Lenovo Neo 50Q Gen 4

I’m with Tom here, I would go with the latest FOS Linux kernel with this new hardware. You didn’t happen to mention what version of the FOS Linux kernel you are using (FOG UI -> FOG Configuration -> Kernel Update). I would go with the latest 6.6.x or later kernel. This is the easiest and quickest way to see if it solves the problem. It would be useful to know what version is your current kernel too so if we see this issue again we can help the next guy.

[Quintin Giesbrecht]

@george1421 Hi there. Thanks to both you and @Tom-Elliott for responding. I believe I am on the latest FOS Linux kernel, see screenshot:

Is that not what you are looking for? If I should be looking elsewhere, let me know.

Thanks!

Q

[Quintin Giesbrecht]

Good afternoon. Wondering if there is a potential solution to this, or if you need further info from me (just tell me what you need, and where to look, and I will get it for you).

Thanks!

Quintin

[Quintin Giesbrecht]

For interest’s sake, I tried a few of the earlier 6.6.x versions of the kernal with slightly better speeds on the earliest 6.6.x version, but still not nearly as fast as the other machines in our fleet get. If anyone has any insight into this, please let me know.

Thanks!

Q

[JJ Fullmer]

@Quintin-Giesbrecht I think there’s potential for a solution for it but it’s tricky to nail down. I imagine you’ve also tested machines that go faster on the same ethernet cable and port? It’s very odd that it would capture fast but not deploy fast. What level of compression are you using on the image? iirc when uploading the image, compression happens on the fog server, when deploying the image decompression happens on the client. So if your compression is too high, it could cause this behavior but it would be more wide spread.

You could play with making a custom kernel (see https://docs.fogproject.org/en/latest/compile-fos-kernel) adding different driver options that might relate, but since the network is already being recognized I’m not sure that would make a difference. It could also be a storage level issue rather than network which could also be something in the kernel.

You can also adjust the kernel log level settings to be higher and you’ll get more output that might point to a different problem.

Also be sure your bios is up to date and navigate through the bios settings for anything that seems like it could relate.

There’s also playing with the different pxe boot files (i.e. ipxe.efi vs snponly.efi) this shouldn’t really make a difference once you’re in part clone and imaging, but I figure it’s worth testing all parts of the process when you’re not sure where the problem lies.

I would also check the fog server dashboard, specifically the bandwidth reports and see if there are lots of dropped packets, perhaps there’s a different network configuration needed for this model?

[Quintin Giesbrecht]

@JJ-Fullmer Thanks for your reply! You have given me some places to at least look. I agree, if it was slowing down because of decompression, you would think it would be more widespread (unless there is some sort of issue/difference with the storage adapter and/or drive in these machines). My compression level is set to 6.

I will enable detailed kernel logging and see if that reveals anything.

This is not end of the world for me. I can still image a machine in about half an hour on these slow ones, but it is nice to be able to just send out an image to a machine if someone’s computer goes squirly and have them running in 10 minutes (plus, I think most of us sys admins have it in us to try and figure out why something isn’t working as expected )

Thanks again!

Q

[Tom Elliott]

@Quintin-Giesbrecht Would there be a potential of finding out if it’s specific to the ssd by removing it from one of these “slow” lenovos, put the drive in one of your normally fast machines, and try to send the image to it?

if it’s slow in the same way, it’s definitely something scoped to the SSD itself, if it’s fast, it’s likely something specific to these Lenovo Neo 50Q Gen 4 devices.

Just my 2 cents.

[Quintin Giesbrecht]

@Tom-Elliott Yeah, that would be a good test to do…thanks for that suggestion.

[Quintin Giesbrecht]

@Tom-Elliott So I moved an SSD from a slow machine to a machine that is running at full speed, and I can confirm that I get full speed out of the drive, so that rules that out.

Can someone tell me where I can enable/adjust logging levels for the kernel? I couldn’t seem to find it in the web gui.

Thanks!

[Quintin Giesbrecht]

@Tom-Elliott @JJ-Fullmer Never mind, I found the logging levels. So now that I have the level maxed, where do I find the logs? Is that still in the web gui somewhere?

Thanks!

[JJ Fullmer]

@Quintin-Giesbrecht It’s been a while since I’ve had to mess with it, if memory serves it may show in the output of the imaging itself. You might need to do a deploy in debug mode on a fast/normal speed machine and then a slow one and compare what you see.

You may also find some logs in the log viewer of fog within the fog configuration page. Where the logs are on the server depend on your fog version and your OS version.

Like for me on Rocky 9 with Fog 1.6 and using apache with php 8 the log with the most information is /var/log/php-fpm/www-error.log but I don’t believe kernel level stuff will output there.

[olivier.bonnici]

Greetings.
Same issue here with LENOVO Neo 50s Gen 5 Desktop series.
Every other model works fine, but not this one.
Capture at 10 Gib/min, but deploy a 600 Mib/min.
The network port is 1 gbit/s during both opération.

Updated fog to the latest 1.5.1660, same issue, testing the beta Linux kernel 6.12.25 / Buildroot 2025.02.0 : same.

Realy stuck here, i’m waiting 200 others same PC on september 2025.

We tested on 3 differents FOG server in different location : same issues.

Thanks for any help.

[george1421]

@olivier-bonnici If you have time to test, I have an idea what maybe causing this. I just saw a video on hacking a chromebox (seemingly unrelated), and they had a similar issue with downloads being unnaturally slow but uploads were OK.

It came down to the energy efficient ethernet built into the realtek nic.

What I want you to do this this:

1. schedule another deployment to this hardware, before you hit the schedule task button tick the debug checkbox then schedule the task.
2. PXE boot the target computer, it should go right into imaging, well it should drop you to a linux command prompt after several screens of text.
3. At the linux command prompt, we need to find the name of the ethernet adapter device. Issue the following command ip a s . Look in the list for the network device that has an IP address. Write down that name for the next stel.
4. Use the linux tool ethtool to disable ‘eee’ on the network adapter, in this case the ethernet adapter name is eno0 yours might be different, fix accordingly ethtool --set-eee eno0 eee off
5. On success of setting green ethernet to off lets start the imaging process. Key in fog and press enter. This will place you in single step mode. You will need to press enter at each breakpoint in the code but you will be able to see the partclone screen to see if the speed has improved.

I don’t know specifically if this is your issue but its worth a shot. If it does work we can do a few things to make this a forever change, but first we need to find out if it works or not.

[olivier.bonnici]

@george1421 Thank you for your help.
When i enter the eee off command, the system tells me it is already disabled.
I try downloading an image, but same low speed at 600 Mib/min.

I have changed the nvme drive by an old SSD : same low speed ><.

[olivier.bonnici]

@olivier-bonnici Edit : Works better with an older Kernel !
Deploy with kernel 5.68 from october 2022 is running at 3-4 Gbits/min.