FOG Very Slow to Deploy Image - Lenovo Neo 50Q Gen 4

[Quintin Giesbrecht]

Good morning! We have several Lenovo Neo 50Q Gen 4 machines that we just got. I used one to create a new image - it captured a 40GB image in about 5 minutes. When I send that same image out to any of these new machines it takes 30 minutes. If I send that image to any of our existing machines, it takes 5 minutes or so. I am thinking it must be a NIC driver issue, or some kind of write issue to the SSD or something like that.

Anyone have any ideas? I did look through the forum, but couldn’t find anything applicable.

Thanks!

Q

[Tom Elliott]

@Quintin-Giesbrecht We have seen this in the past, though I’ve kind of been out of the loop for some time.

I believe this is specific to the kernel and the way it deals with paging and block size.

While this is a very old image now, This was effectively the problem.

Basically, what I found back here was that the page_size value being less than the disk size was causing issues by defaulting to anything less than the 4k sector. So if page_size - 1 was less than 4096, default the page size to 4096 to maintain 4k structures.

Now we may need to revisit this, but i’m not so certain. Even this method could be wrong.

[Tom Elliott]

@Quintin-Giesbrecht basically I think this is something in the kernel, but what it is we don’t have enough information yet. It’s possible a newer kernel might help.

Then again it could be something like:
https://forums.fogproject.org/topic/13620/very-slow-cloning-speed-on-specific-model/105?_=1729271500055

[george1421]

@Quintin-Giesbrecht said in FOG Very Slow to Deploy Image - Lenovo Neo 50Q Gen 4:

Lenovo Neo 50Q Gen 4

I’m with Tom here, I would go with the latest FOS Linux kernel with this new hardware. You didn’t happen to mention what version of the FOS Linux kernel you are using (FOG UI -> FOG Configuration -> Kernel Update). I would go with the latest 6.6.x or later kernel. This is the easiest and quickest way to see if it solves the problem. It would be useful to know what version is your current kernel too so if we see this issue again we can help the next guy.

[Quintin Giesbrecht]

@george1421 Hi there. Thanks to both you and @Tom-Elliott for responding. I believe I am on the latest FOS Linux kernel, see screenshot:

Is that not what you are looking for? If I should be looking elsewhere, let me know.

Thanks!

Q

[Quintin Giesbrecht]

Good afternoon. Wondering if there is a potential solution to this, or if you need further info from me (just tell me what you need, and where to look, and I will get it for you).

Thanks!

Quintin

[Quintin Giesbrecht]

For interest’s sake, I tried a few of the earlier 6.6.x versions of the kernal with slightly better speeds on the earliest 6.6.x version, but still not nearly as fast as the other machines in our fleet get. If anyone has any insight into this, please let me know.

Thanks!

Q

[JJ Fullmer]

@Quintin-Giesbrecht I think there’s potential for a solution for it but it’s tricky to nail down. I imagine you’ve also tested machines that go faster on the same ethernet cable and port? It’s very odd that it would capture fast but not deploy fast. What level of compression are you using on the image? iirc when uploading the image, compression happens on the fog server, when deploying the image decompression happens on the client. So if your compression is too high, it could cause this behavior but it would be more wide spread.

You could play with making a custom kernel (see https://docs.fogproject.org/en/latest/compile-fos-kernel) adding different driver options that might relate, but since the network is already being recognized I’m not sure that would make a difference. It could also be a storage level issue rather than network which could also be something in the kernel.

You can also adjust the kernel log level settings to be higher and you’ll get more output that might point to a different problem.

Also be sure your bios is up to date and navigate through the bios settings for anything that seems like it could relate.

There’s also playing with the different pxe boot files (i.e. ipxe.efi vs snponly.efi) this shouldn’t really make a difference once you’re in part clone and imaging, but I figure it’s worth testing all parts of the process when you’re not sure where the problem lies.

I would also check the fog server dashboard, specifically the bandwidth reports and see if there are lots of dropped packets, perhaps there’s a different network configuration needed for this model?

[Quintin Giesbrecht]

@JJ-Fullmer Thanks for your reply! You have given me some places to at least look. I agree, if it was slowing down because of decompression, you would think it would be more widespread (unless there is some sort of issue/difference with the storage adapter and/or drive in these machines). My compression level is set to 6.

I will enable detailed kernel logging and see if that reveals anything.

This is not end of the world for me. I can still image a machine in about half an hour on these slow ones, but it is nice to be able to just send out an image to a machine if someone’s computer goes squirly and have them running in 10 minutes (plus, I think most of us sys admins have it in us to try and figure out why something isn’t working as expected )

Thanks again!

Q

[Tom Elliott]

@Quintin-Giesbrecht Would there be a potential of finding out if it’s specific to the ssd by removing it from one of these “slow” lenovos, put the drive in one of your normally fast machines, and try to send the image to it?

if it’s slow in the same way, it’s definitely something scoped to the SSD itself, if it’s fast, it’s likely something specific to these Lenovo Neo 50Q Gen 4 devices.

Just my 2 cents.

[Quintin Giesbrecht]

@Tom-Elliott Yeah, that would be a good test to do…thanks for that suggestion.

[Quintin Giesbrecht]

@Tom-Elliott So I moved an SSD from a slow machine to a machine that is running at full speed, and I can confirm that I get full speed out of the drive, so that rules that out.

Can someone tell me where I can enable/adjust logging levels for the kernel? I couldn’t seem to find it in the web gui.

Thanks!

[Quintin Giesbrecht]

@Tom-Elliott @JJ-Fullmer Never mind, I found the logging levels. So now that I have the level maxed, where do I find the logs? Is that still in the web gui somewhere?

Thanks!

[JJ Fullmer]

@Quintin-Giesbrecht It’s been a while since I’ve had to mess with it, if memory serves it may show in the output of the imaging itself. You might need to do a deploy in debug mode on a fast/normal speed machine and then a slow one and compare what you see.

You may also find some logs in the log viewer of fog within the fog configuration page. Where the logs are on the server depend on your fog version and your OS version.

Like for me on Rocky 9 with Fog 1.6 and using apache with php 8 the log with the most information is /var/log/php-fpm/www-error.log but I don’t believe kernel level stuff will output there.