Very Slow

[Jimbohello]

Hello,
Debian Buster, Latest fog
ZFS on Root
SAMSUNG PRO 1 TB mirror

Last mounth, fog was gong arround 12-14 GB/min and thise was the average bitrate for the last years.
since last mounth, the bitrate has lower to 1-3 GB/min

SSD are like 3 mounth old
What’s the problem ??
Can you explain ?
thank’s

[Tom Elliott]

@Jimbohello what changed in your environment?

Just asking that without any context has no value or purpose.

For what it’s worth have you tried restarting the server? We don’t have access so just asking these questions like this doesn’t help you or us!

[Jimbohello]

@Tom-Elliott
We restart !
of course !

the context is simple
since last mounth upgrade ! everything is kind of dead meat !
We test brand new router (corei5 4th gen 8gig ssd evo PFSENSE)
we test with 1 pc and 1 server - brand new switch
same problem !
the only difference since last update i notice is the fact that the database create a new secure user for the node because the user fogproject was not secure enought ! maybe it’s a coiincidence. but that’s the only factor that differ from all update done over the years.

do you need log ?? someting to help ?

[Jimbohello]

@Tom-Elliott
server and node are lenovo P300
16 gig of ram

[Sebastian Roth]

@Jimbohello We have heard a couple of people having issues with speed over the last months. It mostly happens with specific model of SSD drives in the host PCs (not the FOG server!) and is often caused by the newer Linux kernel introducing some kind of energy saving mode for SSDs that is miss-interpreted by some particular drive models.

For some specific drives we found a Linux kernel parameter to fix the slowness. Depending on the FOG version you updated to this is already in use or not. You can either post your FOG version here (web UI, bottom right corner when logged in) or just add the kernel parameter yourself and test: nvme_core.default_ps_max_latency_us=0 (put in hosts settings)

The other thing you can test is scheduling your deploy task as debug (just as normal but before you click “Create Task” there is a checkbox for debug mode). Boot up the host and hit ENTER twice to get to the console. Now you run the commands:

nvme set-feature -f 0x0c -v=0 /dev/nvme0
fog

See if speed if better using one of these two methods. There is a lot more valuable information on this topic here: https://forums.fogproject.org/topic/13620/very-slow-cloning-speed-on-specific-model/

[Jimbohello]

@Sebastian-Roth
You are currently running version: 1.5.7.96
bzImage Version: 4.19.64
bzImage32 Version: 4.19.64

ill try your nvme_core.default_ps_max_latency_us=0
in the kernel
thank’s

[Jimbohello]

@Sebastian-Roth said in Very Slow:

i’m not sure exacly where to put the code ??
Thank’s

[Jimbohello]

@Sebastian-Roth
Found it ! soory
KERNEL ARGS

[george1421]

@Jimbohello After you are able to image the machine, could you provide feedback on if the kernel args works for you or you had to go the debug route and enter the instructions directly into the console? We are trying to collect information on which solution works the best for resolving these issues with these specific nvme drives.

In short these two commands (applied differently) keep the drive from going into low power mode during imaging. The developers are working towards a solution where the imaging engine will adapt these commands without intervention.

[Jimbohello]

@Sebastian-Roth
After analysing the situation, here are my conclusion
i did add your code : nothing change
same slow ridiculous speed
maybe your not looking at the right place for the bug ! of slowness
we use FOG for more than 3 years, alway the same SSD and Same Machine, closely.
so conclusion, has nothing to do with nvme_core.default_ps_max_latency_us=0
nothing has change in our environnement except we upgrade fog every 1-2 mounth.

thank’s

[Tom Elliott]

@Jimbohello Something had to have changed. And if this were a bug in the code, chances are highly likely that we’d have a lot more posts regarding the issue (as it would impact everybody.)

Something changed. I don’t understand why you update fog every 1-3 months. Last official release was 1.5.7 (July 2019). Are you running a dev-branch version? When you say you upgrade fog, do you mean the OS components (e.g. apt-get)?

Have you looked at BIOS upgrades that may be happening? Pictures of the issue would be handy too. (Yes we want to see the blue partclone screen).

I still think this is in your environment. I don’t know where or what, and the fact that things worked okay, then started getting slow, without knowing more this is very " you guys did this now you fix it " and we don’t know how to help.

[Sebastian Roth]

@Jimbohello Have you tried the other suggestion yet??

Schedule a task in debug mode and before you start the job run this command in the command shell on the host PC: nvme set-feature -f 0x0c -v=0 /dev/nvme0

[Quazz]

@Sebastian-Roth He’ll need newer inits that include the NVME package since it wasn’t included in the 1.5.7 package iirc.

[Jimbohello]

@Tom-Elliott
Agree with you !
i’ll get back to you very fast
but for now, i’ve created a samba share and i get full speed of what a gigabit nic can give so arround 112-120 mb/s
with acronis.

FOG 1.5.7.99

[Sebastian Roth]

@Jimbohello said in Very Slow:

but for now, i’ve created a samba share and i get full speed of what a gigabit nic can give so arround 112-120 mb/s
with acronis

We did have reports of another user who had fast speeds with acronis as well. Turns out Acronis is using a fairly old kernel which doesn’t seem to have the issue. If you are keen to give an older kernel a try:
https://fogproject.org/kernels/bzImage-4.9.51
fogproject.org/inits/init-4.9.x.xz

[Jimbohello]

@george1421
obviously i’m not sure i did the right hoses
but i haded
KERNEL ARGS nvme_core.default_ps_max_latency_us=0
and haded postinit.script in the image section = nvme set-feature -f 0x0c -v=0 /dev/nvme0

i’ll try tomorrow the all process debug mode

thank’s

[Jimbohello]

just to let you know that the issue was solve this week buy updating the latest version of fog
seem that you finaly found the problem

thank’s