Very Slow



  • Hello,
    Debian Buster, Latest fog
    ZFS on Root
    SAMSUNG PRO 1 TB mirror

    Last mounth, fog was gong arround 12-14 GB/min and thise was the average bitrate for the last years.
    since last mounth, the bitrate has lower to 1-3 GB/min

    SSD are like 3 mounth old
    What’s the problem ??
    Can you explain ?
    thank’s



  • just to let you know that the issue was solve this week buy updating the latest version of fog
    seem that you finaly found the problem

    thank’s



  • @george1421
    obviously i’m not sure i did the right hoses
    but i haded
    KERNEL ARGS nvme_core.default_ps_max_latency_us=0
    and haded postinit.script in the image section = nvme set-feature -f 0x0c -v=0 /dev/nvme0

    i’ll try tomorrow the all process debug mode

    thank’s


  • Senior Developer

    @Jimbohello said in Very Slow:

    but for now, i’ve created a samba share and i get full speed of what a gigabit nic can give so arround 112-120 mb/s
    with acronis

    We did have reports of another user who had fast speeds with acronis as well. Turns out Acronis is using a fairly old kernel which doesn’t seem to have the issue. If you are keen to give an older kernel a try:
    https://fogproject.org/kernels/bzImage-4.9.51
    fogproject.org/inits/init-4.9.x.xz



  • @Tom-Elliott
    Agree with you !
    i’ll get back to you very fast
    but for now, i’ve created a samba share and i get full speed of what a gigabit nic can give so arround 112-120 mb/s
    with acronis.

    FOG 1.5.7.99


  • Moderator

    @Sebastian-Roth He’ll need newer inits that include the NVME package since it wasn’t included in the 1.5.7 package iirc.


  • Senior Developer

    @Jimbohello Have you tried the other suggestion yet??

    Schedule a task in debug mode and before you start the job run this command in the command shell on the host PC: nvme set-feature -f 0x0c -v=0 /dev/nvme0


  • Senior Developer

    @Jimbohello Something had to have changed. And if this were a bug in the code, chances are highly likely that we’d have a lot more posts regarding the issue (as it would impact everybody.)

    Something changed. I don’t understand why you update fog every 1-3 months. Last official release was 1.5.7 (July 2019). Are you running a dev-branch version? When you say you upgrade fog, do you mean the OS components (e.g. apt-get)?

    Have you looked at BIOS upgrades that may be happening? Pictures of the issue would be handy too. (Yes we want to see the blue partclone screen).

    I still think this is in your environment. I don’t know where or what, and the fact that things worked okay, then started getting slow, without knowing more this is very " you guys did this now you fix it " and we don’t know how to help.



  • @Sebastian-Roth
    After analysing the situation, here are my conclusion
    i did add your code : nothing change
    same slow ridiculous speed
    maybe your not looking at the right place for the bug ! of slowness
    we use FOG for more than 3 years, alway the same SSD and Same Machine, closely.
    so conclusion, has nothing to do with nvme_core.default_ps_max_latency_us=0
    nothing has change in our environnement except we upgrade fog every 1-2 mounth.

    thank’s


  • Moderator

    @Jimbohello After you are able to image the machine, could you provide feedback on if the kernel args works for you or you had to go the debug route and enter the instructions directly into the console? We are trying to collect information on which solution works the best for resolving these issues with these specific nvme drives.

    In short these two commands (applied differently) keep the drive from going into low power mode during imaging. The developers are working towards a solution where the imaging engine will adapt these commands without intervention.



  • @Sebastian-Roth
    Found it ! soory
    KERNEL ARGS



  • @Sebastian-Roth said in Very Slow:

    i’m not sure exacly where to put the code ??
    Thank’s



  • @Sebastian-Roth
    You are currently running version: 1.5.7.96
    bzImage Version: 4.19.64
    bzImage32 Version: 4.19.64

    ill try your nvme_core.default_ps_max_latency_us=0
    in the kernel
    thank’s


  • Senior Developer

    @Jimbohello We have heard a couple of people having issues with speed over the last months. It mostly happens with specific model of SSD drives in the host PCs (not the FOG server!) and is often caused by the newer Linux kernel introducing some kind of energy saving mode for SSDs that is miss-interpreted by some particular drive models.

    For some specific drives we found a Linux kernel parameter to fix the slowness. Depending on the FOG version you updated to this is already in use or not. You can either post your FOG version here (web UI, bottom right corner when logged in) or just add the kernel parameter yourself and test: nvme_core.default_ps_max_latency_us=0 (put in hosts settings)

    The other thing you can test is scheduling your deploy task as debug (just as normal but before you click “Create Task” there is a checkbox for debug mode). Boot up the host and hit ENTER twice to get to the console. Now you run the commands:

    nvme set-feature -f 0x0c -v=0 /dev/nvme0
    fog
    

    See if speed if better using one of these two methods. There is a lot more valuable information on this topic here: https://forums.fogproject.org/topic/13620/very-slow-cloning-speed-on-specific-model/



  • @Tom-Elliott
    server and node are lenovo P300
    16 gig of ram



  • @Tom-Elliott
    We restart !
    of course !

    the context is simple
    since last mounth upgrade ! everything is kind of dead meat !
    We test brand new router (corei5 4th gen 8gig ssd evo PFSENSE)
    we test with 1 pc and 1 server - brand new switch
    same problem !
    the only difference since last update i notice is the fact that the database create a new secure user for the node because the user fogproject was not secure enought ! maybe it’s a coiincidence. but that’s the only factor that differ from all update done over the years.

    do you need log ?? someting to help ?


  • Senior Developer

    @Jimbohello what changed in your environment?

    Just asking that without any context has no value or purpose.

    For what it’s worth have you tried restarting the server? We don’t have access so just asking these questions like this doesn’t help you or us!


Log in to reply
 

412
Online

7.4k
Users

14.5k
Topics

136.7k
Posts