• Register
    • Login
    • Search
    • Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search

    Solved Extremely Slow Deploy to NVME drives

    FOG Problems
    9
    35
    2956
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      KSiig last edited by

      I’m beginning to run into the same problem, also with an HP 840 G6. I’ve tried the kernel arguments, however it just says ‘not a valid indentifier’. I have a Samsung 850 evo M.2 Sata 6GB/s drive laying around which I tested with, along with an HP 830 G5:

      830 + 850 EVO: Success
      830 + Original NVMe SSD: Success
      840 + Original NVMe SSD: Failure
      840 + 850 Evo: Success

      I compared the SSDs from the 840 G6 and the 830 G5, and they are the exact same model. So while it’s of course a very small sample size, it’s pretty clear that’s it has something to do with the 840 G6 combined with NVMe that’s causing the failure.

      One thing that also only happens with the 840 G6 is that it shows the following message:

      udevd[3088]: inotify_add_watch(6, /dev/nvme0n1p2, 10) failed: no such file or directory
      udevd[3089]: inotify_add_watch(6, /dev/nvme0n1p1, 10) failed: no such file or directory
      
      1 Reply Last reply Reply Quote 0
      • george1421
        george1421 Moderator @Guest last edited by

        @Shad0wguy said in Extremely Slow Deploy to NVME drives:

        nvme_core.default_ps_max_latency_us=5500

        This needs to go into the kernel parameters field either the global one in FOG Settings->FOG Configuration field or in the host specific field kernel parameter.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        1 Reply Last reply Reply Quote 0
        • ?
          A Former User last edited by

          I am having this same issue. I posted about this a month ago. I tried the nvme_core.default_ps_max_latency_us=5500 kernel argument but it returned an error about it being not a valid identifier after booting. I’ve updated to 1.5.7.4. Any other suggestions?

          george1421 1 Reply Last reply Reply Quote 0
          • R
            robbit @Sebastian Roth last edited by

            @Sebastian-Roth

            Thank you for that. Our HP laptops were equipped with Western Digital PC SN520 NVMe SSD, but also tested with Toshiba NVMe THNSSN5256GPUK with same results. After the update, it still went to a crawl HOWEVER

            When I went to change the Kernal parameter under FOG Configuration -> FOG Settings -> General -> Kernel args -> nvme_core.default_ps_max_latency_us=5500 -> it’s working now. I was able to deploy the image to a NVMe drive via unicast.

            Test#1 w/ Fog Server 1.5.7.3

            • Same results with the NVMe drives where the rate goes to a crawl of about 10MB/min

            Test#2 w/ nvme_core.default_ps_max_latency_us=5500 + Fog 1.5.7.3

            • Deploying at a solid speed ~5GB/min on isolated networrk

            Whatever it is, the combination of both of those have fixed the issue.

            I want to thank both you @george1421 and @Sebastian-Roth for chiming in! It’s good to know there’s a very active community for this.

            1 Reply Last reply Reply Quote 1
            • S
              Sebastian Roth Moderator last edited by george1421

              @george1421 @robbit As for the commands to get the very latest develpment version (called dev-branch) you need to add one more command to what was posted below:

              sudo -i
              cd /root/fogproject/bin
              git checkout dev-branch
              ./installfog.sh
              

              Leaving that one command out doesn’t breach anything but you’ll end up with current master 1.5.7 again.

              About the slow speed. Do you have Toshiba drives? https://forums.fogproject.org/topic/13620/very-slow-cloning-speed-on-specific-model

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              R 1 Reply Last reply Reply Quote 1
              • R
                robbit @george1421 last edited by

                @george1421
                updating now, I ran ./installfog.sh and it’s taking some time to backup database. I will report back tomorrow

                Thank you for all the help so far! I’ve also just found this thread just now. We pretty much have the same exact laptop
                https://forums.fogproject.org/topic/13733/hp-elitebook-830-gen-6-issues-capturing-images-and-deploying-images/5

                so I may try the kernal parameters as well (unless you suggest otherwise)

                1 Reply Last reply Reply Quote 0
                • george1421
                  george1421 Moderator @robbit last edited by

                  @robbit ok I just assumed since most people install ubuntu for some reason you had that. Let me translate it for centos (what I use)

                  yum update -y
                  yum install git -y
                  git clone https://github.com/FOGProject/fogproject.git /root/fogproject
                  cd /root/fogproject/bin
                  ./installfog.sh
                  

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                  R 1 Reply Last reply Reply Quote 1
                  • R
                    robbit @george1421 last edited by

                    @george1421

                    Thanks for the screenshot. Looks like I don’t have apt-get on CentOS 7. Is there another alternative? It seems like I just need to get Git, right? Tried running yum install git but got a bunch of errors about mirror sites not available.

                    george1421 1 Reply Last reply Reply Quote 0
                    • george1421
                      george1421 Moderator last edited by george1421

                      I’m not going to say this will fix the issue but lets setup the git method, because it will be easier to install future updates this way. FWIW the download method is the tarball way.

                      Run these commands.

                      sudo apt-get update && apt-get install git
                      
                      sudo -i
                      git clone https://github.com/FOGProject/fogproject.git /root/fogproject
                      cd /root/fogproject/bin
                      ./installfog.sh
                      

                      Just rerun through the installer, it will use the answer file you created on the first install. Hopefully that will bring you up to 1.5.7.2, if not we will need to switch to the development branch. But lets see if we can get to the right version this way.

                      FWIW when a new release comes out you would just switch to the /opt/fogproject directory and do a git pull then change to the bin directory and run the installfog.sh script again. No need to download and extract the updated tarball.

                      That 3TB spinning hard disk won’t be a problem until you have 3-4 simultaneous unicast imaging running at the same time, then it will be the bottleneck in your fog server, with that single nic card being a close second.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                      R 1 Reply Last reply Reply Quote 0
                      • R
                        robbit @george1421 last edited by

                        @george1421 said in Extremely Slow Deploy to NVME drives:

                        tarball

                        Tarbell. I went https://fogproject.org/download to download it. Installed it as Normal Server.

                        I forgot to mention, Fog is installed on a standard 3TB 3.5" HDD. The images are being stored under /images.

                        1 Reply Last reply Reply Quote 0
                        • george1421
                          george1421 Moderator @robbit last edited by

                          @robbit The sg300 is a good enough switch. So lets rule out network infrastructure.

                          The 7050 most likely has a NMVe disk in it. So I’m pretty sure we can rule out the FOG server, so that kind of leaves the target computer.

                          I’m trying to remember if I’ve seen this issue before with nvme. I’m thinking yes (but I see a lot of things in the threads so I may be mixing up issues). The latest version of fog is 1.5.7.2 which addresses a few issues that was discovered after 1.5.7 was release. How did you install FOG, was it using the git method or tarball?

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                          R 1 Reply Last reply Reply Quote 0
                          • R
                            robbit @george1421 last edited by

                            @george1421

                            I do, I have a 10port 1Gb managed switch (Cisco SG300-10SFP). I tried it out with the same results. It started out strong for few seconds but rate quickly drops. Elapsed time is about 5 minutes and the rate is at 90MB/min and still dropping. The progress percentage has also froze at 1.21% right now.

                            I’m not sure if it’s a red herring, but I suspect it’s something with the NVMe vs M.2 SSD. It deploys just fine with the M.2 SSD with the exact same laptop, switch, cables. But as soon as I put back the NVMe, this issue comes up. I got a 52port 1Gb managed switch as well if we want to get into medium size business switches as well

                            george1421 1 Reply Last reply Reply Quote 0
                            • george1421
                              george1421 Moderator @robbit last edited by

                              @robbit Just for clarity you are getting 5GB/m deploy times or capture time? I would say that 5GB/m deploy is a tad low but not out of reason. I have seem some of the small (read cheap) 5 or 8 ports (soho) switches perform pretty badly.

                              FWIW, 6.1G/m is about maximum for a well designed 1GbE network. So 5.0GB/m is still acceptable.

                              Do you have an enterprise level switch to test with?

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                              R 1 Reply Last reply Reply Quote 0
                              • R
                                robbit @george1421 last edited by

                                @george1421

                                Hi,

                                Thanks for getting back to me so quickly. Maybe it’s more like 5GB/min? I forgot the exact number but it would only take no more than 10-15 minutes to deploy (at least to my m2 ssd).

                                I have a small unmanaged switch, both the target device and the fog server are plugged into it and nothing else. I’ve tried the basic troubleshooting of using different cables and switches as well.

                                george1421 1 Reply Last reply Reply Quote 0
                                • george1421
                                  george1421 Moderator last edited by

                                  @robbit said in Extremely Slow Deploy to NVME drives:

                                  It captures it just fine, around 7.5GB/min according to the screen. However, when I try to deploy the image to another G6 laptop, it starts up fine around 2.5GB/min then drops to a mere 20MB/min which says will take about 80hours.

                                  I find this interesting, in my experience a capture rate of 7.5GB/m is a little fast. And 2.5GB/m is much slower compared to your capture rates. I’m not saying its wrong only based on my experience capturing should be slower than deploy.

                                  Your fog server “hardware” is more than capable for imaging. Is your target computer on the same subnet or even better the same network switch as your fog server? In fog imaging the target computer does most of the work in imaging. The fog server does really very little other than send the compressed file from the fog server’s hard drive to the target computer. Imaging speed is more dependent on how fast the target computer can intake the image, expand it and then write it to disk.

                                  Lets try to eliminate your network infrastructure by putting the fog server and target computer on the same network switch and see what we get.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                  R 1 Reply Last reply Reply Quote 0
                                  • 1
                                  • 2
                                  • 2 / 2
                                  • First post
                                    Last post

                                  149
                                  Online

                                  10.4k
                                  Users

                                  16.4k
                                  Topics

                                  150.6k
                                  Posts

                                  Copyright © 2012-2023 FOG Project