• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

FOG is very slow to server kernel and init image

Scheduled Pinned Locked Moved Unsolved
FOG Problems
3
10
1.9k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • P
    pberberian
    last edited by Jan 24, 2018, 3:27 PM

    Hello all,
    I am having incredibly slow network speeds when a host is trying to get the kernel and init.xz from the fog server (ie, when it needs to be imaged / captured, when I select “Host information”, etc).
    The two files, roughly 25MB, take a minute or two to transfer.
    I believe the cause of the low throughput is high latency - pings from the FOG server to the host take 5-15 seconds. TCP is very slow over high latency networks.
    So here’s what is making this confusing for me:

    • I have used this FOG server with multiple other hosts, virtual and baremetal without this issue
    • This very high latency only occurs when trying to grab those two files. For example, if I select “Host Information”, grabbing the kernel and init image takes ages, but once I am booted into the host information thing, I can ping the fog server with <.5s latency. Same thing if I am booted from disk. The latency of the host goes up 10x only when at the fog splash screen and trying to grab files from fog
    • The file transfer speeds up significantly if I unplug the other network interfaces. This is an HPE server, with add-in cards that give us a total of 14 network interfaces, 6 of which are utilized. If I unplug everything but the single interface used to PXE boot, the transfer speed goes way up (still not as fast as our 10Gb network would allow, but still fast).

    So its not an issue with the FOG server or the network. And somehow having other network interfaces with link makes it all go much slower.

    Thank you all for any help 🙂

    P 1 Reply Last reply Jan 24, 2018, 4:46 PM Reply Quote 0
    • T
      Tom Elliott
      last edited by Jan 24, 2018, 4:46 PM

      What version of fog are you running?

      What BOOT file are you using? This sounds like you’re using pxelinux.0 still as that limited the network transfer to 10mbps (which would also cause high latency).

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      1 Reply Last reply Reply Quote 1
      • P
        pberberian @pberberian
        last edited by Jan 24, 2018, 4:46 PM

        @pberberian
        After more investigation, the network latency seems to be directly proportional to how many network interfaces are connected. With all six, there are ~5 seconds of latency. With half of them plugged in, it is closer to three seconds. With only the one plugged in, it is still at 800ms, which is faster for sure but still much slower than the .1ms I get when booted in Linux.

        Why does the number of connected interfaces affect the latency?

        T 1 Reply Last reply Jan 24, 2018, 4:51 PM Reply Quote 0
        • P
          pberberian
          last edited by Jan 24, 2018, 4:50 PM

          @tom-elliott I am running FOG 1.4.4

          The hp server grabs ipxe.pxe from the FOG server and boots from there

          1 Reply Last reply Reply Quote 0
          • T
            Tom Elliott @pberberian
            last edited by Jan 24, 2018, 4:51 PM

            @pberberian Are you connected to a switch, or a hub? This difference is significant.

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            P 2 Replies Last reply Jan 24, 2018, 4:57 PM Reply Quote 1
            • P
              pberberian @Tom Elliott
              last edited by Jan 24, 2018, 4:57 PM

              @tom-elliott I actually go through 3 switches, with STP configured properly and everything.
              I know the network topology is good because everything works perfectly when the host is booted in linux

              1 Reply Last reply Reply Quote 0
              • P
                pberberian @Tom Elliott
                last edited by Jan 24, 2018, 5:39 PM

                @tom-elliott
                I did some more digging.
                I can access the iPXE command line by pressing escape at the FOG boot menu. Here’s what I did:

                • ping <fog_server_ip> --> Works fine, perfectly fast
                • imgfetch http://<fog_server>/fog/service/ipxe/init.xz --> Very very slow, manually cancelled
                • ping <fog_server_ip> --> times out. If I try to ping from the fog server, we have the ~5s latency again

                So this definitely seems to be an issue with the networking stack of the iPXE image. It just runs incredibly slow once you try to start a file download, even if the download is cancelled. Any idea how to fix this?

                1 Reply Last reply Reply Quote 0
                • S
                  Sebastian Roth Moderator
                  last edited by Jan 24, 2018, 9:04 PM

                  @pberberian Give undionly.kpxe and undionly.kkpxe a try. That definitely uses the more general UNDI stack instead of the possibly buggy native NIC driver.

                  As well you might tell us which NIC exactly is used in that machine as a starter.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  P 1 Reply Last reply Jan 26, 2018, 4:02 PM Reply Quote 0
                  • P
                    pberberian @Sebastian Roth
                    last edited by Jan 26, 2018, 4:02 PM

                    @sebastian-roth The host fails to boot at all with those files. It must chainload to ipxe first before it can boot (because of HPE’s crap pxe implementation).
                    I am using an intel addin card that gives 4x10G SFP interfaces

                    1 Reply Last reply Reply Quote 0
                    • S
                      Sebastian Roth Moderator
                      last edited by Jan 26, 2018, 10:06 PM

                      @pberberian said in FOG is very slow to server kernel and init image:

                      I am using an intel addin card that gives 4x10G SFP interfaces

                      We need more details than that!

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      1 Reply Last reply Reply Quote 0
                      • 1 / 1
                      1 / 1
                      • First post
                        1/10
                        Last post

                      130

                      Online

                      12.1k

                      Users

                      17.3k

                      Topics

                      155.3k

                      Posts
                      Copyright © 2012-2024 FOG Project