[Seeking Volunteers] Bench Testing! Our trip to the best results!



  • Hi all,

    Intro:
    We’re currently extensively testing FoG and it’s capability’s. And since w’re doing it and gathering data, why not share and spare :)

    Sooo… now what?
    Well… We’ve set-up a FoG Server with a test windows 10 deployment image (25gb). Our goal is to to define the best configuration set-up for FoG and network hardware.

    We needed to apply some configuration changes to our Cisco appliance to increase speed and packet size. i can image more users need to do so.

    If anyone is interested we’re going to test some things and it would be great to be able to compare our results/config with others.

    Bench testing what?
    Since we’re not experienced with FoG, we will play around in some ways:
    –Hardware and bios are exact. in our case. Dell T5810, Dell 3610, Dell 3600.

    • Best Deployment -type, protocol, compression, etc.
      • Deployment difference -Deploying to direct system vs XCP-NG VM. in different settings
      • Multicast: * Multi-casting from Master node (playing with storage groups) to external location through VPN*

    We have our own environment set-up as test corner with it’s own appliance that we use in general. It’s a lot of details. Might be good to set-up a general environment criteria to avoid “dirty” and contradicting results… Feel free to reply if you have any idees or layouts :)

    Goal:

    • Map all criteria to reach highest deployment/capture speed.
    • Define extra configuration requirements of third party appliance of FoG, such as Switch, Router, Firewall, Etc

    Motivated by this wiki page :)
    https://wiki.fogproject.org/wiki/index.php/Image_Compression_Tests
    alt text

    And by the replys –amount of devices the are managing with FoG
    https://forums.fogproject.org/topic/4382/organizations-using-fog/107

    Hoping for a very active thread :) <Fingers-Crossed>

    Cheers,
    Mokerhamer



  • @Junkhacker
    Checking it out!

    We had to move our workspace around the office…

    Edit: We got sick interns, colleges on vacations :( having a hard time scraping time together.


  • Developer

    @Mokerhamer if you’re really wanting to push things to the limit, you might be interested in helping out with testing/development here: https://forums.fogproject.org/topic/13206/the-future-of-partclone-and-therefore-fog-as-it-is/105

    the newest version of partclone will allow us to save images without checksums, decreasing the captured data slightly and increasing compress-ability. my initial testing says it will be about a 10% improvement on compression.



  • @Junkhacker

    Honestly, I’ve not seen multicast this fast. We’re going to seek for the “goldilock” zone IF it exists (Compression level).


  • Developer

    @george1421 i just want to chime in that those speeds seem completely normal to me. that’s what i was getting on a regular basis before i switched the VM host for my FOG server.



  • @george1421
    @Sebastian-Roth

    *ZSTD Compression 16 was used

    Dont worry i am doubting everything. I will post a full report of all hardware used. including ZSTD compression report (linke the screenshot above).

    Just give me 1 more day so our intern can gather all data from compression 0 - 22 on a full 10 GBE network.

    10GBE network can be here any moment :)

    edit: Just received 10GBE Nic. All hardware is now 10GBE

    a52967ff-e01f-42af-8fb9-8dff6050cc20-image.png


  • Moderator

    @Mokerhamer said in [Seeking Volunteers] Bench Testing! Our trip to the best results!:

    Only Client is limited with 1GB network card. Current speed with Multicast (18Gb! - Windows 10 Deployment 1:36 LOL)

    My initial reaction is WOW!. My second reaction is its impossible!! (yet it appears possible).

    With a 10GB server connection and 1GbE target system connection you are getting 18GB/min that’s 3 times wire speed. I might believe 1.6 to 2.4 times wire speed, where the limiting factor is on the client side. You will need to identify all of the hardware you are using to get these speeds.

    Please understand I’m not doubting your results, I’m just having a hard time understand them (plus I’m a bit jealous since I can only get 13.6GB/min with my setup).



  • @george1421 @Sebastian-Roth

    Only Client is limited with 1GB network card. Current speed with Multicast (18Gb! - Windows 10 Deployment 1:36 LOL)

    10GBE network card for client will be here within a hour! Curious what will happen!



  • @Sebastian-Roth

    Oke going to try it now. Crossing fingers :p Server is in a isolated network witouth any internet. Due to Nic driver problems (Ubuntu 16) i’m re-installing it on version 18.


  • Developer

    @Mokerhamer The installer will ask you which interface you want to configure FOG to. The installer is not capable to assign different functionalities of FOG to different network interfaces. Some can be done manually but it’s not officially supported.

    From what I read between the lines I’d suggest you select the interface 192.168.4.1 on installation to have FOG configure all it’s services to this network interface as a start. Using the web UI through 192.168.3.11 should be possible without manual change as Apache does not strictly bind to one interface. Anything else?



  • @george1421

    Hi, i need some help.

    I want FOG to use external card to push images & DHCP (Fog is the DHCP server 192.168.4.1)

    I’ve eenabled and configured Fog as DHCP server when installed. it dident let me control which network card to be dhcp…

    See top of screenshot that Fog DHCP is installed. Any ideas?
    I basically need a simple network 2 devices Fog Server & Fog client on a 10 GBE switch.

    9ec5cf48-fd25-43c4-97e8-0885cafdec36-image.png



  • @george1421

    We will do!

    We will also make an excel sheet as above regarding deploying/capture speeds.
    Hope we get the equipment on time! it’s now 8:33 AM i hope the have the hardware by 12:00


  • Moderator

    @Mokerhamer I have never tried this since we only have 10G in the network core and not to the access layer switches so you will be the first one.

    With zstd there has to be a sweet spot (a point where there is the best of both compressed file size and speed because compression is about compromise) in the compression ratio. For gzip I think its 6 and for zstd its 11 (I have no foundation to say this is accurate because I have not personally tested).

    Understand that the compression ratio is only used during image capture. On image deployment the decompression engine only expands the file that is compressed at full speed. So if you change the compression ratio number you will have to recapture the image to see the results of 1. file size on the FOG server. 2. The speed of the compression engine when deploying.

    At the moment the FOS Linux kernel is restricted to 8 cpu cores and less. If you run FOS Linux (the OS that run on the target computer to capture and deploy images) it will only use up to 8 (v)CPUs even if your workstation has 14 cores.

    FOS Linux is a true linux OS, meaning you could do this to monitor the system during image capture and deployment.

    1. Schedule a debug capture/deployment. Just be for you submit the task on the FOG UI, tick the debug checkbox then schedule the task.

    2. PXE boot the target computer

    3. After a few enter key presses you will be dropped to a linux command prompt.

    4. Key in ip addr show and collect the IP address of the target computer.

    5. Give root a password, something simple like hello with this command passwd

    6. Now you can connect to FOS Linux via ssh or putty

    7. Now you can do this either by the target computer console or via putty key in fog this will start the image capture or deployment. You will then have to hit enter at each step in the task sequence.

    What this will give you is a chance to run top from the other session to see how well partclone is using the system resources. You don’t have to do these steps, but if you are curious about the impact of your changes this is one way to see what the target computer is doing in regards to system resources.



  • @george1421

    Ever tried multi-casting on a full 10 GBE network, including clients having 10 GBE? i am really wondering what ZSTD with high compression on a 10GBE network. Prepared several machines, from 16 GB till 64 GB with strong cpu’s.

    Hardware will come in Monday/Tuesday and we will push the limits curious!


  • Moderator

    @Mokerhamer There are 10GbE modules for it.



  • @george1421

    Procurve 5412zl does not have 10GB poorts? or i might have the wrong product infront of me


  • Moderator

    @Mokerhamer said in [Seeking Volunteers] Bench Testing! Our trip to the best results!:

    What for 10GBe switches are you using?

    Its an older kit, Procurve 5412zl at our core switch and inside data center.

    That’s great it was something simple like spanning tree. The issue is with standard spanning tree is that it doesn’t start forwarding data for 27 seconds once the link is established. Well during the pxe booting process the link “winks” 2 times. The first as iPXE takes over from the PXE rom, and the second time is when FOS Linux takes over from iPXE. FOS Linux boots so fast that by the time the port starts forwarding data FOS Linux has already given up.

    Standard STP listens for a BPU then forwards. Fast-STP forwards first then listens for the BPU.


  • Moderator

    @Mokerhamer I wanted to clear up my first reply. I’m sorry, I didn’t want to seem so negative and I didn’t fully understand what this encompassed. I apologize if I came off as a meanie. I now see you have really good ideas/plans (and the equipment to back it up). Thank you for supporting the community through awareness and testing. I wish you the best for your trials and await the results!



  • @george1421

    Resolved. Spanning-tree was not enabled on the switch port that was directly attached to system.
    spanning-tree portfast on the port solved it.

    What for 10GBe switches are you using? i might purchase the same ones ( we’re seeking for 10GBe switch to test deployment)


  • Moderator

    @Mokerhamer You are probably better to cancel the task on the fog server then reboot. If you were unicasting and wanted to single step through deployment you would enter fog at the FOS linux command prompt. You may be able to do that with a multicast, but I never tried.


Log in to reply
 

375
Online

6.2k
Users

13.5k
Topics

127.5k
Posts