@wt_101 When I say heavy lifting is done by the client computer, I mean all of the work and the actual performance of fog imaging is directly impacted by the target computer’s capabilities and components. While I understand this is technically impossible, but if you have 2 computers that are exactly matched, except for one has DDR3 1600 and the other has DDR4 2133 RAM, the second computer with the faster ram will deploy the image faster because the transferred image is decompressed in ram on the target computer (more on that in Q2).

Q1 To be honest I never paid attention to what the web ui says for size vs what is on the disk. Off the top of my head having a 3:1 compression ratio seems a bit high in my estimation. Is it possible, yes. What really is a metric is what is the size of actual data on the target computer vs the size of the image files. Its possible that the web ui is recording something different that raw source disk vs compressed image file. There is a compression slider in the image definition. This tells the compressor what compression metric to use (not the right words) during compression. The higher the number the more compression methods it uses to compress the data. i think the slider is set for a default of 4 or 6 for gzip that value is a good balance between compression size and speed. For zstd the Goldilocks number is 11. Where the gzip compressor has a range of 0 to 9, zstd has a range of 0 to 22. I don’t think anyone has done any testing to find the actual Goldilocks number in a quantitative way though. I suspect they found a number that worked well for them and called it good.

Q2 Option A is correct. The image is compressed/decompressed on the client so only a compress image is ever communicated with the client. This saves on storage image size on the storage node as well as transfer bandwidth. From a metric standpoint I know that a 25GB target image can be transferred in about 4 minutes. The only way that’s possible on a 1 GbE network is to transfer a compressed image.

Q3 See that is where the magic of FOG is. The developers created a custom version of linux. That version of linux is called FOS (FOG Operating System). That OS has all of the tools built in that FOG uses to image a target computer. Yes FOS has zstd and gzip compressors built in. When you pxe boot a computer during image, first the iPXE boot loader is transferred to the target computer. iPXE is responsible for the FOG iPXE menu. Once you make a menu selection (like registration) you will see two files transferred to the target computer if you have a fast eye. You will see bzImage (the kernel) and init.xz (virtual hard drive) send to the target computer, that IS FOS linux being sent over. The OS is very small and very fast.

For Point 4, that is more of a question for the developers. I don’t look under the hood for statistics settings. I just know that on the Partclone screen what that speed number means. I don’t know if the FOG program as a way to record that speed or not. As for taskelasped time I think that means something else. As I mentioned above, on a 1 GbE network a 25GB image should take about 4 minutes of transfer time. 16 seconds seems a bit quick.

For Point 6, The fog client is used for more than just renaming the client and connecting the target computer to AD. Its also used for application deployment and some rudimentary system management. You do not need to run the FOG Client if you don’t want to manage the target computer after image deployment.

Q1 yes there is a way. On my campus, which is mostly MS Windows based, I don’t use the fog client at all, yet I still have a touchless deployment. I leverage a feature in FOG called a Post Install Script to make changes to MS Windows unattend.xml file just after the image is pushed to the target computer. For a linux client it is just as easy most of the things that configure linux is just in text file, and FOS Linux is… wait for it… linux, so the possibilities are endless. The concept of a post install script is that you would create a bash script on the fog server that is executed by FOS Linux. That bash script would mount the target computer’s hard drive (post image deployment) and make the necessary adjustments to the hostname and any other deployment specific settings. The post install script can have access to fog host definition variable so you can leverage some of the extra fields in the host definition for specific uses (like other1 and other2 fields).

@tesparza I made a post a few years ago to test different hardware and its impact on imaging speed. We may need to reference it as we work through this.

https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast

If your fog server has a sata spinning hard drive that may also be the culprit. But again testing will show us.

@tesparza Its good we can point our finger at low memory as the problem. The small cache space on 4GB is only left 800MB of room so mysql had to commit everything to disk instead of caching it for later. (speculation) I did find it surprising that in the previous top screen shot, you had 4GB of swap space but none was used even when the system was low on ram. It wouldn’t have really helped in your case, but I find it strange that no swap space was used.

Now that you found an acceptable level you can start decreasing your fog client check in time till you find a happy balance between speed and frequency. The only issue I can think of by keeping the check in time to 5 minutes would be during post imaging if the fog client was used to rename the system, connect to AD, or do a scheduled reboot. Its possible the client computer wouldn’t react to the command for about 5 minutes.

@jyost Have you gone through and tried to rebuild the ipxe boot loaders yet? If not here is a tutorial on how to do this: https://forums.fogproject.org/topic/15826/updating-compiling-the-latest-version-of-ipxe

Let see if an updated version of iPXE solves this issue.

@michaeloberg You can test that kernel for sure, but I think the answer is 5.13 and later. I see there is a 5.15.5 that was just created on https://fogproject.org/kernels/

Lets to with this route.

cd /var/www/html/fog/service/ipxe
wget https://fogproject.org/kernels/bzImage-5.15.5
wget https://fogproject.org/kernels/bzImage-no-cdc_ether

That should bring them down to the FOG server without messing up anything else.

Now go into the web ui for that host that you manually registered with the 6C mac address. There is a kernel field. For Sebastians test enter bzImage-no-cdc_ether in that field and schedule a debug capture or deploy doesn’t matter. Make sure you turn mac address pass through back on before you do that.

Now pxe boot it, you should see bzImage-no-cdc_ether get transferred to the target computer. After a few screens of text you will be dropped at the fos linux command pompt. Key in ip a s and look at the mac address for that adapter is it the dongle address or the pass through address?

Now go back into the web ui and change the kernel parameter to bzImage-5.15.5 and save the settings. Now go to the target computer and key in reboot Once again pxe boot into the debug mode (there is still an open deploy task so all you need to do is reboot the target computer and repxe boot it). When running under the 5.15 kernel see if the right mac address is passed with ip a s

@altitudehack First let me say I don’t have an answer to this issue. But i can tell you that php-fpm will only throw an error If there is either a task timeout or there is a problem with the php programming code. I would also inspect the apache error log to see if php-fpm is working like it should but apache is unhappy with the report.

@george1421 On the fog server linux console lets do this.

cd /etc
find ./ -name www.conf

Find should return with some path to the config file
./php-fpm.d/www.conf

Lets edit that file. I can’t give you the exact path because the config file moves depending on the version of php that’s installed.

Search for php_admin_value[memory_limit] and make sure its set to 256MB

php_admin_value[memory_limit] = 256M

Make the change and reboot the server (easy) or restart php-fpm (slightly harder)

Edit: OK I’m too slow between telephone calls and users stopping by. Looks like you hit the right area. Just watch out for RAM exhaustion if you bump the values up too high. The php-fpm process has up to 35 workers defined.

@altitudehack The forum is a bit strange. You have to go into the forum post settings (little gear on original post) then set to ask as question. Once that is done you can mark the entire thread solved or pick the post with the right answer. I don’t know why the forum doesn’t default to every OP needs to be asked as a question. But it is what it is. I took care of it for you.

@michaeloberg OK I have a few kernels I want you to test. Please test each one to see if it reads the pass-thru mac address.

This first one is bzImage-rt-v215 This kernel has the version 2.15 driver from the realtek web site. The version built into the linux kernel is at version 1.11. This is kernel version 5.15.6
https://drive.google.com/file/d/1qxcjC9ZrV8lVh4T6rgBtmtgSp7SWRzL9/view?usp=sharing

This next one is stock FOS linux kernel 5.15.6 but has the additional kernel options listed here: http://linux-hardware.org/?id=usb:0bda-8153 this kernel is called bzImage-rt-opts
https://drive.google.com/file/d/1t4WgWHOv3wIFjnodRtMsRcaO87XIZxM4/view?usp=sharing

Don’t get discouraged if neither one of these works. We are trying to find out where the problem isn’t right now. I want you to run this command after you boot into debug mode and/if if fails grep -i -e firmware /var/log/messages What I’m looking for is if the rt8153 or something network related complains about missing firmware. As I mentioned before I did see several 8153 hardware specific firmware listed but we are not including them by default. The linux kernel will complain if it needs the firmware but none are built into the kernel. This is not a common error, but does happen now and again. Your case is unique so we are casting a wide net here.

For the developers here are the setting changes I found when I compared the FOS Linux default config vs the web site.

# CONFIG_USB_IPHETH is not set
CONFIG_USB_NET_CDCETHER=y
CONFIG_USB_PEGASUS=y
CONFIG_USB_RTL8150=y
CONFIG_USB_USBNET=y
CONFIG_USB_CATC=y
CONFIG_USB_RTL8152=y
# CONFIG_USB_GADGET is not set
CONFIG_USB_PHY=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_NET_DRIVERS=y
# CONFIG_USB_RTL8153_ECM is not set

The not set ones are the deviations between the web site and the default FOS configurations. Personally I think the key is the last realtek settings but I set them all in the above kernel.

Here are the kernel option descriptions.

x Symbol: USB_IPHETH [=n]                                x
x Type  : tristate                                       x
x Defined at drivers/net/usb/Kconfig:580                 x
x   Prompt: Apple iPhone USB Ethernet driver             x
x   Depends on: NETDEVICES [=y] && USB_NET_DRIVERS [=y]  x
x   Location:                                            x
x     -> Device Drivers                                  x
x       -> Network device support (NETDEVICES [=y])      x
x         -> USB Network Adapters (USB_NET_DRIVERS [=y])

x Symbol: USB_GADGET [=n]                                                    x
 x Type  : tristate                                                           x
 x Defined at drivers/usb/gadget/Kconfig:17                                   x
 x   Prompt: USB Gadget Support                                               x
 x   Depends on: USB_SUPPORT [=y]                                             x
 x   Location:                                                                x
 x     -> Device Drivers                                                      x
 x       -> USB support (USB_SUPPORT [=y])                                    x
 x Selects: USB_COMMON [=y] && NLS [=y]                                       x
 x Selected by [n]:                                                           x
 x   - USB_EHCI_TEGRA [=n] && USB_SUPPORT [=y] && USB [=y] && USB_EHCI_HCD [=yx

 
 x Symbol: USB_RTL8153_ECM [=n]                                               x
 x Type  : tristate                                                           x
 x Defined at drivers/net/usb/Kconfig:638                                     x
 x   Prompt: RTL8153 ECM support                                              x
 x   Depends on: NETDEVICES [=y] && USB_NET_DRIVERS [=y] && USB_NET_CDCETHER [x
 x   Location:                                                                x
 x     -> Device Drivers                                                      x
 x       -> Network device support (NETDEVICES [=y])                          x
 x         -> USB Network Adapters (USB_NET_DRIVERS [=y])                     x
 x                                                                            x
 

@altitudehack said in User Tracking search not working:

So the question is where is there a limit of 134MB? Each php-fpm worker should have a 256MB limit. The issue is the size of what is being reported in the report is larger than a defined buffer. Let me look.

@michaeloberg No problem. Since the files are coming from a google drive and not the “official” fog project web site you need to do things a bit differently.

First download them to your windows computer. They will go into the downloads folder.

Now they are in your downloads folder you have 2 options. You can go the gui route or command line by using putty’s pscp program. The GUI route is probably the easiest, by installing winscp Use winscp to log into your FOG server. If you can login as root then do so. My guess it will only let you login as a normal user. Log into your fog server with winscp. If you can log into as root the navigate to /var/www/html/fog/service/ipxe and drop the files. If you have to log in as a normal user then just drop the files in your normal user home directory. Then open a command prompt and issue
sudo cp bzImage-rt-v215 /var/www/html/fog/service/ipxe
and
sudo cp bzImage-rt-opts /var/www/html/fog/service/ipxe

Edit: Tom knows all of the tricks to get things done…

@michaeloberg Well unfortunately we’ve hit a wall. But you have some additional things to now tell HP support.

We are using the latest linux kernel, with the appropriate settings for this nic driver. Yet we are still not seeing the pass thru mac address.

We attempted to use the latest realtek nic driver with the latest linux kernel and no success. (I’m really not liking this approach because its not keeping the linux kernel pure with well tested drivers, but we needed to do it to rule out an old linux kernel driver at fault).

The native windows driver is doing the same thing.

@nick OK so we can say that time fixes the issue related to getting an IP address. Because in the second picture the ethernet adapter did not get an IP address but when you keyed in the same command it picked up an IP address. So the only thing that changed is time??

What I want you to do now is find one of those cheap 5 or 8 port unmanaged switch. Place that between the pxe booting computer and the building enterprise switch. Now see if the target computer gets an IP address every time pxe booting normally.

If an unmanaged switch fixes the issue then have your networking folks look at the enterprise switch and make sure that fast-STP, RSTP, or port-fast (or what ever your switch mfg calls it) is enabled.

@george1421 This post is to just get my thoughts documented.

If we edit the FOS /etc/init.d/S40network startup script and add a flag check after the network interface has been confirmed. If that flag says swap out the mac address with the one provided by iPXE then use the ip program to replace the mac address.

UNDERSTAND: NOTHING has been tested here. I just finished gluing it together from different code snippets.

The heart of the code would be this section.

    for retry in $(seq 3); do
        /sbin/udhcpc -i $iface --now
        ustat="$?"
        curl -Ikfso /dev/null "${web}"/index.php --connect-timeout 5
        cstat="$?"
        # If the udhcp is okay AND we can curl our web
        # we know we have link so no need to continue on.
        # NOTE: the link to web is kind of important, just
        # exiting on dhcp request is not sufficient.
        if [[ $ustat -eq 0 && $cstat -eq 0 ]]; then
            if [[ ! -z $mac0 && ! -z $domacset ]]; then
            	# Update the mac address if the DOMACSET flag
            	# has been raised
            	ip link set dev $iface down
            	ip link set dev $iface address $mac0
            	ip link set dev $iface up
            fi
            exit 0
        fi
        echo "Either DHCP failed or we were unable to access ${web}/index.php for connection testing."
        # If we are on the 2nd loop, lets check too seeif
        # spanning tree is blocking dhcp
        if [ $seq -eq 2 ]; then
            echo "Waiting for Spanning Tree timeout on $iface"
            sleep 27
        else
            sleep 1
        fi
    done

I also added in a bonus that on the 3rd try to init the interface, I added in a 27 second delay for spanning tree to start forwarding data. I figure if we get to the 3rd try the interface will probably not come up cause it either works right away or it doesn’t. One reason why it doesn’t is that spanning tree has the port in a blocking state.

The complete S40network file would be this:

#!/bin/bash
#
# Start the network....
#
if [[ -n $has_usb_nic ]]; then
    echo "Please unplug your device and replug it into the usb port"
    echo -n "Please press enter key to connect [Enter]"
    read -p "$*"
    echo "Sleeping for 5 seconds to allow USB to sync back with system"
    sleep 5
fi

# Geo-Grabbing kernel parameters because the variables are probably not set yet
for var in $(cat /proc/cmdline); do
	var=$(echo "${var}" | awk -F= '{name=$1; gsub(/[+][_][+]/," ",$2); gsub(/"/,"\\\"", $2); value=$2; if (length($2) == 0 || $0 !~ /=/ || $0 ~ /nvme_core\.default_ps_max_latency_us=/) {print "";} else {printf("%s=%s", name, value)}}')
    [[ -z $var ]] && continue;
    eval "export ${var}" 2>/dev/null
done

# Enable loopback interface
echo -e "auto lo\niface lo inet loopback\n\n" > /etc/network/interfaces
/sbin/ip addr add 127.0.0.1/8 dev lo
/sbin/ip link set lo up

sleep 10

# Generated a sorted list with primary interfaces first
read p_ifaces <<< $(/sbin/ip -0 addr show | awk 'ORS=NR%2?FS:RS' | awk -F'[: ]+' 'tolower($0) ~ /link[/]?ether/ && tolower($0) ~ /'$mac'/ {print $2}' | tr '\n' ' ')
read o_ifaces <<< $(/sbin/ip -0 addr show | awk 'ORS=NR%2?FS:RS' | awk -F'[: ]+' 'tolower($0) ~ /link[/]?ether/ && tolower($0) !~ /'$mac'/ {print $2}' | tr '\n' ' ')
ifaces="$p_ifaces $o_ifaces"
for iface in $ifaces; do
    echo "Starting $iface interface and waiting for the link to come up"
    echo -e "auto $iface\niface $iface inet dhcp\n\n" >> /etc/network/interfaces
    /sbin/ip link set $iface up

    # Wait till the interface is fully up and ready (spanning tree)
    timeout=0
    linkstate=0
    until [[ $linkstate -eq 1 || $timeout -ge 35 ]]; do
        let timeout+=1
        linkstate=$(/bin/cat /sys/class/net/$iface/carrier)
        [[ $linkstate -eq 0 ]] && sleep 1 || break
    done
    [[ $linkstate -eq 0 ]] && echo "No link detected on $iface for $timeout seconds, skipping it." && continue
    for retry in $(seq 3); do
        /sbin/udhcpc -i $iface --now
        ustat="$?"
        curl -Ikfso /dev/null "${web}"/index.php --connect-timeout 5
        cstat="$?"
        # If the udhcp is okay AND we can curl our web
        # we know we have link so no need to continue on.
        # NOTE: the link to web is kind of important, just
        # exiting on dhcp request is not sufficient.
        if [[ $ustat -eq 0 && $cstat -eq 0 ]]; then
            # Geo-Checking to see if we should swap out the mac address
            if [[ ! -z $mac && ! -z $domacset ]]; then
            	# Update the mac address if the DoMACSet flag
            	# has been raised
            	ip link set dev $iface down
            	ip link set dev $iface address $mac
            	ip link set dev $iface up
            fi
            exit 0
        fi
        echo "Either DHCP failed or we were unable to access ${web}/index.php for connection testing."
        # Geo-If we are on the 2nd loop, lets check too seeif
        # spanning tree is blocking dhcp
        if [ $seq -eq 2 ]; then
            echo "Waiting for Spanning Tree timeout on ${iface}..."
            sleep 27
        else
            sleep 1
        fi
    done
    echo "No DHCP response on interface $iface, skipping it."
done

# If we end up here something went wrong as we do exit the script as soon as we get an IP!
if [[ -z "$(echo $ifaces | tr -d ' ')" ]]; then  # because ifaces is constructed with a space, we must strip it
    echo "No network interfaces found, your kernel is most probably missing the correct driver!"
else
    echo "Failed to get an IP via DHCP! Tried on interface(s): $ifaces"
fi
echo "Please check your network setup and try again!"
[[ -z $isdebug ]] && sleep 60 && reboot
echo "Press enter to continue"
read
exit 1

@michaeloberg OK I have a new one-off init built. Download this file to /var/www/html/fog/service/ipxe directory. I debugged and replaced the script below in this this init to support mac spoofing like HP is doing. https://drive.google.com/file/d/1cFsPlkrqlwOjblHErCGr-OXHiKZd8jeZ/view?usp=sharing

We are still debugging here. So lets take baby steps. Once this is downloaded go into the host definition for this specific computer. We need to update some values.

Host Init: init_macset.xz
Host Kernel Arguments: domacset=1

This tells this specific computer to use the new init (virtual hard drive). And the kernel parameter flags the mac replacement code to run. That way without that flag the network inits normally.

Schedule another task in debug mode on this dongled computer. Once in the FOS Linux CLI key in ip a s lets see if the nic has the mac address of the pass through mac. If not key in this command cat /proc/cmdline and post the results here.

@michaeloberg Excellent! The MAC pass through (hack) worked. So now it would be interested in seeing if you can get a working capture or deploy.

For full disclosure (for the Devs mainly) I had a problem building the inits with buildroot. gpartdisk (or something like that) failed to apply the patch that was in the package. That patch failed because it said it was already installed. There was another build root package that threw a similar error. In box cases I just removed the patch from the packages directory and then it built correctly.

My confidence level with this init is 90% just because I had errors with the build. The hack to the startup script was simple and as long as it passes syntax checking it should work so that bit is 99% confidence.

@dejv said in cannot find disk on system (getHardDisk):

we still need to change to ahci?

Yes this is an Intel and Linux kernel developer issue. There is no resolution at this time.

@ombreit I think one of use is confused. I will raise my hand here.

The normal process is to:

1. Create an image definition in the FOG Web UI.
2. PXE boot a source computer and register the computer with FOG. The source computer will be used as your golden or master image.
3. You will then in the web ui connect the new computer with the imaged definition.
4. In the web ui you will then schedule a capture task.
5. You will pxe boot the target computer and capture or upload the golden image to the FOG server.

When you do step 5, in the picture you last show, the image size on client value will be set. The picture tells me you have not uploaded an image yet to the FOG server.

Also from the picture you gave us with lsblk the source computer does not have an image on the disk. So there is nothing to upload to the FOG server.

So I don’t understand where is your golden or master image coming from?

@noobfogger Make sure you have secure boot disabled in the firmware. You have the dhcp settings configured correctly for uefi boot. It appears its downloading the ipxe.efi file but its not able to run it because of secure boot (guess). If it wasn’t getting the file you would see other error messages.

@noobfogger Well this is a bit different conditions than your original post. iPXE is being loaded now but when iPXE starts it issues a second dhcp query to pick up its IP address and loads dhcp option 66 once again to find the FOG server. If it fails to get dhcp option 66 it will prompt for the FOG server IP address.

This is still telling me there is a second dhcp server on this subnet. There is a way to find out. Take a third computer and load wirehark on it. Use this exact capture filter port 67 or port 68 Start wireshark and then pxe boot until you get the failure. Stop wireshark and then inspect the captured packets.

In the top section you should see a DISCOVER packet sent from the pxe booting computer. Then you should see one or more OFFER packets. These OFFER packets will be from each dhcp server that hears a DISCOVER packet. In your case you should only hear an OFFER from the FOG server’s IP address. I’m suspecting you will hear a second OFFER. The response sequence is DISCOVER, OFFER, REQUEST, ACK/NACK. If the target computer gets to the prompting for FOG server IP address question you should see two complete sequences of the DORA process. If you are getting the question that second sequence is failing. If you can’t figure out the pcap, upload it to a file share site and share it as you have the videos and I will take a look at it for you. But it should be obvious if you have a second OFFER in response to a DISCOVER something is not as we expect it to be.