• I have a dedicated FOG server on a Proxmox box I’ve been setting up all day today. I get the following when trying to PXE boot a client.

    https://imgur.com/a/1t2osMZ

    I’ve read through all the other TFTP threads and tried all the troubleshooting steps I could find, clean re-installing multiple times along the way but I still have the same issue. I’m able to succesfully pull undionly.kpxe from a separate windows machine so I can see that TFTP is working.

    No clue what’s going on.


  • @sebastian-roth said in Another TFTP timeout issue:

    Sorry my bad! Newer Ubuntu versions seem to log those requests to /var/log/apache2/other_vhosts_access.log…

    Roger that. I tried it again and I do see the 3 requests for those files.

    alt text

  • Senior Developer

    @choppaholic26 said in Another TFTP timeout issue:

    I PXE booted to the failed registration attempt but unless I’m doing it wrong this file is completely empty for me.

    Sorry my bad! Newer Ubuntu versions seem to log those requests to /var/log/apache2/other_vhosts_access.log


  • @sebastian-roth said in Another TFTP timeout issue:

    Can you please run tail -f /var/log/apache2/access.log while PXE booting the VirtualBox VM? You should see requests for boot.php, then bzImage and init.xz and lastly another request as connection check which we see in the picture. Do you see all those?

    I PXE booted to the failed registration attempt but unless I’m doing it wrong this file is completely empty for me. See below:
    alt text


  • This post is deleted!
  • Senior Developer

    @choppaholic26 The new pictures you posted clearly show that it does get an IP from the DHCP. So from the message “Either DHCP failed or we were unable to access http://192.168.70.11/…” I would imagine the later one to fail. Though the URL looks all fine.

    Can you please run tail -f /var/log/apache2/access.log while PXE booting the VirtualBox VM? You should see requests for boot.php, then bzImage and init.xz and lastly another request as connection check which we see in the picture. Do you see all those?

  • Senior Developer

    @choppaholic26 said in Another TFTP timeout issue:

    However, if I wait a couple of seconds while in the FOG menu, the registration will start (I tested waiting a few seconds on memtest and it started running fine) but DHCP will fail.

    Well, that’s awkward. Maybe some rate limiting firewall on the Proxmox host?! Just a wild guess.

    As far as the setup, I have my physical proxmox server with the installed FOG VM setup to use its interfaces. I have a laptop with VirtualBox running that is plugged directly into the back of the server. To PXE boot I have a bridged adapter for VirtualBox to use to PXE in the VirtualBox settings.

    Ok, sounds reasonable. I do understand that using VirtualBox to test can make things easier and in this case it really is of help to see that TFTP actually works.

    Edit: Thought I should clarify though, that the TFTP timeout issue when PXE booting the actual laptop hasn’t changed.

    Ok. As mentioned before some PXE booting devices are happy to use whatever information they get while others are more picky and won’t boot unless you provide exactly what they are after. So from what we’ve seen so far it looks like the physical notebook doesn’t like the missing DHCP options (66/67).

    I have to admit that I haven’t looked at the DHCP offer/ack packets that isc-dhcp sends out in a long time. Seems like we actually miss the options 66 and 67 in our default config. Probably because 99.9% of machines properly PXE boot with the information given in the DHCP body as mentioned below.

    To get the extra options you need to adjust your config and add option tftp-server-name and option bootfile-name to make it look like this:

    ...
        next-server 192.168.70.11;
        option tftp-server-name "192.168.70.11";
        class "Legacy" {
            match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
            filename "undionly.kkpxe";
            option bootfile-name "undionly.kkpxe";
        }
    ...
    

  • @sebastian-roth Disabled the firewall but the issue persisted. I got something else though.

    Here’s the pcap file for the following registration attemp I’ve outlined below: output.pcap

    I tried doing a registration like you suggested and I got this:

    alt text

    However, if I wait a couple of seconds while in the FOG menu, the registration will start (I tested waiting a few seconds on memtest and it started running fine) but DHCP will fail. See here:

    alt text
    alt text

    As far as the setup, I have my physical proxmox server with the installed FOG VM setup to use its interfaces. I have a laptop with VirtualBox running that is plugged directly into the back of the server. To PXE boot I have a bridged adapter for VirtualBox to use to PXE in the VirtualBox settings.

    I’ve tried eliminating the complexity by also by just physically PXE booting the actual laptop. But I keep getting the same results anyways so it’s just faster to use VirtualBox for me without having to restart over and over again.

    Edit: Thought I should clarify though, that the TFTP timeout issue when PXE booting the actual laptop hasn’t changed. Just thought that VirtualBox getting past that might say something about what issue could be.

  • Senior Developer

    @choppaholic26 Wow, seems like an endeavor and I am not exactly sure where to jive in. Although George is right the DHCP packets seem to be missing the actual DHCP options 66 and 67 I still think that some clients can handle this and PXE boot using the information seen in the DHCP base body.

    I would focus on capturing more traffic on the FOG server to see if the TFTP requests actually make it to the server. If you have used the command as suggested in the post by George (tcpdump -w output.pcap port 67 or port 68 or port 69 or port 4011) we should actually see TFTP requests coming in (as from the initial picture you posted we know it tries to).

    You might check if the Ubuntu firewall is blocking it! sudo ufw disable and try again.

    By the way, where did you setup the VirtualBox stuff so it would PXE boot off you FOG server running in a VM on the Proxmox server. I am wondering if it’s more due to complexity in the setup that things don’t work. On the other hand the new screenshots show it actually pull the iPXE binary from the server (so TFTP transfer must work). Beside booting into the memdisk task, have you tried doing a registration through the FOG menu yet?


  • @george1421 I think I might have something.

    1. I can:
      • Ping the fog server (192.168.70.11) successfully.
      • View the fog management web portal in my browser.
      • My workstation picks up an IP (192.168.70.33) so I can see that fog’s DHCP is working.
    2. Tried to PXE boot with several laptops, got the TFTP timeout:

    I tried something and I think I got something. I tried PXE booting using Virtualbox instead. IT DOES SUCCESSFULLY PXE BOOT but take a look at network interface. During the PXE boot you can see that it’s not up initially.

    alt text
    alt text

    After getting to the fog menu this is what happens when you try and pick an item (in this case memtest). You can see that the connection is dropping for whatever unknown reason.

    alt text

    The cables are good. The equipment and ports are good, everything lights up. DHCP is working. Fog management page is accessible.

    I’m out of ideas here.


  • @george1421 I’m using Proxmox 6.2 and Ubuntu Server 20.04 for fog.

    Does the networking inside the VM and in Proxmox look right to you? I’m sorry for bothering but I’m a little out of my element here.

    alt text
    alt text
    alt text

  • Moderator

    @choppaholic26 Well I guess the only thing I can do is try to duplicate it in my home lab. What host OS is the fog server running on?

    When I test it I’ll use a vm on the proxmox server to first eliminate any network interconnect to physical adapter nonsense.

    … this can’t be a proxmox firewall issue because the packets are getting through, they just are not containing all of the details. …


  • @george1421 Don’t really know where to look now. This is the output from ps aux | grep dhcp:

    alt text

  • Moderator

    @choppaholic26 Well this is confusing because the configuration file you provided should not do this. Here is your dhcp server OFFER packet that tells the client about the network.

    dhcp_bad.png

    You will see what is missing in the above screen shot is the dhcp options 66 and 67 which is part of the dhcp pxe boot protocol. The bootp protocol (fields highlighted in the ethernet header) are present. I’m pretty sure that the isc-dhcp server is looking at this config file because it issued the ip address in the right range for the scope. This is what I’m expecting:

    dhcp_good.png

    Is it possible that the isc-dhcp server is looking at a different config file ?? ps aux | grep dhcp I think in the command line it might define a config file.

    Now the only thing I see different between your config file and the example one on the wiki site is the example one has an authoritative; directive. But that shouldn’t cause that much of the a change between what we see in the pcap and what is configured.

    Right now this problem is between/with the isc-dhcp server and not really anything to do with FOG (at the moment)


  • @george1421 Here is the dhcpd.conf

    # DHCP Server Configuration file\n#see /usr/share/doc/dhcp*/dhcpd.conf.sample
    # This file was created by FOG
    #Definition of PXE-specific options
    # Code 1: Multicast IP Address of bootfile
    # Code 2: UDP Port that client should monitor for MTFTP Responses
    # Code 3: UDP Port that MTFTP servers are using to listen for MTFTP requests
    # Code 4: Number of seconds a client must listen for activity before trying
    #         to start a new MTFTP transfer
    # Code 5: Number of seconds a client must listen before trying to restart
    #         a MTFTP transfer
    option space PXE;
    option PXE.mtftp-ip code 1 = ip-address;
    option PXE.mtftp-cport code 2 = unsigned integer 16;
    option PXE.mtftp-sport code 3 = unsigned integer 16;
    option PXE.mtftp-tmout code 4 = unsigned integer 8;
    option PXE.mtftp-delay code 5 = unsigned integer 8;
    option arch code 93 = unsigned integer 16;
    use-host-decl-names on;
    ddns-update-style interim;
    ignore client-updates;
    # Specify subnet of ether device you do NOT want service.
    # For systems with two or more ethernet devices.
    # subnet 136.165.0.0 netmask 255.255.0.0 {}
    subnet 192.168.70.0 netmask 255.255.255.0{
        option subnet-mask 255.255.255.0;
        range dynamic-bootp 192.168.70.32 192.168.70.254;
        default-lease-time 21600;
        max-lease-time 43200;
        option routers 192.168.70.1;
        #option routers 0.0.0.0
        next-server 192.168.70.11;
        class "Legacy" {
            match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
            filename "undionly.kkpxe";
        }
        class "UEFI-32-2" {
            match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
            filename "i386-efi/ipxe.efi";
        }
        class "UEFI-32-1" {
            match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
            filename "i386-efi/ipxe.efi";
        }
        class "UEFI-64-1" {
            match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00007";
            filename "ipxe.efi";
        }
        class "UEFI-64-2" {
            match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00008";
            filename "ipxe.efi";
        }
        class "UEFI-64-3" {
            match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00009";
            filename "ipxe.efi";
        }
        class "SURFACE-PRO-4" {
            match if substring(option vendor-class-identifier, 0, 32) = "PXEClient:Arch:00007:UNDI:003016";
            filename "ipxe.efi";
        }
        class "Apple-Intel-Netboot" {
            match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
            option dhcp-parameter-request-list 1,3,17,43,60;
            if (option dhcp-message-type = 8) {
                option vendor-class-identifier "AAPLBSDPC";
                if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                    # BSDP List
                    option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                    filename "ipxe.efi";
                }
            }
        }
    }
    
  • Moderator

    @choppaholic26 Ah I see it. The dhcp server is only sending out the bootp fields (in the ethernet neader) not the dhcp fields (options 66 and 67).

    Lets compare your current isc-dhcp server config file to the one outlined here. https://wiki.fogproject.org/wiki/index.php/BIOS_and_UEFI_Co-Existence#Example_1 There has to be something missing from your config file.

    Depending on the target computer some only look at the bootp information (in the header) and others look at the dhcp options (66 and 67) and some look at both. So if both fields are not populated then you may have issue with random target computers.

  • Moderator

    @choppaholic26 And this pcap generated the previous error message about no dhcp offers were received?

    This is a physical computer?

    The reason why I asked is because this appears to be a normal pcap. I would have expected the target computer to reach out to the fog server and ask for the file after that.


  • @george1421 The FOG installer did. At least I think so. I never messed with any DHCP settings.

    Here’s the .pcap file.

    output.pcap

  • Moderator

    @choppaholic26 So did you configure the dhcp server or did the FOG Installer do this?

    I understand what the error means and what can cause it. I just want to look at the obvious things first.

    So if we can’t fingure out what is wrong we can use tcpdump on the fog server to see what is flying down your wires.

    https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

    If you have wireshark on your fog server you can use that too, but the tcpdump command will only give us dhcp info. The output of that file output.pcap can be moved to a windows computer and reviewed with wireshark.

    I would expect to see the DORA (DISCOVER, OFFER, REQUEST, ACK) process it should be about 4-6 packets. The client will send out a DISCOVER packet and the FOG server should respond with a OFFER packet. Based on the error message one of the two is not happening. If you don’t see the DISCOVER packet then you need to look into your proxmox networking to ensure the vm is connected to the network adapter. If you see the discover but no offer then check to see if the firewall is turned off and/or the dhcp server is running ps aux|grep dhcp should return the process ID of the dhcp server. If the dhcp server is running then lets look at the configuration.


  • @george1421 When asked if I wanted FOG to handle DHCP I put yes.

    Side note I tried reverting to snapshot and trying another FOG install and now I can’t pull undionly.kpxe, I get this:

    alt text

    And when PXE booting I get this:

    alt text

    I’m going to just wipe the VM and try again. Think I messed something up.

    Edit: Okay. I’m back to just the regular old TFTP timeout issue. Back to square one at least.

273
Online

8.0k
Users

14.9k
Topics

140.5k
Posts