connection timed out chainloading failed


  • Fog 1.5.9 on ubu 20.04.1 VM not handling dhcp, using dnsmasq

    I feel like the flaw is the tftp:// ip address showing the gateway, but fog settings shows the correct ip of fog.

    I’m close to having it up again, but … any help would be great !

    d0360782-6df0-46d2-a310-b1c21c90d047-image.png

    38e78eef-72f5-46c4-ae6a-9fd91d8d404d-image.png

    fog-service-output.txt


  • @sebastian-roth So I’m not sure what has changed, as I’ve had to focus on other things, but now it works. I didn’t follow the most recent direction, but I’m booting various boxes via pxe successfully. No idea. …

  • Senior Developer

    @geardog Yes sure, building your custom iPXE binaries needs manual steps every time you run the FOG installer.


  • @sebastian-roth
    What does this get me into? Will I have to recreate/replace these when I reinstall fog? I may be ham fisted, as I’ve probably reinstalled fog 15 times during my use of it. I’d like to be aware of complications in SOP, moving forward.

  • Senior Developer

    @geardog Too bad. Well then the next step is to build your custom iPXE binary. Use our script as base and just change those lines:

    :proxycheck
    isset ${proxydhcp/next-server} && set next-server ${proxydhcp/next-server} || goto nextservercheck
    

    to

    :proxycheck
    dhcp net0
    isset ${proxydhcp/next-server} && set next-server ${proxydhcp/next-server} || goto nextservercheck

  • @sebastian-roth
    Well, I made the following edit in ltsp.conf

    
    # Set the root directory for files available via FTP.
    tftp-root=/tftpboot/10secdelay
    
    

    rebooted the fog vm
    no apparent change in behavior…
    Chainloading still fails, and shell>config output shows no proxydhcp section

  • Senior Developer

    @geardog I am at a loss on why it would not populate the proxydhcp information on the first (successful) round of DHCP. May it’s something to do with timing. We have iPXE binaries that wait for 10 seconds before it does DHCP. Find those in /tftpboot/10secdelay/ directory on your FOG server. Either modify your dnsmasq configuration to point to the iPXE binary in that subdirectory or rename the one in /tftpboot/ and copy the delay binary there for testing.


  • @sebastian-roth

    aye, confirmed
    20210310_145208.jpg
    sticky slash key

  • Senior Developer

    @geardog said in connection timed out chainloading failed:

    In short, after chainloading fails and I run config, there is no proxydhcp listed
    when I run dhcp net0 from shell
    I then get fog popping up in proxydhcp with the correct ip

    So it looks like proxydhcp is not set on the first round of DHCP done by iPXE. Quite strange. Could you please verify by printing the variable when you get to the iPXE shell right after chainloading fails:

    iPXE> echo ${proxydhcp/next-server}
    ...
    iPXE> dhcp net0
    ...
    iPXE> echo ${proxydhcp/next-server}
    ...
    

  • @sebastian-roth

    so a montage of my poking around
    In short, after chainloading fails and I run config, there is no proxydhcp listed
    when I run dhcp net0 from shell
    I then get fog popping up in proxydhcp with the correct ip

    What does this mean?

    1.jpg
    3.jpg
    7.jpg
    8.jpg
    net0
    9.jpg
    11.jpg
    12.jpg
    net0
    13-.jpg
    proxydhcp
    15.jpg


  • @sebastian-roth
    No adjustments have been made. I was waiting for this to play out.

  • Senior Developer

    @geardog said in connection timed out chainloading failed:

    No mention of next server.

    So I would expect tcpdump to just not print it on the direct verbose text output. We did see it in the PCAP file. Or you have played with things so much that clearOS doesn’t send it anymore. But then I would expect the PXE boot to work just fine (with dnsmasq on the FOG server).


  • @sebastian-roth said in connection timed out chainloading failed:

    tcpdump -i enp5s0 port 67 or port 68 -e -n -vv

    No mention of next server.

    [root@Clear0 etc]# tcpdump -i enp17s0f0 port 67 or port 68 -e -n -vv
    tcpdump: listening on enp17s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
    11:24:14.566873 1c:6f:65:83:a0:95 > Broadcast, ethertype IPv4 (0x0800), length 590: (tos 0x0, ttl 20, id 1, offset 0, flags [none], proto                                                                                                    UDP (17), length 576)
        0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 1c:6f:65:83:a0:95, length 548, xid 0x6783a095, secs 8,                                                                                                    Flags [Broadcast] (0x8000)
              Client-Ethernet-Address 1c:6f:65:83:a0:95
              Vendor-rfc1048 Extensions
                Magic Cookie 0x63825363
                DHCP-Message Option 53, length 1: Discover
                Parameter-Request Option 55, length 36:
                  Subnet-Mask, Time-Zone, Default-Gateway, Time-Server
                  IEN-Name-Server, Domain-Name-Server, RL, Hostname
                  BS, Domain-Name, SS, RP
                  EP, RSZ, TTL, BR
                  YD, YS, NTP, Vendor-Option
                  Requested-IP, Lease-Time, Server-ID, RN
                  RB, Vendor-Class, TFTP, BF
                  Option 128, Option 129, Option 130, Option 131
                  Option 132, Option 133, Option 134, Option 135
                MSZ Option 57, length 2: 1260
                GUID Option 97, length 17: 0.49.67.54.70.54.53.56.51.65.48.57.53.255.255.255.255
                ARCH Option 93, length 2: 0
                NDI Option 94, length 3: 1.2.1
                Vendor-Class Option 60, length 32: "PXEClient:Arch:00000:UNDI:002001"
    11:24:14.567192 00:15:17:c4:a9:92 > Broadcast, ethertype IPv4 (0x0800), length 366: (tos 0xc0, ttl 64, id 2182, offset 0, flags [none], pr                                                                                                   oto UDP (17), length 352)
        172.17.17.1.bootps > 255.255.255.255.bootpc: [bad udp cksum 0xbe6f -> 0x1006!] BOOTP/DHCP, Reply, length 324, xid 0x6783a095, secs 8,                                                                                                    Flags [Broadcast] (0x8000)
              Your-IP 172.17.17.191
              Server-IP 172.17.17.1
              Client-Ethernet-Address 1c:6f:65:83:a0:95
              Vendor-rfc1048 Extensions
                Magic Cookie 0x63825363
                DHCP-Message Option 53, length 1: Offer
                Server-ID Option 54, length 4: 172.17.17.1
                Lease-Time Option 51, length 4: 43200
                RN Option 58, length 4: 21600
                RB Option 59, length 4: 37800
                Domain-Name Option 15, length 22: "xxx"
                Domain-Name-Server Option 6, length 12: 172.17.17.16,172.17.17.17,172.17.17.1
                Default-Gateway Option 3, length 4: 172.17.17.1
                BR Option 28, length 4: 172.17.17.255
                Subnet-Mask Option 1, length 4: 255.255.255.0
    11:24:14.574606 00:15:5d:02:0a:16 > Broadcast, ethertype IPv4 (0x0800), length 374: (tos 0xc0, ttl 64, id 10347, offset 0, flags [none], p                                                                                                   roto UDP (17), length 360)
        172.17.17.82.bootps > 255.255.255.255.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 332, xid 0x6783a095, secs 8, Flags [Broadcast] (0                                                                                                   x8000)
              Server-IP 172.17.17.82
              Client-Ethernet-Address 1c:6f:65:83:a0:95
              file "undionly.kpxe"[|bootp]
    11:24:14.575555 00:15:5d:02:0a:16 > Broadcast, ethertype IPv4 (0x0800), length 374: (tos 0xc0, ttl 64, id 10348, offset 0, flags [none], p                                                                                                   roto UDP (17), length 360)
        172.17.17.82.bootps > 255.255.255.255.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 332, xid 0x6783a095, secs 8, Flags [Broadcast] (0                                                                                                   x8000)
              Server-IP 172.17.17.82
              Client-Ethernet-Address 1c:6f:65:83:a0:95
              file "undionly.kpxe"[|bootp]
    11:24:22.585346 1c:6f:65:83:a0:95 > Broadcast, ethertype IPv4 (0x0800), length 590: (tos 0x0, ttl 20, id 2, offset 0, flags [none], proto                                                                                                    UDP (17), length 576)
    
    
  • Senior Developer

    @geardog Interesting that Nick doesn’t see the next server filed being populated. I am wondering if it’s just something tcpdump doesn’t print on the console? So can you please run tcpdump -i enp5s0 port 67 or port 68 -e -n -vv (maybe need a different network interface name) on your clearOS while PXE booting a machine? Post output here so we can all compare to what Nick posted in the other forum.


  • While it was discussed that fog may be the issue, talking to Sebastian had me thinking there was some debate.

    Here is the clearOS side.
    https://www.clearos.com/clearfoundation/social/community/clearos-dnsmasq-seems-to-step-on-other-nextserver-broadcasts

    His tcp dump isn’t showing a next server broadcast. I’m feeling like I may have done something wrong. hmm

  • Moderator

    @geardog said in connection timed out chainloading failed:

    George, should I email you the PCAP?

    Sebastian was able to explain what he saw well enough to understand where the problem is and isn’t.


  • Thank you guys. If there is anything I can do to facilitate the big brains, let me know.
    George, should I email you the PCAP?

  • Senior Developer

    @george1421 said in connection timed out chainloading failed:

    Well then the FOG iPXE script is at fault (not laying actual blame here). It is responsible for taking over once iPXE starts. We may need to look at this exception case here to see if there is a weakness in the script.

    Good point. Now that I think about it a little more I have a faint memory of trying to detect such a situation in the iPXE script but failed. Though I don’t remember the details but it might be iPXE just not using the information from the proxy DHCP in case it receives next server from the other one. Not sure though.

  • Moderator

    @sebastian-roth said in connection timed out chainloading failed:

    sending “Next Server IP” in the DHCP body and therefore wants to load default.ipxe from the wrong IP.

    So from the pxe rom perspective dnsmasq is working as intended. Well then the FOG iPXE script is at fault (not laying actual blame here). It is responsible for taking over once iPXE starts. We may need to look at this exception case here to see if there is a weakness in the script.

    My initial concern was that dnsmasq was behaving in an unexpected way causing the initial pxe boot to fail. That is why I wanted to see the pcap.

  • Senior Developer

    @george1421 said in connection timed out chainloading failed:

    dnsmasq (proxydhcp) should override any settings for both bootp and dhcp pxe booting.

    It kind of does as we see in the screenshot that PXE ROM is actually able to find and load the iPXE binary. Though it seems like iPXE is getting confused by the clearOS dnsmasq (not in proxy mode) sending “Next Server IP” in the DHCP body and therefore wants to load default.ipxe from the wrong IP.

    One check is to see if at the end of the dora process the target computer reaches out to the fog server on udp port 4011. If that dialog is not happening then the target computer will go with what is in the OFFER packet.

    Yes it does on the first DORA round (PXE ROM) but not on the second round (iPXE).

318
Online

8.1k
Users

15.0k
Topics

141.3k
Posts