connection timed out chainloading failed

[Sebastian Roth]

@geardog In that case we’ll probably need to look at the actual packets on the wire: https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

[Sebastian Roth]

Had a look at the PCAP and we clearly see the clearOS dnsmasq sending next server information in the DHCP offer/ack packet body. I think it shouldn’t do that in this setup because it’s confusing clients that receive the IP from DHCP and PXE boot information from the proxy DHCP.

Maybe this is just the default behavior of dnsmasq even if it’s not setup as DHCP proxy or full PXE bootp service. Not sure.

[george1421]

@sebastian-roth Would you send me a link to the pcap? I have ubuntu 20.04 running in my lab and have dnsmasq running because my router does the same thing. I can compare it to what I get here. It sends itself as the next server for bootp. dnsmasq (proxydhcp) should override any settings for both bootp and dhcp pxe booting.

One check is to see if at the end of the dora process the target computer reaches out to the fog server on udp port 4011. If that dialog is not happening then the target computer will go with what is in the OFFER packet.

[Sebastian Roth]

@george1421 said in connection timed out chainloading failed:

dnsmasq (proxydhcp) should override any settings for both bootp and dhcp pxe booting.

It kind of does as we see in the screenshot that PXE ROM is actually able to find and load the iPXE binary. Though it seems like iPXE is getting confused by the clearOS dnsmasq (not in proxy mode) sending “Next Server IP” in the DHCP body and therefore wants to load default.ipxe from the wrong IP.

One check is to see if at the end of the dora process the target computer reaches out to the fog server on udp port 4011. If that dialog is not happening then the target computer will go with what is in the OFFER packet.

Yes it does on the first DORA round (PXE ROM) but not on the second round (iPXE).

[george1421]

@sebastian-roth said in connection timed out chainloading failed:

sending “Next Server IP” in the DHCP body and therefore wants to load default.ipxe from the wrong IP.

So from the pxe rom perspective dnsmasq is working as intended. Well then the FOG iPXE script is at fault (not laying actual blame here). It is responsible for taking over once iPXE starts. We may need to look at this exception case here to see if there is a weakness in the script.

My initial concern was that dnsmasq was behaving in an unexpected way causing the initial pxe boot to fail. That is why I wanted to see the pcap.

[Sebastian Roth]

@george1421 said in connection timed out chainloading failed:

Well then the FOG iPXE script is at fault (not laying actual blame here). It is responsible for taking over once iPXE starts. We may need to look at this exception case here to see if there is a weakness in the script.

Good point. Now that I think about it a little more I have a faint memory of trying to detect such a situation in the iPXE script but failed. Though I don’t remember the details but it might be iPXE just not using the information from the proxy DHCP in case it receives next server from the other one. Not sure though.

[geardog]

Thank you guys. If there is anything I can do to facilitate the big brains, let me know.
George, should I email you the PCAP?

[george1421]

@geardog said in connection timed out chainloading failed:

George, should I email you the PCAP?

Sebastian was able to explain what he saw well enough to understand where the problem is and isn’t.

[geardog]

While it was discussed that fog may be the issue, talking to Sebastian had me thinking there was some debate.

Here is the clearOS side.
https://www.clearos.com/clearfoundation/social/community/clearos-dnsmasq-seems-to-step-on-other-nextserver-broadcasts

His tcp dump isn’t showing a next server broadcast. I’m feeling like I may have done something wrong. hmm

[Sebastian Roth]

@geardog Interesting that Nick doesn’t see the next server filed being populated. I am wondering if it’s just something tcpdump doesn’t print on the console? So can you please run tcpdump -i enp5s0 port 67 or port 68 -e -n -vv (maybe need a different network interface name) on your clearOS while PXE booting a machine? Post output here so we can all compare to what Nick posted in the other forum.

[geardog]

@sebastian-roth said in connection timed out chainloading failed:

tcpdump -i enp5s0 port 67 or port 68 -e -n -vv

No mention of next server.

[root@Clear0 etc]# tcpdump -i enp17s0f0 port 67 or port 68 -e -n -vv
tcpdump: listening on enp17s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:24:14.566873 1c:6f:65:83:a0:95 > Broadcast, ethertype IPv4 (0x0800), length 590: (tos 0x0, ttl 20, id 1, offset 0, flags [none], proto                                                                                                    UDP (17), length 576)
    0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 1c:6f:65:83:a0:95, length 548, xid 0x6783a095, secs 8,                                                                                                    Flags [Broadcast] (0x8000)
          Client-Ethernet-Address 1c:6f:65:83:a0:95
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Discover
            Parameter-Request Option 55, length 36:
              Subnet-Mask, Time-Zone, Default-Gateway, Time-Server
              IEN-Name-Server, Domain-Name-Server, RL, Hostname
              BS, Domain-Name, SS, RP
              EP, RSZ, TTL, BR
              YD, YS, NTP, Vendor-Option
              Requested-IP, Lease-Time, Server-ID, RN
              RB, Vendor-Class, TFTP, BF
              Option 128, Option 129, Option 130, Option 131
              Option 132, Option 133, Option 134, Option 135
            MSZ Option 57, length 2: 1260
            GUID Option 97, length 17: 0.49.67.54.70.54.53.56.51.65.48.57.53.255.255.255.255
            ARCH Option 93, length 2: 0
            NDI Option 94, length 3: 1.2.1
            Vendor-Class Option 60, length 32: "PXEClient:Arch:00000:UNDI:002001"
11:24:14.567192 00:15:17:c4:a9:92 > Broadcast, ethertype IPv4 (0x0800), length 366: (tos 0xc0, ttl 64, id 2182, offset 0, flags [none], pr                                                                                                   oto UDP (17), length 352)
    172.17.17.1.bootps > 255.255.255.255.bootpc: [bad udp cksum 0xbe6f -> 0x1006!] BOOTP/DHCP, Reply, length 324, xid 0x6783a095, secs 8,                                                                                                    Flags [Broadcast] (0x8000)
          Your-IP 172.17.17.191
          Server-IP 172.17.17.1
          Client-Ethernet-Address 1c:6f:65:83:a0:95
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Offer
            Server-ID Option 54, length 4: 172.17.17.1
            Lease-Time Option 51, length 4: 43200
            RN Option 58, length 4: 21600
            RB Option 59, length 4: 37800
            Domain-Name Option 15, length 22: "xxx"
            Domain-Name-Server Option 6, length 12: 172.17.17.16,172.17.17.17,172.17.17.1
            Default-Gateway Option 3, length 4: 172.17.17.1
            BR Option 28, length 4: 172.17.17.255
            Subnet-Mask Option 1, length 4: 255.255.255.0
11:24:14.574606 00:15:5d:02:0a:16 > Broadcast, ethertype IPv4 (0x0800), length 374: (tos 0xc0, ttl 64, id 10347, offset 0, flags [none], p                                                                                                   roto UDP (17), length 360)
    172.17.17.82.bootps > 255.255.255.255.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 332, xid 0x6783a095, secs 8, Flags [Broadcast] (0                                                                                                   x8000)
          Server-IP 172.17.17.82
          Client-Ethernet-Address 1c:6f:65:83:a0:95
          file "undionly.kpxe"[|bootp]
11:24:14.575555 00:15:5d:02:0a:16 > Broadcast, ethertype IPv4 (0x0800), length 374: (tos 0xc0, ttl 64, id 10348, offset 0, flags [none], p                                                                                                   roto UDP (17), length 360)
    172.17.17.82.bootps > 255.255.255.255.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 332, xid 0x6783a095, secs 8, Flags [Broadcast] (0                                                                                                   x8000)
          Server-IP 172.17.17.82
          Client-Ethernet-Address 1c:6f:65:83:a0:95
          file "undionly.kpxe"[|bootp]
11:24:22.585346 1c:6f:65:83:a0:95 > Broadcast, ethertype IPv4 (0x0800), length 590: (tos 0x0, ttl 20, id 2, offset 0, flags [none], proto                                                                                                    UDP (17), length 576)

[Sebastian Roth]

@geardog said in connection timed out chainloading failed:

No mention of next server.

So I would expect tcpdump to just not print it on the direct verbose text output. We did see it in the PCAP file. Or you have played with things so much that clearOS doesn’t send it anymore. But then I would expect the PXE boot to work just fine (with dnsmasq on the FOG server).

[geardog]

@sebastian-roth
No adjustments have been made. I was waiting for this to play out.

[geardog]

@sebastian-roth

so a montage of my poking around
In short, after chainloading fails and I run config, there is no proxydhcp listed
when I run dhcp net0 from shell
I then get fog popping up in proxydhcp with the correct ip

What does this mean?

net0



net0

proxydhcp

[Sebastian Roth]

@geardog said in connection timed out chainloading failed:

In short, after chainloading fails and I run config, there is no proxydhcp listed
when I run dhcp net0 from shell
I then get fog popping up in proxydhcp with the correct ip

So it looks like proxydhcp is not set on the first round of DHCP done by iPXE. Quite strange. Could you please verify by printing the variable when you get to the iPXE shell right after chainloading fails:

iPXE> echo ${proxydhcp/next-server}
...
iPXE> dhcp net0
...
iPXE> echo ${proxydhcp/next-server}
...

[geardog]

@sebastian-roth

aye, confirmed

sticky slash key

[Sebastian Roth]

@geardog I am at a loss on why it would not populate the proxydhcp information on the first (successful) round of DHCP. May it’s something to do with timing. We have iPXE binaries that wait for 10 seconds before it does DHCP. Find those in /tftpboot/10secdelay/ directory on your FOG server. Either modify your dnsmasq configuration to point to the iPXE binary in that subdirectory or rename the one in /tftpboot/ and copy the delay binary there for testing.

[geardog]

@sebastian-roth
Well, I made the following edit in ltsp.conf

# Set the root directory for files available via FTP.
tftp-root=/tftpboot/10secdelay

rebooted the fog vm
no apparent change in behavior…
Chainloading still fails, and shell>config output shows no proxydhcp section

[Sebastian Roth]

@geardog Too bad. Well then the next step is to build your custom iPXE binary. Use our script as base and just change those lines:

:proxycheck
isset ${proxydhcp/next-server} && set next-server ${proxydhcp/next-server} || goto nextservercheck

to

:proxycheck
dhcp net0
isset ${proxydhcp/next-server} && set next-server ${proxydhcp/next-server} || goto nextservercheck

[geardog]

@sebastian-roth
What does this get me into? Will I have to recreate/replace these when I reinstall fog? I may be ham fisted, as I’ve probably reinstalled fog 15 times during my use of it. I’d like to be aware of complications in SOP, moving forward.