SVN 5209 Not able to pxe boot

[Hanz]

Ipxe will show initializing devices and then a very brief glimpse of where it typically shows all supported ipxe “processes” ie HTTP and so on.

The systemctl status httpd shows over 4500 connections, but I only have the client on about 590 machines. I believe it’s getting timed out due to the large number of connections, but this is only a guess.

httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2015-11-03 10:36:58 EST; 2min 55s ago
Main PID: 1036 (httpd)
Status: “Total requests: 4522; Idle/Busy workers 15/84;Requests/sec: 27.4; Bytes served/sec: 48KB/sec”
CGroup: /system.slice/httpd.service
├─1036 /usr/sbin/httpd -DFOREGROUND
├─1237 /usr/sbin/httpd -DFOREGROUND
├─1239 /usr/sbin/httpd -DFOREGROUND
├─1240 /usr/sbin/httpd -DFOREGROUND
├─1305 /usr/sbin/httpd -DFOREGROUND
├─1320 /usr/sbin/httpd -DFOREGROUND
├─1360 /usr/sbin/httpd -DFOREGROUND
├─1361 /usr/sbin/httpd -DFOREGROUND
├─1362 /usr/sbin/httpd -DFOREGROUND
├─1447 /usr/sbin/httpd -DFOREGROUND
├─1450 /usr/sbin/httpd -DFOREGROUND
├─1456 /usr/sbin/httpd -DFOREGROUND
├─1479 /usr/sbin/httpd -DFOREGROUND
├─1500 /usr/sbin/httpd -DFOREGROUND
├─1506 /usr/sbin/httpd -DFOREGROUND
├─2809 /usr/sbin/httpd -DFOREGROUND
├─2854 /usr/sbin/httpd -DFOREGROUND
├─2855 /usr/sbin/httpd -DFOREGROUND
├─2898 /usr/sbin/httpd -DFOREGROUND
├─2899 /usr/sbin/httpd -DFOREGROUND
├─2900 /usr/sbin/httpd -DFOREGROUND
├─2910 /usr/sbin/httpd -DFOREGROUND
├─2963 /usr/sbin/httpd -DFOREGROUND
├─2965 /usr/sbin/httpd -DFOREGROUND
├─2973 /usr/sbin/httpd -DFOREGROUND
├─2982 /usr/sbin/httpd -DFOREGROUND
├─2987 /usr/sbin/httpd -DFOREGROUND
├─2991 /usr/sbin/httpd -DFOREGROUND
├─2992 /usr/sbin/httpd -DFOREGROUND
├─3286 /usr/sbin/httpd -DFOREGROUND
├─3287 /usr/sbin/httpd -DFOREGROUND
├─3372 /usr/sbin/httpd -DFOREGROUND
├─3373 /usr/sbin/httpd -DFOREGROUND
├─3436 /usr/sbin/httpd -DFOREGROUND
├─3438 /usr/sbin/httpd -DFOREGROUND
├─3504 /usr/sbin/httpd -DFOREGROUND
├─3516 /usr/sbin/httpd -DFOREGROUND
├─3520 /usr/sbin/httpd -DFOREGROUND
├─3529 /usr/sbin/httpd -DFOREGROUND
├─5302 /usr/sbin/httpd -DFOREGROUND
├─5331 /usr/sbin/httpd -DFOREGROUND
├─5332 /usr/sbin/httpd -DFOREGROUND
├─5365 /usr/sbin/httpd -DFOREGROUND
├─5366 /usr/sbin/httpd -DFOREGROUND
├─5367 /usr/sbin/httpd -DFOREGROUND
├─5368 /usr/sbin/httpd -DFOREGROUND
├─5400 /usr/sbin/httpd -DFOREGROUND
├─5402 /usr/sbin/httpd -DFOREGROUND
├─5403 /usr/sbin/httpd -DFOREGROUND
├─5404 /usr/sbin/httpd -DFOREGROUND
├─5408 /usr/sbin/httpd -DFOREGROUND
├─5906 /usr/sbin/httpd -DFOREGROUND
└─6085 /usr/sbin/httpd -DFOREGROUND

Nov 03 10:36:53 fog-server systemd[1]: Starting The Apache HTTP Server…
Nov 03 10:36:58 fog-server systemd[1]: Started The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.

[Hanz]

within a couple minutes the total requests has tripled.

[Hanz]

fixed per SVN 5211, but pxe problems still exist

[Sebastian Roth]

What do you see in apache error and access log when trying PXE boot?

[Joseph Hales]

Where is the access log under ubuntu 14.

[Hanz]

@Sebastian-Roth I’m not seeing anything actually pertaining to the ip assigned to machine while pxe booting

[Sebastian Roth]

/var/log/apache2/access.log and /var/log/apache2/error.log

[Sebastian Roth]

@Hanz Well, without an error message or any other hints we won’t be able to help!

Try capturing DHCP and TFTP traffic on the server while booting one particular client. Please make sure only one client is booting at that time so the packet dump won’t be several MB big. Install package tcpdump and then:

tcpdump -i eth0 -w pxe.pcap port 67 or port 68 or port 69

Leave it there and PXE boot the client. After it failed stop tcpdump (Ctrl-c) and upload the PCAP file.

[Hanz]

@Sebastian-Roth pxe.pcap

[Raymond Bell]

Also unable to iPXE boot with r4285

[Sebastian Roth]

@Tom-Elliott Seams like the current undionly.kpxe binary (possibly also the other iPXE binaries too) is corrupt. Can you update again. Don’t think this is related to the script change (proxydhcp) as it completely crashes in my test QEMU setup. Script error wouldn’t crash I suppose.

[Joseph Hales]

I also saw this issue but couldn’t trouble shoot as I needed it up for production I also didn’t see anything weird in the logs. Right now I’m reverted to 4978.

[Wayne Workman]

@Sebastian-Roth said:

@Tom-Elliott Seams like the current undionly.kpxe binary (possibly also the other iPXE binaries too) is corrupt. Can you update again. Don’t think this is related to the script change (proxydhcp) as it completely crashes in my test QEMU setup. Script error wouldn’t crash I suppose.

I just tested and you are correct, something is corrupt/not right at all. I just did a test upload using the latest svn and it failed exactly as described here:
https://forums.fogproject.org/topic/6050/imaging-isseus-new-host-svn-5201

[Tom Elliott]

PXE Booting should be fixed.

We had a bad caller in the ipxe script.

Particularly, the code segment:

...
isset(${proxydhcp/next-server}) && set next-server ${proxydhcp/next-server}
...

It doesn’t fail out propery (releases back to BIOS if you will) if the proxydhcp/next-server variable is not set. The fix was simple, just make the segment as:

...
isset(${proxydhcp/next-server}) && set next-server ${proxydhcp/next-server} ||
...

[Hanz]

@Tom-Elliott Thank you sir…I’m sorry I didn’t have much info to help diagnose. Appreciate everyone’s help.