X1 AIO Desktop - i7 vPro network issue with Intel I219-LM [was: Make new bzImage...]

[Sebastian Roth]

@mandrade Checking out your PCAP file right now. I see that the client (x.x.x.142) first requests undionly.kpxe from the FOG/TFTP server. This seems to work fine. Then there is no packets for more than 80 seconds (!) - I guess you stopped the client, right? After that the client requests undionly.kkpxe from the server. Please tell me this is because you where changing things while capturing. It’s not a problem. I just really hope that it’s not the client requesting two different files on it’s own. Next the client requests default.ipxe - fine - and then boot.php. Here I see that the server responds with a chunked answer. In theory iPXE should be able to handle chunked HTTP transfer properly but I am not sure if I’ve ever seen this on one of my machines. Maybe check your apache and PHP configuration to see why it would send chunked HTTP answer.

POST /fog/service/ipxe/boot.php HTTP/1.1
Content-Length: 152
Content-Type: application/x-www-form-urlencoded
Connection: keep-alive
User-Agent: iPXE/1.0.0+ (9f91d)
Host: x.x.x.14

mac0=aa%3Abb%3Acc%3Add%3Aee%3Aff&arch=x86_64&platform=pcbios&product=20FB001XAU&manufacturer=20FB001XAU&ipxever=1.0.0+%20(9f91d)&filename=undionly.kkpxe

Above the request directly followed by the answer:

HTTP/1.1 200 OK
Date: Fri, 27 May 2016 02:30:55 GMT
Server: Apache/2.4.7 (Ubuntu)
Connection: close
X-Frame-Options: sameorigin
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000
Set-Cookie: PHPSESSID=53k5223j5h32kjh5kj3h453345432kh; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Transfer-Encoding: chunked
Content-Type: text/plain;charset=UTF-8

a18
#!ipxe
set fog-ip x.x.x.14
set fog-webroot fog
set boot-url http://${fog-ip}/${fog-webroot}
...
:fog.reg
kernel bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=x.x.x.14/fog/ consoleblank=0 rootfstype=ext4 loglevel=4 mode=autoreg
imgfetch init.xz
boot || goto MENU
...

0

Why do I think this has something to do with the chunked transfer? Well I am not really sure. What I see in the PCAP is that the server sends the full answer (in two chunks, first 532 bytes and second 2596 bytes) plus the final FIN,ACK packet. The client would need to ACK(nowledge) those 3128 bytes via FIN,ACK as well to finish and close the TCP connection. But the client only sends an ACK for 1981 bytes and is silent from then on as if it would have died - maybe it actually freezes at this stage?!? Are you able to ping the client from the server? Please give that a try while the PXE booting is going on. It should work up to the point where it tries to request boot.php.

If this would be a network problem - like packets gone lost - we would see the client requesting the missing bytes via TCP retransmit. But there are only TCP retransmits from the server side as the client has most probably died.

Would you be able to build/download a iPXE binary as described in https://wiki.fogproject.org/wiki/index.php/IPXE#rom-o-matic.eu. Right at the end you want to add http,httpcore,httpconn into the “Enable Debug” field. Please take a picture or video of what you see on screen then.

[Sebastian Roth]

@mandrade Ok, I found out what’s causing the chunked HTTP transfer. Possibly disabling this for the moment might fix your issue. Please edit /var/www/fog/service/ipxe/boot.php (or /var/www/html/fog/service/ipxe/boot.php) and delete (or comment) the line flush(); (yes only that single one).

No server or service restart needed. Simply boot up your client and see if it works.

@Tom-Elliott The flush() call seems to really push the data to be send to the client - forcing the apache webserver to send it as chunked response because it is not allowed to wait and calculate the full content length before sending the data. Let’s wait and see if this is actually causing the trouble here. Then we might think about reverting this change partly. We don’t need to push it that hard - using flush() - I’d say

[Tom Elliott]

@Sebastian-Roth change made to remove all the extra flushing.

[mandrade]

ok so removing line flush(); from /var/www/html/fog/service/ipxe/boot.php means now it no longer hangs at the bootp.php but now hangs at:

http://X.X.X.X/fog/service/ipxe/bg.png

[mandrade]

oh and for the record, for this machine ipxe will only boot up with undionly.kkpxe. If I use undionly.kpxe it hangs.

[mandrade]

I tried renaming the bg.png file and it went further, only now it hangs here:


[mandrade]

woops forgot to resize:

[Sebastian Roth]

@mandrade On the picture I see that it seems to pop back to the iPXE shell after timing out in the bzImage. Does it hang there or are you able to type in commands? Please run the two iPXE commands ifstat and nstat to see more information about the state of the network (please take a picture). As well run kernel http://x.x.x.142/fog/service/ipxe/bzImage (use your FOG server IP instead of x.x.x) to see if it is able to load the linux kernel via HTTP.

At this stage you have the same two options. Either generate a debugging enabled iPXE binary (see my message below) or you take another packet dump with tcpdump and I can have a look!

[mandrade]

@Sebastian-Roth said in Make new bzImage...:

ither generate a debugging enabled iPXE binar

I am able to type commands I type ‘S’ to be taken to the iPXE shell. So I ran ifstat, nstat and also tried to load the kernel. This was the result:

[Tom Elliott]

@mandrade if undionly.kkpxe works why not just use it? Those systems that work on undionly.kpxe shouldn’t have any issues using undionly.kkpxe and the systems that are currently hanging should work as well.

[mandrade]

@Tom-Elliott ,

I meant the undionly.kkpxe boots past the ipxe boot screens and into the menus where I can inventory etc. The problem then comes when I make select a menu item like inventory for instance. The result you can see in the image below. I cannot inventory or take an image of this host using either undionly.kpxe or undionly.kkpxe only that the undionly.kkpxe gets further in the process.

[Sebastian Roth]

@mandrade Thanks for the new screenshot. I guess I need another tcpdump capture of exactly this! Can you to another one (same command as mentioned before) and send it to me?

[mandrade]

@Sebastian-Roth ,

Hi, I had already sent a second TCPDUMP to the supplied e-mail. Let me know if you didn’t get it I’ll re-send if need be.

[Tom Elliott]

@mandrade Based on what I see of the image as found in the reply link It appears the place where it’s trying to download the file is unreachable. Maybe this is due to firewall? The IP address looks correct, but what’s the IP address of the system? Maybe a DHCP command is needed?

[mandrade]

@Tom-Elliott ,

Thanks for the response Tom. Other workstations work fine, I am able to image them without issue. I did mention earlier that this setup is slightly different from the others in that this laptop doesn’t have an onboard NIC. It uses a OneLink+ mini dock. The thing is I can see it gets an IP from the DHCP server before it starts to the boot process. Somewhere along the line it loses it’s config somehow and then Fog server is no longer reachable.

[Tom Elliott]

@mandrade The boot process is a bit strange.

Most likely where you’re seeing the issue is one of a couple potential pitfalls. The biggest one, I believe, is that the tftp files you’re using appear to be from a newer (developmental) version of fog. As I understand this is a debug generated set of files, I suppose this isn’t the underlying issue, rather a symptom of something else.

The boot process, in simple terms, is not as simple as the initial DHCP. The first DHCP is for PXE to grab information from. Then it hands the system off to the bootfile, in your case undionly.{k,kk}pxe. When it is in undonly side, it needs to reestablish the DHCP protocol. Do you see the system getting an IP from there? Normally it would reboot, but if it’s just skipping and trying to boot, maybe the bootfile is doing something differently as well? Issues that could be happening, I suppose, are STP (Spanning Tree Protocol), firewall blocking this series of data, system is on a vlan that can’t reach the fog server to download the http data, or possibly (unlikely) all of the above.

[mandrade]

@Tom-Elliott ,

I am indeed using a trunked version of FOG. There is no firewall, VLAN separation between the host and FOG. It does get an IP otherwise I would not be able to get to the menu to select an option am I right?

When I select inventory from the menu list it starts to attempt to load the bzImage but fails with connection timed out. All just very strange! I then run the same thing only this time on another host that has an onboard NIC and BOOM it works fine. So my take is that it’s something to do with the mini DOCK. Perhaps drivers? I dunno pure speculation from my part.

[Tom Elliott]

@mandrade Totally understand, just trying to clarify too. Yes, if it’s getting to the menu it should be fine. What if you change the boot file (undionly) to ipxe.pxe?

[Wayne Workman]

This thread has gotten pretty long. Have you tried all the things Sebastian posted? Can we summarize where things are?

[mandrade]

@Tom-Elliott ,

this is what happens if I boot from ipxe.pxe:

It just sits there and eventually reboots