NBP filesize is 0 Bytes; PXE-E18: Server response timeout

[frazzor123]

Hi Guys,

first of all, let me tell you that you already helped so much! Great forum and awesome community.
Unfortunately, I have an issue I didn’t find a solution in the forum or wiki.

I’ve set up a fog server yesterday. Actually it was a bit of a pain cause of my nvme drives on host and server, UEFI, firewall/gateway, etc. but finally, I figured everything out with your help and all of the previous topics in this forum and got it up and running. Next step was to relocate the server back to the VLAN it belongs and everything stopped working. (Actually it was in a different VLAN than the clients all the time, but it was the wrong one. I know… Stupid mistake…).

My configuration:
FOG Server - 1.5.7
Ubuntu 18.04 LTS
Sophos Firewall between the VLANs (also my DHCP)
40ish clients, all with the same specs (AMI Motherboard, Intel i7, nvme, etc.)

I’m getting the error message:
NBP filename is ipxe.efi
NBP filesize is 0 bytes
PXE-E18: Server response timeout.

I had the same issue yesterday because I forgot to set up the “next-server” in my DHCP settings (I just had 66 and 67).

What I tried to fix it:

• Check DHCP settings - 66, 67 and next-server all seem to fit
• restarted TFTP service
• logged in via FileZilla (everything is up and running and the filesize of the ipxe.efi is something around 8MB if I recall correctly)
• tried to get ipxe.efi via TFTP in windows CLI (didn’t work though)
• checked the FTP credentials everywhere
• I didn’t change anything on the client-side from the working state, so I didn’t touch their configuration since the error
• I did a tcpdump and checked it in Wireshark - the client sends the request but the server doesn’t answer
• reinstalled the FOG server (just in case)

The clients I checked with worked perfectly fine (after changing half of the BIOS setting, including AHCI or turning off secure boot, and changing the host’s disk to /dev/nvme0n1), and it only worked with the ipxe.efi so far. But I tried the undionly.kpxe first.
I hope, that I didn’t forget any of my tries to fix it.

But right now I have no more idea what to do.

Does anybody have a clue on that one? Is there any more data I can provide?

Thanks a lot in advance,

Mo

[frazzor123]

@george1421
Yes, I just figured it out. It had something to do with a little Tool called “TFTP Helper” I needed it to activate in the firewall.
Now everything is going great.
Thanks a lot for your help!

[george1421]

OK so is the fog server and pxe target computer on the same subnets?

If yes then I might think the tftp service on the fog server isn’t working or configured correctly.
If no then I would ask your networking team if they are filtering tftp (udp port 69) traffic between the subnets (vlans).

[frazzor123]

Hey George,

thanks for your fast reply.
No, the VLANs are not on the same subnet, but - for testing purposes - no traffic is blocked between the VLANs and all ports are open.
When it was working clients and server hasn’t been on the same subnet either (it was just a different one).
Do you have anything in mind what could have been misconfigured on the tftp service on the server?

[frazzor123]

While triple checking everything I found that the TFTP PXE KERNEL DIR has been set to /var/www/fog//service/ipxe/
I changed it and deleted the second “/”, but it didn’t change anything. Since this is a fairly fresh installation and I’m pretty sure I didn’t change the directory, I’m thinking if it maybe was on purpose, but I don’t see a reason why.

[george1421]

@frazzor123 Ok what I want you to do this is to follow this tutorial to create a pcap of the pxe booting process. This will capture what the target computer is being told to do. Its a little complicated but the quickest way to give us a direction.

https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

Upload the pcap to a file share site (i.e. google drive) and paste the link here. Once I review it you can remove the file from the fileshare site.

[frazzor123]

@george1421
I already did the tcpdump some hours ago, but i did a new one right now. You’ll find it here: https://drive.google.com/open?id=1BGH3VrgHNrPYtt9DzUF2BUEI3YL59mkf
192.168.110.100 is the client (before ip address was assigned through our dhcp)

[george1421]

@frazzor123 will you share the file as public so I can get it

[frazzor123]

@george1421 Sorry about that!! Should be public now

[george1421]

@frazzor123 That isn’t the complete pcap its only the tftp bits.

But what I can see from it is the FOG server and pxe booting client computer are on different subnets. This will mask the dhcp part of the pcap. That’s ok.

What I’m seeing is the target computer 192.168.110.100 keeps asking 192.168.140.176 for the file size but I don’t see an answer from the FOG server.

So on the fog server key in the following command to see if the tftp server is listening.

netstat -an | grep :69

The results of that command should look like this if the tftp server is running.

udp        0      0 0.0.0.0:69              0.0.0.0:*

If its listening make sure the ubuntu firewall is turned off.

The next check you should be able to do with a windows computer, you will need to add the tftp client feature into windows 10. Then issue a tftp -i <fog_server_ip> get ipxe.efi Note you may have to monetarily drop the windows firewall or create an exception for the tftp.exe client program to get the transfer to happen. tftp is akin to ftp where it opens 2 channels one for commands and one for data. We are only interested in seeing if the download works.

[frazzor123]

@george1421
Yes, they are are on different subnets. That worked earlier.

The netstat output is:

udp      0    0 0.0.0.0:69             0.0.0.0*
udp6     0    0 :::69                  :::*

The linux firewall (ufw) is disabled and inactive.

I one more time turned off the windows firewall and tried to get the ipxe.efi over tftp in windows command line. There was a request timeout.
Do have anything else I could check?

[george1421]

@frazzor123 So your test tftp computer is on a different subnet as your FOG server?

If your test computer was on a different subnet then do you have another computer on the same subnet as the FOG server? I’m suspecting there is some type of screening firewall between the two subnets. If we can test tftp from a device on the same subnet as the FOG server that will tell us where the problem isn’t

If the above test fails then you should install the tftp client on your FOG server (understand we keep moving towards the source with these tests). Then issue the following command from your fog server tftp -4 192.168.140.176 -c get ipxe.efi If that fails then the problem is surely in the FOG server host OS.

Edit: BTW the output from the netstat commands means that the tftp service is listening on the proper port and its up.

[frazzor123]

@george1421
If I put the two computers on the same subnet it does work and I can get the ipxe.efi file with windows…
I guess that is progress and we can be pretty sure that it is a firewall issue.

Do you have any thoughts on that? I implemented a rule that allows all traffic through all ports between the two networks.

[george1421]

@frazzor123 said in NBP filesize is 0 Bytes; PXE-E18: Server response timeout:

Do you have any thoughts on that? I implemented a rule that allows all traffic through all ports between the two networks.

Well since they are on different subnets, can the workstations on the remote subnet ping the FOG server or connect to the web ui? (realize I know nothing about how your network is designed)

If yes then network routing is setup correctly. If no then you need to look into how the fog server’s networking is setup. Is a default route defined? I’m assuming yes since you were able to install fog, because the fog installer needs to reach the internet to download the binary package for fog.

[frazzor123]

@george1421
Yes, I can ping the FOG server and even open its web ui. The server has also access to the internet.
I’m pretty convinced that it is a firewall issue and I will head out to search for the problem on that side.
Thanks a lot for the help and I will come back with the solution as soon as I got it. Maybe someone else experiences the problems.

[george1421]

@frazzor123 So just to be clear the firewall you speak of should be between the vlans on your campus. So you will need your infrastructure team to look at it.

FOG uses a number of ports/protocols that need to be open in firewalls to make imaging work.

tftp
ftp
http
nfs

[frazzor123]

@george1421
Yes, I just figured it out. It had something to do with a little Tool called “TFTP Helper” I needed it to activate in the firewall.
Now everything is going great.
Thanks a lot for your help!

[foggymind]

This just started happening for me too, completely randomly. Any idea what would cause it? I didn’t change anything on the server or domain controller. Any help would be greatly appreciated!

Thanks!

[foggymind]

This post is deleted!

[george1421]

@foggymind Please open a new thread because your conditions are probably different than the OP of this thread. In his case it was a setting on their firewall that was causing this issue.

BUT confirm that you only have 1 dhcp server on your network. I’ve seen rouge dhcp servers cause random pxe boot errors.

[foggymind]

@george1421 Thanks so much for your quick reply! I do have 3 DHCP servers that sync with each other but all the settings seem to be correct. I will start a new topic.