TFTP not working (Fog 1.1.1 on Ubuntu 12.04 Server)

[BBBBlaine]

I installed a fresh instance of Ubuntu server 12.04 on a VM, then downloaded and installed Fog 1.1.1.

We already have a dhcp server on the network, so I chose not to use Fog as the dhcp server. I added the dhcp options 66 and 67 as follows on our Windows dhcp server:
66 10.1.0.21
67 undionly.kpxe

When attempting to PXE boot a server, it gets an IP address… however, it doesn’t seem to do anything after that. It doesn’t even show that it’s timing out or anything.

When I try connecting with a Windows TFTP client using the command “tftp 10.1.0.21 get undionly.kpxe” it times out.

I checked to make sure ufw is disabled on the Fog server, and there is no iptables.

[CODE]root@PCI-FOG:~# netstat -a | grep ftp
tcp 0 0 *:ftp : LISTEN
udp 0 0 *:tftp :
[/CODE]
So is TFTP not running correctly?

What could the problem be?

[Tom Elliott]

what happens if you run:
[code]sudo service tftpd-hpa restart[/code]

[BBBBlaine]

I saw that restarting TFTP was a common fix for this type of issue, however, it didn’t fix it in my case.

I even added a 30 second delay on tftp-hpa service startup, but this did not resolve it.

[Tom Elliott]

Did you add 30 second delay to tftp-hpa, or tftpd-hpa?

[BBBBlaine]

Sorry, meant to type tftpd-hpa. I added “sleep 30” to the beginning of /etc/default/tftpd-hpa

[ianabc]

Can you tftp the file on the fog server itself? (I mean run the tftp get above on the fog server). If so I would suspect the client or an intervening firewall: from memory TFTP might uses 69 as a control port but the actual transfer takes place on other port(s).

EDIT Also, if you use tcpwrappers, take a look in your hosts.allow and hosts.deny

[BBBBlaine]

ianabc: Yes, downloading the file works from the fog server itself with the command “tftp 10.1.0.21 -c get undionly.kpxe”

I can SSH to this server from my workstation, and the hosts.allow/hosts.deny files don’t have anything in them but comments.

[ianabc]

Given that, my best guess would be a firewall somewhere between the server and your client. As I mentioned above, TFTP does weird things with it’s port choices, disabling the firewall(s) for a few seconds could save you a lot of time in debugging.

If you have another machine (e.g. another linux server on the same switch as your fog server), you could try tftp from there and rule out networking/firewall issues on the fog server.

[BBBBlaine]

Okay I noticed something strange… TFTP’ing into the server from the local shell downloads the file, but it is 0 kb in size:
[CODE]root@PCI-FOG:~# tftp 10.1.0.21 -c get undionly.kpxe
root@PCI-FOG:~# ls -al undionly.kpxe
-rw-r–r-- 1 root root 0 Jun 19 11:51 undionly.kpxe
[/CODE]

I checked the original file and it also appears to be 0 kb:
[CODE]root@PCI-FOG:/tftpboot# ls -al
total 2496
drwxr-xr-x 2 fog root 4096 Jun 17 13:35 .
drwxr-xr-x 25 root root 4096 Jun 18 08:11 …
-rw-r–r-- 1 fog root 840 Jun 17 13:35 boot.txt
-rw-r–r-- 1 root root 291 Jun 17 13:35 default.ipxe
-rw-r–r-- 1 fog root 389702 Jun 17 13:35 ipxe.kkpxe
-rw-r–r-- 1 fog root 389750 Jun 17 13:35 ipxe.kpxe
-rw-r–r-- 1 fog root 391231 Jun 17 13:35 ipxe.krn
-rw-r–r-- 1 fog root 389766 Jun 17 13:35 ipxe.pxe
-rw-r–r-- 1 fog root 25340 Jun 17 13:35 memdisk
-rw-r–r-- 1 fog root 16794 Jun 17 13:35 pxelinux.0.old
-rw-r–r-- 1 fog root 165088 Jun 17 13:35 snponly.efi
-rw-r–r-- 1 fog root 102777 Jun 17 13:35 undionly.kkpxe
-rw-r–r-- 1 fog root 0 Jun 18 11:51 undionly.kpxe
-rw-r–r-- 1 fog root 382650 Jun 17 13:35 undionly.kpxe.INTEL
-rw-r–r-- 1 fog root 102841 Jun 17 13:35 undionly.pxe
-rw-r–r-- 1 fog root 147728 Jun 17 13:35 vesamenu.c32
[/CODE]

So I deleted undionly.kpxe and renamed undionly.kkpxe to undionly.kpxe now.

I tried PXE booting another server and it comes up with an error when trying to boot:
[CODE]
iPXE 1.0.0+ (7fe0) – Open Source Network Boot Firmware – http://ipxe.org
Features: NFS FTP HTTP HTTPS iSCSI DNS TFTP VLAN AoE bzImage ELF MBOOT PXE PXEXT Menu
Configuring (net0 00:1e:c9:d7:ee:de)… ok
Could not start download: Operation not supported (http://ipxe.org/3c092003)
PXE-M0F: Exiting Broadcom PXE ROM.[/CODE]

Now what? I’m wondering if I’m using the correct boot file, or if it got messed up somehow during installation o_O

[ianabc]

Yeah, it sounds like you’re on the right track. my undonly.kpxe looks slightly different
[CODE]
$ cd /tftpboot
$ ls -l undionly.kpxe
-rw-r–r–. 1 fog root 102825 Jun 17 14:46 undionly.kpxe
$ md5sum undionly.kpxe

2967e86058a2ed5fbc159f722a62e211 undionly.kpxe
[/CODE]

If you untar a copy of the fog installer somewhere then you can copy out the file, it should be under packages/tftp/undionly.kpxe. Copy that into place and see if you are back in business.

[BBBBlaine]

[CODE]root@PCI-FOG:~/fog_1.1.1/packages/tftp# cp undionly.kpxe /tftpboot/undionly.kpxe
root@PCI-FOG:~/fog_1.1.1/packages/tftp# chmod 644 /tftpboot/undionly.kpxe
root@PCI-FOG:~/fog_1.1.1/packages/tftp# ls -al /tftpboot/
total 2600
drwxr-xr-x 2 fog root 4096 Jun 19 14:39 .
drwxr-xr-x 25 root root 4096 Jun 18 08:11 …
-rw-r–r-- 1 fog root 840 Jun 17 13:35 boot.txt
-rw-r–r-- 1 root root 291 Jun 17 13:35 default.ipxe
-rw-r–r-- 1 fog root 389702 Jun 17 13:35 ipxe.kkpxe
-rw-r–r-- 1 fog root 389750 Jun 17 13:35 ipxe.kpxe
-rw-r–r-- 1 fog root 391231 Jun 17 13:35 ipxe.krn
-rw-r–r-- 1 fog root 389766 Jun 17 13:35 ipxe.pxe
-rw-r–r-- 1 fog root 25340 Jun 17 13:35 memdisk
-rw-r–r-- 1 fog root 16794 Jun 17 13:35 pxelinux.0.old
-rw-r–r-- 1 fog root 165088 Jun 17 13:35 snponly.efi
-rw-r–r-- 1 fog root 102777 Jun 17 13:35 undionly.kkpxe
-rw-r–r-- 1 root root 102825 Jun 19 14:39 undionly.kpxe
-rw-r–r-- 1 fog root 382650 Jun 17 13:35 undionly.kpxe.INTEL
-rw-r–r-- 1 fog root 102841 Jun 17 13:35 undionly.pxe
-rw-r–r-- 1 fog root 147728 Jun 17 13:35 vesamenu.c32
[/CODE]

So now it looks like I have the same size pxe boot file as you, so I tried PXE booting the server again and…
[CODE]Could not configure console: Input/output error (http://ipxe.org/1d8c2139)
Could not boot: Input/output error (http://ipxe.org/1d8c2139)
Could not boot: Input/output error (http://ipxe.org/1d8c2139)[/CODE]

The link in the error message says that the error originated from vesafb.c (line 114)
[CODE]/**
103 * Convert VBE status code to iPXE status code
104 *
105 * @v status VBE status code
106 * @ret rc Return status code
107 */
108 static int vesafb_rc ( unsigned int status ) {
109 unsigned int code;
110
111 if ( ( status & 0xff ) != 0x4f )
112 return -ENOTSUP;
113 code = ( ( status >> 8 ) & 0xff );
114 return ( code ? -EIO_VBE ( code ) : 0 );[/CODE]

I have no idea how to resolve this. Any ideas?

[ianabc]

I don’t know iPXE (or PXE) well so I’m not sure what the sequence is, but just comparing the file sizes in the directories, my default.ipxe looks different from yours. I’ll md5sum everything in that directory to let you compare
[CODE]
$ md5sum *
e98752ef921d9ae727e8056550a6c9a5 boot.txt
418b6eb51122fcdab90ec8c7ed80ef85 default.ipxe
48f6c6d1b580a4a8b64f20c13d0dd683 ipxe.kkpxe
326f7de6b826275725cd0728e4f5c194 ipxe.kpxe
828b8ccf956cf63ef4d20b46c4a13698 ipxe.krn
565956b5a03307ba52b38a45edc61291 ipxe.pxe
2c44adda31db993665f51a8717af3985 memdisk
3794629cd0a52cf5d20df3984319de1b pxelinux.0.old
ffb7b9176e3d40eca5903364def6753d snponly.efi
418b7656f88e07e591947e6a0f11694f undionly.kkpxe
2967e86058a2ed5fbc159f722a62e211 undionly.kpxe
dc15ad8391f21429be2db7d6922aed41 undionly.kpxe.INTEL
b9c71e3130f2a5963c15ab157fcf695b undionly.pxe

2e3f0be8f5e63ff99812a140795beffc vesamenu.c32
[/CODE]
And here is my default.ipxe
[CODE]
#!ipxe
cpuid --ext 29 && set arch x86_64 || set arch i386
params
param mac0 ${net0/mac}
param arch ${arch}
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme

:bootme
[/CODE]

The missing (and corrupt??) files are weird. A reinstall might fix it, but it’s probably better if we can figure out what has gone wrong this time.

[Tom Elliott]

[quote=“ianabc, post: 30876, member: 24548”]And here is my default.ipxe
[CODE]
#!ipxe
cpuid --ext 29 && set arch x86_64 || set arch i386
params
param mac0 ${net0/mac}
param arch ${arch}
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme

:bootme
[/CODE]

The missing (and corrupt??) files are weird. A reinstall might fix it, but it’s probably better if we can figure out what has gone wrong this time.[/quote]

The default.ipxe you’re seeing looks wrong to me, unless you cut out the last line because of IP?
The last line should be set to something like:
chain 192.168.1.1/fog/service/ipxe/boot.php##params

[ianabc]

[quote=“Tom Elliott, post: 30877, member: 7271”]The default.ipxe you’re seeing looks wrong to me, unless you cut out the last line because of IP?
The last line should be set to something like:
chain 192.168.1.1/fog/service/ipxe/boot.php##params[/quote]

Yes, sorry! that line is indeed in my default.ipxe - lazy copy pasting!

[CODE]
#!ipxe
cpuid --ext 29 && set arch x86_64 || set arch i386
params
param mac0 ${net0/mac}
param arch ${arch}
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:bootme
chain http://MY IP HERE/fog/service/ipxe/boot.php##params
[/CODE]

Actually BBBBLaine, now that I think about it, your default.ipxe IS going to be different from mine, it’ll have your fog server’s IP address in it. Can you verify that the contents of your ipxe file that is chain loaded look sane? (point a browser at [url]http://MY[/url] IP HERE/fog/service/ipxe/boot.php##params). Here’s mine
[CODE]
[COLOR=#000000]#!ipxe
cpuid --ext 29 && set arch x86_64 || set arch i386
colour --rgb 0xff6600 2
cpair --foreground 7 --background 2 2
console --picture http://MY IP HERE/fog/service/ipxe/bg.png --left 100 --right 80
prompt --key 0x1b --timeout 3000 Booting… (Press Escape to access the menu) && goto menuAccess || sanboot --no-describe --drive 0x80
:menuAccess
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param menuaccess 1
param debug 0
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:bootme
chain http://MY IP HERE/fog/service/ipxe/boot.php##params[/COLOR]
[/CODE]

[BBBBlaine]

In this [URL=‘http://fogproject.org/forum/threads/dell-optiplex-320.10872/’]other thread[/URL] where the same issue seems to be happening, somebody suggested trying different boot files.

Tried undionly.kpxe.INTEL:
[CODE]http://10.1.0.21/fog/service/ipxe/boot.php… ok
http://10.1.0.21/fog/service/ipxe/bg.png...transmit timed out
transmit timed out
38%transmit timed out
transmit timed out
transmit timed out
51%[/CODE]

Tried undionly.pxe:
This boot file only reboots the server immediately after it loads. I don’t see anything useful come up on the screen.

Tried vesamenu.c32:
PXE doesn’t do anything after getting an IP from DHCP… just sits there.

Tried ipxe.kkpxe:
Just like with undionly.pxe… server reboots

Tried ipxe.kpxe:
Server reboots

Tried ipxe.pxe:
Server reboots

Tried ipxe.krn:
[CODE]This image cannot be loaded from a floppy disk.[/CODE] <-- What?? o.O

Tried pxelinux.0.old (it’s worth a shot, I guess):
[CODE]Trying to load: pxelinux.cfg/
Trying to load: pxelinux.cfg/44454c4c-5100-1057-804d-b2c04f4e4731
Trying to load: pxelinux.cfg/01-00-1e-c9-d7-ee-de
Trying to load: pxelinux.cfg/0A010106
Trying to load: pxelinux.cfg/0A01010
Trying to load: pxelinux.cfg/0A0101
Trying to load: pxelinux.cfg/0A010
Trying to load: pxelinux.cfg/0A01
Trying to load: pxelinux.cfg/0A0
Trying to load: pxelinux.cfg/0A
Trying to load: pxelinux.cfg/0
Trying to load: pxelinux.cfg/default
Unable to locate configuration file

Boot failed: press a key to retry, or wait for reset…
[/CODE]

At this point, I’m stuck.

Does anybody have a WORKING installation of Fog 1.1.1? If so, what distro of linux is it running on? Maybe I’m just running into a bug?

[ianabc]

I’m running it on RHEL 6.5 and also Centos 5. Do you have any other clients you can try to iPXE boot? the VESA thing might be a hardware issue.

[BBBBlaine]

[quote=“ianabc, post: 30881, member: 24548”]
Actually BBBBLaine, now that I think about it, your default.ipxe IS going to be different from mine, it’ll have your fog server’s IP address in it. Can you verify that the contents of your ipxe file that is chain loaded look sane? (point a browser at [url]http://MY[/url] IP HERE/fog/service/ipxe/boot.php##params). Here’s mine
[CODE]
[COLOR=#000000]#!ipxe
cpuid --ext 29 && set arch x86_64 || set arch i386
colour --rgb 0xff6600 2
cpair --foreground 7 --background 2 2
console --picture http://MY IP HERE/fog/service/ipxe/bg.png --left 100 --right 80
prompt --key 0x1b --timeout 3000 Booting… (Press Escape to access the menu) && goto menuAccess || sanboot --no-describe --drive 0x80
:menuAccess
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param menuaccess 1
param debug 0
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:bootme
chain http://MY IP HERE/fog/service/ipxe/boot.php##params[/COLOR]
[/CODE][/quote]

Here is what I see when I browse to [url]http://10.1.0.21/fog/service/ipxe/boot.php##params[/url]
[CODE]#!ipxe
cpuid --ext 29 && set arch x86_64 || set arch i386
colour --rgb 0xff6600 2
cpair --foreground 7 --background 2 2
console --picture http://10.1.0.21/fog/service/ipxe/bg.png --left 100 --right 80
:MENU
menu
colour --rgb 0xff0000 0
cpair --foreground 0 3
item --gap Host is NOT registered!
item --gap – -------------------------------------
item fog.local Boot from hard disk
item fog.memtest Run Memtest86+
item fog.reginput Perform Full Host Registration and Inventory
item fog.reg Quick Registration and Inventory
item fog.sysinfo Client System Information (Compatibility)
choose --default fog.local --timeout 3000 target && goto ${target}
:fog.local
exit || goto MENU
:fog.memtest
kernel memdisk iso raw
initrd memtest.bin
boot || goto MENU
:fog.reginput
kernel bzImage32 root=/dev/ram0 rw ramdisk_size=127000 ip=dhcp dns=10.1.0.3 keymap= web=10.1.0.21/fog/ consoleblank=0 loglevel=4 mode=manreg
imgfetch init_32.xz
boot || goto MENU
:fog.reg
kernel bzImage32 root=/dev/ram0 rw ramdisk_size=127000 ip=dhcp dns=10.1.0.3 keymap= web=10.1.0.21/fog/ consoleblank=0 loglevel=4 mode=autoreg
imgfetch init_32.xz
boot || goto MENU
:fog.sysinfo
kernel bzImage32 root=/dev/ram0 rw ramdisk_size=127000 ip=dhcp dns=10.1.0.3 keymap= web=10.1.0.21/fog/ consoleblank=0 loglevel=4 mode=sysinfo
imgfetch init_32.xz
boot || goto MENU
autoboot
[/CODE]

[Tom Elliott]

The only files you should be trying are:
ipxe.pxe
ipxe.kpxe
ipxe.kkpxe
undionly.pxe
undionly.kpxe
undionly.kkpxe

I’d first start by trying the (same named) files in:
[url]https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/[/url]

Just replace the file name with undionly.kpxe (so you don’t have to change dhcp stuff).
Example:
[code]wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/undionly.pxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/undionly.kpxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/undionly.kkpxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/ipxe.pxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/ipxe.kpxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/undionly.kkpxe[/code]

[BBBBlaine]

[quote=“Tom Elliott, post: 30885, member: 7271”]The only files you should be trying are:
ipxe.pxe
ipxe.kpxe
ipxe.kkpxe
undionly.pxe
undionly.kpxe
undionly.kkpxe

I’d first start by trying the (same named) files in:
[url]https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/[/url]

Just replace the file name with undionly.kpxe (so you don’t have to change dhcp stuff).
Example:
[code]wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/undionly.pxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/undionly.kpxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/undionly.kkpxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/ipxe.pxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/ipxe.kpxe
wget -O /tftpboot/undionly.kpxe https://svn.code.sf.net/p/freeghost/code/trunk/packages/tftp/undionly.kkpxe[/code][/quote]

Okay I’m trying each of those one by one…
With undionly.pxe I get:
Reboot after “Initialising devices”

With undionly.kpxe I get:
[CODE]Could not start download: Operation not supported (http://ipxe.org/3c092003)[/CODE]
With undionly.kkpxe I get:
[CODE]Could not configure console: Input/output error (http://ipxe.org/1d8c2139)
Could not boot: Input/output error (http://ipxe.org/1d8c2139)
Could not boot: Input/output error (http://ipxe.org/1d8c2139)[/CODE]
With ipxe.pxe I get:
[CODE]transmit timed out…
transmit timed out…
transmit timed out…[/CODE]
With ipxe.kpxe I get:
[CODE]Configuring (net0 00:1e:c9:d7:ee:de)…
[/CODE] then the server reboots with no error message

EDIT maybe I will try this on a different machine to see if maybe there’s just some problem with this server… It is a Dell PowerEdge 2900 - refurbished (I mean the server I’m using to try PXE boot at least. The Fog server is a Virtual Machine on ESXi 5.5)

[ianabc]

[quote=“BBBBlaine, post: 30888, member: 24687”]

EDIT maybe I will try this on a different machine to see if maybe there’s just some problem with this server… It is a Dell PowerEdge 2900 - refurbished (I mean the server I’m using to try PXE boot at least. The Fog server is a Virtual Machine on ESXi 5.5)[/quote]

That sounds like a good idea, it would be informative to see some different PXE clients. Good luck!