Since the apache crashes did not correlate with the no directory or file errors I double checked everything attached to our workbench and…
Found out one of my engineers had a test box attached to our workbench that was set for no reply to ICMP so I did not catch it had been static’d to 10.0.0.250 as well. Flipped it over to DHCP and now PXE boot works consistently.
Sorry and thank you for your time!
Since the apache crashes did not correlate with the no directory or file errors I double checked everything attached to our workbench and…
Found out one of my engineers had a test box attached to our workbench that was set for no reply to ICMP so I did not catch it had been static’d to 10.0.0.250 as well. Flipped it over to DHCP and now PXE boot works consistently.
Sorry and thank you for your time!
From a couple of tries then a restart error.log.1
[Thu Oct 17 12:53:39.[Thu Oct 17 12:56:22.309094 2019] [mpm_prefork:notice] [pid 2036] AH00169: caught SIGTERM, shutting down
[Thu Oct 17 13:15:53.987103 2019] [mpm_prefork:notice] [pid 2158] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured – resuming normal operations
[Thu Oct 17 13:15:54.098310 2019] [core:notice] [pid 2158] AH00094: Command line: ‘/usr/sbin/apache2’
[Thu Oct 17 13:04:26.059866 2019] [mpm_prefork:notice] [pid 2158] AH00169: caught SIGTERM, shutting down
[Thu Oct 17 13:24:01.222333 2019] [mpm_prefork:notice] [pid 2023] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured – resuming normal operations
[Thu Oct 17 13:24:01.414886 2019] [core:notice] [pid 2023] AH00094: Command line: ‘/usr/sbin/apache2’
[Thu Oct 17 13:14:33.786132 2019] [mpm_prefork:notice] [pid 2023] AH00169: caught SIGTERM, shutting down
[Thu Oct 17 13:34:03.037187 2019] [mpm_prefork:notice] [pid 2074] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured – resuming normal operations
[Thu Oct 17 13:34:03.227593 2019] [core:notice] [pid 2074] AH00094: Command line: ‘/usr/sbin/apache2’
[Fri Oct 18 00:06:40.122273 2019] [mpm_prefork:notice] [pid 2074] AH00171: Graceful restart requested, doing restart
Then fresh after successful host registration and no boot.php again
foggy@fog:/var/log/apache2$ cat error.log
[Fri Oct 18 00:06:40.167102 2019] [mpm_prefork:notice] [pid 2074] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured – resuming normal operations
[Fri Oct 18 00:06:40.167132 2019] [core:notice] [pid 2074] AH00094: Command line: ‘/usr/sbin/apache2’
This is really odd as boot works consistently if fog has just been rebooted - but only on the first operation (image registration or deploy) then it goes back to being intermittent.
If I leave the client bootlooping after 10 or so cycles it will continue through to the fog registration and imaging phase - most of the time.
I have verified I am able to access http://10.0.0.250/fog/service/ipxe/boot.php consistently from linux/windows/OSX without fail. This issue occurs with physical clients and virtualized clients.
The fog server is running DHCP and is on an isolated vlan with portfast enabled - connecting directly to a dummy switch does not resolve this issue.
Verified Ubuntu firewall is disabled
foggy@fog:~$ sudo ufw status verbose
Status: inactive
Fog Version: 1.5.7
Ubuntu Version: Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-65-generic x86_64)
Server specs:
8 CPU @ 2.53GHz (was an older DL360 G6 that was just sitting around)
24 Gigs of DDR3 2133mhz ECC RAM
1 NIC attached (no errors on switch port its attached to - replaced cable anyway, tried switching NIC’s)
Here is the boot.php content
#!ipxe
set fog-ip 10.0.0.250
set fog-webroot fog
set boot-url http://${fog-ip}/${fog-webroot}
cpuid --ext 29 && set arch x86_64 || set arch i386
goto get_console
:console_set
colour --rgb 0x00567a 1 ||
colour --rgb 0x00567a 2 ||
colour --rgb 0x00567a 4 ||
cpair --foreground 7 --background 2 2 ||
goto MENU
:alt_console
cpair --background 0 1 ||
cpair --background 1 2 ||
goto MENU
:get_console
console --picture http://10.0.0.250/fog/service/ipxe/bg.png --left 100 --right 80 && goto console_set || goto alt_console
:MENU
menu
colour --rgb 0xff0000 0 ||
cpair --foreground 1 1 ||
cpair --foreground 0 3 ||
cpair --foreground 4 4 ||
item --gap Host is NOT registered!
item --gap – -------------------------------------
item fog.local Boot from hard disk
item fog.memtest Run Memtest86+
item fog.reginput Perform Full Host Registration and Inventory
item fog.reg Quick Registration and Inventory
item fog.deployimage Deploy Image
item fog.multijoin Join Multicast Session
item fog.sysinfo Client System Information (Compatibility)
choose --default fog.local --timeout 3000 target && goto ${target}
:fog.local
sanboot --no-describe --drive 0x80 || goto MENU
:fog.memtest
kernel memdisk initrd=memtest.bin iso raw
initrd memtest.bin
boot || goto MENU
:fog.reginput
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 web=http://10.0.0.250/fog/ consoleblank=0 rootfstype=ext4 storage=10.0.0.250:/images/ storageip=10.0.0.250 loglevel=4 mode=manreg
imgfetch init_32.xz
boot || goto MENU
:fog.reg
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 web=http://10.0.0.250/fog/ consoleblank=0 rootfstype=ext4 storage=10.0.0.250:/images/ storageip=10.0.0.250 loglevel=4 mode=autoreg
imgfetch init_32.xz
boot || goto MENU
:fog.deployimage
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param qihost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.multijoin
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param sessionJoin 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.sysinfo
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 web=http://10.0.0.250/fog/ consoleblank=0 rootfstype=ext4 storage=10.0.0.250:/images/ storageip=10.0.0.250 loglevel=4 mode=sysinfo
imgfetch init_32.xz
boot || goto MENU
:bootme
chain -ar http://10.0.0.250/fog/service/ipxe/boot.php##params ||
goto MENU
autoboot!
Screenshot from ESXI pxe boot - Exact same error on physical clients
Any help or direction would be appreciated