macOS update broke iPXE

[sysadminatelier]

Server

• FOG Version:
  Running Version 1.4.2
  SVN Revision: 6072
• OS:
  CentOS Linux release 7.3.1611

Client

• Service Version: iPXE 1.0.0+ (7156)
• OS: macOS 10.11

Description

Hi everyone !

Been using Fog for a while here in our studio however we’ve recently hit some issue with regards to Mac’s.

A few weeks ago, I was able to setup Fog to work with mac’s, once I was done capturing and deploying was flawless on every computer, both older and newer models ( all intel based ).

Here is the apple specific part of dhcpd.conf that used to work :

class "Apple-Intel-Netboot" {
        match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                filename "ipxe.efi";
            }
        }
    }
  class "Apple-Intel-Netboot-Alt" {
    match if substring (option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
    option dhcp-parameter-request-list 1,3,17,43,60;
    if (option dhcp-message-type = 8) {
        option vendor-class-identifier "AAPLBSDPC";
        if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
            # BSDP List
            option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
        }
        elsif (substring(option vendor-encapsulated-options, 0, 3) = 01:01:02) {
            # BSDP Select
            option vendor-encapsulated-options 01:01:02:08:04:81:00:05:2a:82:0a:4e:65:74:42:6f:6f:74:30:30:31;
            filename "ipxe7156.efi";
            next-server 10.20.80.1;
        }
    }

So I was able to capture base images from El Capitan fresh installs but then I updated and prepared our clones and a system update broke everything, this doesn’t seem specific to a particular mac model, every system that had the update stopped working in PXE. This includes Mac minis and iMacs.

This is the error that I get :

Waiting for link-up on net0................. Down (http://ipxe.org/38086193)
Could not open net1: Input/output error (http://ipxe.org/1d6a4698)

From the research I’ve done it seems to be a firmware update on the network cards that caused this.
(ref : http://forum.ipxe.org/archive/index.php/thread-8296.html )
This does seem to be the cause since I still get the same issue after another fresh install.

I tried every possible .efi file as an alternative but none worked, I also tried booting to PXE from a usb key using both these methods :

• https://wiki.fogproject.org/wiki/index.php?title=USB_Bootable_Media#USB_Boot_UEFI_client_into_FOG_menu_.28easy_way.29
• https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image/4

The first method seems to work as I get the ipxe screen but I get the same error. ( net0 Down )

The second method didn’t work as I only got a blank screen with a flashing _ , did this one on a ubuntu laptop.

The problem isn’t with the DHCP as it works great on all our Linux workstations and every Mac computer that hasn’t got the system update.

Any help would be appreciated as I really don’t know where to look at next.

[Warget]

This problem is same to mine ? Check :

@george1421 said in NetBoot problem, new mac..!:

@Warget This should be working!!

You are getting a dhcp address so the network adapter is working, but FOS can’t reach the FOG server. The error message is a bit misleading but I remember this was an issue because of a bad check.

Just as an idea, did you remember to change / update the grub.conf file on the usb flash drive. You needed to give the IP address of the fog server so FOS knows how to find the FOG server. It was in the link I provided before: https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image/4 in the section called: The final steps you MUST complete

[sysadminatelier]

@Warget
Hi Warget,

I wasn’t successful with the USB method in the thread.

I did modify the script to change the ip for the ip of our DHCP server.

All I got was a blank screen with a flashing _

Could it be that it’s because I did it on a Ubuntu Live USB and not a complete install ?

Thanks again.

[Tom Elliott]

@sysadminatelier What did you put for the kernel arg string? You MUST have the initrd=init.xz line in the efi string for booting to work properly. I’m only guessing here that this is part of the problem, but think seeing the whole string would go a lot further.

[x23piracy]

@sysadminatelier said in macOS update broke iPXE:

10.11

is this maybe related to your problem?
http://forum.ipxe.org/archive/index.php/thread-8296.html

Sorry one page 2 you are referencing to this thread, saw it to late sorry.

[Sebastian Roth]

@sysadminatelier Building Georg’s FOS USB stick should also work from a Ubuntu Live system I reckon. The only issue I can see might be grub-install being different on the Live system - not sure though.

Would you mind sharing details about your MAC model(s) that have an issue with booting iPXE? Please read through this, compare the version numbers and let us know which ones you have. (exact MAC model, NIC ROM and SMC number) I really hope we can get this issue fixed at some point.

[george1421]

@sysadminatelier I have a fresh made fos usb boot image I created for another project if you want to test it. PM me for the details.

[x23piracy]

@george1421 can i have that image too please?

[sysadminatelier]

Good news everyone,

The image that George sent me worked great so it must have been the usb key I made that didn’t quite work. Will investigate on my side and try to redo it to see what went wrong.

Thanks to all for your help

[george1421]

@Sebastian-Roth So then that tells us that FOG is working as it should. So the developers should then either look at the uefi hand off from iPXE or iPXE itself as being a bit confused (??)

[Tom Elliott]

@george1421 It’s some firmware update Apple applied between 10.11 to 10.12. The same iPXE version works in 10.11, upgrade macbook and same iPXE version fails.

[Sebastian Roth]

@sysadminatelier Please take a look at the other thread I posted a couple of minutes ago and tell us about your MAC model(s), and NIC firmware/ROM versions. Just to confirm this is exactly the same issue we are talking about…

I hope to be able to get hold of some MAC devices where I can debug this issue soon. Keep your fingers crossed.

[sysadminatelier]

@Sebastian-Roth

Hi Sebastian,

iMac Late 2012
Ethernet card :
Apple 57766-A1
Firmware : 57766a-v1.13, 0x0faa56c8
Version : 10.2.0

Mac Mini Late 2014
Ethernet card :
Broadcom 57766-A1
Firmware : 57766a-v1.15, 0xad0d59c9
Version : 10.2.0

Both of these have been updated so they only work with the USB method.

I might have the time to check out other macs this week if you need it as I am pretty sure we have 2015 models and they likely weren’t updated.

If you need some other info I’ll be happy to provide what I can.

Thanks

[Sebastian Roth]

@sysadminatelier Thanks for sharing the details. Could you please add Boot ROM Version and SMC Version from your MACs. Find a screenshot of where to find those details here.
We are still not exactly sure but it looks like an update of the Boot ROM version is causing the issue. As mentioned here IM143.0118.B13 was alright but IM143.0118.B14 has the issue.

[sysadminatelier]

@Sebastian-Roth

Hi Sebastian,

For the iMac Late 2012 :
Model Identifier : iMac13,2
Boot ROM : IM131.010A.B11
SMC Version : 2.11f14

For the Mac Mini :
Model Identifier : Macmini7,1
Boot ROM Version : MM71.0220.B14
SMC Version : 2.24f32

[Sebastian Roth]

@sysadminatelier Thanks for posting those as well. We have more and more but it just doesn’t add up for me. The boot ROM version numbers seem to differ from model to model and I can’t see where the issue was probably introduced.

As well I am not able to spin up a Sierra (10.12) installation on an old MacBook. Too bad I can’t get a test setup going here.

Maybe we can tackle this from a different angle. Hope you are keen to get into debugging iPXE and test things out. I compiled an iPXE binary with debug enabled (DEBUG=netdevice,nii,snpnet). Please download 01_ipxe.efi here and try it out. Post a picture or video of the output on screen.

[sysadminatelier]

@Sebastian-Roth

Original pictures (in reverse order): 4_1497885826459_IMG_0189.JPG 3_1497885826459_IMG_0172.JPG 2_1497885826458_IMG_0120.JPG 1_1497885826458_IMG_0112.JPG 0_1497885826458_IMG_0108.JPG

[Sebastian Roth]

@sysadminatelier Thanks heaps for testing and posting the pictures. I forgot to ask you if you could do exactly the same on a MAC that is still working so we have an idea of how it is supposed to look like. Could you please?

As well could you pay attention to the status LED on the switch where the client is connected to. Does it come up at all, turns off at some point?

I put together a combined picture which is easier to follow and updated your post. Hope you don’t mind. All the green and yellow messages in this picture are about the WLAN adapter (net1) which we don’t need at this point. Let’s hope this does not get in the way here. Will be interesting to see what’s different with the older firmware.

[sysadminatelier]

@Sebastian-Roth

Hi Sebastian,

Luckily I’ve had to work on a Mac that happens to work with PXE, keep in mind it’s a Mac Pro with 2 ethernet ports so net1 might be the other port.

This is the info I collected before I did the updates :

Model Identifier : MacPro6,1
Boot Rom Version : MP61.0116.B17
SMC Version : 2.20f18

Ethernet card : Broadcom 57762-A0
Firmware : 57762-a1.15, 0x525fa2f2
Version : 10.1.11

After I did the updates :

Model Identifier : MacPro6,1
Boot Rom Version : MP61.0116.B25
SMC Version : 2.20f18

Ethernet card : Broadcom 57762-A0
Firmware : 57762-a1.15, 0x525fa2f2
Version : 10.2.0

The updates did break PXE once again, the pictures below are from when the PXE was working as you requested.

Hope this helps !

[Sebastian Roth]

@sysadminatelier Interesting stuff you posted, thanks! After researching and reading for hours I am still missing a brilliant idea. Maybe we can take this from a different perspective.

• I found this in the iPXE forums where one had an issue with link down on another Broadcom NIC (BCM5764m). He was able to work around the issue by using a dump 100 MBit switch to connect the client to. Could you give that a try?
• The driver used within iPXE is tg3. A similar one is included in the linux kernel. Please go back to George’s USB FOS… does the network come up properly on one of the updated/broken MACs?