macOS update broke iPXE



  • Server
    • FOG Version:
      Running Version 1.4.2
      SVN Revision: 6072
    • OS:
      CentOS Linux release 7.3.1611
    Client
    • Service Version: iPXE 1.0.0+ (7156)
    • OS: macOS 10.11
    Description

    Hi everyone !

    Been using Fog for a while here in our studio however we’ve recently hit some issue with regards to Mac’s.

    A few weeks ago, I was able to setup Fog to work with mac’s, once I was done capturing and deploying was flawless on every computer, both older and newer models ( all intel based ).

    Here is the apple specific part of dhcpd.conf that used to work :

    class "Apple-Intel-Netboot" {
            match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
            option dhcp-parameter-request-list 1,3,17,43,60;
            if (option dhcp-message-type = 8) {
                option vendor-class-identifier "AAPLBSDPC";
                if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                    # BSDP List
                    option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                    filename "ipxe.efi";
                }
            }
        }
      class "Apple-Intel-Netboot-Alt" {
        match if substring (option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
            }
            elsif (substring(option vendor-encapsulated-options, 0, 3) = 01:01:02) {
                # BSDP Select
                option vendor-encapsulated-options 01:01:02:08:04:81:00:05:2a:82:0a:4e:65:74:42:6f:6f:74:30:30:31;
                filename "ipxe7156.efi";
                next-server 10.20.80.1;
            }
        }
    

    So I was able to capture base images from El Capitan fresh installs but then I updated and prepared our clones and a system update broke everything, this doesn’t seem specific to a particular mac model, every system that had the update stopped working in PXE. This includes Mac minis and iMacs.

    This is the error that I get :

    Waiting for link-up on net0................. Down (http://ipxe.org/38086193)
    Could not open net1: Input/output error (http://ipxe.org/1d6a4698)
    

    From the research I’ve done it seems to be a firmware update on the network cards that caused this.
    (ref : http://forum.ipxe.org/archive/index.php/thread-8296.html )
    This does seem to be the cause since I still get the same issue after another fresh install.

    I tried every possible .efi file as an alternative but none worked, I also tried booting to PXE from a usb key using both these methods :

    The first method seems to work as I get the ipxe screen but I get the same error. ( net0 Down )

    The second method didn’t work as I only got a blank screen with a flashing _ , did this one on a ubuntu laptop.

    The problem isn’t with the DHCP as it works great on all our Linux workstations and every Mac computer that hasn’t got the system update.

    Any help would be appreciated as I really don’t know where to look at next.


  • Developer


  • Developer

    @sysadminatelier Ok, I got some new advice from the iPXE devs. Debugging the actual driver issue won’t be easy and will therefore take too many round trips of “try this binary and report again”.

    Nevertheless we ought to try out using the SNP binary (debug enabled - DEBUG=snp:3,nii:3 to start with). So find 01_snp.efi here. Just give it a try and post pictures or video. Make sure when you take those to place the camera or phone on a stack of books in front of the screen so we have a good picture.


  • Developer

    @sysadminatelier Did you get to test the stuff I posted last? @Warget Maybe you are keen to try this as well?

    As well I am trying to revive the discussion on this in the iPXE forums, so hopefully we’ll push this a little further.


  • Developer

    @sysadminatelier Interesting stuff you posted, thanks! After researching and reading for hours I am still missing a brilliant idea. Maybe we can take this from a different perspective.

    • I found this in the iPXE forums where one had an issue with link down on another Broadcom NIC (BCM5764m). He was able to work around the issue by using a dump 100 MBit switch to connect the client to. Could you give that a try?
    • The driver used within iPXE is tg3. A similar one is included in the linux kernel. Please go back to George’s USB FOS… does the network come up properly on one of the updated/broken MACs?


  • @Sebastian-Roth

    Hi Sebastian,

    Luckily I’ve had to work on a Mac that happens to work with PXE, keep in mind it’s a Mac Pro with 2 ethernet ports so net1 might be the other port.

    This is the info I collected before I did the updates :

    Model Identifier : MacPro6,1
    Boot Rom Version : MP61.0116.B17
    SMC Version : 2.20f18

    Ethernet card : Broadcom 57762-A0
    Firmware : 57762-a1.15, 0x525fa2f2
    Version : 10.1.11

    After I did the updates :

    Model Identifier : MacPro6,1
    Boot Rom Version : MP61.0116.B25
    SMC Version : 2.20f18

    Ethernet card : Broadcom 57762-A0
    Firmware : 57762-a1.15, 0x525fa2f2
    Version : 10.2.0

    The updates did break PXE once again, the pictures below are from when the PXE was working as you requested.

    Hope this helps !

    0_1497979855042_IMG_0242.JPG 0_1497979868929_IMG_0255.JPG 0_1497979882277_IMG_0325.JPG 0_1497979900062_IMG_0355.JPG 0_1497979913784_IMG_0378.JPG 0_1497979925013_IMG_0394.JPG 0_1497979937605_IMG_0435.JPG 0_1497979956805_IMG_0453.JPG 0_1497979978693_IMG_0465.JPG


  • Developer

    @sysadminatelier Thanks heaps for testing and posting the pictures. I forgot to ask you if you could do exactly the same on a MAC that is still working so we have an idea of how it is supposed to look like. Could you please?

    As well could you pay attention to the status LED on the switch where the client is connected to. Does it come up at all, turns off at some point?

    I put together a combined picture which is easier to follow and updated your post. Hope you don’t mind. All the green and yellow messages in this picture are about the WLAN adapter (net1) which we don’t need at this point. Let’s hope this does not get in the way here. Will be interesting to see what’s different with the older firmware.




  • Developer

    @sysadminatelier Thanks for posting those as well. We have more and more but it just doesn’t add up for me. The boot ROM version numbers seem to differ from model to model and I can’t see where the issue was probably introduced.

    As well I am not able to spin up a Sierra (10.12) installation on an old MacBook. Too bad I can’t get a test setup going here.

    Maybe we can tackle this from a different angle. Hope you are keen to get into debugging iPXE and test things out. I compiled an iPXE binary with debug enabled (DEBUG=netdevice,nii,snpnet). Please download 01_ipxe.efi here and try it out. Post a picture or video of the output on screen.



  • @Sebastian-Roth

    Hi Sebastian,

    For the iMac Late 2012 :
    Model Identifier : iMac13,2
    Boot ROM : IM131.010A.B11
    SMC Version : 2.11f14

    For the Mac Mini :
    Model Identifier : Macmini7,1
    Boot ROM Version : MM71.0220.B14
    SMC Version : 2.24f32


  • Developer

    @sysadminatelier Thanks for sharing the details. Could you please add Boot ROM Version and SMC Version from your MACs. Find a screenshot of where to find those details here.
    We are still not exactly sure but it looks like an update of the Boot ROM version is causing the issue. As mentioned here IM143.0118.B13 was alright but IM143.0118.B14 has the issue.



  • @Sebastian-Roth

    Hi Sebastian,

    iMac Late 2012
    Ethernet card :
    Apple 57766-A1
    Firmware : 57766a-v1.13, 0x0faa56c8
    Version : 10.2.0

    Mac Mini Late 2014
    Ethernet card :
    Broadcom 57766-A1
    Firmware : 57766a-v1.15, 0xad0d59c9
    Version : 10.2.0

    Both of these have been updated so they only work with the USB method.

    I might have the time to check out other macs this week if you need it as I am pretty sure we have 2015 models and they likely weren’t updated.

    If you need some other info I’ll be happy to provide what I can.

    Thanks


  • Developer

    @sysadminatelier Please take a look at the other thread I posted a couple of minutes ago and tell us about your MAC model(s), and NIC firmware/ROM versions. Just to confirm this is exactly the same issue we are talking about…

    I hope to be able to get hold of some MAC devices where I can debug this issue soon. Keep your fingers crossed.


  • Senior Developer

    @george1421 It’s some firmware update Apple applied between 10.11 to 10.12. The same iPXE version works in 10.11, upgrade macbook and same iPXE version fails.


  • Moderator

    @Sebastian-Roth So then that tells us that FOG is working as it should. So the developers should then either look at the uefi hand off from iPXE or iPXE itself as being a bit confused (??)



  • Good news everyone,

    The image that George sent me worked great so it must have been the usb key I made that didn’t quite work. Will investigate on my side and try to redo it to see what went wrong.

    Thanks to all for your help :grinning:



  • @george1421 can i have that image too please?


  • Moderator

    @sysadminatelier I have a fresh made fos usb boot image I created for another project if you want to test it. PM me for the details.


  • Developer

    @sysadminatelier Building Georg’s FOS USB stick should also work from a Ubuntu Live system I reckon. The only issue I can see might be grub-install being different on the Live system - not sure though.

    Would you mind sharing details about your MAC model(s) that have an issue with booting iPXE? Please read through this, compare the version numbers and let us know which ones you have. (exact MAC model, NIC ROM and SMC number) I really hope we can get this issue fixed at some point.



  • @sysadminatelier said in macOS update broke iPXE:

    10.11

    is this maybe related to your problem?
    http://forum.ipxe.org/archive/index.php/thread-8296.html

    Sorry one page 2 you are referencing to this thread, saw it to late sorry.


 

531
Online

5.4k
Users

12.6k
Topics

118.7k
Posts