macOS update broke iPXE

Sebastian Roth · Jun 13, 2017, 8:14 AM

@sysadminatelier Building Georg’s FOS USB stick should also work from a Ubuntu Live system I reckon. The only issue I can see might be grub-install being different on the Live system - not sure though.

Would you mind sharing details about your MAC model(s) that have an issue with booting iPXE? Please read through this, compare the version numbers and let us know which ones you have. (exact MAC model, NIC ROM and SMC number) I really hope we can get this issue fixed at some point.

george1421 · Jun 13, 2017, 2:28 PM

@sysadminatelier I have a fresh made fos usb boot image I created for another project if you want to test it. PM me for the details.

x23piracy · Jun 13, 2017, 2:29 PM

@george1421 can i have that image too please?

sysadminatelier · Jun 13, 2017, 2:55 PM

Good news everyone,

The image that George sent me worked great so it must have been the usb key I made that didn’t quite work. Will investigate on my side and try to redo it to see what went wrong.

Thanks to all for your help

george1421 · Jun 13, 2017, 3:02 PM

@Sebastian-Roth So then that tells us that FOG is working as it should. So the developers should then either look at the uefi hand off from iPXE or iPXE itself as being a bit confused (??)

Tom Elliott · Jun 13, 2017, 3:04 PM

@george1421 It’s some firmware update Apple applied between 10.11 to 10.12. The same iPXE version works in 10.11, upgrade macbook and same iPXE version fails.

Sebastian Roth · Jun 13, 2017, 3:17 PM

@sysadminatelier Please take a look at the other thread I posted a couple of minutes ago and tell us about your MAC model(s), and NIC firmware/ROM versions. Just to confirm this is exactly the same issue we are talking about…

I hope to be able to get hold of some MAC devices where I can debug this issue soon. Keep your fingers crossed.

sysadminatelier · Jun 14, 2017, 3:07 PM

@Sebastian-Roth

Hi Sebastian,

iMac Late 2012
Ethernet card :
Apple 57766-A1
Firmware : 57766a-v1.13, 0x0faa56c8
Version : 10.2.0

Mac Mini Late 2014
Ethernet card :
Broadcom 57766-A1
Firmware : 57766a-v1.15, 0xad0d59c9
Version : 10.2.0

Both of these have been updated so they only work with the USB method.

I might have the time to check out other macs this week if you need it as I am pretty sure we have 2015 models and they likely weren’t updated.

If you need some other info I’ll be happy to provide what I can.

Thanks

Sebastian Roth · Jun 15, 2017, 1:42 AM

@sysadminatelier Thanks for sharing the details. Could you please add Boot ROM Version and SMC Version from your MACs. Find a screenshot of where to find those details here.
We are still not exactly sure but it looks like an update of the Boot ROM version is causing the issue. As mentioned here IM143.0118.B13 was alright but IM143.0118.B14 has the issue.

sysadminatelier · Jun 15, 2017, 7:41 AM

@Sebastian-Roth

Hi Sebastian,

For the iMac Late 2012 :
Model Identifier : iMac13,2
Boot ROM : IM131.010A.B11
SMC Version : 2.11f14

For the Mac Mini :
Model Identifier : Macmini7,1
Boot ROM Version : MM71.0220.B14
SMC Version : 2.24f32

Sebastian Roth · Jun 16, 2017, 2:04 PM

@sysadminatelier Thanks for posting those as well. We have more and more but it just doesn’t add up for me. The boot ROM version numbers seem to differ from model to model and I can’t see where the issue was probably introduced.

As well I am not able to spin up a Sierra (10.12) installation on an old MacBook. Too bad I can’t get a test setup going here.

Maybe we can tackle this from a different angle. Hope you are keen to get into debugging iPXE and test things out. I compiled an iPXE binary with debug enabled (DEBUG=netdevice,nii,snpnet). Please download 01_ipxe.efi here and try it out. Post a picture or video of the output on screen.

sysadminatelier · Jun 19, 2017, 3:26 PM

@Sebastian-Roth

Original pictures (in reverse order): 4_1497885826459_IMG_0189.JPG 3_1497885826459_IMG_0172.JPG 2_1497885826458_IMG_0120.JPG 1_1497885826458_IMG_0112.JPG 0_1497885826458_IMG_0108.JPG

Sebastian Roth · Jun 19, 2017, 3:34 PM

@sysadminatelier Thanks heaps for testing and posting the pictures. I forgot to ask you if you could do exactly the same on a MAC that is still working so we have an idea of how it is supposed to look like. Could you please?

As well could you pay attention to the status LED on the switch where the client is connected to. Does it come up at all, turns off at some point?

I put together a combined picture which is easier to follow and updated your post. Hope you don’t mind. All the green and yellow messages in this picture are about the WLAN adapter (net1) which we don’t need at this point. Let’s hope this does not get in the way here. Will be interesting to see what’s different with the older firmware.

sysadminatelier · Jun 20, 2017, 5:33 PM

@Sebastian-Roth

Hi Sebastian,

Luckily I’ve had to work on a Mac that happens to work with PXE, keep in mind it’s a Mac Pro with 2 ethernet ports so net1 might be the other port.

This is the info I collected before I did the updates :

Model Identifier : MacPro6,1
Boot Rom Version : MP61.0116.B17
SMC Version : 2.20f18

Ethernet card : Broadcom 57762-A0
Firmware : 57762-a1.15, 0x525fa2f2
Version : 10.1.11

After I did the updates :

Model Identifier : MacPro6,1
Boot Rom Version : MP61.0116.B25
SMC Version : 2.20f18

Ethernet card : Broadcom 57762-A0
Firmware : 57762-a1.15, 0x525fa2f2
Version : 10.2.0

The updates did break PXE once again, the pictures below are from when the PXE was working as you requested.

Hope this helps !

Sebastian Roth · Jun 20, 2017, 8:49 PM

@sysadminatelier Interesting stuff you posted, thanks! After researching and reading for hours I am still missing a brilliant idea. Maybe we can take this from a different perspective.

I found this in the iPXE forums where one had an issue with link down on another Broadcom NIC (BCM5764m). He was able to work around the issue by using a dump 100 MBit switch to connect the client to. Could you give that a try?
The driver used within iPXE is tg3. A similar one is included in the linux kernel. Please go back to George’s USB FOS… does the network come up properly on one of the updated/broken MACs?

Sebastian Roth · Jul 22, 2017, 5:00 AM

@sysadminatelier Did you get to test the stuff I posted last? @Warget Maybe you are keen to try this as well?

As well I am trying to revive the discussion on this in the iPXE forums, so hopefully we’ll push this a little further.

Sebastian Roth · Jul 28, 2017, 4:16 PM

@sysadminatelier Ok, I got some new advice from the iPXE devs. Debugging the actual driver issue won’t be easy and will therefore take too many round trips of “try this binary and report again”.

Nevertheless we ought to try out using the SNP binary (debug enabled - DEBUG=snp:3,nii:3 to start with). So find 01_snp.efi here. Just give it a try and post pictures or video. Make sure when you take those to place the camera or phone on a stack of books in front of the screen so we have a good picture.

Sebastian Roth · Aug 3, 2017, 9:33 PM

Just cross-linking this here: https://forums.fogproject.org/topic/10615/ipxe-booting-possibly-broken-on-os-x-sierra-update

macOS update broke iPXE

190

12.0k

17.3k

155.2k