Mac support

[Tom S]

[quote=“Tom Elliott, post: 34265, member: 7271”]Tom S,

Can you post a diff -u of your fog.upload/fog.download? You said you already had it going. What OSID are you comparing it with? I’ve added osid 8 for Apple MAC OS. I just don’t know the best approach for the fog.down/upload files.[/quote]
Of course Tom… I left my source files at work today…oops. I can get those to you tomorrow.

If I might digress for a moment please.
I have been working for school districts in a state where we provide a laptop for every 6, 7 and 8 grade students (State Law). I have seen lots of imaging solutions (ghost,Deploy Studio, JAMF amongst many others) and I really like fog. One thing that does drive me a little bonkers is multicasting. Making/editing groups and etc while you have a thousand devices to image is rather cumbersome. Whereas deploy studio and ghost allow you to just join a session.

I recently stumbled across a fog wiki post explaining how to bypass the host registration so that one could join a multicast session by supplying its name ( just like ghost) but it was designed for version 0.32. I reworked the hack to be 1.1.2 compatible. Then I added hfs+ support which was adding 10 lines of code to func.sh and fog.upload. But really fog was setup to do the work it just needed to point partclone to use hfsp rather than imager. MPS disk works great. If finds GPT, creates disk1.mbr, and subsequent
img files. The restoring process is the same old business. So long as you are not changing drives the firmware remembers default boot device

To my point: I can send you the diff files but they contain more than just the os x support. I guess my question is, do you want to remove all but the changes needed to get os x support or would you like to see all my changes?

Either way I am extremely happy to be able to help out with this.

As far as the name… I have developer access to apple and I can tell you that the name scheme for Yosemite is OS X 10.10.x and already talks about OS X 10.11 are popping up. I asked one of the Apple Engineers assigned to our states 1 to 1 laptop program and he stated “the name is OS X Mavericks and OS X Yosemite versus just mountain lion, lion or leopard” So your naming of Apple Mac OS (8) is good!!

Thanks

-TS

[Tom Elliott]

If you know how to adapt the Multicast Session stuff, that’d be interesting and feature I’m sure other’s wouldn’t mind.

I didn’t/don’t have the mental capacity for any level of multicast myself, so I appreciate the work and effort if you’re willing to provide and allow us to use it.

I’m sure others will appreciate that as well. I don’t mind getting the code and I can probably figure out what’s happening to get multicast working for you.

[Tom Elliott]

And for what it’s worth, SVN 1999 is when I added hfsp, so that means !.2.0 already has the hfsp file format in use.

[fractal13]

Update on Mac support:

The developers at iPXE (mcb in particular) have been very good to devote time to the issues with supporting Apple’s EFI. At this time, the latest linux kernels (3.10) and above have the kernel patch necessary to be loaded correctly from iPXE. The latest master version of iPXE has the patch necessary to launch the linux kernel with Apple’s EFI.

I’ve only been able to test this on the mid-2011 iMac hardware. However, the late-2012 and late-2013 iMac hardware all report the same EFI version.

There is still an issue with using the SNP network interface with the Apple EFI from iPXE. So, if the network interface in your system is supported by a card specific driver in iPXE, you should be able to use FOG. That’s the case with the mid-2011 iMac we’ve been testing on. Note that even the mid-2011 iMac doesn’t work with the SNP driver.

So, if you’re really hankering to try out imaging on Apple hardware, get the latest version of FOG, build the latest linux kernel, and build the latest iPXE. Configure your DHCP server to talk Apple Netboot, and give it a try with “mps” or “mpa” style captures. GPT and HFS support have been in FOG for a while now.

If you’re a little more cautious, give us a little more time to get FOG polished and tested on several Apple hardware versions.

[Tom S]

Great news!!! I will try out the ipxe source and let you all know!! Once I confirm the patches on my end I will test on macbook from 2007-2010, Mac pro 2009-2012, and macbook airs over WiFi.

I will then pull from git the latest branch and diff my changes to submit to fog (multicast tweaks)

I have most of my scripts ready for renaming and binding to domain as well. Just need to see how fog records those tasks. Then create the service.

I love open source…look at what you have accomplish in such a sort time!!! Great job everyone

Thanks,
TS

BTW
Boot camp imaging works on Maverick and Windows XP. I will try 7 this weekend

[Tom S]

fractal13,

I am still getting the VFS syncing error. Are you using the TomElliott.configs? If so are you still using the firmware patch Fog recommends to use when building the kernel?

Thanks,

Tom

[Tom Elliott]

[quote=“Tom S, post: 34506, member: 25305”]fractal13,

I am still getting the VFS syncing error. Are you using the TomElliott.configs? If so are you still using the firmware patch Fog recommends to use when building the kernel?

Thanks,

Tom[/quote]

The VFS syncing error, how does this come out? Are you using the correct pairing? init.xz and bzImage should both be 64 bit. init_32.xz and bzImage32 should both be 32 bit. If you load 32 bit bzImage with 64 bit init.xz, or 64 bit bzImage with 32 bit init.xz, you will see this error.

[Tom S]

[quote=“Tom Elliott, post: 34517, member: 7271”]The VFS syncing error, how does this come out? Are you using the correct pairing? init.xz and bzImage should both be 64 bit. init_32.xz and bzImage32 should both be 32 bit. If you load 32 bit bzImage with 64 bit init.xz, or 64 bit bzImage with 32 bit init.xz, you will see this error.[/quote]

Thanks for reply Tom,
No my versions are correct, I have tested with the new files last night. The same pairing also works on cd boot of mac and the 30 Dell pcs I am imaging right now.

I believe it has to do with ipxe and the bzImage. I changed the log level to 6 and can see that it sees the macbook devices and board info. It also can see my hard drive. It also talks to dhcp and gets an ip address. I think it just can’t see the /dev/ram0. Fractal13 said this had to do with a patch for the kernel. I have tried numerous variations of the config file for the kernel, adding and removing efi support settings, etc. Currently I am working on kernel 3.10.50.

Thinking that is a memory issue I tried to add the ipxe Mem_map settings but the compiling fails. If at all possible, Fractal13 if you could provide your config files for the kernel and ipxe???

(I should also mention I am testing on MacBook 5,2 and 5,1)

I will continue to tinker…thanks,

TS

[Tom S]

OK…after hours of researching the efi stubs and the ipxe boot process, I got macbooks 2007-2011 netbooting so far. It maybe common knowledge, but it was news to me.

The VFS syncing error was totally the way iPXE handles memory mapping and the hand off of the init.xz file system.

I was using the syntax to load the image in the boot menu as:
[CODE]:img1
kernel bzImage root=/dev/ram0 rw ramdisk_size=127000 ip=dhcp dns=< DNS IP> web=${fog-ip}/fog/ consoleblank=0 loglevel=4 type=down img=<IMAGE NAME> ftp=${fog-ip} imgType=n osid=7 storage=${fog-ip}:/images capone=1 imgFormat=2
imgfetch init.xz
boot || goto MENU[/CODE]

But the only way I could get it to work was as follows:
[CODE]:Img1
imgfetch init.xz
kernel bzImage.efi initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 ip=dhcp dns=${fog-ip} web=${fog-ip}/fog/ consoleblank=0 loglevel=4 type=down img= ftp=${fog-ip} imgType= osid=99 storage=${fog-ip}:/images mc=yes bypass=1 mSession=“”
boot || goto MENU[/CODE]

imgfetch needs to be first so that is add to memory first. Then add the kernel parameter with initrd=. The bzImage.efi is just a renamed bzImage. I think the previous pattern did not supply the name of the init file system. I could see that it was using the room in memory but the kernel just could not grab it…strange

Nonetheless so far, no driver issues, no kernel issues. Current grab from git provides bootable 32 and 64 bit images and ipxe efi(64 bit only,I compiled the 32 Bit needed for 2007 and 2008 macbooks). I feel this is a good start.

Thanks Tom and Fractal13 for getting the new binary up!!

TS

[fractal13]

TS,

Tom did all of the work on the binary. I’m sorry I forgot about the initrd= kernel parameter. I completely spaced it. Tom do you know if the initrd= kernel parameter will give problems to non-efi versions of ipxe? If it doesn’t, we should probably add it to BootMenu.class.php for the $this->kernel specification.

The ipxe team (mcb30) is still looking at the Apple EFI SNP network driver issues for unsupported NICs.

Keep up the good work.

[Tom Elliott]

[quote=“fractal13, post: 34588, member: 24300”]TS,

Tom did all of the work on the binary. I’m sorry I forgot about the initrd= kernel parameter. I completely spaced it. Tom do you know if the initrd= kernel parameter will give problems to non-efi versions of ipxe? If it doesn’t, we should probably add it to BootMenu.class.php for the $this->kernel specification.

The ipxe team (mcb30) is still looking at the Apple EFI SNP network driver issues for unsupported NICs.

Keep up the good work.[/quote]

I can readd the initrd= parameter as it used to do that. Just when we switched, I left it out as all seemed to work perfectly. So I’ll add that back. Thank you much for the info.

[Tom Elliott]

And it’s done.

[Dave Smith]

I would Like to thanks Tom and fractal13 for all there hard work on getting mac support in fog. Can’t wait for the day I can Bin my deploy studio server. The day is surely drawing near. that no matter what OS comes through the door (Linux mac or windows). i will be able to image it using ONE imaging system FOG.:D:):D

[fractal13]

Great news folks!

Today Michael Brown finished the changes to iPXE to support Apple’s EFI implementation with the SNP (generic) network driver.

This makes at least 3 hurdles that iPXE has cleared so that those of us who have to support Apple hardware will be able to use FOG for imaging. 1) Switch into text mode so we can see menus; 2) Get the linux kernel fixed so it doesn’t write to memory it doesn’t own; 3) Generic networking (important for iMacs in the last couple of years.).

Kudos to Michael and the rest of the iPXE team!

If you’re interested in being an alpha tester, follow this thread. In a few days we expect to need a variety of Apple hardware to finish testing the iPXE fixes (only tested on iMacs mid-2011, late-2012 so far), and begin testing the FOG features. We’ll make an additional post when ready.

[Tom S]

OK so I have extensively tested out mac support for Fog. So far so good. Macbook Airs over WiFi no go though. Obviously… ipxe works but the kernel and ram disk do not have support for wifi support, maybe someday…:). The funcs.sh file still needs the Apple Mac OS added to the determine os function for the current fog init file to work, but that aside, everything works!!

I finished the multicast patches I spoke of earlier and once I find the correct forum I will upload the diff files and link it back here.

I did want to talk about the Fog service for Mac support. In my current profession I have written scripts that do what I can tell the Windows Fog service does, but for OS X. They are primarily bash scripts that can easily be piped from a C or another language. My question to everyone here is this:

What language do we want the service built on???

The obvious answer would be Obj-C, Apple’s native IDE platform. At the same time an argument could be for straight up C. My experience working for schools and businesses that I feel are closely related to the people of Fog, is that the device OS’s are all over the place, Windows XP, 7, Vista, 8, 8.1, OSX and linux thus requiring a new app project for each platform. I know that I find it a real pain writing for a half dozen os versions.

Please forgive me as I say this next statement. Some people will want to shoot me on the spot but…what about a cross platform language like Java??? (Please do not hurt me;)) Please entertain an example.

I wrote a bit of software for our devices that is written in Java called “iInstall.” This software calls out to my php server probing for a task, if it finds one it will download and install the software. The users can also authenticate against LDAP and install software provided to their group. We piloted this on Macs. It was written on Obj-C at first. Worked awesome, I was instructed to rewrite it for windows and nix. I looked at the actual tasks and thought “really the only change between os versions is the commands, not the logic. So why do I have to write the same logic 3 or 4 times?” I decided to rewrite the app in java. I used calls to get the os version and then adjust the commands for the verious versions. ( I sorry, I do not want to be known as the long email guy)

So as we are looking at expanding fog to support Macs and in need for a Service to support OS X what do we do? I am happy to write in Obj-C or Java. It doesn’t matter to me, I just wanted to point my past experience in writing apps for the cross platform world.

Thanks for the time,

TS

[Tom S]

[quote=“fractal13, post: 34774, member: 24300”]Great news folks!

Today Michael Brown finished the changes to iPXE to support Apple’s EFI implementation with the SNP (generic) network driver.

This makes at least 3 hurdles that iPXE has cleared so that those of us who have to support Apple hardware will be able to use FOG for imaging. 1) Switch into text mode so we can see menus; 2) Get the linux kernel fixed so it doesn’t write to memory it doesn’t own; 3) Generic networking (important for iMacs in the last couple of years.).

Kudos to Michael and the rest of the iPXE team!

If you’re interested in being an alpha tester, follow this thread. In a few days we expect to need a variety of Apple hardware to finish testing the iPXE fixes (only tested on iMacs mid-2011, late-2012 so far), and begin testing the FOG features. We’ll make an additional post when ready.[/quote]
I can supply you with the list of the macs I have confirmed working. Later this month as we finish up imaging our several thousand macs

[Tom Elliott]

@Tom S, you can post the diff’s right here.

[Tom S]

So here are the changes made from 2124. I have been told in the past to omit comments in my diffs in other projects I have contributed to. Please let me know if you need more comments.

So this is a serious hack. Most of the SQL calls do not use the fog api rather they use the php mysql commands. The Fog api kept denying my request to create tasks because of invalid MAC address issues. Also the main post function includes some sql “Alter” structure commands in order to edit the tables to fit the needs of the script.

I am not proud of the overall structure of the patch files so if someone knows how to do it with the fog api…awesome.

Please let me know if you have questions.

Thank you,

TS

[url=“/_imported_xf_attachments/1/1254_diff.zip?:”]diff.zip[/url]

[Tom Elliott]

Hey Tom,

I’ve taken a look at these, and some have been relatively simple to add, but there’s a lot of confusion in the files.

For example, When I look at the diff file for fog.checkin here’s what I see:
[code]— new/SubMenu.class.php2014-08-04 10:10:50.000000000 -0400
+++ old/SubMenu.class.php2014-07-10 18:11:12.000000000 -0400
@@ -192,7 +192,6 @@
$this->subMenu[$this->node][‘search’] = $this->foglang[‘NewSearch’];
$this->subMenu[$this->node][‘list’] = sprintf($this->foglang[‘ListAll’],$this->foglang[‘Images’]);
$this->subMenu[$this->node][‘add’] = sprintf($this->foglang[‘CreateNew’],$this->foglang[‘Image’]);
-$this->subMenu[$this->node][‘Multicast’] = sprintf($this->foglang[‘Multicast’],$this->foglang[‘Image’]);
if ($_REQUEST[‘id’])
{
$this->subMenu[$this->node][‘id’][$linkformat.‘#image-gen’] = $this->foglang[‘General’];
[/code]
fog.download:
[code]— new/fog.checkin2014-08-05 18:11:58.000000000 -0400
+++ old/fog.checkin2014-07-24 14:32:42.000000000 -0400
@@ -61,7 +61,7 @@
mkdir /images 2>/dev/null;
mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime $storage /images;
echo “Done”;
-elif [ “$type” == “down” ] && [ “$capone” != “1” ] && [ “$joinmc” != “1” ]; then
+elif [ “$type” == “down” ] && [ “$capone” != “1” ]; then
mac64=ifconfig | grep HWaddr | head -n1 | base64;
dots “Attempting to send inventory”;
doInventory 2>/dev/null;
@@ -153,81 +153,6 @@
debugPause;
fi
done
-elif [ “$joinmc” == “1” ] && [ “$mc” == “yes” ]; then
-echo “”;
-echo " #############################################################################“;
-echo " # #”;
-echo " # Joining Multicast Session! #“;
-echo " # #”;
-echo " #############################################################################";
-echo “”;
-canGo=“no”;
-while [ “$mSession” != “exit” ] && [ “$canGo” != “yes” ]; do
-echo -n “Enter in Session Name:[Type “exit” to quit]”;
-echo “”;
-read mSession;
-multicastValues=wget -O - "http://${web}service/mcTask.php?task=join&mSession=$mSession" 2>/dev/null
-if [ “$multicastValues” != “” ]; then
-arr=$(echo $multicastValues | tr “;” “\n”);
-i=0;
-for x in $arr
-do

• array[i]=$x;
  -i=$i+1;
  -done
  -img=${array[0]};
  -imgTypeId=${array[1]};
  -osid=${array[2]};
  -port=${array[3]};
  -storageip=${array[4]};
  -sessionPwd=${array[5]};
  -if [ “$imgTypeId” = “0” ]; then
  -imgType=“n”;
  -fi
  -if [ “$imgTypeId” = “2” ]; then
  -imgType=“mps”;
  -fi
  -if [ “$imgTypeId” = “3” ]; then
  -imgType=“mpa”;
  -fi
  -if [ “$sessionPwd” != “” ]; then
  -echo “”;
  -echo -n " This Session Requires A Password To Join,";
  -didAuth=“0”;
  -for I in 1 2 3;do
  -echo “Please enter in password:”;
  -read password;
  -if [ “$password” == “$sessionPwd” ]; then
  -echo “Password Accepted!!”;
  -didAuth=“1”;
  -break;
  -else
  -echo “Incorrect Password, Try again”;
  -fi
  -done
  -if [ “$didAuth” != “1” ]; then
  -echo “Incorrect password. To many attempts made, now rebooting”;
  -exit 0;
  -fi
  -fi
  -determineOS $osid;
  -canGo=“yes”;

•       echo -n "Multicast Configuration Complete";
  

• echo "";
  

•     echo "Joining Session: $mSession
  

-Using Image: $img
-OS: $osname
-Image Type: $imgType
-Session Port: $port";
-dots “Mounting File System”;
-mkdir /images $debugstring 2>/dev/null;
-mount -o nolock,proto=tcp,rsize=32768,intr,noatime $storage /images 2>/tmp/mntfail;
-echo “Done”;
-else
-echo -n “No mulicast sesion with that name was found!! Try again.\n”;
-fi
-done

else
echo “Done”;
dots “Mounting File System”;[/code]
fog.mcupdate:
[code]#!/bin/sh
IFS=’
';
web=$1;
mSession=$2;
sleep 3;
while true; do
fle=“/tmp/status.fog”
status=tail -n 2 $fle 2>/dev/null | head -n 1 2>/dev/null
cat /dev/null > $fle 2>/dev/null

status=echo $status | base64;
res=wget -T 3 -O - --post-data="status=${status}" "http://${web}service/mcTask.php?task=progress&mSession=${mSession}" 2>/dev/null;
sleep 3;
done[/code]
ImageManagementPage.class.php:
[code]<?php

@error_reporting(0);
require(‘…/commons/base.inc.php’);

$conn = @mysql_connect( DATABASE_HOST, DATABASE_USERNAME, DATABASE_PASSWORD);

if ( $conn ){
if ( ! @mysql_select_db( DATABASE_NAME, $conn ) ) die( mysql_error() );

if( isset($_GET[‘task’]) && isset($_GET[‘mSession’])){
$task = $_GET[‘task’];
$mSession = $_GET[“mSession”];

if($task==“join”){
$sql = “select max(msID) as msID from multicastSessions where msName = ‘$mSession’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
while( $ar = mysql_fetch_array( $res ) ){
$msID = $ar[“msID”];
}
$storageNodeID=“”;
$storageNodeIP=“”;
$sql = “select * from multicastSessions where msID = ‘$msID’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
if($res !=“”){
while( $ar = mysql_fetch_array( $res ) ){
$msImage = $ar[“msImage”];
$msIsDD = $ar[“msIsDD”];
$port = $ar[“msBasePort”];
$sessionPwd = $ar[“sessionPwd”];
$storageNodeID=$ar[‘msNFSGroupID’];
}
$imageType=“”;
$sql = “select * from images where imageID = ‘$msImage’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
while( $ar = mysql_fetch_array( $res ) ){
$hostOS = $ar[“imageOSID”];
$imageType = $ar[‘imageTypeID’];
$msImage = $ar[‘imageName’];

}
$sql = “select * from nfsGroupMembers where ngmID = ‘$storageNodeID’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
while( $ar = mysql_fetch_array( $res ) ){
$storageNodeIP = $ar[“ngmHostname”];

}

echo “$msImage”.“;”.“$imageType”.“;”.“$hostOS”.“;”.“$port”.“;”.“$storageNodeIP”.“;”.“$sessionPwd”;

}
}else if($task==“progress”){
$str = explode(‘@’,base64_decode($_REQUEST[‘status’]));
$percent=trim($str[5]);
mysql_query("UPDATE multicastSessions set msPercent=‘$percent’,msState=‘3’ WHERE msName=‘$mSession’ ");
}else if($task==“finish”){
$sql = “select * from multicastSessions where msName = ‘$mSession’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
$clientCount=0;
while( $ar = mysql_fetch_array( $res ) ){
$clientCount = $ar[“msClients”];
}
if($clientCount>1){
$clientCount–;
mysql_query("UPDATE multicastSessions set msClients=‘$clientCount’ WHERE msName=‘$mSession’ ");
}else{
mysql_query("DELETE FROM tasks WHERE taskName=‘$mSession’ ");
mysql_query("DELETE FROM hosts WHERE hostName=‘$mSession’ ");
$sql = “SELECT * FROM multicastSessions WHERE msName=‘$mSession’ “;
$res = mysql_query( $sql, $conn );
$thisMSID=””;
while( $row = mysql_fetch_array( $res ) ){
$thisMSID= $row[‘msID’];
}
mysql_query("DELETE FROM multicastSessions WHERE msName=‘$mSession’ ");
mysql_query("DELETE FROM multicastSessionsAssoc WHERE msID=‘$mSession’ ");
}
echo “##”;
}

}else{
echo “invalid syntax”;
}

}else{
echo “connecting to DB”;
}
?>
[/code]
mcTask.php appears to be a binary file.
MulticastTask.class.php:
[code]— new/funcs.sh2014-08-02 23:36:10.000000000 -0400
+++ old/funcs.sh2014-07-15 17:16:38.000000000 -0400
@@ -508,9 +508,6 @@
osname=“Windows 8.1”;
mbrfile=“/usr/share/fog/mbr/win8.mbr”;
defaultpart2start=“368050176B”;
-elif [ “$1” = “8” ]; then
-osname=“Apple Mac OS”;
-mbrfile=“”;
elif [ “$1” = “50” ]; then
osname=“Linux”;
mbrfile=“”[/code]
fog.checkin is the diff for SubMenu.class.php, fog.download is the diff for fog.checkin, fog.mcupdate might be correct? ImageManagementPage.class.php is supposed to be mcTask.php? mcTask appears to be a binary file. MulticastTask.class.php appears to be funcs.sh.

It also is backwards in the diff, It’s using the old files as yours and new files as the ones from svn.

I imagine, you could also make this a single diff with:
[code]diff -rupN <originalTrunk> <yourEditedTrunk> > mychanged.diff[/code]

This will give a single file and give all the changes compared to what mine looks like.

[Tom S]

Sorry Tom. I did diff -u on my Mac. Maybe that’s the issue. I will reupload tonight