Mac support

[Tom S]

Great news!!! I will try out the ipxe source and let you all know!! Once I confirm the patches on my end I will test on macbook from 2007-2010, Mac pro 2009-2012, and macbook airs over WiFi.

I will then pull from git the latest branch and diff my changes to submit to fog (multicast tweaks)

I have most of my scripts ready for renaming and binding to domain as well. Just need to see how fog records those tasks. Then create the service.

I love open source…look at what you have accomplish in such a sort time!!! Great job everyone

Thanks,
TS

BTW
Boot camp imaging works on Maverick and Windows XP. I will try 7 this weekend

[Tom S]

fractal13,

I am still getting the VFS syncing error. Are you using the TomElliott.configs? If so are you still using the firmware patch Fog recommends to use when building the kernel?

Thanks,

Tom

[Tom Elliott]

[quote=“Tom S, post: 34506, member: 25305”]fractal13,

I am still getting the VFS syncing error. Are you using the TomElliott.configs? If so are you still using the firmware patch Fog recommends to use when building the kernel?

Thanks,

Tom[/quote]

The VFS syncing error, how does this come out? Are you using the correct pairing? init.xz and bzImage should both be 64 bit. init_32.xz and bzImage32 should both be 32 bit. If you load 32 bit bzImage with 64 bit init.xz, or 64 bit bzImage with 32 bit init.xz, you will see this error.

[Tom S]

[quote=“Tom Elliott, post: 34517, member: 7271”]The VFS syncing error, how does this come out? Are you using the correct pairing? init.xz and bzImage should both be 64 bit. init_32.xz and bzImage32 should both be 32 bit. If you load 32 bit bzImage with 64 bit init.xz, or 64 bit bzImage with 32 bit init.xz, you will see this error.[/quote]

Thanks for reply Tom,
No my versions are correct, I have tested with the new files last night. The same pairing also works on cd boot of mac and the 30 Dell pcs I am imaging right now.

I believe it has to do with ipxe and the bzImage. I changed the log level to 6 and can see that it sees the macbook devices and board info. It also can see my hard drive. It also talks to dhcp and gets an ip address. I think it just can’t see the /dev/ram0. Fractal13 said this had to do with a patch for the kernel. I have tried numerous variations of the config file for the kernel, adding and removing efi support settings, etc. Currently I am working on kernel 3.10.50.

Thinking that is a memory issue I tried to add the ipxe Mem_map settings but the compiling fails. If at all possible, Fractal13 if you could provide your config files for the kernel and ipxe???

(I should also mention I am testing on MacBook 5,2 and 5,1)

I will continue to tinker…thanks,

TS

[Tom S]

OK…after hours of researching the efi stubs and the ipxe boot process, I got macbooks 2007-2011 netbooting so far. It maybe common knowledge, but it was news to me.

The VFS syncing error was totally the way iPXE handles memory mapping and the hand off of the init.xz file system.

I was using the syntax to load the image in the boot menu as:
[CODE]:img1
kernel bzImage root=/dev/ram0 rw ramdisk_size=127000 ip=dhcp dns=< DNS IP> web=${fog-ip}/fog/ consoleblank=0 loglevel=4 type=down img=<IMAGE NAME> ftp=${fog-ip} imgType=n osid=7 storage=${fog-ip}:/images capone=1 imgFormat=2
imgfetch init.xz
boot || goto MENU[/CODE]

But the only way I could get it to work was as follows:
[CODE]:Img1
imgfetch init.xz
kernel bzImage.efi initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 ip=dhcp dns=${fog-ip} web=${fog-ip}/fog/ consoleblank=0 loglevel=4 type=down img= ftp=${fog-ip} imgType= osid=99 storage=${fog-ip}:/images mc=yes bypass=1 mSession=“”
boot || goto MENU[/CODE]

imgfetch needs to be first so that is add to memory first. Then add the kernel parameter with initrd=. The bzImage.efi is just a renamed bzImage. I think the previous pattern did not supply the name of the init file system. I could see that it was using the room in memory but the kernel just could not grab it…strange

Nonetheless so far, no driver issues, no kernel issues. Current grab from git provides bootable 32 and 64 bit images and ipxe efi(64 bit only,I compiled the 32 Bit needed for 2007 and 2008 macbooks). I feel this is a good start.

Thanks Tom and Fractal13 for getting the new binary up!!

TS

[fractal13]

TS,

Tom did all of the work on the binary. I’m sorry I forgot about the initrd= kernel parameter. I completely spaced it. Tom do you know if the initrd= kernel parameter will give problems to non-efi versions of ipxe? If it doesn’t, we should probably add it to BootMenu.class.php for the $this->kernel specification.

The ipxe team (mcb30) is still looking at the Apple EFI SNP network driver issues for unsupported NICs.

Keep up the good work.

[Tom Elliott]

[quote=“fractal13, post: 34588, member: 24300”]TS,

Tom did all of the work on the binary. I’m sorry I forgot about the initrd= kernel parameter. I completely spaced it. Tom do you know if the initrd= kernel parameter will give problems to non-efi versions of ipxe? If it doesn’t, we should probably add it to BootMenu.class.php for the $this->kernel specification.

The ipxe team (mcb30) is still looking at the Apple EFI SNP network driver issues for unsupported NICs.

Keep up the good work.[/quote]

I can readd the initrd= parameter as it used to do that. Just when we switched, I left it out as all seemed to work perfectly. So I’ll add that back. Thank you much for the info.

[Tom Elliott]

And it’s done.

[Dave Smith]

I would Like to thanks Tom and fractal13 for all there hard work on getting mac support in fog. Can’t wait for the day I can Bin my deploy studio server. The day is surely drawing near. that no matter what OS comes through the door (Linux mac or windows). i will be able to image it using ONE imaging system FOG.:D:):D

[fractal13]

Great news folks!

Today Michael Brown finished the changes to iPXE to support Apple’s EFI implementation with the SNP (generic) network driver.

This makes at least 3 hurdles that iPXE has cleared so that those of us who have to support Apple hardware will be able to use FOG for imaging. 1) Switch into text mode so we can see menus; 2) Get the linux kernel fixed so it doesn’t write to memory it doesn’t own; 3) Generic networking (important for iMacs in the last couple of years.).

Kudos to Michael and the rest of the iPXE team!

If you’re interested in being an alpha tester, follow this thread. In a few days we expect to need a variety of Apple hardware to finish testing the iPXE fixes (only tested on iMacs mid-2011, late-2012 so far), and begin testing the FOG features. We’ll make an additional post when ready.

[Tom S]

OK so I have extensively tested out mac support for Fog. So far so good. Macbook Airs over WiFi no go though. Obviously… ipxe works but the kernel and ram disk do not have support for wifi support, maybe someday…:). The funcs.sh file still needs the Apple Mac OS added to the determine os function for the current fog init file to work, but that aside, everything works!!

I finished the multicast patches I spoke of earlier and once I find the correct forum I will upload the diff files and link it back here.

I did want to talk about the Fog service for Mac support. In my current profession I have written scripts that do what I can tell the Windows Fog service does, but for OS X. They are primarily bash scripts that can easily be piped from a C or another language. My question to everyone here is this:

What language do we want the service built on???

The obvious answer would be Obj-C, Apple’s native IDE platform. At the same time an argument could be for straight up C. My experience working for schools and businesses that I feel are closely related to the people of Fog, is that the device OS’s are all over the place, Windows XP, 7, Vista, 8, 8.1, OSX and linux thus requiring a new app project for each platform. I know that I find it a real pain writing for a half dozen os versions.

Please forgive me as I say this next statement. Some people will want to shoot me on the spot but…what about a cross platform language like Java??? (Please do not hurt me;)) Please entertain an example.

I wrote a bit of software for our devices that is written in Java called “iInstall.” This software calls out to my php server probing for a task, if it finds one it will download and install the software. The users can also authenticate against LDAP and install software provided to their group. We piloted this on Macs. It was written on Obj-C at first. Worked awesome, I was instructed to rewrite it for windows and nix. I looked at the actual tasks and thought “really the only change between os versions is the commands, not the logic. So why do I have to write the same logic 3 or 4 times?” I decided to rewrite the app in java. I used calls to get the os version and then adjust the commands for the verious versions. ( I sorry, I do not want to be known as the long email guy)

So as we are looking at expanding fog to support Macs and in need for a Service to support OS X what do we do? I am happy to write in Obj-C or Java. It doesn’t matter to me, I just wanted to point my past experience in writing apps for the cross platform world.

Thanks for the time,

TS

[Tom S]

[quote=“fractal13, post: 34774, member: 24300”]Great news folks!

Today Michael Brown finished the changes to iPXE to support Apple’s EFI implementation with the SNP (generic) network driver.

This makes at least 3 hurdles that iPXE has cleared so that those of us who have to support Apple hardware will be able to use FOG for imaging. 1) Switch into text mode so we can see menus; 2) Get the linux kernel fixed so it doesn’t write to memory it doesn’t own; 3) Generic networking (important for iMacs in the last couple of years.).

Kudos to Michael and the rest of the iPXE team!

If you’re interested in being an alpha tester, follow this thread. In a few days we expect to need a variety of Apple hardware to finish testing the iPXE fixes (only tested on iMacs mid-2011, late-2012 so far), and begin testing the FOG features. We’ll make an additional post when ready.[/quote]
I can supply you with the list of the macs I have confirmed working. Later this month as we finish up imaging our several thousand macs

[Tom Elliott]

@Tom S, you can post the diff’s right here.

[Tom S]

So here are the changes made from 2124. I have been told in the past to omit comments in my diffs in other projects I have contributed to. Please let me know if you need more comments.

So this is a serious hack. Most of the SQL calls do not use the fog api rather they use the php mysql commands. The Fog api kept denying my request to create tasks because of invalid MAC address issues. Also the main post function includes some sql “Alter” structure commands in order to edit the tables to fit the needs of the script.

I am not proud of the overall structure of the patch files so if someone knows how to do it with the fog api…awesome.

Please let me know if you have questions.

Thank you,

TS

[url=“/_imported_xf_attachments/1/1254_diff.zip?:”]diff.zip[/url]

[Tom Elliott]

Hey Tom,

I’ve taken a look at these, and some have been relatively simple to add, but there’s a lot of confusion in the files.

For example, When I look at the diff file for fog.checkin here’s what I see:
[code]— new/SubMenu.class.php2014-08-04 10:10:50.000000000 -0400
+++ old/SubMenu.class.php2014-07-10 18:11:12.000000000 -0400
@@ -192,7 +192,6 @@
$this->subMenu[$this->node][‘search’] = $this->foglang[‘NewSearch’];
$this->subMenu[$this->node][‘list’] = sprintf($this->foglang[‘ListAll’],$this->foglang[‘Images’]);
$this->subMenu[$this->node][‘add’] = sprintf($this->foglang[‘CreateNew’],$this->foglang[‘Image’]);
-$this->subMenu[$this->node][‘Multicast’] = sprintf($this->foglang[‘Multicast’],$this->foglang[‘Image’]);
if ($_REQUEST[‘id’])
{
$this->subMenu[$this->node][‘id’][$linkformat.‘#image-gen’] = $this->foglang[‘General’];
[/code]
fog.download:
[code]— new/fog.checkin2014-08-05 18:11:58.000000000 -0400
+++ old/fog.checkin2014-07-24 14:32:42.000000000 -0400
@@ -61,7 +61,7 @@
mkdir /images 2>/dev/null;
mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime $storage /images;
echo “Done”;
-elif [ “$type” == “down” ] && [ “$capone” != “1” ] && [ “$joinmc” != “1” ]; then
+elif [ “$type” == “down” ] && [ “$capone” != “1” ]; then
mac64=ifconfig | grep HWaddr | head -n1 | base64;
dots “Attempting to send inventory”;
doInventory 2>/dev/null;
@@ -153,81 +153,6 @@
debugPause;
fi
done
-elif [ “$joinmc” == “1” ] && [ “$mc” == “yes” ]; then
-echo “”;
-echo " #############################################################################“;
-echo " # #”;
-echo " # Joining Multicast Session! #“;
-echo " # #”;
-echo " #############################################################################";
-echo “”;
-canGo=“no”;
-while [ “$mSession” != “exit” ] && [ “$canGo” != “yes” ]; do
-echo -n “Enter in Session Name:[Type “exit” to quit]”;
-echo “”;
-read mSession;
-multicastValues=wget -O - "http://${web}service/mcTask.php?task=join&mSession=$mSession" 2>/dev/null
-if [ “$multicastValues” != “” ]; then
-arr=$(echo $multicastValues | tr “;” “\n”);
-i=0;
-for x in $arr
-do

• array[i]=$x;
  -i=$i+1;
  -done
  -img=${array[0]};
  -imgTypeId=${array[1]};
  -osid=${array[2]};
  -port=${array[3]};
  -storageip=${array[4]};
  -sessionPwd=${array[5]};
  -if [ “$imgTypeId” = “0” ]; then
  -imgType=“n”;
  -fi
  -if [ “$imgTypeId” = “2” ]; then
  -imgType=“mps”;
  -fi
  -if [ “$imgTypeId” = “3” ]; then
  -imgType=“mpa”;
  -fi
  -if [ “$sessionPwd” != “” ]; then
  -echo “”;
  -echo -n " This Session Requires A Password To Join,";
  -didAuth=“0”;
  -for I in 1 2 3;do
  -echo “Please enter in password:”;
  -read password;
  -if [ “$password” == “$sessionPwd” ]; then
  -echo “Password Accepted!!”;
  -didAuth=“1”;
  -break;
  -else
  -echo “Incorrect Password, Try again”;
  -fi
  -done
  -if [ “$didAuth” != “1” ]; then
  -echo “Incorrect password. To many attempts made, now rebooting”;
  -exit 0;
  -fi
  -fi
  -determineOS $osid;
  -canGo=“yes”;

•       echo -n "Multicast Configuration Complete";
  

• echo "";
  

•     echo "Joining Session: $mSession
  

-Using Image: $img
-OS: $osname
-Image Type: $imgType
-Session Port: $port";
-dots “Mounting File System”;
-mkdir /images $debugstring 2>/dev/null;
-mount -o nolock,proto=tcp,rsize=32768,intr,noatime $storage /images 2>/tmp/mntfail;
-echo “Done”;
-else
-echo -n “No mulicast sesion with that name was found!! Try again.\n”;
-fi
-done

else
echo “Done”;
dots “Mounting File System”;[/code]
fog.mcupdate:
[code]#!/bin/sh
IFS=’
';
web=$1;
mSession=$2;
sleep 3;
while true; do
fle=“/tmp/status.fog”
status=tail -n 2 $fle 2>/dev/null | head -n 1 2>/dev/null
cat /dev/null > $fle 2>/dev/null

status=echo $status | base64;
res=wget -T 3 -O - --post-data="status=${status}" "http://${web}service/mcTask.php?task=progress&mSession=${mSession}" 2>/dev/null;
sleep 3;
done[/code]
ImageManagementPage.class.php:
[code]<?php

@error_reporting(0);
require(‘…/commons/base.inc.php’);

$conn = @mysql_connect( DATABASE_HOST, DATABASE_USERNAME, DATABASE_PASSWORD);

if ( $conn ){
if ( ! @mysql_select_db( DATABASE_NAME, $conn ) ) die( mysql_error() );

if( isset($_GET[‘task’]) && isset($_GET[‘mSession’])){
$task = $_GET[‘task’];
$mSession = $_GET[“mSession”];

if($task==“join”){
$sql = “select max(msID) as msID from multicastSessions where msName = ‘$mSession’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
while( $ar = mysql_fetch_array( $res ) ){
$msID = $ar[“msID”];
}
$storageNodeID=“”;
$storageNodeIP=“”;
$sql = “select * from multicastSessions where msID = ‘$msID’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
if($res !=“”){
while( $ar = mysql_fetch_array( $res ) ){
$msImage = $ar[“msImage”];
$msIsDD = $ar[“msIsDD”];
$port = $ar[“msBasePort”];
$sessionPwd = $ar[“sessionPwd”];
$storageNodeID=$ar[‘msNFSGroupID’];
}
$imageType=“”;
$sql = “select * from images where imageID = ‘$msImage’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
while( $ar = mysql_fetch_array( $res ) ){
$hostOS = $ar[“imageOSID”];
$imageType = $ar[‘imageTypeID’];
$msImage = $ar[‘imageName’];

}
$sql = “select * from nfsGroupMembers where ngmID = ‘$storageNodeID’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
while( $ar = mysql_fetch_array( $res ) ){
$storageNodeIP = $ar[“ngmHostname”];

}

echo “$msImage”.“;”.“$imageType”.“;”.“$hostOS”.“;”.“$port”.“;”.“$storageNodeIP”.“;”.“$sessionPwd”;

}
}else if($task==“progress”){
$str = explode(‘@’,base64_decode($_REQUEST[‘status’]));
$percent=trim($str[5]);
mysql_query("UPDATE multicastSessions set msPercent=‘$percent’,msState=‘3’ WHERE msName=‘$mSession’ ");
}else if($task==“finish”){
$sql = “select * from multicastSessions where msName = ‘$mSession’”;
$res = mysql_query( $sql, $conn ) or die( mysql_error() );
$clientCount=0;
while( $ar = mysql_fetch_array( $res ) ){
$clientCount = $ar[“msClients”];
}
if($clientCount>1){
$clientCount–;
mysql_query("UPDATE multicastSessions set msClients=‘$clientCount’ WHERE msName=‘$mSession’ ");
}else{
mysql_query("DELETE FROM tasks WHERE taskName=‘$mSession’ ");
mysql_query("DELETE FROM hosts WHERE hostName=‘$mSession’ ");
$sql = “SELECT * FROM multicastSessions WHERE msName=‘$mSession’ “;
$res = mysql_query( $sql, $conn );
$thisMSID=””;
while( $row = mysql_fetch_array( $res ) ){
$thisMSID= $row[‘msID’];
}
mysql_query("DELETE FROM multicastSessions WHERE msName=‘$mSession’ ");
mysql_query("DELETE FROM multicastSessionsAssoc WHERE msID=‘$mSession’ ");
}
echo “##”;
}

}else{
echo “invalid syntax”;
}

}else{
echo “connecting to DB”;
}
?>
[/code]
mcTask.php appears to be a binary file.
MulticastTask.class.php:
[code]— new/funcs.sh2014-08-02 23:36:10.000000000 -0400
+++ old/funcs.sh2014-07-15 17:16:38.000000000 -0400
@@ -508,9 +508,6 @@
osname=“Windows 8.1”;
mbrfile=“/usr/share/fog/mbr/win8.mbr”;
defaultpart2start=“368050176B”;
-elif [ “$1” = “8” ]; then
-osname=“Apple Mac OS”;
-mbrfile=“”;
elif [ “$1” = “50” ]; then
osname=“Linux”;
mbrfile=“”[/code]
fog.checkin is the diff for SubMenu.class.php, fog.download is the diff for fog.checkin, fog.mcupdate might be correct? ImageManagementPage.class.php is supposed to be mcTask.php? mcTask appears to be a binary file. MulticastTask.class.php appears to be funcs.sh.

It also is backwards in the diff, It’s using the old files as yours and new files as the ones from svn.

I imagine, you could also make this a single diff with:
[code]diff -rupN <originalTrunk> <yourEditedTrunk> > mychanged.diff[/code]

This will give a single file and give all the changes compared to what mine looks like.

[Tom S]

Sorry Tom. I did diff -u on my Mac. Maybe that’s the issue. I will reupload tonight

[fractal13]

[quote=“Tom S, post: 34775, member: 25305”]

I did want to talk about the Fog service for Mac support. In my current profession I have written scripts that do what I can tell the Windows Fog service does, but for OS X. They are primarily bash scripts that can easily be piped from a C or another language. My question to everyone here is this:

What language do we want the service built on???

TS[/quote]

TS: The FOG service for Windows is under active development in C#. So, we won’t be using that code for OS X/Linux. I would encourage you to consider C. That will be at least a portable as Java, if we static compile it. It will be easy to write a simple service like you’re talking about being a wrapper for shell scripts. I could give you some boiler-plate networking code and give advice on invoking shell scripts from within, if you want it. This is something I’ve done in previous projects many times.

Thanks for all of your help.

[Tom Elliott]

I’ve begun work on this and added the necessary GUI elements already.

I can create the session and cancel the session. There is no need for the storage group or the os parameters in scheduling this as the group and os are stored with the image.

I’m not going to create the fog.mcupdate or mcTask.php file as I can do all of this from the ipxe menu.

This is just a checklist for me to follow, the multicast stuff is not operational from a name joining setup yet.

[Tom S]

Ok so here is the requested diff file. I had to zip it in order to upload the file. So I hope this helps. I look forward in seeing the final product!!

[url=“/_imported_xf_attachments/1/1260_joinMC.diff.zip?:”]joinMC.diff.zip[/url]

[Tom S]

[quote=“fractal13, post: 34855, member: 24300”]TS: The FOG service for Windows is under active development in C#. So, we won’t be using that code for OS X/Linux. I would encourage you to consider C. That will be at least a portable as Java, if we static compile it. It will be easy to write a simple service like you’re talking about being a wrapper for shell scripts. I could give you some boiler-plate networking code and give advice on invoking shell scripts from within, if you want it. This is something I’ve done in previous projects many times.
Thanks for all of your help.[/quote]
Fractal13
I get what you are saying about using C. My only point was if I where to write it in java and got a working app, adding the windows version would a piece of cake and then fog would have a single executable for all os types one Maven or eclipse file rather than a VS project and an xcode project and a another for the various flavors of nix. I know a lot of people are not very happy with java so thats why I asked.

Nonetheless I can write it in C and keep you all informed. Is an xCode project acceptable for submittal?

I feel this forum proves why open source Rocks!!!

Thanks,
TS