Script/Program to Batch Copy ISO Contents for PXE Boot?

[Zer0Cool]

This isn’t really FOG specific, though it will help me in my use of FOG (and maybe others). If it isn’t appropriate to ask here please feel free to close the thread, my apologies.

I was wondering if anyone knows of a script or tool for automating the process of preparing the OS dirs/content needed to PXE install and OS.

What I mean is, before ever getting to creating the iPXE menu entry, one would typically mount an OS iso and copy its contents to a directory on FOG.

What I am finding is that I often need to repeat this process as new versions of OS’s come out and generally check every couple of weeks/months so I end up finding multiple OS’s with new versions. Its then a laborious process to manually create each destination dir/subdirs, mount the iso, copy the contents, unmount the iso and repeat.

I am considering trying to write a bash script to help automate this but figured I would ask before diving in in case someone else has already addressed this.

So, essentially what I would like to accomplish is to point something to a dir and have it process all the iso’s in the dir such that:

• Each iso has a folder created for it according to its OS/ver. IE: win/7 or esxi/6.5 A01
• Each iso is mounted and its contents copied to the right directory for it then unmounted.

From my perspective, the tricky part in a bash script would be how to identify each iso and properly “categorize” it to create the desired dir/subdirs.

I have searched online to see if a tool for this exists but couldnt find one, maybe its just not feasible?

I dont mind updating the iPXE menu entries (or creating new ones) manually in FOG, but processing the isos is starting to be a chore I feel automation could address.

Thanks

[Wayne Workman]

@Zer0Cool said in Script/Program to Batch Copy ISO Contents for PXE Boot?:

From my perspective, the tricky part in a bash script would be how to identify each iso and properly “categorize” it to create the desired dir/subdirs.

That is exactly the most difficult part. Let’s use the Debian DVD ISOs as an example, and let’s use Python instead of bash.

Here’s the debian DVD download page, it’s just basic HTML. This won’t be the same for other sites. You’ll need to code your own parsing for each site. You can see the ISO links at the bottom. https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/

Scripting to find the links in here is somewhat challenging because it’s html instead of something designed to be programmatically read like JSON.

But anyway, install python2 and pip on your fog server. Then use pip to install requests like pip install requests Then create this file:

import requests
import os

result = requests.get('https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/')

# Get just the text from the page.
text = result.text

# Split text on new lines into a list.
lines = text.splitlines()

# Iterate through each line, see if it has .iso in it or not. If so, add it to the ISOs list.
isos = []
for line in lines:
    if '.iso' in line:
        isos.append(line)

# Done with the lines list, delete it.
del lines

# Further splitting. Cut off first unneeded text from the link.
trimmedIsoText = []
for iso in isos:
    iso = iso.split('<a href="')[1]
    # Cut off the last unnecessary part of the line.
    iso = iso.split('">')[0]
    trimmedIsoText.append(iso)

# Done with the isos list, delete it.
del isos

# Define a downloads directory.
downloadDir = "/root"

# Create a list called isos.
isos = []
# Going to put objects into the list.
for iso in trimmedIsoText:
    
    # Create a full link
    link = 'https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/' + iso
    # Add the link to the list, along with it's filename.
    downloadPath = os.path.join(downloadDir,iso)
    isos.append({"link":link,"filename":downloadPath})

# Done with the trimmed isos text, delete it.
del trimmedIsoText

for iso in isos:
    # Download the link to the filesystem.
    response = requests.get(iso["link"], stream=True)

    total_size = int(response.headers.get('content-length', 0))
    print "Downloading '" + str(total_size) + "' bytes from " + iso["link"]
    with open(iso["filename"], 'wb') as writer:
        for chunk in response.iter_content(chunk_size=128):
            writer.write(chunk)

Run that file with python, and it should start downloading to root’s home directory. The way you’d run it with python would be python <file>

Of course, there is more logic needed to determine if you already have a particular ISO or not, putting it into the right spot, mounting the ISO (use python’s subprocess.popen() for that), and adding it to FOG’s boot menu maybe. All that other stuff, I leave up to you. Go learn some python and enjoy the fruits it’ll bring you.

[Zer0Cool]

I actually made some progress on this front. It became a bit more clear when I thought it over and made some considerations.

My setup is such that I have mirrored (via rsync) local repos of CentOS and Fedora. I then symlink the ISO’s within those repos to another place to make them easy for people to download the ISO. In this scenario I don’t have to do anything with the ISO to get the installation files need by PXE as they are available in the repo as I have syncd them.

With the above in mind, I should be able to accomplish the same for any Linux distro that I can mirror/rsync locally which nullifies the needs/goals of my OP.

I then thought about what OS’s this need actually applied to, and in my case it came down to Windows 10 and ESXi (various versions). I don’t mind downloading the ISO manually. What I was looking to accomplish was being able to take an iso or a batch of them in a dir and identify their; OS, version, build, revision and be able to use that to create a unique directory structure for said iso and then dumb the iso contents into the created dir structure for it.

The part I was hung up on was how to get the information I needed for this and that part, at least for Windows 10 (really any Windows ISO) and ESXi I have figured out at least as a proof of concept. I realized that the iso file name would not be the most reliable method of doing so and found ways unique to each OS family to identify the metadata I needed.

ESXi (having checked multiple Dell isos from 6.5-6.7):

I found that they had a /upgrade folder with 2 note worthy files in them. metadata.xml contains the version number and build number. profile.xml has the full name (so even if the file name of the iso is changed this should stay the same) from which I could extract things like the Dell revision and U1/U2 monikers used for Update 1 and Update 2, etc. It also has a tag for “creator” so I should be able to differentiate between Dell isos and non-dell isos.

To parse out the xml I found that xmllint provided by libxml2 would allow me to reliably pull this info from the xml files. Coincidentally, libxml2 was needed by wimlib anyhow (mentioned below for Windows).

Windows 10:

I installed wimlib and using the wiminfo command on the install.wim (combined with grep) I am able to pull information about the ISO as well as deduce if the ISO is a Windows ISO. I can get name, version, build number in this fashion.

I may expand this to Server 2016/2019 if they start releasing new iso builds “frequently” like they do with Windows 10, but older versions the iso hasnt changed for a long time and I have these setup already.

So basically it comes down to checking the ISO’s for the structure/files that match the OS family. In this case /upgrade/metadata.xml being ESXi and /sources/install.wim being Windows. Then I either parse out the XML for ESXi or using wiminfo pull the info from the wim file for Windows.

After that should be “easy”. Create a path using that info, check if it already exists, dump the iso contents, unmount, etc. I am also considering putting a fail safe kind of option in so that if the ISO doesnt match any conditions I am checking to “classify” it, I can prompt the user for manual input if they want to process the ISO allowing them to determine the path to dump it into.

Follow Up Question:

I hadnt considered it before, but generally speaking I assume it would be possible via the bash script to also update or add to the FOG’s iPXE menu by manipulating the “config” file that stores the menu? Or is there a database in between that would make that problematic?

I am not stuck on automating the menu creation but if its a relatively simple addition might as well add it.

Conclusion:

Once I get this written up into a moderately working fashion I intend to post it on my Github and would be happy to share the link back here.

Thanks