Posts made by RobPomeroy

@fogman4 Cool - good result!

@fogman4 As Tom says, I wouldn’t use FTP. It’s a painful protocol. I’ve favoured HTTP because of past experiences with TFTP (dropped packets and lower quality networks, corruption…). TFTP would be marginally faster at certain points though. And the HTTP setup leaves you with a clean directory structure.

@george1421 Yep, thanks, that’s exactly the approach I have in mind for my centralised systems.

@george1421 Right, thanks George. Interesting to see a personal vote for Veeam. It’s on my PoC list, alongside ReaR and SystemImager.

@george1421 - Embarrassingly I’m a bit out of my depth to take this any further. I’ve been furiously reading up on specifics of MD and EFI, but I don’t have the breadth of OS installation experience and deep filesystem knowledge to pull it all together. (I tend to spend most of my time higher up the application stack.)

What might work for me, is to turn my attention to a backup/recovery solution - still with FOG’s assistance. As I’ve posted elsewhere, ReaR is looking like a strong contender for that. I need to run a PoC to confirm, but that might give me the higher speeds I’m looking for, as well as being more forgiving of my knowledge deficit.

With apologies for the necropost, I’m looking at exactly this question at the moment. My thinking is you could possibly pair FOG with Relax-and-Recover or SystemImager for a complete backup/image/bare metal restore solution.

Looks like you would be able to add a Relax-and-Recover (ReaR) menu item to FOG’s menu for a full network-based restore, in the event of disaster. Since FOG can be used as a scheduler and general-purpose task executer, it is entirely possible to run one of these open source “live backup” solutions alongside FOG - i.e. on the same server.

I also note that ReaR can deal with UEFI/MD RAID situations, which seems to be a rare gift.

RST gives me the heebie jeebies.

@tom-elliott - yes, exactly. I believe the slowness in deploying an image could also be down to the lack of insight Partclone has into the contents of the MD RAID partitions? I imagine it is faithfully writing long sequences of zeroes, because we’re in dd/raw mode.

I’m hoping that, soon if not yet, Partclone will be able to “lift the veil” on MD RAID 1 partitions and clone in a more efficient manner. Since we have all the data in that partition, we don’t need to worry about faithfully recreating striping/checksums, etc. I’d be perfectly happy with a solution that imaged only one drive, and left the other drive to be populated by a RAID re-sync, if that improved speed to boot.

Are there any other tools FOG might have at its disposal (he asks, plaintively)?

Hey George - I’m using Linux MD RAID, not Intel RST. FWIW, the devices under Linux are at /dev/md0, etc. as you would expect.

The ESP partitions are always formatted as VFAT. I imagine the MD RAID physical partitions and their child partitions will be visible provided you load the mdraid kernel driver.

Partclone’s documentation mentions nothing about handling MD RAID (that I can find).

Does anyone have tips for improving speed & efficiency when imaging MD RAID systems?

I’m working with UEFI boot machines with dual 1TB drives. UEFI throws us a curve ball in that you can’t make the EFI System Partition part of a RAID array; instead you need duplicated normal ESPs on both drives (a shortcoming of the EFI specification, apparently). The remainder of the drives consist of RAID 1 arrays, e.g.:

As far as I can tell, FOG’s only imaging option here is multi-partition, non-resizable. And that sets Partclone into dd mode, which as you can imagine on 1TB (SATA) drives is slow. Taking the image (yesterday) took 5 or 6 hours. Restoring an image which I’m doing today looks like it will take circa four hours (for both drives).

I’ve scoured the wiki (and the internet) for ideas to speed up this process, but I’m drawing a blank. It seems highly inefficient to have Partclone write nearly 1TB worth of zeroes.

Sidenote: in case anyone wonders why the machines are set up this way - they will be imaged centrally but then deployed to remote locations with no on-site technical resource. RAID 1 is one of our strategies for reducing down time on drive failure.

@george1421 Done.

With the assistance of @george1421 in this thread, I arrived at the following process.

Features

• Unattended installation, preseeded (using a preseed.cfg)
• Completely hands-off, after selecting the appropriate FOG menu item on PXE boot
• Based on Debian’s netboot - no need to download the Debian ISO
• Uses HTTP only (though you could swap in NFS, TFTP, etc., if you prefer)
• The three files required for the installation can reside in a single location on the FOG server
• Process can be generalised to other versions of Debian and other OSes (though you’d need to work out the Kickstart part for RedHat distros, etc.)
• Works with UEFI - I repeat, works with UEFI!!!

Process

On the FOG server, Unpack netboot.tar.gz to a folder under the Apache root, e.g.:

mkdir -p /home/fogproject/www/os/debian/10.7N
cd /home/fogproject/www/os/debian/10.7N
wget http://ftp.nl.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/netboot.tar.gz
tar zxf netboot.tar.gz
rm netboot.tar.gz
chown -R apache:apache /home/fogproject/www/os/debian/10.7N

Note that I relocated my www root to /home/fogproject. Yours may still be at /var/www. or similar.

The above unpacks:

debian-installer/
pxelinux.0@
pxelinux.cfg@
version.info

We only really need two files, which are in the debian-installer directory: linux and initrd.gz. Still, I leave this folder structure as it is.

Place your preseed file (normally preseed.cfg) in the same directory as the netboot files above (/home/fogproject/www/os/debian/10.7N in my example). The content of your preseed file is entirely up to you. See the official example, to get started.

Create a FOG menu entry as follows:

Menu item: os.Debian.10.7.auto.UEFI
Description: Unattended Debian 10.7 installation (UEFI)
Parameters:

kernel http://${fog-ip}/os/debian/10.7N/debian-installer/amd64/linux auto=true url=http://${fog-ip}/os/debian/10.7N/preseed.cfg interface=auto hostname=debian-10 domain=local initrd=initrd.gz vga=788 noprompt quiet
imgfetch http://${fog-ip}/os/debian/10.7N/debian-installer/amd64/initrd.gz
boot || goto MENU

Menu Show with: All hosts

Notes

If you need an option for manual installation, simply drop the auto parameters from the kernel line:

kernel http://${fog-ip}/os/debian/10.7N/debian-installer/amd64/linux initrd=initrd.gz vga=788 quiet

The clue to getting this working was to include initrd=initrd.gz in the kernel line and then use imgfetch to make initrd.gz available.

@robpomeroy said in Fully automated UEFI-based O/S installation (Debian 10):

three required

*the required

Yep, HTTP worked perfectly too, which means I could put the three required files in a single location under the web root. FOG menu parameters:

kernel http://${fog-ip}/os/autoinstall/debian-10/debian-installer/amd64/linux auto=true url=http://${fog-ip}/os/autoinstall/debian-10/preseed.cfg interface=auto hostname=debian-10 domain=local initrd=initrd.gz vga=788 noprompt quiet
imgfetch http://${fog-ip}/os/autoinstall/debian-10/debian-installer/amd64/initrd.gz
boot || goto MENU

This is clean & efficient. I like it. Probably not as quick as using a Live image, but I’m doing this over a fast internet connection and it’s a minimal install. So the whole thing is done in ten minutes or so.

Again, I’m more than willing to write this up somewhere if useful for others.

@george1421 George, firstly thanks again for the all the work you’ve put into this!

This steered me in exactly the right direction. The following are the parameters I used for a completely automated netboot install (no Live ISO required). This is the point I was hoping to get to. Brilliant.

kernel tftp://${fog-ip}/debian-installer/amd64/linux auto=true url=http://${fog-ip}/os/autoinstall/debian-10/preseed.cfg interface=auto hostname=debian-10 domain=local initrd=initrd.gz vga=788 noprompt quiet
imgfetch tftp://${fog-ip}/debian-installer/amd64/initrd.gz
boot || goto MENU

So this removes the need to mess around with grub.cfg or boot to an intermediate (grub) menu. All you need are the netboot.tar.gz files and a preseed.cfg. Next I’ll try doing this all via HTTP (rather than TFTP), to avoid polluting /tftpboot. I’m mindful that I would like to make the backup and restoration of FOG itself as straightforward as possible.

I know I’ve said this a few times in this thread (!) but once more: thank you. If I can help by writing up a wiki page for this process, do point me in the right direction.

@george1421 - okay, thanks George.

This boots beautifully though:

kernel tftp://${fog-ip}/grubx64.efi
boot || goto MENU

Is there no way to squirt additional parameters into that Grub EFI loader?

Sorry - I figured that this might be simple. I should know better by now!

@george1421 It’s a physical server (Dell T40) that literally cannot boot BIOS-based OSes, I’m afraid. It has no internal legacy boot capability. I was amazed, but then I haven’t worked with physical hardware in a while.

@george1421 - Thanks George. My problem is understanding all the options!

Following your latest instructions unfortunately results in a kernel panic (presumably initram related?). The kernel and init disk both transferred correctly via TFTP. Incompatibility with UEFI perhaps?

kernel tftp://${fog-ip}/os/debian/Server10.7/linux
initrd tftp://${fog-ip}/os/debian/Server10.7/initrd.gz

Have you tried leaning on IPXE’s HTTP support here rather than TFTP? I’m just wondering if that would make it cleaner (keeping all installer files in one location under the web root).

@george1421 said in Fully automated UEFI-based O/S installation (Debian 10):

@robpomeroy With the exception of testing the preceed file these instructions work as advertized: https://forums.fogproject.org/post/140720

On UEFI? I’ll try that now! Thanks.