Topics created by george1421

@tom-elliott said in FOG 1.5.9.57 on Debian 10 mysql root password is blank:

We are a fully clean install. Meaning no packages except git are installed. No Apache, php, or MySQL. Why should the installer ask for root password here? It has never been setup before this point. There would be no password. FOG, in my opinion, should not be defining the root user password here either.

I think it should force the user to have a DB root password unless it’s a setup with local socket access as described below. That was one of the major points of re-writing that part of the installer. I tested a lot and would hope that the installer does what I say on all officially supported systems.

@george1421 I’ve been able to boot into ipxe now, register the system with the fog server and capture an image from a raspberry pi. I have not tried to deploy an image to the pi because I’m using the sd card to boot the pi into ipxe and if it doesn’t deploy correctly I’ll have a broken boot device. I’m working on some other options right now, but basically FOG is imaging a raspberry pi running an ARM processor.

I need to look into this a bit more, but when partclone ran the background was RED and not normally BLUE. I don’t know what this means but that is the only thing out of normal I found so far. The rest of the imaging environment seemed. Normal. The FOG iPXE menu screen displayed the background image normally and the text was colored appropriately. So the RED in partclone is the only abnormal thing I found so far.

I did have to build a custom FOS Kernel for this Pi because the standard linux kernel could not see the USB keyboard, but using the Pi version of the linux source code did. I also had to build a ARM version of init.xz with no changes except one.

For both bzImage and init.xz I needed to remove the compression on the files. For bzImage the bzip compression is only an x86 feature. For other architectures you need to use gzip to compress the kernel. The other thing, especially for the PIs, is that they are somewhat CPU bound so this dynamic compression takes longer to download and run than just transferring a larger uncompressed image and running it. So for both the kernel and initrd.img I’m not compressing the images. For the initrd.img I’m sending it in cpio format to the Pi.

Development naming convention I’m using is
kernel: ImagePiARM64
initrd: init_arm64.cpio

The kernel is unique to the Raspberry Pi and not intended for any other ARM architecture.

For FOG 1.6.x we probably should add 2 new fields in the FOG Configuration -> FOG settings for the ARM kernels too just like we have for bzImage and bzImage32. But in this case ARM64 and ARM32. I can see one fog server imaging both x86 and ARM processors so we will need to make accommodations. I’m not sure how to handle the kernel field in the host configuration manager though.

@Sebastian-Roth said in Feature request for FOG 1.6.x - Support GRUB as UEFI exit mode:

So back to my point: Why would you chainload to another iPXE binary again for exiting?

I don’t have a specific use case at the moment. So adding additional features without a specific reason may not be as valuable as I initially throught.

Also I had a setback with grub. For some reason grub isn’t loading the config file that is uploaded, but using tftp to connect back to {next-server} (guess) and picking up the grub config file I had there. It is picking up the config file from there, I’m just not sure how it knows where to get the file from. Possibly from dhcp request. I’m still working with it, but grub isn’t quite as dynamic as I would have hoped to find uefi boot partitions.

@Tom-Elliott Good thoughts about not focusing on bash scripts. My thoughts on this…

At work, if you can point your peers to documentation saying “this project only supports this distribution” Generally your peers accept it.

It’s been a topic that has come up before: Dropping installer support down to one or two distributions. CentOS and/or Debian. This probably deserves another forums topic.

At this point, I think supporting one distribution is best. I don’t care which one it is, though Debian probably has the best shot at longevity. I fear CentOS will slowly become irrelevant to many as Red Hat focuses on supporting IBM (their parent company), giving less focus to everything else.

As I continue down this rabbit hole I find that in buildroot the nfs-utils in nfs-utils.mak nfsv4 is specifically disabled.

NFS_UTILS_CONF_OPTS = \ --disable-nfsv4 \ --disable-nfsv41 \ --disable-gss \ --disable-uuid \ --enable-tirpc \ --enable-ipv6 \ --without-tcp-wrappers \ --with-statedir=/run/nfs \ --with-rpcgen=$(HOST_DIR)/bin/rpcgen HOST_NFS_UTILS_CONF_OPTS = \ --disable-nfsv4 \ --disable-nfsv41 \ --disable-gss \ --disable-uuid \ --disable-ipv6 \ --without-tcp-wrappers \ --with-statedir=/run/nfs \ --disable-caps \ --disable-tirpc \ --without-systemd \

Need to be changed to enable nfsv4 to this

NFS_UTILS_CONF_OPTS = \ --enable-nfsv4 \ --enable-nfsv41 \ --disable-gss \ --disable-uuid \ --enable-tirpc \ --enable-ipv6 \ --without-tcp-wrappers \ --with-statedir=/run/nfs \ --with-rpcgen=$(HOST_DIR)/bin/rpcgen HOST_NFS_UTILS_CONF_OPTS = \ --enable-nfsv4 \ --enaable-nfsv41 \ --disable-gss \ --disable-uuid \ --disable-ipv6 \ --without-tcp-wrappers \ --with-statedir=/run/nfs \ --disable-caps \ --disable-tirpc \ --without-systemd \

Once enabled the package and initrd need to be recompiled.

In the initrd in /bin/fog.mount in lines 17 and 19
ref: mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime “$storage” /images >/tmp/mount-output 2>&1

The mount command needs to be updated to create an nfsv4 mount.
from this

up) mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime "$storage" /images >/tmp/mount-output 2>&1 ;; down) mount -o nolock,proto=tcp,rsize=32768,intr,noatime "$storage" /images >/tmp/mount-output 2>&1 ;;

to this

up) mount -o nolock,nfsvers=4,proto=tcp,rsize=32768,wsize=32768,intr,noatime "$storage" /images >/tmp/mount-output 2>&1 ;; down) mount -o nolock,nfsvers=4,proto=tcp,rsize=32768,intr,noatime "$storage" /images >/tmp/mount-output 2>&1 ;;

So far the FOG web code needs to be updated because the nfsv4 shares are presented differently than the >nfsv4.

nfsv3 share structure

/images/ /images/dev/

nfsv4 shares structure

/ /images/ /capture/

./lib/fog/bootmenu.class.php starting at 1497 https://github.com/FOGProject/fogproject/blob/171d63724131c396029992730660497d48410842/packages/web/lib/fog/bootmenu.class.php#L1497

Replacing

$storage = escapeshellcmd( sprintf( '%s:/%s/%s', $ip, trim($StorageNode->get('path'), '/'), ( $TaskType->isCapture() ? 'dev/' : '' ) ) );

to this

$storage = escapeshellcmd( sprintf( '%s:/%s', $ip, ( $TaskType->isCapture() ? 'capture/' : trim($StorageNode->get('path'), '/') ) ) );

Its still not clear if there is a benefit to the cost of modifying the code to support NFS v4 over just keeping everything the same.

@Tom-Elliott said in Feature request for FOG 1.6.x - FOG Installer instll DNSMASQ:

I’m on board for this as well, though wouldn’t mind some help in implementing.

Sure, I will! There is no rush yet. I am working on making the move from 1.5.x to 1.6.x first and then we’ll look into all the feature requests opened lately.

Let me clarify, it turns off the limitation preventing the update to move all tables to innodb.

Fog 1.6 already does this.

@Wayne-Workman said in Feature request for FOG 1.6.x - Scheduled database maintenance:

I should add I’m not talking about history. I’m referring to the issues that literally break fog. This stuff:

Definitely good you phrase this more explicitely!! We should discuss those two things separately (maybe not in different topics though).

While we tell people to sometimes try a cleanup of the DB I am not sure how much issues in the code are still causing this or if this is related to people coming older versions and have not done a cleanup in a long time. Don’t get me wrong, I am not saying the code is perfectly fine, I just don’t know as we don’t have enough evidence that this happens all the time. We would expect way more people to ask about this in the forums.

On the other hand I do agree that running those cleanup (not the history ones!) once a week or even daily wouldn’t hurt:

DELETE FROM `hosts` WHERE `hostID` = '0'; DELETE FROM `hostMAC` WHERE hmID = '0' OR `hmHostID` = '0'; DELETE FROM `groupMembers` WHERE `gmID` = '0' OR `gmHostID` = '0' OR `gmGroupID` = '0'; DELETE FROM `snapinGroupAssoc` WHERE `sgaID` = '0' OR `sgaSnapinID` = '0' OR `sgaStorageGroupID` = '0'; DELETE FROM `snapinAssoc` WHERE `saID` = '0' OR `saHostID` = '0' OR `saSnapinID` = '0'; DELETE FROM `hosts` WHERE `hostID` NOT IN (SELECT `hmHostID` FROM `hostMAC` WHERE `hmPrimary` = '1'); DELETE FROM `hosts` WHERE `hostID` NOT IN (SELECT `hmHostID` FROM `hostMAC`); DELETE FROM `hostMAC` WHERE `hmhostID` NOT IN (SELECT `hostID` FROM `hosts`); DELETE FROM `snapinAssoc` WHERE `saHostID` NOT IN (SELECT `hostID` FROM `hosts`); DELETE FROM `groupMembers` WHERE `gmHostID` NOT IN (SELECT `hostID` FROM `hosts`);

You don’t want to run those on a regular basis I find because it can kill running tasks/sessions if those were setup to run over the weekend for example (state ID see here)

DELETE FROM `tasks` WHERE `taskStateID` IN ("1","2","3"); DELETE FROM `snapinTasks` WHERE `stState` in ("1","2","3"); TRUNCATE TABLE multicastSessions; TRUNCATE TABLE multicastSessionsAssoc; DELETE FROM tasks WHERE taskTypeId=8;

And history is another story altogether…

TRUNCATE TABLE history; TRUNCATE TABLE userTracking;

The API and UI could be separated while still using the same port. The virtualhost config would look something like this:

<VirtualHost *:80> ServerName dev.localhost DocumentRoot /home/projects/smk/cms ErrorLog /var/log/apache2/smk-cms-error.log </VirtualHost> <VirtualHost *:80> ServerName my-project.localhost DocumentRoot /home/projects/smk/deploy ErrorLog /var/log/apache2/smk-deploy-error.log </VirtualHost>

Reference:
https://stackoverflow.com/questions/6069892/different-virtualhosts-with-the-same-port

If this separation were done, I would think the installer could ask you if you want to install the API (defaulting to yes) and ask if you want to install the UI (defaulting to yes). This would allow the admin to break stuff apart.

I also think the database portion should be broken out, and the installer should ask if you want to install that or not (defaulting to yes).

There should be installation arguments for these things too.

Also, not wanting to delay 1.6 as noted by others. I would like to see this separation in a later release. As George has noted, if these things are separated it becomes easier to work on each one independently, meaning the learning curve to contribute is lower. Particularly with using popular frameworks.

Just my 2 cents.

@george1421 We use the certificate store. Though this is something else I was hoping to change in the future because Mono on Linux and Mac OS X have a long history of issues with the certificate store and we might think about keeping the certs just in files on the disk. I have not had the time to think this through. Might be a dead road…

Testing systems Dell o7010 both fog server and client computer. Both systems have local ssd sata drives. The target computer is running a customized linux kernel 5.6.18 and a customized init but both as based on FOG 1.5.9. The customization was done to aid in debugging and bench-marking the systems.

Testing script

mkdir /mnt/locdsk mount /dev/sda1 /mnt/locdsk mkdir /images mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime "192.168.10.1:/images/dev" /images #Test 1 creation of local and remote file by target computer time dd if=/dev/zero of=/mnt/locdsk/L10gb.img count=1024 bs=10485760 time dd if=/dev/zero of=/images/R10gb.img count=1024 bs=10485760 #Test 2 cp files to and from server time cp /mnt/locdsk/L10gb.img /images time cp /mnt/locdsk/L10gb.img /images/L10gb-1.img time cp /images/R10gb.img /mnt/locdsk time cp /images/R10gb.img /mnt/locdsk/R10gb-1.img #Test 3 scp files to and from server time scp /mnt/locdsk/L10gb.img root@192.168.10.1:/images/L10gb-2.img time scp /mnt/locdsk/L10gb.img root@192.168.10.1:/images/L10gb-3.img time scp root@192.168.10.1:/images/dev/R10gb.img /mnt/locdsk/R10gb-2.img time scp root@192.168.10.1:/images/dev/R10gb.img /mnt/locdsk/R10gb-3.img #Test 4 ssh pipeline to and from server time cat /mnt/locdsk/L10gb.img | ssh root@192.168.10.1 "cat > /images/L10gb-4.img" time cat /mnt/locdsk/L10gb.img | ssh root@192.168.10.1 "cat > /images/L10gb-5.img" time ssh root@192.168.10.1 "cat /images/dev/R10gb.img" | cat > /mnt/locdsk/L10gb-6.img time ssh root@192.168.10.1 "cat /images/dev/R10gb.img" | cat > /mnt/locdsk/L10gb-7.img

Testing results as captured.

## Building the test files both local and remote # time dd if=/dev/zero of=/mnt/locdsk/L10gb.img count=1024 bs=10485760 10737418240 bytes (11 GB, 10 GiB) copied, 20.2216 s, 531 MB/s **real 0m20.223s user 0m0.001s sys 0m6.460s # time dd if=/dev/zero of=/images/R10gb.img count=1024 bs=10485760 10737418240 bytes (11 GB, 10 GiB) copied, 93.3867 s, 115 MB/s **real 1m33.390s user 0m0.003s sys 0m5.369s ## Confirm that files exist and are properly sized # ls -la /mnt/locdsk/ total 10485785 drwxr-xr-x 3 root root 4096 Oct 9 08:25 . drwxr-xr-x 3 root root 1024 Oct 9 08:23 .. -rw-r--r-- 1 root root 10737418240 Oct 9 08:26 L10gb.img drwx------ 2 root root 16384 Jan 10 2013 lost+found # ls -la /images/ total 10519109 drwxrwxrwx 3 sshd root 63 Oct 9 2020 . drwxr-xr-x 19 root root 1024 Oct 9 08:23 .. -rwxrwxrwx 1 sshd root 0 Sep 28 13:36 .mntcheck -rw-r--r-- 1 root root 10737418240 Oct 9 2020 R10gb.img drwxrwxrwx 2 sshd root 26 Sep 28 13:36 postinitscripts ### Copy Local to Remote ### # time cp /mnt/locdsk/L10gb.img /images ** real 1m34.821s user 0m0.083s sys 0m7.314s # time cp /mnt/locdsk/L10gb.img /images/L10gb-1.img **real 1m34.759s user 0m0.046s sys 0m6.801s

### Copy Remote to Local ### # time cp /images/R10gb.img /mnt/locdsk **real 1m41.710s user 0m0.084s sys 0m11.327s # time cp /images/R10gb.img /mnt/locdsk/R10gb-1.img **real 1m41.520s user 0m0.095s sys 0m11.392s

### SCP Local to Remote ### # time scp /mnt/locdsk/L10gb.img root@192.168.10.1:/images/L10gb-2.img The authenticity of host '192.168.10.1 (192.168.10.1)' can't be established. ECDSA key fingerprint is SHA256:OpIsFYWVDCr/ovMlmPPSl46jpT332P3+BHnchdxzTCI. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.10.1' (ECDSA) to the list of known hosts. root@192.168.10.1's password: L10gb.img 100% 10GB 110.0MB/s 01:33 **real 1m40.007s user 0m44.460s sys 0m13.378s # time scp /mnt/locdsk/L10gb.img root@192.168.10.1:/images/L10gb-3.img root@192.168.10.1's password: L10gb.img 100% 10GB 109.5MB/s 01:33 **real 1m37.404s user 0m44.420s sys 0m13.068s

### SCP Remote to Local ### # time scp root@192.168.10.1:/images/dev/R10gb.img /mnt/locdsk/R10gb-2.img root@192.168.10.1's password: R10gb.img 100% 10GB 101.9MB/s 01:40 **real 1m44.166s user 0m43.986s sys 0m22.887s # time scp root@192.168.10.1:/images/dev/R10gb.img /mnt/locdsk/R10gb-3.img root@192.168.10.1's password: R10gb.img 100% 10GB 102.0MB/s 01:40 **real 1m44.620s user 0m43.437s sys 0m23.061s

### SSH Pipeline Local to Remote ### # time cat /mnt/locdsk/L10gb.img | ssh root@192.168.10.1 "cat > /images/L10gb-4.img" root@192.168.10.1's password: **real 1m35.562s user 0m42.701s sys 0m12.975s # time cat /mnt/locdsk/L10gb.img | ssh root@192.168.10.1 "cat > /images/L10gb-5.img" root@192.168.10.1's password: **real 1m35.749s user 0m43.478s sys 0m11.166s

### SSH Pipeline Remote to Local ### # time ssh root@192.168.10.1 "cat /images/dev/R10gb.img" | cat > /mnt/locdsk/L10gb-6.img root@192.168.10.1's password: **real 1m43.745s user 0m44.738s sys 0m20.828s # time ssh root@192.168.10.1 "cat /images/dev/R10gb.img" | cat > /mnt/locdsk/L10gb-7.img root@192.168.10.1's password: **real 1m43.564s user 0m43.976s sys 0m21.966s

@Wayne-Workman The replication services provided by FOG use their very own logic implemented in PHP to sync files via FTP protocol. I vote for getting rid of this!

I would hope we can tunnel rsync through SSH to not have to open up another port in the firewall. Together with this I would want to get rid of FTP altogether by using SSH for the other places in the code where FTP is used at the moment (move images directory after upload and kernel after download).

Works like a charm! Thanks george1421!