Posts made by george1421

@EduardoTSeoane It would be helpful if you could document the changes you have made to your fog installation. The developers don’t have FOG installations as large as what you have so its hard to simulate the workloads you are seeing. It would be helpful for both the Developers as well as other large campus FOG installations if you could share what you have learned. @fry_p does have a large campus network where he was working on a similar issue to what you have.

So it would be very helpful if you could share what you learned from your testing for everyone.

@Jimbohello After you are able to image the machine, could you provide feedback on if the kernel args works for you or you had to go the debug route and enter the instructions directly into the console? We are trying to collect information on which solution works the best for resolving these issues with these specific nvme drives.

In short these two commands (applied differently) keep the drive from going into low power mode during imaging. The developers are working towards a solution where the imaging engine will adapt these commands without intervention.

This may take a few times discussion to find the best solution for you.

Lets start with basics of FOG.

The operating system that runs on the target computer to capture and deploys images is based on linux, for this talk we will call it FOS (Fog Operating System) Linux. This is a customized version of linux designed just for imaging. So you have to remember that FOS is Linux and can only run linux program not windows programs. There are a few linux programs that let you interact with windows registry but not many.

In FOG there are 2 places where the fog admin can add in custom linux bash scripts to interact with the target computer. The first place is a POSTINIT script. This script is run before imaging begins. In the POSTINIT script the fog admin can run commands that maybe needed to connect to disk arrays or configure the system needed before imaging starts.

The second place is a POSTINSTALL script. This bash script is called after the image has been written to the storage media and before the system reboots. At this place you have the ability to connect to the target disk and interact with the contents of the disk using linux commands. You may NOT run any windows commands in a POSTINSTALL because FOS is Linux not Windows.

If you want to interact with the UEFI bios boot order FOS Linux has the linux command efibootmgr that you can call from your POSTINSTALL script.

Your POSTINSTALL script can edit text files on the target computer by mounting the proper partition and using sed or other linux script commands to change the text file values. So if you need to edit the grub.conf file you can do it in the post install script.

@mattlong01 said in Boot problems on Lenovo M720s with M.2 drive & UEFI:

The M.2 drive isn’t even listed

So this tells me that the firmware can’t find the uefi boot loader typically in /EFI/Boot directory on the uefi boot partition.

I didn’t ask you this last time, what version of FOG are you using? If its not 1.5.7 then what kernel version are you using (as seen under the fog configuration menu)?

@EduardoTSeoane said in Bottleneck on DATABASE:

Yes i have the database virtual server (mariadb 10.4) and fog web virtual server are different virtual machines.

Ok good this would have been the first step.

The Virtual Sysadmin told me than the database server is stored on a mechanical sas.

As long as its a sas array then its OK. A single sas disk is not good for FOG imaging. But since you said its a VM then you probably have a proper VM Host server.

On myISAM i have implemented all the recommendations, now i have innodb on basic configuration, I’ll do optimizations next week with a few days working.

The mysqltuner script is very informative on its recommendations. Its good to let your mysql server run for a few days so it collects good run time data for the mysqltuner script to review.

I use FOG client, with a checkin time about 15min as test, currently workload average is about 15-17, but im still doing verifications.

For a 15 minute check in time and 17000 fog client computers, that gives you (In an ideal environment) 18 check-ins per second. I wonder if you increased the number of php-fpm worker threads to help with this load.

I do this changes because i detect a lot of table locks, that make a poor performance.

First let me say that FOG isn’t designed for this large of an installation. But I’m sure there are things we can do to optimized the installation. The first step is to get the database running well. Then we can look into other tweaks to help productivity.

Have you already moved the mysql server (service) to its own server instance and not running directly on the FOG server?

Is the disk where the database stored been moved to a flash based media (SSD or NVMe)?

Have you run the mysqltuner script against your mysql server? Have you implemented any of its recommendations?

In your installation, do you use the FOG client? If so what is the check in time you set?

Did you sysprep the image before capture?

How did you build your reference image? Did you build it on the same model computer, capture and then deploy to a different computer of the same model, or did you create a generic reference image on a VM and then deployed it to the target hardware.

For uefi computers, if the boot device has a fat32 partition with the boot file in the proper location the uefi firmware will let you set it as a bootable device. If the boot file isn’t found that drive won’t be selectable to boot.

So this tells me that refind can find the proper uefi boot loader, but the firmware can’t??

Does the lenovo have something similar to the Dell F12 boot manager? If so can you select the m.2 drive from that menu and have it boot?

@mpatel OK a few more question. What is your plan for imaging at the remote site (physical action plan)? Will you send a IT Tech to the remote site and image all of the computers or do you want to do everything remotely?

Do you have hardware you can send to the remote site to act as a FOG server? Understand that a circa 2016 laptop with 4GB of RAM and a SSD is more than enough computer for a FOG server for less than 50 computers.

@stu Once you get the system registered, I’d like you to do something for us.

1. Schedule a capture/deploy it doesn’t matter, but before you hit the schedule task button tick the debug checkbox. Then schedule the task.
2. PXE boot the target computer. Plug a traditional keyboard in for this test.
3. After several screens of text on the target computer that you need to clear by pressing the enter key, you will be dropped to the FOS Linux command prompt in debug mode.
4. At the command prompt key in lspci and take a clear picture of the numbers it prints.
5. Plug in the KVM switch (but leave the traditional keyboard connected). Confirm that the KVM is selected towards the target computer.
6. Run the same lspci command again. Hopefully you have 1 or more new PCI devices listed. Take a picture of the new output (you can probably get both outputs in one screen shot as long as I can read the numbers we will be good.

The difference between the two commands will be the hardware IDs of the kvm switch. We can then take those IDs and confirm that the linux kernel supports the KVM.

Yes, windows is in a dirty state. Google “windows dirty bit” for more details on this issue. The issue isn’t with FOG but Windows and how it closes the files on the disk.

To fix this you will need to restart the workstation and then:

1. Use the proper sysprep command switch to power off the computer (best)
2. Sysprep the system then use shutdown.exe -s -t 0 to power off the computer.

Since Windows 8 windows start button shutdown isn’t really a shutdown, but an enhanced sleep state (S5). Using one of the above methods will ensure the system is properly closed and ready for imaging.

@jhumpf All I can say is that based on the two to three years of using that configuration for different fog admins, we haven’t needed to deviate from that configuration since dnsmasq is not giving out dhcp addresses the mask “should be” irrelevant. The remote subnets should just work as long as dnsmasq is being informed of them requesting a dhcp address.

If you want to debug deeper into the matter, I would suggest that you reset the configuration back to what I have in the article then follow this procedure to see if we can capture a bad remote proxydhcp request and then capture a good remote proxydhcp request so we can compare. There also may be some value in taking a computer on the remote “bad” subnet and loading wireshark on it. Start the capture filter of port “67 or port 68”. That would capture what the dhcp and dnsmasq are sending to the target computer, from the target computer’s point of view. https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

Upload those 3 captures to a share file site and share them as public. Either DM me directly or post the links here and I will review them.

My gut feeling is that you have something not configured right in your infrastructure not related to FOG or dnsmasq.

@jhumpf I can say from past experiences that you only need the dhcp pool statement for the local subnet to where dnsmasq is running. The only other thing you need to do is to update your dhcp-relay/helper service on your (vlan) router to include the dnsmasq server as the last helper in the list. This way dnsmasq server is informed of the remote client requesting a dhcp address and allow it to respond to the dhcp request with the ProxyDHCP response.

edit: Wait a minute lets make sure we are talking about the same setting. If you used my dnsmasq configuration exactly this should be your last line.

dhcp-range=<fog_server_ip>,proxy

You only need to add in the fog server’s IP a comma and “proxy”. Don’t define any other values that what I have in my configuration. You don’t want dnsmasq turning into a full dhcp server here.

@jhumpf I’m almost of a mind that you have more than one dhcp server or the target computer isn’t seeing the dnsmasq response.

@alesser said in PXE-Booting a Modern Ubuntu live CD via UEFI mode:

I simply want to boot a linux live CD in EFI mode when the option is selected, run a script, and then stop.

Reading between the lines here, I wonder if another approach would be better/faster/greener?

Let me tell you a bit how fog images computers. The iPXE menu in conjunction with the boot.php program send 2 files to the target computer with some kernel parameters that tell the target computer what to do. Those two files are bzImage (the kernel) and init.xz (virtual hard drive a.k.a initrd). Those two files make up the os known as FOS (Fog Operating System). FOG uses buildroot to construct the initrd as well as compiles bzImage.

So one might wonder, do you need a full commercial OS run your script or just a linux OS? Do you need a persistent disk storage or once the script runs can the virtual machine just disappear?

So the grey beard hacker in me wonders if you could take FOS Linux delivered by the FOG server to do your bidding? Unless you are doing something crazy (like vlans, wireless, packet filtering) FOG’s bzImage should do what you need. It supports both uefi and bios modes, as well as init.xz. FOG supplies both a 64 bit and 32 bit version of FOS Linux.

So how do you hack init.xz to not image your target computer? Simply unpack the inits https://wiki.fogproject.org/wiki/index.php/Modifying_the_Init_Image Once you unpack the inits then look in the /etc/init.d directory for S99fog and delete it. Repack the inits and then pxe boot the computer again. Now you have a command line linux OS to play with that boots in under 15 seconds and disappears when powered off. The entire OS fits into 512MB of RAM, so its very VM friendly. If you need persistence memory just map an NFS share and grab your persistent data from there.

You might want to look into the /etc/init.d/S99fog file a bit deeper since you want to run a script the S99fog to do your task. https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/etc/init.d/S99fog

Now what do you do if FOS Linux doesn’t contain the linux programs you need. Welp, then you need to turn to build root and build your own init.xz (initrd) file. No worries since the FOG Project also provides the buildroot config file and fs overlay you need in the github site. https://github.com/FOGProject/fos/tree/master/Buildroot

Ubuntu 1910 Desktop

Under development 17-Jan-2020

1. First we’ll create the required directories:

mkdir -p /images/os/ubuntu/Desk19.10
mkdir -p /tftpboot/os/ubuntu/Desk19.10

1. Now we’ll mount the Ubuntu 19.10 installer over the loop directory. Then we’ll copy the contents of the DVD to the directory we built above.

mount -o loop -t iso9660 /{full path where you have the iso stored}/ubuntu-19.10-desktop-amd64.iso /mnt/loop

cp -R /mnt/loop/* /images/os/ubuntu/Desk19.10
umount /mnt/loop

1. Finally we’ll copy the pxe boot kernel and intfs to the tftpboot directory. We’ll need to download the netboot version from here: http://archive.ubuntu.com/ubuntu/dists/eoan/main/installer-amd64/current/images/netboot/netboot.tar.gz This version of bzlinuz.efi and initrd.lz support booting over an NFS share instead of the local DVD Drive.

wget http://archive.ubuntu.com/ubuntu/dists/eoan/main/installer-amd64/current/images/netboot/netboot.tar.gz
tar -zxf netboot.tar.gz
cp ./netboot/vmlinuz.efi /tftpboot/os/ubuntu/Desk19.10
cp ./netboot/initrd.lz /tftpboot/os/ubuntu/Desk19.10

1. The last bit of magic we need to do is setup a new FOG iPXE boot menu entry for this OS.
2. In the fog WebGUI go to FOG Configuration->iPXE New Menu Entry
   Set the following fields
   Menu Item: os.Ubuntu.Desktop.19.10
   Description: Ubuntu Desktop 19.10
   Parameters:
   kernel tftp://${fog-ip}/os/ubuntu/Desk19.10/vmlinuz.efi
   initrd tftp://${fog-ip}/os/ubuntu/Desk19.10/initrd.lz
   imgargs vmlinuz.efi root=/dev/nfs boot=casper netboot=nfs nfsroot=${fog-ip}:/images/os/ubuntu/Desk19.10/ locale=en_US.UTF-8 keyboard-configuration/layoutcode=us quiet splash ip=dhcp rw
   boot || goto MENU
   Menu Show with: All Hosts
3. That’s it, just pxe boot your target system and pick Ubuntu Desktop 19.10 from the FOG iPXE boot menu.

References:

@alesser Those instructions will/should work for both bios and uefi systems. The only one that won’t work is the one where you load the iso directly with memtest which is bios only.

https://forums.fogproject.org/topic/10944/using-fog-to-pxe-boot-into-your-favorite-installer-images/6

For the unbuntu distro it list 16.04 but the concepts can be applied to 18.04. The transfer of both the kernel and virtual hard drive (initrd) to the target computer is done by iPXE, but once that kernel is launched its in control of the hardware so uefi/bios compatibility is up to the kernel not fog.

You might need to describe how the remote offices are connected to HQ and what link speeds you have.

How many computers are at the remote offices.

Do you have any IT support at the remote locations?

@nextechinc I can’t answer that. If you can tweak 5.4 with the newer driver then go for it. 5.5rc may also work. I guess the question would be how many NUCs do you have and what would take the least amount of effort to do (probably 5.5). Understand that you can define a custom kernel for each host definition. SO what ever direction you go in keep the kernel name unique and only assign that kernel to the systems that need it.

@nextechinc

8086:02f0 is the intel wireless adapter.
8086:0d4f is the intel 1000e network adapter.

Looking at the kernel database 8086:0d4f is in the linux 5.5+ kernel and not before. ref: https://cateee.net/lkddb/web-lkddb/E1000E.html So that is why building linux 5.4 did not work for you (you might try adding that hardware ID to the existing 5.4 intel nic driver and it might work or might not). You ARE running on the bleeding edge with that hardware.

So what can you do? Well a USB3 or USB2 network adapter will work in a pinch. You will need to have both network adapters plugged in. The LOM nic for pxe booting and USB nic for FOS Linux imaging. Its kind of janky but until the linux kernel can catch up to the hardware, sometimes you have to just be janky.

@nextechinc Ok let me look up the codes.