Best posts made by george1421

@Nabeel Did you configure your dhcp server to supply dhcp options 66 and 67? For dhcp option 66 that needs to be the IP address of your fog server. For dhcp option 67 that should be undionly.kpxe for bios based computers and ipxe.efi for UEFI based systems.

If you need to support both systems in your environment then once you prove that the above options work you can review this article: https://wiki.fogproject.org/wiki/index.php/BIOS_and_UEFI_Co-Existence#Using_Windows_Server_2012_.28R1_and_later.29_DHCP_Policy

@androsszit So just to be clear, deploy image from the iPXE menu doesn’t expand the disk correctly, but registering the host then and then from the web ui deploying the image to it works?

I know this is asking a bit more from you, but if you do a iPXE menu full registration at the end of the full registration it asks you if you want to deploy the image, if you answer yes to this does it expand the disk correctly? This is the first I’ve heard about a different reaction from the iPXE Deploy image and from the web ui deploy image.

@Sebastian-Roth I also did testing of iPXE Deploy image vs the web ui Deploy image and the instructions (kernel parameters passed) to FOS are identical. So at least what FOS knows about there should be no difference in the way it deploys the image.

@siara Lets have you rerun the fog installer script. It almost sounds like the fog program code didn’t get installed correctly.

@KennethJ said in Image sharing between sites:

Our network is setup in a way that makes it impossible for the remote-node to contact the master-node

Top down is fine as long as the master node can make http calls as well as ftp to the remote site. Remember that ftp is a two way street the master will open a command channel to the remote FTP server, and the remote FTP server will open a data channel back to the master FOG server (ftp client) to send the file over. This should be allowed through your site to site links since ftp is a well known communications protocol that has these 2 channels. If this is a limitation for your network design, you will loose a little flexibility but you could use rsync (outside of FOG) to send files from the master node to the remote storage nodes you will loose the ability to control when an image is replicated but you can achieve this top down communications.

There are FOG replicator logs in /opt/fog/logs that might give you an understanding why replication isn’t working too. I would begin with the logs then see where that leads you.

@Sebastian-Roth said in Deployment and Quick Delete Host not working from iPXE menu:

ave you seen UEFI clients behaving differently on the iPXE menu/login/params stuff yet

No I haven’t. This is a unique case. So both Deploy Image (quick image) and Quick Delete function by sending a single parameter (delhost or qihost) to boot.php. Its the same for the registration commands too. But registration works. I haven’t looked at the boot.php flow but what might make FOS fall through both qihost and delhost to end up executing the compat test code.

Possibly we could set a global kernel parameter of isdebug=yes then get this menu to appear by picking delhost or qihost. At them menu break out and confirm that the kernel parameters sent from FOG are correct. If they are correct then we could focus on the FOS code (/bin/fog) to why it might fail through to compat test mode. When testing is done then isdebug=yes should be removed so FOG runs correctly.

You have to remember that the FOG Client runs as SYSTEM, and SYSTEM has full authority on the LOCAL COMPUTER. SYSTEM does not have access or authority to any external resources. If you want the batch file to work that way you will need to map a drive (using a proper domain account) within the batch file to the remote share and then run the installer from that mapped drive. The alternative method is to deploy the Authy package as an executable if everything is in a compressed executable or use a snapin package (zip file) to include all of your files to be copied to the target computer.

Yes you can delete the image, FOG will not care about the missing image until you go to redeploy to those hosts again. I have not tried this, but I suspect you will get an invalid image ID if you try to redeploy those hosts without first assigning a new image to them. But the only time you will have a problem in the future is when you are deploying to these systems and the image is unlinked.

@jflippen how are your storage nodes setup? Are you using the location plugin to direct clients to specific storage nodes? -OR- are they setup just for load sharing (i.e. first one available gets the imaging job)?

I see from your picture that your fog server is at 10.59.10.12 and the storage server you are using is at 10.59.181.12. Are they on different subnets? What subnet is the pxe booting client on when attempting to connect to 10.59.181.12? Do you have any screening/firewall routers in between that might be blocking the NFS mount?

@KennethJ said in Image sharing between sites:

Have tried a bit now and it seems the FTP port is closed by default on all our networks

Is ssh port open? While FOG doesn’t currently support it, you may need to use rsync in place of FOG Replication to sync your image files. You will loose some capability with this option, but you might bypass the ftp port blockage.

@sehume Well this is an interesting issue…

The next steps are to see what the pxe booting client is seeing, because something is going on there. I have a suspicion, but lets test it.

1. Manually register this host from the picture.
2. Schedule a debug capture or deploy (don’t care). Before you schedule the task, tick the debug checkbox, then schedule the task.
3. PXE boot the target computer. After a few screens of text where you have to clear with the enter key you should be dropped to a linux command prompt. You may get the same error message about connecting to the fog server, just press enter like you did before.
4. At the FOS Linux command prompt, on the target computer, key in ip addr show confirm that eth0 has an IP address. Please post a clear picture of the output of that command here.
5. Ping the fog server from the target computer. Post the results of the test.
6. Next we are going to do the http test from the target computer. Key in the following command curl -Ikfs http://10.245.220.30/fog/management/index.php --connect-timeout 5
   The success response will start out with

HTTP/1.1 200 OK

Lets see what info those tests provide.

@frankv24 said in tftp server issue:

when adding capture filter 67 and 68 no data was found for .203

Just to be clear, if you setup a filter for .203 in addition to udp ports 67 and 68, then that is the problem. The dhcp process uses broadcast messages to communicate. Since they are broadcast messages all client computers on the 10.15 subnet will hear this communication. That is why as long as the witness computer is on the 10.15 subnet and listening it should hear the dhcp communications. At the point of the dhcp communications, the target computer doesn’t know its IP address.

@faresalandlos Well at this point I don’t think its a firewall issue, but to answer your question for ubuntu this is the command to disable the firewall sudo ufw disable

So what do I think:

1. You need to understand why nfs is not starting on that system.
2. your exports are right and once nfs is running the when you run the showmount command you should see what is posted in the /etc/exports file. The exports file is configured correctly.
3. Your /images/dev directory is not complete. You are missing the post init scripts directory.
   Your directory should look like this:

[root@sonic ~]# ls -la /images/dev
total 428
drwxrwxrwx  5 fog  fog    4096 Jun 24 13:11 .
drwxrwxrwx 18 fog  root   4096 May 28 11:11 ..
-rwxrwxrwx  1 fog  fog       0 Mar 16  2016 .mntcheck
drwxrwxrwx  2 fog  root   4096 Oct  4  2018 postinitscripts

You may need to just rerun the fog installer and it should fix this file path.

4. For your nfs service there are a number of sub services like portmapper and such that nfs depends on. Does running sudo systemctl status nfs.service give you any clue to why nfs is not starting?

@jmason If you can still duplicate this random switching of device and linux dev node names we could use your help trying to debug this. We have the idea to query the nvme controller directly to see what its seeing as to order vs what linux is detecting as the order. We need a system that has 2 nvme drives in it that are exhibiting the switching of nvme drives issue. Do you have time to test this?

@Sebastian-Roth

@Sebastian-Roth I agree we need a system with a dual nvme drive so we can compare. I have a pdf of what all the short names really mean that I’ll link in too. I’m suspecting the field you pointed to will tell us how many and then there are other fields that will tell use more details about the drive. Also in the nvme list command I’m interested in seeing if the name space field changes if there are more drives as well as the interaction when these disks change ordinal position in respect to what udev assigns them.

I’m sure your fog server is setup correctly. I want to focus on your target computer. More specifically the network connection. I feel this is a spanning tree issue. As a quick test, do you have a (dumb) unmanaged switch? The dumber the better, like a cheap $30 amazon switch? If you do insert that between your building switch and your pxe booting computer. See if that resolves your issue.

If it does then you will need to get your networking group to look into that building switch. They need to change the STP (spanning tree protoocol) from standard STP to one of the fast STP protocols like RSTP, MSTP, fast-STP, port-fast or whatever your switch vendor calls it.

@tlehrian Ok here is what we are seeing. When FOS Linux boots, the nvme drives initialize (become ready to the os) at different times. Sometimes drive A is ready first and other times drive B is ready first. Well when linux boots what ever drive inits first becomes /dev/nvme01 and the second one becomes /dev/nvme02. This is not an issue with FOG or linux, its an issue between the linux OS and the hardware.

So what we need is to run a utility with a few commands to help us detect which drive is which in each state. The switch is totally at random so we can’t predict the order using linux. So what I need you (as a tester) to do is to pxe boot the target computer multiple times to record the settings when the drives are in a normal and then reversed order. If we are lucky you will see this swap within 10 pxe boots.

Here is what I want you to do:

1. Download this updated init from here: https://fogproject.org/inits/init_nvme-cli.xz
2. Rename the original inits in `/var/www/html/fog/service/ipxe init.xz to init.xz.sav
3. Move the downloaded file to that directory and save as init.xz
4. Pick one of these dual drive nvme computers and schedule a deploy task to it. But before you hit the schedule task button tick the debug checkbox then schedule the task.
5. PXE boot the target computer. After a few screens of text where you need to press enter to clear you will be dropped to the FOS Linux command prompt.
6. At the FOS Linux command prompt run this command lsblk to note the size and order of the nvme disk. Use disk size to be your guide in determining the order. So this is state 1.
   6.1 You can use these steps if you want to setup remote debugging. Its easier to do the copy and paste of commands from putty. You don’t need to, its just one option.
   6.2 At the FOS Linux command prompt key in ip addr show and collect the IP address of the FOS Linux computer.
   6.3 Give root a password with passwd. Just give it a simple password like hello. The password will be reset on the next reboot. So don’t worry.
   6.4 From a windows computer use putty to ssh into the FOS Linux computer. Login as root and the password you created in step 6.3
7. At the FOS Linux command prompt key in the following and post the results here nvme list. If the nvme command isn’t known then the downloaded inits are not in the right spot.
8. Key in the following command and post the result(s) here nvme id-ctrl /dev/nvme0n1 -H and (I’m guessing at the name since I don’t have a dual nvme system, use the name from the lsblk command above) nvme id-ctrl /dev/nvme0n2 -H
9. Now reboot the FOS Linux computer with ctrl-alt-del or key in reboot at the FOS Linux command prompt. The system should PXE boot right back into FOS Linux in debug mode.
10. Use the lsblk command to determine the disk order. We are looks for the order of the drives when they switch places. If you can’t get them to switch then power off the system instead of rebooting to see if we can get them to switch. The key is to capture the output of the nvme command in both states.

@Luc-Novales Ok lets take a step back because I haven’t seen a picture yet from iPXE. Chain booting from syslinux to iPXE won’t work because you say that iPXE is hanging. Using syslinux to call boot.php won’t work since boot.php will make a menu for iPXE and not Syslinux. I suspect that pxelinux.0 (syslinux) will not like an iPXE style menu.

Lets first set dhcp options 67 to ipxe.kkpxe (this exact boot file) and pxe boot the target system. Using a mobile phone, take a screen shot of the error produced by ipxe (I know you have done this many times, but I want to see the error as well as the context of the error).

FOS Linux: This is the capture engine that runs on the target computer to capture and deploy images on the target hardware. Normally it is sent to the target hardware using iPXE via the menu produced by the boot.php page. FOS Linux is constructed out of 2 parts bzImage32 is the Linux 32 bit kernel and init_32.xz is the 32 bit virtual hard drive. Its iPXE’s job to transfer them to the target computer.

Today we have a usb flash drive that can be used to transfer bzImage32 and init_32.xz to the target computer using a usb flash drive and a grub boot loader.

Now with that said, we should be able to duplicate that grub boot menu using syslinux (pxelinux.0) Its not an ideal solution but if you have no choice, sometimes you have to do what you have to do to make things work. So if we can’t get things working with iPXE, then we can attempt to use syslinux to at least get FOS Linux to the target computer.

I know we are focusing on the CPU in this thread, but something I have to ask How much RAM does this device have?

Only for reference so I don’t forget and lose it.

default menu.c32
prompt 0
timeout 300
ONTIMEOUT local

MENU TITLE FOG PXE Menu

LABEL 1. FOG Image Deploy/Capture
        MENU LABEL 1. FOG Image Deploy/Capture
        KERNEL bzImage32
	APPEND loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4

LABEL 2. Perform Full Host Registration and Inventory
        MENU LABEL 2. Perform Full Host Registration and Inventory
        KERNEL bzImage32
        APPEND loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=manreg

LABEL 3. Quick Registration and Inventory
        MENU LABEL 3. Quick Registration and Inventory
        KERNEL bzImage32
        APPEND loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=autoreg

LABEL 4. Client System Information (Compatibility)
        MENU LABEL 4. Client System Information (Compatibility)
        KERNEL bzImage32
        APPEND loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=sysinfo

LABEL 5. FOG Debug Kernel
        MENU LABEL 5. FOG Debug Kernel
        KERNEL bzImage32
        APPEND loglevel=7 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= boottype=usb consoleblank=0 rootfstype=ext4 isdebug=yes

Just as a follow up on this. The bzImage32 is configured to require a 686 as the minimum processor: https://github.com/FOGProject/fos/blob/master/configs/kernelx86.config#L251

I’ve reconfigured my build environment and changed the minimum processor to 486 and I’m rebuilding the bzImage32 file for i486. Understand two things. 1. This is not an official FOG Project kernel since its not coming from one of the Developers. 2. I have no idea if it will work because I don’t have a 486 machine to test it on.

When its done compiling I’ll post a link so you can download it and try it.

Also on my syslinux configuration, Its been almost 8 years since I’ve worked with syslinux so I guessed at most of the entries. You WILL need to adjust it to replace the entire variable $myfogip with the IP address of your fog server to make it work correctly. It will take about 20 minutes to finish the new kernel build.

@tlehrian Ok so this IS something that the Linux kernel developers are going to have to address. Its not something that only impacts FOG, but all distros of Linux.