Posts made by george1421

@stokehall ok, let me start out with neither the developers or myself have an ARM system (other than an RPi, but that is a different beast) so we will rely on the forum members to help us debug this.

With that said, let me tell you how it should work. When you pxe boot the computer it sends out to the dhcp server “hey I’m this kind of computer, please configure me” message. We need to see this message but more on that later. When the fog server or dhcp server sees this message it will send two optional dhcp fields. One is the boot server’s ip address and the second is the name of the file to bootstrap the computer. The boot loader/strap file that FOG uses is iPXE. The default ipxe files need to be hardware specific. For bios x86 systems the default should be undionly.kpxe, For uefi x86 systems ipxe.efi (or snp.efi) The first issue here is that the fog server or dhcp server always assumes your computer is x86 based. What you need to do to get past this part is to tell your dhcp server to send the ARM boot loader (full disclosure I don’t know what its called, but its on the fog server). Look on the fog server in the /tftpboot directory. Search for all files that end in efi ls -la /tftpboot/*.efi one of those efi files should have arm in the title. Get that file name and key it into the dhcp server/s option 67 value (understand this is a global setting and will send the arm file name to all pxe booting computers, but right now its just a test). With that arm boot loader in place you should get to the FOG iPXE menu. If you do then wonderful. Try to get into the hardware compatibility checker. Hopefully the ARM version of bzImage will be sent to the target computer. If you get into the hardware compatibility checker. That tests the FOS engine, if it runs it should image.

@Sab First I would have to say both fog server and fog storage node have to be on the same version. The fog storage node doesn’t have its own database, it uses the fog server’s database. There could be schema changes in the database between 1.5.10 and 1.5.9.

Unless you change the storage group around, its not possible to capture to a storage node (in the default configuration). Only the master node in a storage group captures the image, then it replicates the captured image to all storage nodes. Both the master node and storage node can deploy images.

If I had to guess, I would think that in the storage group the nas fog server is not defined as the master node. If I think I understand what you did to configure this setup.

On the fog error screen it should print out the kernel parameters sent to the FOS imaging engine. There should be a filed marked storage or storage_ip, that will point to the fog server it will try to get the image from. Make sure that is correct for your configuration.

@LLamaPie Well keep an eye on it, my concern is that there is a bug in php that allowed this to happen some how. Its in the http path and not the nfs path. Hopefully you will scan and or keep an eye on the cpu usage. When you detect a change surely review the apache logs both error and access as well as login logs to abnormal activity.

@Tom-Elliott The key is that its post the security update for FOG.

The question I have is:

1. How did that file/malware get onto the server
2. Why did it pick that specific path to hide in.
3. When was the server compromised. The date on the files in that directory may give us a clue.
4. Could it happen again? We don’t know because we don’t know how it was installed.

It almost seems intentional and deliberate to pick that specific path. I don’t think apache normally has write access to that path.

@OP is your fog server exposed directly to the internet?

@LLamaPie

/var/www/html/fog/management/.sys

Is not something that FOG creates. ref: https://github.com/FOGProject/fogproject/tree/stable/packages/web/management

.sys file / directory name means the directory is hidden unless you use the command ls -la /var/www/html/fog/management also .systmd is a hidden file made to represent systemd application.

I did find this article: https://sarperavci.com/ironshade-writeup-tryhackme/

So the question is how was this server compromised and if we don’t know it will probably happen again. What version of FOG did you have installed?

@MarkG OK let us know because there is some info that you can collect on these devices to help the developers validate the code.

@JamiesonCA092 ok it shows you have some debugging skills, so I’ll add a few new tools into your toolbox.

1. on the fos linux (target computer) when you schedule a task and tick the debug checkbox, when you pxe boot the target computer you will be dropped to the fos linux command prompt.
2. You can start the imaging process in single step mode by simply keying in fog at the command prompt.
3. The script will run until it hits a debugPause; command in the script. You can add that command in your custom script with echo statements to understand how your script is running. Pressing the enter key will cause the script to run again until the next debugPause command.
4. At the fos linux command prompt if you get the ip address of the fos linux (target computer) with ip a s and then give root a password with the passwd command (make it simple like hello, it will be reset at the next reboot). Once you have those 2 bits of into you can use putty or ssh from a second computer to the fos linux computer for debugging. Using this method you don’t have to sit in front of the target computer plus you can copy and paste using the putty/ssh session.
5. If you need to stop the fog script during hit ctrl-c to skip out of the script. At this point you can verify the environment, check the available variables, etc. When you are done looking around, then you can enter fog again to begin the imaging process once again. No reboot needed.
6. In the fog script the kernel parameters sent to fos linux are converted to bash script variables in the functs.sh script between lines 9 and 13 https://github.com/FOGProject/fos/blob/0efdd68f1783a06f3214607fc313db50747fbc43/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh#L9

I’m pretty sure Wayne is right that the varialbles in the postinit script are local and will be erased when the script terminates. But I would test to be sure.

One last hack I can think of is that you can hack the fog.mount script https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.mount Put the logic to decide the target’s IP address into that script. The concept of dynamically patching the fog.mount script can be found in this tutorial (which addressing the fog.man.reg script) https://forums.fogproject.org/topic/14278/creating-custom-hostname-default-for-fog-man-reg but the concept and actions are the same.

@DBCountMan In the post init or post install scripts the $mac variable contains the mac address of the target computer

To get the ip address this script will work

myip=`ip route get 8.8.8.8 | awk 'NR==1 {print $NF}' | cut -d "." -f1-2`;

Getting the user id of who did it will be a bit harder since they never log into the web ui. The user ID and password they key in during the deploy image stays in ipxe and is not sent onto FOS Linux. I’d like to say that the username field can be trapped from iPXE and then pass that variable as a kernel parameter onto fos linux. Once that is done then the $username variable should exist in your fog scripts. Getting that info into the fog database is another challenge. In the post install script the /ntfs directory is mapped over to the fog servers /images/dev where you could append an echo statement into a text file.

Understand I’m taking in programming pseudo code. But I see an path here and it should work.

@MarkG In general there is basic support for ARM processors in FOG. But the realities are that there hasn’t been a viable ARM based workstation to validate the FOG configuration against. Yes there is the Raspberry Pi, but that is more of an embedded device and not a general computing computer. Since the developers don’t have one of these computers to verify the FOG code against it will take community involvement to work out the details.

When I was first testing the R-Pis I did find a few assumptions in the code that if FOG couldn’t determine the processor, it assumed a 32 bit X86 processor. Those issues have been corrected, but I’m sure there are others hidden away in 15 year old code.

So the short answer is, it might work or might not.

@DBCountMan The issue/explanation is that the unregistered hosts deployed via the ipxe menu doesn’t go through the normal task creation process so there is no database interaction. When you call the deploy image task the fog server creates the ipxe menu needed for imaging but that’s it. The fog server never “really knows” and unregistered guest was imaged.

Now what can you do on the back end? Probably a post install script could open a file in the /images/dev folder and write the mac address into a text file with the current date/time. This is possible, but I don’t have an example I can point to. If FOS Linux had a ail client it could send an email when it was done imaging too.

@JamiesonCA092 I can’t really give you a solid solution because fog isn’t designed to work the way you want it with unregistered target computers.

But I know how FOG works. There are 2 “hook” scripts. One is before imaging starts and one after the image push is complete. The bash script that is called before imaging starts is the postinit script. When this script runs all of the kernel parameters have been expanded to variables, including storageip value. This value as well as most others could be reset by this postinit script.

I have a tutorial on a postinstall script that uses the IP address of the target computer to decide what OU the target computer will connect to. That mechanics of the bash script could be the basis to update the storage IP variable. I don’t know if it works, but if you are that deep into modifying the ipxe script, creating your own bash script shouldn’t be to complicated.

Edit: Post on point. https://forums.fogproject.org/post/69725

@mentaluproar said in Could not select: Exec format error:

/var/www/html/dban/test.html

try it in /var/www/dban/test.html

different distros have a different apache2 document root.

@t-schuurmans Looking at the error message I can tell you that /dev/vda is the wrong answer in regards to a software raid array.

There was just another thread in the forum where someone else was doing the exact thing as you.

Edit: https://forums.fogproject.org/topic/17626/issues-with-capturing-an-image-with-a-raid0-array

I would suggest the same thing here. Set the global flag for mdraid=truethen schedule a debug capture. At the fos linux command prompt key in lsblk and then see if fos linux assembled the array for you. If yes, find out the correct device name it is for the array. And finally back in the fog web ui, in the host definition set the hard drive to use the same name as what was discovered in the debug console.

@mentaluproar The mount -loop or simply copying the files there does the same thing. So the url you should be using is http://<fog_server_ip>/dban/<filename>

There could be a number of things going on here. the path of the files may be /var/www/dban or /var/www/html/dban depending on the apache servers doc root settings.

As a test I would create a file in the dban directory, lets call it test.html in that file put the following.

<html><head></head><body>It works!</body></html>

Now try to download the file at http://<fog_server_ip>/dban/test.html Does it load the file?

If no then the fog configuration for the web server is getting in the way.

There are other options, but lets first see if

@sega I don’t know enough to answer your question intelligently.

I can tell you that for multicast imaging FOG uses the linux commands udpsend and udpreceive.

I don’t see any command line switches to pick which igmp mode is being used.
https://users.soe.ucsc.edu/~kent/test/udpcast.html

I can say its rare that there is ever a question on the revision of igmp being used.

@45lightrain said in Issues with capturing an image with a raid0 array.:

Also is fog able to capture both the OS SSD data and raid array data on the NVMe drives?

FOG captures disks in block mode. It doesn’t care about partitions (mostly). Also make sure that md0 is the true device for these drives. When you are in the FOS linux shell you can / should create a mount point and then mount /dev/mdX partition to see if you can read the content. I have seen where sometimes md0 is created but the real raid array is /dev/md126.

Also FOG calls a script before imaging starts. Sometimes its needed to assemble the array in this postinit script. Or do other things to prep the system for imaging. So the postinit script (found in the /images directory on the fog server) is a the place to put that code.

Also when you start in debug mode (when debugging the imaging process) you can actually start the imaging process from the command line by keying in fog. The imaging will stop at each debugPause; command in the imaging code as a breakpoint. If you notice something wrong, you can hit ctrl-C to exit back to the command prompt. To restart the imaging process again just rekey in fog.

While it doesn’t exactly apply here I wrote an article 8 years ago about imaging using the intel rst adapter that may contain a nugget of help: https://forums.fogproject.org/topic/7882/capture-deploy-to-target-computers-using-intel-rapid-storage-onboard-raid

@45lightrain how did you create the raid 0 array? From within linux or using the intel rst controller?

FWIW: I have not been able to make the vroc arrays work with FOG, but if you have ubuntu loaded on a vroc array then it has to work and we need to talk.

If you created the array with intel rst or using a software raid created by ubuntu then you have some luck. The first thing I would do is go into the fog global settings and look for kernel parameters field. Then add mdraid-true into that field. That tells the fos linux kernel to look for a raid array.

Next I would schedule a capture or deploy image, but before you hit the schedule task button, tick the debug checkbox.

PXE boot the target computer, after a few screens of text you need to clear with the enter key you will be dropped to the fos linux command prompt. Key in lsblk to see if FOS Linux assembled your array for you. It will probably be listed as /dev/md0, or /dev/md126 or 127. If its there THAT is the device name you need to key into the host record for this target computer in the hard drive field. This way FOG will not try to use one of the nvme drive, but the raid device.

If the device is not automatically detected you may need to assemble the array, but lets first start with making sure you have the kernel parameter set and that fos linux sees the array.

@jfernandz Would you post a screen shot of the configuration for the ldap plugin? Be sure to blank out only the bits that could make the picture content identifiable.

@mentaluproar so this dban.bzi is located in the /tftpboot directory on the fog server?

The exec format error is iPXE saying the kernel file is not bootable.

Locate that file on the fog server (should be in the /tftpboot directory) and then issue file dban.bzi does it say its a linux kernel file?

@mentaluproar I have seen this when you try to boot an bios based image on a uefi based computer. How are you starting dban? Are you loading the kernel and init.rd file or are you trying to boot from an iso image?