@george1421 said in FOG speed problem:
The FOG server only moves disk blocks from its local storage and sends them out the network adapter. The fog server does monitor the imaging process but does not perform heavy computational load
What about in the case of using storage nodes? Does the Fog server tell the target where the data blocks are then the target only talks to the storage node?
@Sebastian-Roth Sorry I didnt see your first question. This is what I downloaded:
https://github.com/FOGProject/fos/releases/download/20230331/init.xz
I put my backup file back into the original place, /var/www/fog/service/ipxe, file perms identical, init.xz doesn’t load after download. So far on physical PCs and VirtualBox vms I saw the kernel panic error, but on my Hyper-v VM after init.xz downloads to 100% I see that EFI Stub message then black screen. Nothing happens after that.
EDIT: Here’s a screenshot from a physical PC
I’m receiving this error as soon as init.xz loads after selecting any item on the FOG boot menu booting a device via UEFI. Legacy boot loads just fine. I downloaded init.xz from github and set permissions accordingly but the issue persists.
@george1421 That would be the next step. When I said “replicate” I meant that I have a cron sync from primary to secondary every night, not instantly. So lets say I set up both servers to accept ipxe req’s and a field tech loads a new image on the primary at 10am and wants to deploy them to 20 PCs in the field. Primary FOG server takes a dump. The secondary FOG server won’t have the image until midnight. So there’s one issue. Unless I set cron to sync every hour or 30min. Another issue I found with rsync as much as I love it, is that if the primary server goes offline, the secondary, which has the /images nfs share from FOG1 mounted, will appear empty, and rsync will sync an empty dir to its own /images, thus wiping out that folder. I read that rsync can be tuned to not sync if a directory is empty but I have to research and test.
@Sebastian-Roth @george1421
Guess what? The key ID of 5e…c9 belongs to my secondary FOG server. Somehow it was getting mixed into the pxe request chain. My DHCP server is configured to send pxe requests to my primary FOG server only. I have no reference to my secondary server anywhere in the tftp configuration. Very strange. I use dnsmasq because for some reason tftpd didn’t play nice with our DHCP server. I had dnsmasq running on the secondary server and thus is somehow received pxe requests.
Disabled the service on the secondary FOG server and now pxe works flawlessly.
The only reason I left dnsmasq running on the secondary is for redundancy. I replicate my primary to my secondary. Since any time I use the USB boot method outside the subnet of any FOG server, the ipxe boot process will ask for an IP. What I thought I could have is the option to boot from the secondary in case the primary was out of service. I also incorrectly thought that dnsmasq is required but since the USB boot method takes care of ipxe, its just an HTTPS connection from there.
So again I thank you guys for helping me figure this out.
Found something interesting. When I first boot the vm and get the initial error, then drop to prompt, I run certstat. The first certstat shows this:
The addressing in .59 is the fog server. Now look at certstat after a successful chain of boot.php:
So I don’t know where that “5e…c9” CA key is from or why it appears initially. Could explain the initial permission denied because the key doesn’t match? BTW the “81…0c” is the SHA key of the cert that is actually being used, I matched it with the cert issued to my browser.
@george1421 I probably shouldn’t have re-run the installer but I wanted to see if that would fix it.
ls /tftpboot -lah
total 6.5M
drwxr-xr-x 5 fogproject root 4.0K Aug 23 09:13 .
drwxr-xr-x 29 root root 4.0K Aug 23 08:24 ..
drwxr-xr-x 4 fogproject root 4.0K Mar 16 2021 10secdelay
drwxr-xr-x 2 fogproject root 4.0K Mar 16 2021 arm64-efi
-rw-r-xr-x 1 fogproject root 39 Aug 23 09:13 autoexec.ipxe
-rw-r-xr-x 1 fogproject root 868 Apr 27 2022 boot.txt
-rw-r-xr-x 1 fogproject root 459 Aug 23 08:36 default.ipxe
-rw-r-xr-x 1 fogproject root 454 Mar 16 2021 default.ipxe.bak
-rw-r-xr-x 1 fogproject root 458 Jun 10 2021 default_usb.ipxe
drwxr-xr-x 2 fogproject root 4.0K Mar 16 2021 i386-efi
-rw-r-xr-x 1 fogproject root 270K Aug 23 08:36 intel.efi
-rw-r-xr-x 1 fogproject root 103K Aug 23 08:36 intel.kkpxe
-rw-r-xr-x 1 fogproject root 103K Aug 23 08:36 intel.kpxe
-rw-r-xr-x 1 fogproject root 103K Aug 23 08:36 intel.pxe
-rw-r-xr-x 1 fogproject root 1.1M Aug 23 08:36 ipxe.efi
-rw-r-xr-x 1 fogproject root 888K Aug 23 08:36 ipxe.iso
-rw-r-xr-x 1 fogproject root 363K Aug 23 08:36 ipxe.kkpxe
-rw-r-xr-x 1 fogproject root 363K Aug 23 08:36 ipxe.kpxe
-rw-r-xr-x 1 fogproject root 362K Aug 23 08:36 ipxe.krn
-rw-r-xr-x 1 fogproject root 362K Aug 23 08:36 ipxe.lkrn
-rw-r-xr-x 1 fogproject root 363K Aug 23 08:36 ipxe.pxe
-rw-r-xr-x 1 fogproject root 400K Aug 23 08:36 ipxe.usb
-rw-r-xr-x 1 fogproject root 26K Aug 23 08:36 memdisk
-rw-r-xr-x 1 fogproject root 301K Aug 23 08:36 ncm--ecm--axge.efi
-rw-r-xr-x 1 fogproject root 268K Aug 23 08:36 realtek.efi
-rw-r-xr-x 1 fogproject root 104K Aug 23 08:36 realtek.kkpxe
-rw-r-xr-x 1 fogproject root 104K Aug 23 08:36 realtek.kpxe
-rw-r-xr-x 1 fogproject root 104K Aug 23 08:36 realtek.pxe
-rw-r-xr-x 1 fogproject root 268K Aug 23 08:36 snp.efi
-rw-r-xr-x 1 fogproject root 268K Aug 23 08:36 snponly.efi
-rw-r-xr-x 1 fogproject root 102K Aug 23 08:36 undionly.kkpxe
-rw-r-xr-x 1 fogproject root 102K Aug 23 08:36 undionly.kpxe
-rw-r-xr-x 1 fogproject root 102K Aug 23 08:36 undionly.pxe
-rw-r-xr-x 1 fogproject root 51K Mar 17 2021 wimboot
@george1421 Since I re-ran the fog installer using the -S switch, it rebuilt ipxe and the corresponding certs were regenerated. The file dates themselves are marked with today’s date when they were recreated.
openssl x509 -in /var/www/fog//management/other/ssl/srvpublic.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
*redacted*
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = FOG Server CA
Validity
Not Before: Aug 23 12:18:57 2023 GMT
Not After : Aug 20 12:18:57 2033 GMT
Subject: CN = <fogserverip>
openssl x509 -in /var/www/fog//management/other/ca.cert.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
*redacted*
Signature Algorithm: sha512WithRSAEncryption
Issuer: CN = FOG Server CA
Validity
Not Before: Apr 27 20:15:47 2022 GMT
Not After : Apr 24 20:15:47 2032 GMT
Subject: CN = FOG Server CA
@george1421 As far as the part of the environment I have control over, no. Not sure if there was a network change. However that being said nothing changed with apache. I ran the installer again using the -S option and it completed but the issue persists. Just seems odd that it won’t chain during the first attempt but works after running the autoboot command. Can I PM you a webm video recording of the vm I’m testing with? It is about 400KB.
EDIT: I added the fog server ip to the apache2.conf file at the bottom of the file as “ServerName <fogip>” and got rid of that AH00558 error message, but it didn’t fix the issue. Also autoboot doesn’t always work the first try, eventually it does though.
apache2ctl -S
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using <fogserverhostname>. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:80 <fogserverip> (/etc/apache2/sites-enabled/001-fog.conf:1)
*:443 <fogserverip> (/etc/apache2/sites-enabled/001-fog.conf:16)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex mpm-accept: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
/etc/apache2/sites-enabled/001-fog.conf
<VirtualHost *:80>
<FilesMatch "\.php$">
SetHandler "proxy:fcgi://127.0.0.1:9000/"
</FilesMatch>
KeepAlive Off
ServerName <fogserverip>
ServerAlias <fogserverhostname>
DocumentRoot /var/www/
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]
RewriteRule /management/other/ca.cert.der$ - [L]
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}/$1 [R,L]
</VirtualHost>
<VirtualHost *:443>
KeepAlive Off
<FilesMatch "\.php$">
SetHandler "proxy:fcgi://127.0.0.1:9000/"
</FilesMatch>
ServerName <fogserverip>
ServerAlias <fogserverhostname>
DocumentRoot /var/www/
SSLEngine On
SSLProtocol all -SSLv3 -SSLv2
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GC>
SSLHonorCipherOrder On
SSLCertificateFile /var/www/fog//management/other/ssl/srvpublic.crt
SSLCertificateKeyFile /opt/fog/snapins/ssl//.srvprivate.key
SSLCACertificateFile /var/www/fog//management/other/ca.cert.pem
<Directory /var/www/fog/>
DirectoryIndex index.php index.html index.htm
</Directory>
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-d
RewriteRule ^/fog/(.*)$ /fog/api/index.php [QSA,L]
</VirtualHost>
@george1421 I did run the fog installer using https and it had been working up until yesterday. If ipxe doesn’t have the correct cert then why would the ipxe still loaded properly after running the autoboot command at the ipxe shell? Should I re-run the installer anyway?
I used wireshark and this is the result right after the ipxe.efi file is downloaded. .138 is the client device, .59 is the fog server.
When pxe booting either a physical or vm device, all goes as it should, until it tries
https://<fog server IP>/fog/service/ipxe/boot.php
At this step I see permission denied, so I hit the S key to drop to a prompt, then I enter autoboot, and the process restarts and succeeds. Same result happens on a Dell optiplex, two hyper-v vms, and a virtualbox vm. I can access the boot.php from my browser. Can’t tell where the fault is. Any ideas?
@entr0py said in Odd performance issue:
Second, playing with the settings in the NIC on the host device made little difference, but playing with the ones on the actual Hyper-V switch did. Mainly some of the offloading and VLAN filtering settings
Curious, which Hyper-V settings did you play and found success with?
@Tom-Elliott Based on my tests, partclone hasn’t worked faster on a 10Gbe link vs a 1Gbe link. This is on a 64-core hyper-v host with 768MB RAM and 12TB SSD storage.
On that note, does FOG use pigz or just regular gzip compression?
@Tom-Elliott said in Odd performance issue:
get, decompress, process, and write back to the drive.
Right this is what I was getting at about partclone. I don’t think it is capable of performing those tasks at a rate that would saturate a 10Gbe channel. Assuming system components aren’t the bottleneck, software and how it is coded becomes the determining bottleneck factor.
I tested 10Gbe FOG imaging on my Hyper-V server (from FOG Server VM to Client PC VM, both connected at 10Gbe via internal switch) and imaging speed was exactly the same as 1Gbe. The Hyper-V host has all SSD storage too. After some research, it seems that partclone (clonezilla) does not scale well with faster network connections. What I haven’t tested yet was imaging several 1Gbe workstations on a 10Gbe backbone to a 10Gbe FOG server. Just don’t have the infrastructure for it.
@anwoke8204 Well your first screenshot indicates that you already successfully booted UEFI PXE, but it stops at chaining boot.php via HTTPS.
First thing to try is browse to https://10.4.47.15/fog/service/ipxe/boot.php from a desktop. If it loads and you can see the script, then you know the web service is working correctly. Since you are using HTTPS which uses a self-signed cert that has a validity period, I suggested you check the system date and time in your BIOS.
Another thing to check is the system time in the BIOS. I’ve seen instances where the BIOS data/time was reset, causing the date to be prior to the FOG certificate’s valid period, causing the permission denied error.
Is it possible to schedule a snap-in task then a capture task? I tried to add a snap-in task then a capture but the capture task overrode the snapin task. The snapin task is a powershell that clears the Event Viewer logs, and the time it takes can vary. The way I do it normally is after the logs are cleared the PC shuts down to be ready for a capture. The goal is to clear the logs, reboot, then begin capture, automatically.
I have no experience with snapins and was wondering how can I use them to run a powershell script before rebooting to capture. I have a script that clears the Windows logs and in that script (right now) at the very end shuts the computer down, preparing it for manual capture. I have the FOG client installed on my test system.
:EDIT: Sorry I think I figured it out.
Might as well think out loud in case anyone else can use this info. I got the snapin to run my clearalllogs.ps1 script, now the next challenge is to have the script run then immediately after it reboots, start the capture.