Best posts made by george1421

@joe-gill Ah OK, when you get back to it, we may need to up the memory settings in php-fpm since the php.ini settings are ignored by php-fpm. Understand we are still tuning the php-fpm settings. We need real world feedback to get the most appropriate settings for everyone.

@jflippen Each fog server has a local user account called fog with its own password. Have you tried to log into the remote fog server using the appropriate linux user fog password as defined in the storage node configuration for that storage node and done by ftp?

I know that was a mouth full, but the basic premise is to ftp to the remote storage node using the appropriate fog account. Change to the /images directory and ensure that you can put and delete files over ftp.

@quazz said in fogcontroller.class.php on line 260 : Allowed memory size of 3145728000 bytes exhausted:

you are running a reverse proxy Nginx

If I had to guess (unless the OP is doing something strange) that ngix might be the FOG servers. I believe that is where the FOG management gui reaches out to the FOG Project servers to get the information, both on the login and configuration pages.

The memory exhaustion issue could be php-fpm but that issue should have been fixed with FOG 1.5.4. We can test if it is php-fpm by turning it off in the configuration file to see if the memory error is coming from there. I can see issue #1 could be related to the tighter (more exact) restrictions on php code than apache php.

@howtogravity OK to test if your updates are correct. Please post the output of these commands.

sudo lsblk
and
sudo df -h

The first command shows the block devices (i.e. storage devices) that your fog server knows about.

The second command shows how your storage devices are connected.

@k-hays said in Issues with Fog 1.5.2:

. would 90 be a stretch?

Don’t do that much you could run into a situation of resource exhaustion. Under normal imaging you shouldn’t see more than 10 worker thread. For max children I would set it to 40.

If you are using FOG 1.5.2 or later here is what I would change in the www.conf file for php-fpm.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 40
pm.min_spare_servers = 6
pm.start_servers = 5

Update those settings then restart php-fpm.

Queue up your multicast then on the fog server linux console start top then press P to sort by CPU usage. Watch, you should have 5-7 php-fpm worker threads running, they should be the top cpu users.

@k-hays Between FOG 1.5.0 and 1.5.1 (or 1.5.1 and 1.5.2 sorry I can’t remember) the developers switched from using the built in php engine in apache to using a dedicated php engine (php-fpm). This was done to address an issue with GUI slowness reported with the new 1.5.x Web GUI on certain distributions. Baseline testing then showed an excellent improvement in GUI responsiveness as well as client check ins were not causing an issue with heavy web server load as they were before.

After 1.5.2 was released the developers started seeing other reports of issues with multicasting and unicasting in larger environment that wasn’t see during development. So certain tweaks to the php-fpm config files were made in 1.5.3 and 1.5.4. Increasing the memory limit even more to 256MB resolved a multicasting issue was discovered after 1.5.4. was released. That setting will be in 1.5.5 when its released in a few months. With these new settings you should see a pretty stable multicasting deployment.

I wonder if its related to a php time being off. Since you are PDT, I might have expected it to be a day ahead not behind if php is using UTC time.

Is the system timezone set correctly?
Do you have the timezone set correctly inside the fog gui (FOG Settings)?

Lets collect a bit more information here.

1. What version of FOG are you using?
2. You captured an image from a reference computer, what hardware was that reference image created on?
3. Is the hardware you are deploying that image to the same as the source computer?
4. Did you capture the image as single disk resizable?
5. Is the target hardware running in the same mode as the source computer (i.e. bios and bios, or uefi and uefi?)

Something you might consider.

login
kernel nfs://10.100.0.55/winimgs/iso/Winpe/wimboot gui
iseq ${platform} pcbios && goto mem_bios || goto no_mem
:mem_bios
initrd -n bootmgr.exe    nfs://10.100.0.55/winimgs/iso/Winpe/bootmgr.exe bootmgr.exe
goto mem_cont
:no_mem
initrd -n bootx64.efi   nfs://10.100.0.55/winimgs/iso/Winpe/bootx64.efi                 bootx64.efi    
:mem_cont
initrd -n bcd           nfs://10.100.0.55/winimgs/iso/Winpe/BCD                         bcd
...

Also realize that ${fog-ip} is a variable that expands out to the IP address of your fog server. If you use the variable it makes your pxe menus portable between FOG servers.

@mparlette Ok you have a few things to work on.

First I would set your system timezone correctly (not UTC). Understand this is the linux timezone setting of your linux distro. Once that is set, then in the fog gui under Fog Settings->Fog Configuration expand all menus and search for time make sure that is right. Once all are set correctly reboot the fog server. From then on your times should be correct.

@mparlette said in Date off in dashboard #253 - Github:

CentOS 7 w/ FOG 1.5.4
/etc/php.ini date.timezone entry was commented out.
corrected with:
date.timezone =“America/Vancouver”

Technically that may not help you. In a normal world it would. But in FOG’s case its using php-fpm which has its own opinion of things and if you have php 7.x installed then the /etc/php.ini file is typically not referenced since php 7 has this file in a different path.

But having the OS and fog menus in agreement should get you to where you want to be. So good job!!

Why did you install fog 1.5.1 instead of 1.5.4?

I can say I think there is an issue with Debian 9.4 installer for FOG that does not create the config file properly for php-fpm. If this is the case you will have a very slow UI in 1.5.2 and later version of FOG.

There is a quick test to see if php-fpm is configured correctly. Open a linux console and run top then press P to sort by CPU usage. You should have php-fpm at the top of the processes. There should be 3 or 4 instances running. If apache is at the top then php-fpm is not configured correctly on your FOG server.

There are a few tweaks needed to FOG 1.5.4 that was discovered after 1.5.4 was released. Once we are sure that php-fpm is running then we can look at those settings.

@jflippen Just as an idea (first let me say I’m not a programmer), if you look about in the code where you can find an example of the replication agent writing to a log file. Clone that and place it in the correct location in the code to write both md5 hash codes into the log. Once the fog server has restarted then it should log that information into the replicator log file. I’ve had to do somethings similar in the past to reverse engineer some of the magic Tom does with his code.

Lets assume is the issue we’ve found after FOG 1.5.4 has been released. Similar posts have addressed other multicasting issues with FOG 1.5.4. What the developers have seen is that under certain conditions php-fpm runs out of usable memory during a multicast. Probably the most useful value to you is bumping the memory from the default of 32MB to 256MB.

1. Change to the /etc directory from the fog server linux command prompt.
2. Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
   find /etc -name www.conf (hopefully you will only find one)
3. Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except php_admin_value[memory_limit] = 256M you will need to add that entry.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 35
pm.min_spare_servers = 5
pm.start_servers = 5

4. Save and exit your text editor.
5. Reboot the fog server.
6. See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.

Also we found there is something strange going on in the linux kernels after 4.15.2, I’m going to recommend that you downgrade your FOG/FOS kernel to 4.15.2. The issue with later kernels is that its taking 3-5 minutes to create the disk structure under certain circumstances, where with 4.15.2 and older its only seconds to create the structure.

Now the kernel will not impact your issue, but processing is incomplete might be related to the missing php-fpm configuration setting.

@stucraster I have a tutorial around moving your imaging files off the root partition here: https://forums.fogproject.org/topic/11048/moving-fog-s-images-files-off-the-root-partition-2017-edition

Usefull commands are lsblk to show block devices (i.e. storage devices) and df -h to show mounted disks.

Do you have an unmanged (dumb) switch to place between the pxe booting computer and your building network switch.

Your condition sounds like a spanning tree issue, where you are running standard spanning tree and not one of the fast spanning tree protocols (RSTP, MSTP, fast-STP, etc). I specifically think this because time corrects the iPXE dhcp process. Placing an unmanged switch between the pxe booting computer and the building network switch keeps the building switch port from dropping during the pxe booting process and is a quick indicator of traditional spanning tree protocol.

Side note: roll your FOS kernel back to 4.15.2 to have a better experience with UEFI / GPT disks. There is a regression bug in 4.17.0 that the developers haven’t been able to correct just yet.

I need you to watch the computer at the point where you post install scripts run. One of the very last steps after the post install scripts finish is “Updating database” Make sure that step is completed.

Where we see this issue, is there is a error on the target computer at the step of updating the database. But normally on capture image, because someone has changed the password on the FOG service account in linux called fog. In your case it appears you are deploying an image. Make sure your post install scripts are not accidentally rebooting the target computer before the Update Database command is executed.

If things fly by too fast for you to see, you can always setup a debug deploy. Tick the debug checkbox before you submit the deploy task. Then pxe boot the target computer. On the target computer you will see several screens of text and then eventually get dropped to a linux command prompt on the target computer. At the linux command prompt key in fog. This will start a debug deploy, requiring you to press enter at each breakpoint. It will give you a chance to see what is going on with the deployment.

Your best bet to have a standalone solution is to deploy the image to a temp computer and then capture the image with clonezilla. Both FOG and Clonezilla use partclone to capture the image, but FOG uses an image compression format not understood by partclone. You won’t have the FOG service capabilities with clonezilla but imaging will work standalone.

@fry_p Well the first thing I noticed is that you have way too many php-fpm processes running. Typically we see 5-7 process running at one time.

Below is my standard response when it comes to FOG between version 1.5.2 and 1.5.4. Ensure your settings are consistent with these:

Lets assume is the issue we’ve found after FOG 1.5.4 has been released.

1. Change to the /etc directory from the fog server linux command prompt.
2. Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
   find /etc -name www.conf (hopefully you will only find one)
3. Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except php_admin_value[memory_limit] = 256M you will need to add that entry.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 35
pm.min_spare_servers = 5
pm.start_servers = 5

4. Save and exit your text editor.
5. Reboot the fog server.
6. See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.

Also we found there is something strange going on in the linux kernels after 4.15.2, I’m going to recommend that you downgrade your FOG/FOS kernel to 4.15.2. The issue with later kernels is that its taking 3-5 minutes to create the disk structure under certain circumstances, where with 4.15.2 and older its only seconds to create the structure.

Now the kernel will not impact your issue, but processing is incomplete might be related to the missing php-fpm configuration setting.

///

For completeness, how many computers that have the fog client installed, connect to this fog server. (round numbers are good).

@Brenth453 Well there are only a handful of reasons for it to fail here.

1. The dhcp server can’t be reached
2. The FOG server url can’t be reached from that subnet
3. FOS does not support the network adapter used

I’m kind of going to rule out #1, since the device is pxe booting and starting FOS. Typically a spanning tree issue will show up where iPXE can’t get an IP address. But it has happened at this point too.

For #2. You should confirm that you can reach the URL defined in the error screen from a computer on the same subnet as the pxe booting computer so we can rule out any kind of screening router blocking the call to the fog server.

For #3, this one is going to be a bit harder to diagnose, but its surely possible.

1. Manually register this device.
2. Go in and schedule an image deployment or capture to this device (no worries because we are not going to do either). But before you submit the task, tick the check box that says debug.
3. PXE boot the target computer. It should jump into FOS right away. You may see the same warnings about IP address, but FOS will contine.
4. After several enter key presses and pages of text FOS should drop you to a linux command prompt. At that command prompt key in
   ip addr show
5. If you get an IP address then try to ping the fog server.
6. If you don’t get an ip address then we will need to dig into what hardware you are using.