Deployment stops/waits very long with: Reattempting to update database ....

kratkale

I’m using FOG Version 1.5.4. A brand new FOG installation on a brand new net installation of Debian 9.5. The host Computer is a HP7700 with 2GB Ram (about 500MB used) and the cpu is idle with 99%.

When a multicast deployment is nearly finished I get the message:

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.4.25 (Debian) Server at 192.168.0.196 Port 80

“Reattempting to update database …”
…
This message repeats, repeats …

Then the PC’s are rebooting.

The deployed image is working.

While the Deploymentprocess is stucked I can also not use the web interface of FOG. It is loaaaaaaaaaading.

If I deploy more then 12 PC’s some of the windows systems do not join the domain … I tried multicast with 56 PC’s

george1421

Please install FOS kernel 4.15.2 for both x64 and x32 bit systems. That will address the log delay in partition creation.

kratkale

@george1421
can I see which kernel is used at the moment?

kratkale

@kratkale

found it …
just click on "DefaultMember FOG Version: (1.5.4)

kratkale

@george1421
It did not solve the problem;
still waiting

updating Database …
Reattemting to update database …
…
what to do?

george1421

@kratkale Sorry I did not fully read the body of you OP. The subject line and body have two different fixes.

Installing the 4.15.2 is the right answer for the delay in creation of the disk image.

The multicast issue related to something else. Lets start with my standard response.

///
Lets assume is the issue we’ve found after FOG 1.5.4 has been released.

Change to the /etc directory from the fog server linux command prompt.
Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
find /etc -name www.conf (hopefully you will only find one)
Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except php_admin_value[memory_limit] = 256M you will need to add that entry.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 35
pm.min_spare_servers = 5
pm.start_servers = 5

Save and exit your text editor.
Reboot the fog server.
See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.

Also we found there is something strange going on in the linux kernels after 4.15.2, I’m going to recommend that you downgrade your FOG/FOS kernel to 4.15.2. The issue with later kernels is that its taking 3-5 minutes to create the disk structure under certain circumstances, where with 4.15.2 and older its only seconds to create the structure.

Now the kernel will not impact your issue, but processing is incomplete might be related to the missing php-fpm configuration setting.
///

Update the www.conf file and reboot the server. If that still doen’t fix the issue, the developers working on the multicast code in the 1.5.5 beta code. But lets see if we can fix it before going down that path.

kratkale

@george1421
No Problem - thank you for your help. On Monday school starts again and last week I got 87 new Computers.

I will make some comments - maybe it helps someone …

Well I use Debian net install 9.5 (installed 20180826)

vi /etc/php/7.0/fpm/pool.d/www.conf
php_admin_value[memory_limit] = 256M
This entry is commeted out and with a value of 32M
pm.max_requests = 2000
This entry is in the Debian Version like you suggest
pm.max_childrend = 35
Here the Debian value is 5
pm.min_spare_servers = 5
Here the Debian value is 1
pm.start_servers = 5
Here the Debian value is2

I can already confirm that the 4.15.2 Kernel is better. With the kernel after the FOG installation it took several minutes to delete the MPR Partition.

Thank you so much for you help. Funny I use the FOG Server since many years - everything worked all the time out of the box.

Now I cannot open the FOG Webinterface:

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.4.25 (Debian) Server at 192.168.0.196 Port 80

mhm

george1421

@kratkale phpf-pm is something new in FOG 1.5.2 and later. The developers have found some fine tuning settings once 1.5.4 was release (with 1.5.5 being delayed because of the summer holidays) we have to manually patch. I would expect that 1.5.5 to be released with all of the hot fixes soon .

One thing you need to be sure is that you only have one www.conf file. We’ve see users having multiple version of php on their computer (I saw one user that have all 3 copies of php7 running at the same time. This caused a mess).

So is your system working now (the 256MB ram should really help things).

kratkale

@george1421

No - it is not working …

There is only one file:
root@FOG:/home/kratkale# find /etc -name www.conf
/etc/php/7.0/fpm/pool.d/www.conf
root@FOG:/home/kratkale#

george1421

@kratkale Are you willing to try 1.5.5 beta? You may be running into the issue that was already addressed in the working branch.

If so from the linux console change to the git repository you downloaded using the git clone command.

cd /root/fogproject
git pull
git checkout working
git pull
cd bin

Then rerun the installer

./installfog.sh

To switch back to the main repository you need to do the following.

cd /root/fogproject
git pull
git checkout main
git pull
cd bin

Then rerun the installer

./installfog.sh

kratkale

@george1421

yes I can do that - but there is no /root/fogproject
shall I create it?

george1421

@kratkale How did you install fog 1.5.4 to start with? From git or download the tarball?

kratkale

@george1421
download from the FOG Webpage

sorry I’ve to wait …

Error

As a new user, you can only post once every 120 second(s) until you have earned 3 reputation - please wait before posting again

george1421

@kratkale OK lets create the standard way to install and upgrade fog. Note: Your FOG server needs internet access to download the installer files.

sudo apt-get update && apt-get install git

sudo -i
git clone https://github.com/FOGProject/fogproject.git /root/fogproject
cd /root/fogproject

That will create the required files on your fog server, then follow the steps I set out before.

kratkale

@george1421
installation working fine;
Multicast is not working …
I tried the installed Kernel and also 4.15.2
The process stops when all devices are in the first partclone blue screen

Deploying to a single computer works fine.

george1421

@kratkale Interesting… On the fog server in /opt/fog/logs is there anything helpful in the multicast log? Or in /var/log/php-fpm logs? There has to be something unique going on there.

kratkale

@george1421

ls -trl
…
-rw-r–r-- 1 root root 81348 Aug 31 20:52 multicast.log.udpcast.68
-rw-r–r-- 1 root root 1950 Aug 31 23:42 fogimagesize.log
-rw-r–r-- 1 root root 2306 Sep 1 00:12 fogscheduler.log
-rw-r–r-- 1 root root 3175 Sep 1 00:12 pinghost.log
-rw-r–r-- 1 root root 5501 Sep 1 00:12 fogsnapinhash.log
-rw-r–r-- 1 root root 8243 Sep 1 00:12 fogreplicator.log
-rw-r–r-- 1 root root 150997 Sep 1 00:12 groupmanager.log
-rw-r–r-- 1 root root 12194 Sep 1 00:12 fogsnapinrep.log
-rw-r–r-- 1 root root 8733 Sep 1 00:13 multicast.log
-rw-r–r-- 1 root root 49638 Sep 1 00:13 servicemaster.log

the multicast log ends with:

[08-31-18 10:12:45 pm] * No tasks found!
[08-31-18 10:12:55 pm] * No tasks found!
[08-31-18 10:13:05 pm] * No tasks found!
[08-31-18 10:13:15 pm] * No tasks found!
[08-31-18 10:13:25 pm] * No tasks found!
[08-31-18 10:13:35 pm] * No tasks found!

php7.0-fpm.log:

[31-Aug-2018 19:03:49] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 20:35:25] NOTICE: Terminating …
[31-Aug-2018 20:35:26] NOTICE: exiting, bye-bye!
[31-Aug-2018 20:36:10] NOTICE: fpm is running, pid 449
[31-Aug-2018 20:36:10] NOTICE: ready to handle connections
[31-Aug-2018 20:36:10] NOTICE: systemd monitor interval set to 10000ms
[31-Aug-2018 20:36:15] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 20:36:30] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 20:37:21] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 20:52:24] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 21:10:46] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 23:34:18] NOTICE: Terminating …
[31-Aug-2018 23:34:18] NOTICE: exiting, bye-bye!
[31-Aug-2018 23:35:10] ALERT: [pool www] pm.max_spare_servers(3) must not be less than pm.min_spare_servers(5)
[31-Aug-2018 23:35:10] ERROR: failed to post process the configuration
[31-Aug-2018 23:35:10] ERROR: FPM initialization failed
[31-Aug-2018 23:42:30] ALERT: [pool www] pm.max_spare_servers(3) must not be less than pm.min_spare_servers(5)
[31-Aug-2018 23:42:30] ERROR: failed to post process the configuration
[31-Aug-2018 23:42:30] ERROR: FPM initialization failed
[01-Sep-2018 00:15:48] NOTICE: fpm is running, pid 7989
[01-Sep-2018 00:15:48] NOTICE: ready to handle connections
[01-Sep-2018 00:15:48] NOTICE: systemd monitor interval set to 10000ms
[01-Sep-2018 00:16:45] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 1 idle, and 14 total children

george1421

@kratkale said in Deployment stops/waits very long with: Reattempting to update database ....:

-rw-r–r-- 1 root root 81348 Aug 31 20:52 multicast.log.udpcast.68

This is probably the log you want to review.

[31-Aug-2018 20:36:15] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 23:42:30] ALERT: [pool www] pm.max_spare_servers(3) must not be less than pm.min_spare_servers(5)
[01-Sep-2018 00:16:45] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 1 idle, and 14 total children

You need to double check your settings in the php-fpm file. If these settings are not right and php-fpm does not start correctly, you won’t have a web gui.

kratkale

@george1421
Good Morning Georg,
I’ve a working web GUI.
FOG was deploying 30 PC over night but without multicast. The image size is about 50GB.

Sorry I don’t know what to change.
/home/kratkale# cat /etc/php/7.0/fpm/pool.d/www.conf | grep -v ‘;’| grep -v ^$

[www]
user = www-data
group = www-data
listen = 127.0.0.1:9000
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 2000
php_admin_value[memory_limit] = 256M

So what to do? Multicast stopped at the beginning of partclone;

Thank you Alexander

kratkale

@george1421
another interesting point is:
I cannot deploy an image direct on the client Computer:
If I start the PC I get the normal FOG Welcome screen.
If I select Deploy Image, I’m asked for the password. But it does not Deploy the image; immediately I get the FOG Welcome screen again. Another attempt: The User and the password is still on the FOG Welcome screen???

Another attempt with a different user: same behavior.

I can deploy the image from the Web Interface …

Deployment stops/waits very long with: Reattempting to update database ....

173

12.1k

17.3k

155.4k