Deployment stops/waits very long with: Reattempting to update database ....

kratkale

@kratkale

found it …
just click on "DefaultMember FOG Version: (1.5.4)

kratkale

@george1421
It did not solve the problem;
still waiting

updating Database …
Reattemting to update database …
…
what to do?

george1421

@kratkale Sorry I did not fully read the body of you OP. The subject line and body have two different fixes.

Installing the 4.15.2 is the right answer for the delay in creation of the disk image.

The multicast issue related to something else. Lets start with my standard response.

///
Lets assume is the issue we’ve found after FOG 1.5.4 has been released.

Change to the /etc directory from the fog server linux command prompt.
Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
find /etc -name www.conf (hopefully you will only find one)
Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except php_admin_value[memory_limit] = 256M you will need to add that entry.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 35
pm.min_spare_servers = 5
pm.start_servers = 5

Save and exit your text editor.
Reboot the fog server.
See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.

Also we found there is something strange going on in the linux kernels after 4.15.2, I’m going to recommend that you downgrade your FOG/FOS kernel to 4.15.2. The issue with later kernels is that its taking 3-5 minutes to create the disk structure under certain circumstances, where with 4.15.2 and older its only seconds to create the structure.

Now the kernel will not impact your issue, but processing is incomplete might be related to the missing php-fpm configuration setting.
///

Update the www.conf file and reboot the server. If that still doen’t fix the issue, the developers working on the multicast code in the 1.5.5 beta code. But lets see if we can fix it before going down that path.

kratkale

@george1421
No Problem - thank you for your help. On Monday school starts again and last week I got 87 new Computers.

I will make some comments - maybe it helps someone …

Well I use Debian net install 9.5 (installed 20180826)

vi /etc/php/7.0/fpm/pool.d/www.conf
php_admin_value[memory_limit] = 256M
This entry is commeted out and with a value of 32M
pm.max_requests = 2000
This entry is in the Debian Version like you suggest
pm.max_childrend = 35
Here the Debian value is 5
pm.min_spare_servers = 5
Here the Debian value is 1
pm.start_servers = 5
Here the Debian value is2

I can already confirm that the 4.15.2 Kernel is better. With the kernel after the FOG installation it took several minutes to delete the MPR Partition.

Thank you so much for you help. Funny I use the FOG Server since many years - everything worked all the time out of the box.

Now I cannot open the FOG Webinterface:

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.4.25 (Debian) Server at 192.168.0.196 Port 80

mhm

george1421

@kratkale phpf-pm is something new in FOG 1.5.2 and later. The developers have found some fine tuning settings once 1.5.4 was release (with 1.5.5 being delayed because of the summer holidays) we have to manually patch. I would expect that 1.5.5 to be released with all of the hot fixes soon .

One thing you need to be sure is that you only have one www.conf file. We’ve see users having multiple version of php on their computer (I saw one user that have all 3 copies of php7 running at the same time. This caused a mess).

So is your system working now (the 256MB ram should really help things).

kratkale

@george1421

No - it is not working …

There is only one file:
root@FOG:/home/kratkale# find /etc -name www.conf
/etc/php/7.0/fpm/pool.d/www.conf
root@FOG:/home/kratkale#

george1421

@kratkale Are you willing to try 1.5.5 beta? You may be running into the issue that was already addressed in the working branch.

If so from the linux console change to the git repository you downloaded using the git clone command.

cd /root/fogproject
git pull
git checkout working
git pull
cd bin

Then rerun the installer

./installfog.sh

To switch back to the main repository you need to do the following.

cd /root/fogproject
git pull
git checkout main
git pull
cd bin

Then rerun the installer

./installfog.sh

kratkale

@george1421

yes I can do that - but there is no /root/fogproject
shall I create it?

george1421

@kratkale How did you install fog 1.5.4 to start with? From git or download the tarball?

kratkale

@george1421
download from the FOG Webpage

sorry I’ve to wait …

Error

As a new user, you can only post once every 120 second(s) until you have earned 3 reputation - please wait before posting again

george1421

@kratkale OK lets create the standard way to install and upgrade fog. Note: Your FOG server needs internet access to download the installer files.

sudo apt-get update && apt-get install git

sudo -i
git clone https://github.com/FOGProject/fogproject.git /root/fogproject
cd /root/fogproject

That will create the required files on your fog server, then follow the steps I set out before.

kratkale

@george1421
installation working fine;
Multicast is not working …
I tried the installed Kernel and also 4.15.2
The process stops when all devices are in the first partclone blue screen

Deploying to a single computer works fine.

george1421

@kratkale Interesting… On the fog server in /opt/fog/logs is there anything helpful in the multicast log? Or in /var/log/php-fpm logs? There has to be something unique going on there.

kratkale

@george1421

ls -trl
…
-rw-r–r-- 1 root root 81348 Aug 31 20:52 multicast.log.udpcast.68
-rw-r–r-- 1 root root 1950 Aug 31 23:42 fogimagesize.log
-rw-r–r-- 1 root root 2306 Sep 1 00:12 fogscheduler.log
-rw-r–r-- 1 root root 3175 Sep 1 00:12 pinghost.log
-rw-r–r-- 1 root root 5501 Sep 1 00:12 fogsnapinhash.log
-rw-r–r-- 1 root root 8243 Sep 1 00:12 fogreplicator.log
-rw-r–r-- 1 root root 150997 Sep 1 00:12 groupmanager.log
-rw-r–r-- 1 root root 12194 Sep 1 00:12 fogsnapinrep.log
-rw-r–r-- 1 root root 8733 Sep 1 00:13 multicast.log
-rw-r–r-- 1 root root 49638 Sep 1 00:13 servicemaster.log

the multicast log ends with:

[08-31-18 10:12:45 pm] * No tasks found!
[08-31-18 10:12:55 pm] * No tasks found!
[08-31-18 10:13:05 pm] * No tasks found!
[08-31-18 10:13:15 pm] * No tasks found!
[08-31-18 10:13:25 pm] * No tasks found!
[08-31-18 10:13:35 pm] * No tasks found!

php7.0-fpm.log:

[31-Aug-2018 19:03:49] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 20:35:25] NOTICE: Terminating …
[31-Aug-2018 20:35:26] NOTICE: exiting, bye-bye!
[31-Aug-2018 20:36:10] NOTICE: fpm is running, pid 449
[31-Aug-2018 20:36:10] NOTICE: ready to handle connections
[31-Aug-2018 20:36:10] NOTICE: systemd monitor interval set to 10000ms
[31-Aug-2018 20:36:15] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 20:36:30] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 20:37:21] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 20:52:24] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 21:10:46] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 23:34:18] NOTICE: Terminating …
[31-Aug-2018 23:34:18] NOTICE: exiting, bye-bye!
[31-Aug-2018 23:35:10] ALERT: [pool www] pm.max_spare_servers(3) must not be less than pm.min_spare_servers(5)
[31-Aug-2018 23:35:10] ERROR: failed to post process the configuration
[31-Aug-2018 23:35:10] ERROR: FPM initialization failed
[31-Aug-2018 23:42:30] ALERT: [pool www] pm.max_spare_servers(3) must not be less than pm.min_spare_servers(5)
[31-Aug-2018 23:42:30] ERROR: failed to post process the configuration
[31-Aug-2018 23:42:30] ERROR: FPM initialization failed
[01-Sep-2018 00:15:48] NOTICE: fpm is running, pid 7989
[01-Sep-2018 00:15:48] NOTICE: ready to handle connections
[01-Sep-2018 00:15:48] NOTICE: systemd monitor interval set to 10000ms
[01-Sep-2018 00:16:45] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 1 idle, and 14 total children

george1421

@kratkale said in Deployment stops/waits very long with: Reattempting to update database ....:

-rw-r–r-- 1 root root 81348 Aug 31 20:52 multicast.log.udpcast.68

This is probably the log you want to review.

[31-Aug-2018 20:36:15] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[31-Aug-2018 23:42:30] ALERT: [pool www] pm.max_spare_servers(3) must not be less than pm.min_spare_servers(5)
[01-Sep-2018 00:16:45] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 1 idle, and 14 total children

You need to double check your settings in the php-fpm file. If these settings are not right and php-fpm does not start correctly, you won’t have a web gui.

kratkale

@george1421
Good Morning Georg,
I’ve a working web GUI.
FOG was deploying 30 PC over night but without multicast. The image size is about 50GB.

Sorry I don’t know what to change.
/home/kratkale# cat /etc/php/7.0/fpm/pool.d/www.conf | grep -v ‘;’| grep -v ^$

[www]
user = www-data
group = www-data
listen = 127.0.0.1:9000
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 2000
php_admin_value[memory_limit] = 256M

So what to do? Multicast stopped at the beginning of partclone;

Thank you Alexander

kratkale

@george1421
another interesting point is:
I cannot deploy an image direct on the client Computer:
If I start the PC I get the normal FOG Welcome screen.
If I select Deploy Image, I’m asked for the password. But it does not Deploy the image; immediately I get the FOG Welcome screen again. Another attempt: The User and the password is still on the FOG Welcome screen???

Another attempt with a different user: same behavior.

I can deploy the image from the Web Interface …

george1421

@kratkale This is really strange because the errors don’t align with what you have in your config file.

The logs clearly state:
[31-Aug-2018 20:37:21] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

But your config file has
pm.max_children = 50

Are you SURE you don’t have 2 www.conf files? Or another .conf file in /etc/php/7.0/fpm/pool.d

If you stop php-fpm, nuke php7.0-fpm.log and then restart php-fpm are those error still present?

I don’t think the php-fpm errors are related to multicast stopping at the beginning of partclone. Is there anything on the screen of the target computers that indicate why its stopped? I clear screen shot of the error taken with a mobile phone might give us a clue to what is going on.

Also have you reviewed the multicast logs multicast.log and multicast.log.udpcast to see of there is any useful info in there?

And for some background information.

The target computers you are deploying to, deploy OK with unicast, but fail with multicast?
How many systems are you trying to multicast at one time?
Are alll of the computers on the same subnet?

george1421

@kratkale said in Deployment stops/waits very long with: Reattempting to update database ....:

If I select Deploy Image, I’m asked for the password. But it does not Deploy the image; immediately I get the FOG Welcome screen again. Another attempt: The User and the password is still on the FOG Welcome screen???

For clarity what hardware are you trying to deploy to? (manufacturer and model). I can’t remember the normal sequence off the top of my head, in that when you create a direct deployment if it deploys immediately or automatically schedules a deployment, reboots, then deploys. I can see logic to both.

kratkale

@george1421
Hi,
Are you SURE you don’t have 2 www.conf files?
Yes I think so:
-rw-r–r-- 1 root root 19K Sep 1 00:15 www.conf

I think it is ok! I changed the www.conf on the 1 Sep - the log entry is on 31-Aug-2018 at 20:37:21
So the log file informs about the old config?!
Also have you reviewed the multicast logs
Can I upload the log files? Yes there are a lot of informations - meaningless to me.
And for some background information.
1) The target computers deploy OK with unicast. The stop deploying with multicast at the first screen of partclone. It looks to me they are waiting that all computers get ready. But they never do … Finally all the PC’s show the same picture. I can do a picture when my deployment is finished.
I was able to deploy multicast with FOG 1.4.4 and 1.5.4. The Version 1.5.4 had the trouble with “Update Database”. The Deployment istself worked fine. On all PC’s the image was completely. Some of them didn’t join my domain. But all of them where running the windows 10 image

2) I have one class with 25 PCs, the second with 36 PCs. There are three (a, b, c) D-Link Switches DGS1210
a) the FOG Server, the Green Interface from IP Fire, 25 PCs and the connection to the other two switches
b) 18 Clients of the 36 PCs
c) 18 Clients of the 36 PCs

3) Yes all Computer are in one subnet
192.168.0.0\24
The Client PCs get their IP via DHCP form an IPFire. The IP adresses are fixed.

All of them are nearly identical hardware
Lenovo I5 with SSD and 8GB RAM delivered 201808
The only difference is the size of the second hard disc - I think irrelevant to FOG.

Deployment stops/waits very long with: Reattempting to update database ....

259

11.7k

17.1k

154.6k