After update of Fog Trunk - PHP Errors

Wayne Workman · Apr 26, 2016, 5:50 PM

@RipAU said in After update of Fog Trunk - PHP Errors:

@Wayne-Workman
The server is the 64bit version of Fedora.

if the fog VM only has one core assigned, I promise you it’s not a 64 bit operating system. It installed as 32 bit. Why? Only one core. Impossible to execute 64 bit code with one 32 bit core.

-edit-
You’re basically executing 64 bit code on a 32 bit processor.

Wayne Workman · Apr 26, 2016, 11:52 PM

@RipAU Try to bump it to two cores. See what happens.

RipAU · Apr 26, 2016, 11:55 PM

@Wayne-Workman

Okay I’m confused.

So you are saying even though the ESXi host is 64bit CPU and I installed the Fedora x86_64 version of the OS
and the kernel is listed as x86_64 GNU/Linux.
Because I only allocated 1 virtual CPU It installed as 32bit?

Wayne Workman · Apr 26, 2016, 5:59 PM

@RipAU x86_64 compiled code can run on 32 bit or 64 bit systems. Because you have one single core assigned, you have one processor. One single 32 bit core. Meaning that one single core is running 64 bit os because of how that os is compiled, and the instruction set that allows a 32 bit core to run 64 bit code. It’s slower, but works. If you intend to take real advantage of a 64 bit OS, you need two cores.

Plus, besides all that, if your load is high, wouldn’t it make sense to have more cores? And why are you being stingy with cores? I have an old core 2 duo in my basement I use as a VM host - it runs 4 VMs, each with 2 cores assigned. It does fine for my purposes. But point being, cores can be shared.

RipAU · Apr 27, 2016, 12:06 AM

@Wayne-Workman

I’m happy to try a re-install and try to migrate the current database and images to an updated OS as I was going update to CentOS7 in future as I would like to be able to keep everything up to date.

I guess I generally will only give it a single CPU unless required as that was what I was told by a VMware engineer at a training session a few years ago and I can’t say I’ve even heard that only allocating a single 64bit capable CPU ends up running as a 32bit CPU on VMware ESXi.

Wayne Workman · Apr 27, 2016, 12:07 AM

@RipAU Pretty sure you can’t install CentOS 7 on a machine with one CPU. I’ve tried on an old Pentium 4 - didn’t work. I also tried once at work when I accidentally forgot to bump the cores from 1 to 4 for a VM - and CentOS 7 threw a nasty error when I tried to boot.

RipAU · Apr 27, 2016, 12:23 AM

@Wayne-Workman

No worries, I’ll look at spinning up an updated CentOS 7 machine to test the SVN with.

That said, I currently have a working version of CentOS 7 64bit with 1 Virtual CPU on the same ESXi and that didn’t have any problems being installed and everything is listed as 64bit by vmware etc.

I’ll give this a go over the next few days and report back.

Cheers,

Wayne Workman · Apr 27, 2016, 12:23 AM

@RipAU Well that is very interesting.

RipAU · Apr 26, 2016, 7:53 PM

Just tested again with a full update from the working version.

I’m still convinced it is an issue with the update in FOG with my server as the server load is so much lower before the update.

last hit: 01:32:48         atop runtime:  0 days, 00:00:35             01:32:49
All:         2351 reqs (  67.2/sec)       1470.0K (   42.0K/sec)     640.3B/req
2xx:    2351 ( 100%) 3xx:       0 ( 0.0%) 4xx:     0 ( 0.0%) 5xx:     0 ( 0.0%)
R ( 30s):    2018 reqs (  67.3/sec)       1265.5K (   42.2K/sec)     642.2B/req
2xx:    2018 ( 100%) 3xx:       0 ( 0.0%) 4xx:     0 ( 0.0%) 5xx:     0 ( 0.0%)

 REQS REQ/S    KB KB/S URL
  658 21.93 205.1  6.8*/fog/management/index.php
  644 21.47  1056 35.2 /fog/management/other/ssl/srvpublic.crt
  566 18.87   2.8  0.1 /fog/service/jobs.php
   79  2.63   0.4  0.0 /fog/service/greenfog.php
   46  1.53   0.2  0.0 /fog/service/servicemodule-active.php
   23  0.79   1.1  0.0 /fog/service/Printers.php
    2  0.10   0.0  0.0 *

Also from mysql SHOW FULL PROCESSLIST;

MariaDB [(none)]> SHOW FULL PROCESSLIST;
+-----+------+-----------------+------+---------+------+-------+-----------------------+----------+
| Id  | User | Host            | db   | Command | Time | State | Info                  | Progress |
+-----+------+-----------------+------+---------+------+-------+-----------------------+----------+
|  14 | root | localhost:60778 | fog  | Sleep   |    1 |       | NULL                  |    0.000 |
|  19 | root | localhost:60790 | fog  | Sleep   |    1 |       | NULL                  |    0.000 |
|  39 | root | localhost:60856 | fog  | Sleep   |    0 |       | NULL                  |    0.000 |
|  42 | root | localhost:60862 | fog  | Sleep   |    0 |       | NULL                  |    0.000 |
|  43 | root | localhost:60890 | fog  | Sleep   |    5 |       | NULL                  |    0.000 |
|  44 | root | localhost:60902 | fog  | Sleep   |  514 |       | NULL                  |    0.000 |
|  45 | root | localhost:60914 | fog  | Sleep   |  537 |       | NULL                  |    0.000 |
|  46 | root | localhost:60950 | fog  | Sleep   |   55 |       | NULL                  |    0.000 |
|  47 | root | localhost:32782 | fog  | Sleep   |  152 |       | NULL                  |    0.000 |
|  49 | root | localhost:33290 | fog  | Sleep   |    1 |       | NULL                  |    0.000 |
|  51 | root | localhost:33474 | fog  | Sleep   |    0 |       | NULL                  |    0.000 |
|  52 | root | localhost:33914 | fog  | Sleep   |    1 |       | NULL                  |    0.000 |
|  53 | root | localhost:33924 | fog  | Sleep   |    0 |       | NULL                  |    0.000 |
|  56 | root | localhost:33968 | fog  | Sleep   |    0 |       | NULL                  |    0.000 |
|  58 | root | localhost:33984 | fog  | Sleep   |    0 |       | NULL                  |    0.000 |

Just FYI there is about 120 lines of this that is the same.

I’ve also upped the CPU and the Memory just to test but it is the same result, which is why I’m thinking its more an issue with the FOG update than 32bit vs 64bit.

As well as errors in the HTTPD log listing PHP warnings.

[Wed Apr 27 11:08:21.707362 2016] [:error] [pid 6824] [client 10.254.3.15:50021] PHP Warning:  implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67
[Wed Apr 27 11:08:21.728167 2016] [:error] [pid 7001] [client 10.254.1.171:60754] PHP Warning:  implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67

Thanks everyone so far for the help

Edit: forgot the CPU from htop

Wayne Workman · Apr 26, 2016, 9:00 PM

I’m linking all forum threads I can find about high CPU utilization right here for this thread and future reference. I also put them into the wiki here:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_Web_Interface

https://forums.fogproject.org/topic/6020/fog-svn-5020-and-above-cpu-hammered-thread/20?page=2

https://forums.fogproject.org/topic/6469/tons-of-httpd-processes

https://forums.fogproject.org/topic/6277/not-sure-if-its-a-bug-or-a-feature-high-fog-server-cpu-on-dashboard/22

https://forums.fogproject.org/topic/6901/high-cpu-usage-from-multiple-httpd-processes-version-5800?page=1

https://forums.fogproject.org/topic/6940/high-cpu-fog-services-after-update-r5029-v6759

https://forums.fogproject.org/topic/7234/after-update-of-fog-trunk-php-errors

https://forums.fogproject.org/topic/7215/high-cpu-php-errors-after-update-to-trunk-github-7234

https://forums.fogproject.org/topic/4929/fog-3396-high-cpu-utilization-when-auto-update-is-running-on-the-active-tasks-page

RipAU · Apr 27, 2016, 4:05 AM

Awesome, thanks.

Also just FYI after doing a fresh install of CentOS 7 that has been updated with the latest SVN version of Fog Version: 7332
I’m also getting the same errors from httpd

[Wed Apr 27 13:30:18.351161 2016] [:error] [pid 12236] [client 10.254.14.101:53436] PHP Warning:  session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/user.class.php on line 113
[Wed Apr 27 13:30:28.607672 2016] [:error] [pid 9846] [client 10.254.14.101:53470] PHP Warning:  implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67
[Wed Apr 27 13:30:28.665747 2016] [:error] [pid 9846] [client 10.254.14.101:53470] PHP Warning:  session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/user.class.php on line 113
[Wed Apr 27 13:30:38.938836 2016] [:error] [pid 12215] [client 10.254.14.101:53507] PHP Warning:  implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67
[Wed Apr 27 13:30:38.996816 2016] [:error] [pid 12215] [client 10.254.14.101:53507] PHP Warning:  session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/user.class.php on line 113
[Wed Apr 27 13:30:49.257933 2016] [:error] [pid 9864] [client 10.254.14.101:53541] PHP Warning:  implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67
[Wed Apr 27 13:30:49.316223 2016] [:error] [pid 9864] [client 10.254.14.101:53541] PHP Warning:  session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/user.class.php on line 113

RipAU · Apr 27, 2016, 4:47 AM

Hopefully its ok I am loading this thread up with screenshots of issues?

Ok, with some unscientific testing on the new server, I’m also getting higher CPU with only 2 hosts that have been registered.
Before I registered the hosts the CPU was near 0% now it is quite a bit higher.

I am suspecting the Fog Clients are basically flooding the FOG server as it seems to do a request every few seconds.
The Fog client logs are also filling up pretty quickly.

I’m basically getting the following over and over and over again until the fog.log starts again.

27/04/2016 2:09 PM Middleware::Communication Response: Invalid host certificate
 27/04/2016 2:09 PM Middleware::Communication URL: http://10.254.14.101/fog/management/other/ssl/srvpublic.crt
 27/04/2016 2:09 PM Data::RSA FOG Server CA cert found
 27/04/2016 2:09 PM Middleware::Authentication Cert OK
 27/04/2016 2:09 PM Middleware::Communication POST URL: http://10.254.14.101/fog/management/index.php?sub=authorize
 27/04/2016 2:09 PM Middleware::Communication Response: Success
 27/04/2016 2:09 PM Middleware::Authentication Authenticated
 27/04/2016 2:09 PM Middleware::Communication URL: http://10.254.14.101/fog/service/hostname.php?moduleid=hostnamechanger&mac=98:90:96:B3:A6:5D||00:00:00:00:00:00:00:E0&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1

This is with a fresh Centos 7 64bit with 4 vCPUs with Fog Version: 7332

The following is just from 2 fog clients on a standard Centos 7

from apachetop

last hit: 04:46:03         atop runtime:  0 days, 00:08:15             04:46:04
All:         5675 reqs (  11.5/sec)       3691.2K ( 7636.0B/sec)     666.0B/req
2xx:    5674 (100.0%) 3xx:       1 ( 0.0%) 4xx:     0 ( 0.0%) 5xx:     0 ( 0.0%)
R ( 30s):     350 reqs (  11.7/sec)        225.4K ( 7695.3B/sec)     659.6B/req
2xx:     350 ( 100%) 3xx:       0 ( 0.0%) 4xx:     0 ( 0.0%) 5xx:     0 ( 0.0%)

 REQS REQ/S    KB KB/S URL
  115  3.83   0.6  0.0*/fog/service/greenfog.php
  115  3.83 188.6  6.3 /fog/management/other/ssl/srvpublic.crt
  114  3.80  36.3  1.2 /fog/management/index.php
    3  0.10   0.0  0.0 /fog/service/servicemodule-active.php
    1  0.03   0.0  0.0 /fog/service/snapins.checkin.php
    1  0.03   0.0  0.0 /fog/service/Printers.php
    1  0.03   0.0  0.0 /fog/service/printerlisting.php

Hopefully this helps narrow it down?

RipAU · Apr 28, 2016, 2:37 AM

Hey guys.

Just FYI I updated to the latest trunk version and so far the CPU load has dropped quite substantially.
Between 2% and 20% depending on the traffic from the Clients.

The only remaining errors I have is a PHP Warnings

[Thu Apr 28 11:29:42.884801 2016] [:error] [pid 1551] [client 127.0.0.1:53970] PHP Warning:  session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/user.class.php on line 113

[Thu Apr 28 11:30:48.950325 2016] [:error] [pid 1551] [client 10.55.14.100:15431] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/html/fog/lib/fog/fogbase.class.php on line 451

Aside from this everything else seems to be working correctly including the client not looping and stuck on authentication.

Fog Version: 7380

Thanks again guys.
I’m guessing this could be marked as solved as of Fog Version: 7380 for me.

Cheers.

Wayne Workman · Apr 28, 2016, 1:35 PM

@RipAU said in After update of Fog Trunk - PHP Errors:

Hopefully its ok I am loading this thread up with screenshots of issues?

It’s perfectly fine as long as it’s related to FOG.

After update of Fog Trunk - PHP Errors

161

12.0k

17.3k

155.1k