After update of Fog Trunk - PHP Errors
-
Just checked then.
I’m not sure I can see an issue? with the following.
last hit: 03:35:45 atop runtime: 0 days, 00:02:05 03:35:46 All: 2043 reqs ( 16.3/sec) 467.5K ( 3829.8B/sec) 234.3B/req 2xx: 1817 (88.9%) 3xx: 52 ( 2.5%) 4xx: 2 ( 0.1%) 5xx: 172 ( 8.4%) R ( 30s): 476 reqs ( 15.9/sec) 116.9K ( 3990.2B/sec) 251.5B/req 2xx: 424 (89.1%) 3xx: 6 ( 1.3%) 4xx: 0 ( 0.0%) 5xx: 46 ( 9.7%) REQS REQ/S KB KB/S URL 144 4.80 0.7 0.0*/fog/service/servicemodule-active.php 94 3.13 4.1 0.1 /fog/management/index.php 66 2.20 2.4 0.1 /fog/status/bandwidth.php 49 1.63 80.3 2.7 /fog/management/other/ssl/srvpublic.crt 43 1.43 1.5 0.0 /fog/service/Printers.php 14 0.48 0.1 0.0 /fog/service/greenfog.php 13 0.43 27.5 0.9 /fog/service/printerlisting.php 13 0.43 0.1 0.0 /fog/service/snapins.checkin.php 11 0.37 0.1 0.0 /fog/service/hostname.php 10 0.36 0.0 0.0 /fog/service/jobs.php 10 0.36 0.1 0.0 /fog/service/getversion.php 6 0.21 0.0 0.0 /fog/index.php 3 0.10 0.0 0.0 *looks like most requests are being sent to servicemodule-active.php ?
-
Hi, Guys.
As a bit of a hopeful move I updated to the latest SVN update (FOG Version: 7330)
I’ve found the web interface near unusable because of the load that is now being put on the server(Huge number of httpd processes are now being spawned taking up most of the CPU and quite allot of mysqld processes as well)I’ll roll back to a working version of the SVN with a snapshot. Happy to try to update and test if this helps
I’ve taken several screenshots that hopefully help.
just to clarify this is with Fedora 22 Server, PHP7 from Remi Repository with FOG version 7330 on an VMware ESXi server.
Previously it was working perfectly
Cheers.
-
@RipAU You are seeing a lot of requests to srvpublic.crt which makes me think that this might be just a peek load while a lot of the clients redo the certificate stuff for secure communication to the FOG server.
Not sure if this was asked already but do you know which version you updated from that was perfectly working for you?
-
Yep just did a revert back to the previous version then. It is Fog Version: 6207
The CPU usage is a little higher than I’d like (fluctuates from 20% to 65% depending on the amount of traffic to the server, which is one of the reasons I was going to update) but otherwise has been working perfectly with no real issues.
Server load generally seems fine as well - load average: 0.88, 1.74, 1.55 -
@RipAU While the load is high, go into MySQL and issue this command:
SHOW FULL PROCESSLISTand give us the output.
Maybe it’s time for you to tell us about the ESXi hosting machine, and the resources you have assigned to FOG.
Specifically, the host’s core speed and processor type. and how many cores you assigned to FOG, and how much RAM.
-
@Wayne-Workman
Yep no worries.The server is ESXi 6.
At this stage it has 1vCPU has been assigned with 2GB of RAM.
Server CPUS are Intel Xenon CPU E5-2630 @ 2.4ghzFog Server at this stage only has about 300 hosts (as soon as everything is smoothed out I was going to migrate the rest of our hosts over and this will be about 1100)
I can revert to the updated fog version of the snapshot after It was updated and run that command.
I’ll do that now.Cheers,
-
@RipAU Oh… well there’s your problem. Fog sucks with one core.
give it at least two.
But don’t just bump it to two. Now you need to tear the whole thing down and start over - because your using a 32 bit operating system.
give it 2, and then re-install the OS.
-
I can easily up the CPU and RAM - The server is the 64bit version of Fedora.
4.2.7-200.fc22.x86_64 #1 SMP Thu Dec 10 03:28:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux -
@RipAU said in After update of Fog Trunk - PHP Errors:
@Wayne-Workman
The server is the 64bit version of Fedora.if the fog VM only has one core assigned, I promise you it’s not a 64 bit operating system. It installed as 32 bit. Why? Only one core. Impossible to execute 64 bit code with one 32 bit core.
-edit-
You’re basically executing 64 bit code on a 32 bit processor. -
@RipAU Try to bump it to two cores. See what happens.
-
Okay I’m confused.
So you are saying even though the ESXi host is 64bit CPU and I installed the Fedora x86_64 version of the OS
and the kernel is listed as x86_64 GNU/Linux.
Because I only allocated 1 virtual CPU It installed as 32bit? -
@RipAU x86_64 compiled code can run on 32 bit or 64 bit systems. Because you have one single core assigned, you have one processor. One single 32 bit core. Meaning that one single core is running 64 bit os because of how that os is compiled, and the instruction set that allows a 32 bit core to run 64 bit code. It’s slower, but works. If you intend to take real advantage of a 64 bit OS, you need two cores.
Plus, besides all that, if your load is high, wouldn’t it make sense to have more cores? And why are you being stingy with cores? I have an old core 2 duo in my basement I use as a VM host - it runs 4 VMs, each with 2 cores assigned. It does fine for my purposes. But point being, cores can be shared.
-
I’m happy to try a re-install and try to migrate the current database and images to an updated OS as I was going update to CentOS7 in future as I would like to be able to keep everything up to date.
I guess I generally will only give it a single CPU unless required as that was what I was told by a VMware engineer at a training session a few years ago and I can’t say I’ve even heard that only allocating a single 64bit capable CPU ends up running as a 32bit CPU on VMware ESXi.
-
@RipAU Pretty sure you can’t install CentOS 7 on a machine with one CPU. I’ve tried on an old Pentium 4 - didn’t work. I also tried once at work when I accidentally forgot to bump the cores from 1 to 4 for a VM - and CentOS 7 threw a nasty error when I tried to boot.
-
No worries, I’ll look at spinning up an updated CentOS 7 machine to test the SVN with.
That said, I currently have a working version of CentOS 7 64bit with 1 Virtual CPU on the same ESXi and that didn’t have any problems being installed and everything is listed as 64bit by vmware etc.
I’ll give this a go over the next few days and report back.
Cheers,
-
@RipAU Well that is very interesting.
-
Just tested again with a full update from the working version.
I’m still convinced it is an issue with the update in FOG with my server as the server load is so much lower before the update.
last hit: 01:32:48 atop runtime: 0 days, 00:00:35 01:32:49 All: 2351 reqs ( 67.2/sec) 1470.0K ( 42.0K/sec) 640.3B/req 2xx: 2351 ( 100%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%) R ( 30s): 2018 reqs ( 67.3/sec) 1265.5K ( 42.2K/sec) 642.2B/req 2xx: 2018 ( 100%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%) REQS REQ/S KB KB/S URL 658 21.93 205.1 6.8*/fog/management/index.php 644 21.47 1056 35.2 /fog/management/other/ssl/srvpublic.crt 566 18.87 2.8 0.1 /fog/service/jobs.php 79 2.63 0.4 0.0 /fog/service/greenfog.php 46 1.53 0.2 0.0 /fog/service/servicemodule-active.php 23 0.79 1.1 0.0 /fog/service/Printers.php 2 0.10 0.0 0.0 *Also from mysql SHOW FULL PROCESSLIST;
MariaDB [(none)]> SHOW FULL PROCESSLIST; +-----+------+-----------------+------+---------+------+-------+-----------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +-----+------+-----------------+------+---------+------+-------+-----------------------+----------+ | 14 | root | localhost:60778 | fog | Sleep | 1 | | NULL | 0.000 | | 19 | root | localhost:60790 | fog | Sleep | 1 | | NULL | 0.000 | | 39 | root | localhost:60856 | fog | Sleep | 0 | | NULL | 0.000 | | 42 | root | localhost:60862 | fog | Sleep | 0 | | NULL | 0.000 | | 43 | root | localhost:60890 | fog | Sleep | 5 | | NULL | 0.000 | | 44 | root | localhost:60902 | fog | Sleep | 514 | | NULL | 0.000 | | 45 | root | localhost:60914 | fog | Sleep | 537 | | NULL | 0.000 | | 46 | root | localhost:60950 | fog | Sleep | 55 | | NULL | 0.000 | | 47 | root | localhost:32782 | fog | Sleep | 152 | | NULL | 0.000 | | 49 | root | localhost:33290 | fog | Sleep | 1 | | NULL | 0.000 | | 51 | root | localhost:33474 | fog | Sleep | 0 | | NULL | 0.000 | | 52 | root | localhost:33914 | fog | Sleep | 1 | | NULL | 0.000 | | 53 | root | localhost:33924 | fog | Sleep | 0 | | NULL | 0.000 | | 56 | root | localhost:33968 | fog | Sleep | 0 | | NULL | 0.000 | | 58 | root | localhost:33984 | fog | Sleep | 0 | | NULL | 0.000 |Just FYI there is about 120 lines of this that is the same.
I’ve also upped the CPU and the Memory just to test but it is the same result, which is why I’m thinking its more an issue with the FOG update than 32bit vs 64bit.
As well as errors in the HTTPD log listing PHP warnings.
[Wed Apr 27 11:08:21.707362 2016] [:error] [pid 6824] [client 10.254.3.15:50021] PHP Warning: implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67 [Wed Apr 27 11:08:21.728167 2016] [:error] [pid 7001] [client 10.254.1.171:60754] PHP Warning: implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67Thanks everyone so far for the help
Edit: forgot the CPU from htop
-
I’m linking all forum threads I can find about high CPU utilization right here for this thread and future reference. I also put them into the wiki here:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_Web_Interfacehttps://forums.fogproject.org/topic/6020/fog-svn-5020-and-above-cpu-hammered-thread/20?page=2
https://forums.fogproject.org/topic/6469/tons-of-httpd-processes
https://forums.fogproject.org/topic/6940/high-cpu-fog-services-after-update-r5029-v6759
https://forums.fogproject.org/topic/7234/after-update-of-fog-trunk-php-errors
https://forums.fogproject.org/topic/7215/high-cpu-php-errors-after-update-to-trunk-github-7234
-
Awesome, thanks.
Also just FYI after doing a fresh install of CentOS 7 that has been updated with the latest SVN version of Fog Version: 7332
I’m also getting the same errors from httpd[Wed Apr 27 13:30:18.351161 2016] [:error] [pid 12236] [client 10.254.14.101:53436] PHP Warning: session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/user.class.php on line 113 [Wed Apr 27 13:30:28.607672 2016] [:error] [pid 9846] [client 10.254.14.101:53470] PHP Warning: implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67 [Wed Apr 27 13:30:28.665747 2016] [:error] [pid 9846] [client 10.254.14.101:53470] PHP Warning: session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/user.class.php on line 113 [Wed Apr 27 13:30:38.938836 2016] [:error] [pid 12215] [client 10.254.14.101:53507] PHP Warning: implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67 [Wed Apr 27 13:30:38.996816 2016] [:error] [pid 12215] [client 10.254.14.101:53507] PHP Warning: session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/user.class.php on line 113 [Wed Apr 27 13:30:49.257933 2016] [:error] [pid 9864] [client 10.254.14.101:53541] PHP Warning: implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67 [Wed Apr 27 13:30:49.316223 2016] [:error] [pid 9864] [client 10.254.14.101:53541] PHP Warning: session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/user.class.php on line 113 -
Hopefully its ok I am loading this thread up with screenshots of issues?
Ok, with some unscientific testing on the new server, I’m also getting higher CPU with only 2 hosts that have been registered.
Before I registered the hosts the CPU was near 0% now it is quite a bit higher.I am suspecting the Fog Clients are basically flooding the FOG server as it seems to do a request every few seconds.
The Fog client logs are also filling up pretty quickly.I’m basically getting the following over and over and over again until the fog.log starts again.
27/04/2016 2:09 PM Middleware::Communication Response: Invalid host certificate 27/04/2016 2:09 PM Middleware::Communication URL: http://10.254.14.101/fog/management/other/ssl/srvpublic.crt 27/04/2016 2:09 PM Data::RSA FOG Server CA cert found 27/04/2016 2:09 PM Middleware::Authentication Cert OK 27/04/2016 2:09 PM Middleware::Communication POST URL: http://10.254.14.101/fog/management/index.php?sub=authorize 27/04/2016 2:09 PM Middleware::Communication Response: Success 27/04/2016 2:09 PM Middleware::Authentication Authenticated 27/04/2016 2:09 PM Middleware::Communication URL: http://10.254.14.101/fog/service/hostname.php?moduleid=hostnamechanger&mac=98:90:96:B3:A6:5D||00:00:00:00:00:00:00:E0&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1&newService=1This is with a fresh Centos 7 64bit with 4 vCPUs with Fog Version: 7332
The following is just from 2 fog clients on a standard Centos 7
from apachetop
last hit: 04:46:03 atop runtime: 0 days, 00:08:15 04:46:04 All: 5675 reqs ( 11.5/sec) 3691.2K ( 7636.0B/sec) 666.0B/req 2xx: 5674 (100.0%) 3xx: 1 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%) R ( 30s): 350 reqs ( 11.7/sec) 225.4K ( 7695.3B/sec) 659.6B/req 2xx: 350 ( 100%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%) REQS REQ/S KB KB/S URL 115 3.83 0.6 0.0*/fog/service/greenfog.php 115 3.83 188.6 6.3 /fog/management/other/ssl/srvpublic.crt 114 3.80 36.3 1.2 /fog/management/index.php 3 0.10 0.0 0.0 /fog/service/servicemodule-active.php 1 0.03 0.0 0.0 /fog/service/snapins.checkin.php 1 0.03 0.0 0.0 /fog/service/Printers.php 1 0.03 0.0 0.0 /fog/service/printerlisting.phpHopefully this helps narrow it down?
