After update of Fog Trunk - PHP Errors

[RipAU]

Hi, Guys.

This is more an FYI I guess.

I have found that after successfully updating the FOG via SVN to FOG Version: 7254
That I am seeing the following error:

[Fri Apr 22 15:25:28.814288 2016] [:error] [pid 899] [client 10.254.13.91:62916] PHP Warning:  Cannot modify header information - headers already sent by (output started at /var/www/html/fog/lib/fog/page.class.php:176) in /var/www/html/fog/lib/fog/page.class.php on line 179, referer: http://fog2/fog/management/index.php?node=report&sub=imaging-log
[Fri Apr 22 15:25:28.814349 2016] [:error] [pid 899] [client 10.254.13.91:62916] PHP Warning:  Cannot modify header information - headers already sent by (output started at /var/www/html/fog/lib/fog/page.class.php:176) in /var/www/html/fog/lib/fog/page.class.php on line 180, referer: http://fog2/fog/management/index.php?node=report&sub=imaging-log

I am on Fedora 22 with PHP7 from the remi-repo but I also saw this error on the previous php version (just updated it) which was php5 I think.

As a side note I’ve also found the CPU usage had jumped quite a bit since the previous trunk version I used (back in december It was updated)

I’m not sure all of this is related?

Thanks guys.

[Wayne Workman]

A jump in CPU load is definitely of concern. I have marked this thread as Unsolved.

Can you please provide details about your environment? How many hosts? What fog client are you using, old or new?

Also - if you are using the new client, you could try throwing all hosts into a group, and using that group to reset encryption for them all. This tends to solve the load issues here lately.

[Tom Elliott]

Please update to the latest. The headre issue is already corrected for.

[RipAU]

Perfect, I’ll update that now, thanks
This is the screenshot prior to the update. As you can see if mainly the httpd process spawning 8 or so processes at 8 or 9% CPU usage each.

I’ll let you know how it goes.

Cheers,

[RipAU]

Ok, Just updated and did a reset on the encryption data.
We only have about 300 hosts on this version of FOG at this stage.

The CPU usage is still really high and I am now getting the following Error in the HTTPD logs.

[Tue Apr 26 09:03:40.630623 2016] [:error] [pid 1225] [client 10.254.13.91:61830] PHP Warning:  curl_setopt_array(): Array keys must be CURLOPT constants or equivalent integer values in /var/www/html/fog/lib/fog/fogurlrequests.class.php on line 99, referer: http://fog2/fog/management/index.php?node=home

This is being repeated over and over.

[RipAU]

Sorry I tried having the image in the same post but it was showing up broken.

This is the current load since updating to the latest Trunk.

There are about 300 Hosts being managed by FOG at this stage on Fedora 22 with PHP 7 from the Remi Repository.
They are all using the new fog client as well. (I also reset the encryption data on all hosts after the update, this didn’t seem to change the load at all)
Server is running on VMware ESXi.

Cheers,

[Wayne Workman]

@RipAU What does the number in the cloud say?

[RipAU]

Fog Version 7302

[Tom Elliott]

@RipAU errors should be no more. I can only hope the cpu will settle down as it doesn’t have to try writing the error to the log as well.

[RipAU]

Ok, Just did an update to the latest trunk (in fog version its listed as 7306)
I have lost the graph that lists the amount of storage left and when clicking on the name it has access denied and wrong user name and password listed. I’m also receiving the following errors in HTTPD (different ones this time )

[Tue Apr 26 11:13:35.131109 2016] [:error] [pid 1499] [client 10.254.1.59:55287] PHP Fatal error:  Uncaught Error: Using $this when not in object context in /var/www/html/fog/lib/fog/fogpage.class.php:743\nStack trace:\n#0 /var/www/html/fog/commons/init.php(141): FOGPage::authorize()\n#1 /var/www/html/fog/commons/base.inc.php(10): require_once('/var/www/html/f...')\n#2 /var/www/html/fog/management/index.php(2): require('/var/www/html/f...')\n#3 {main}\n  thrown in /var/www/html/fog/lib/fog/fogpage.class.php on line 743

CPU usage is still hovering very high.

Thanks again for the help

[RipAU]

I decided to check some fog.log files on one of my hosts and I am also seeing this.

26/04/2016 11:27 AM Client-Info Version: 0.9.12
 26/04/2016 11:27 AM Middleware::Communication URL: http://fog2/fog/management/other/ssl/srvpublic.crt
 26/04/2016 11:27 AM Data::RSA FOG Server CA cert found
 26/04/2016 11:27 AM Middleware::Authentication Cert OK
 26/04/2016 11:27 AM Middleware::Communication POST URL: http://fog2/fog/management/index.php?sub=authorize
 26/04/2016 11:27 AM Middleware::Communication ERROR: Failed to POST data
 26/04/2016 11:27 AM Middleware::Communication ERROR: The remote server returned an error: (500) Internal Server Error.
 26/04/2016 11:27 AM Middleware::Authentication ERROR: Could not authenticate
 26/04/2016 11:27 AM Middleware::Authentication ERROR: Object reference not set to an instance of an object.
 26/04/2016 11:27 AM Service Sleeping for 120 seconds```

[Wayne Workman]

install apachetop and then run it.
apachetop

It will give you the request count of every file served by Apache, and the amount of data transmitted. This will help us at least narrow down exactly what file is causing your problems.

sample output:

[RipAU]

Just checked then.

I’m not sure I can see an issue? with the following.

last hit: 03:35:45         atop runtime:  0 days, 00:02:05             03:35:46
All:         2043 reqs (  16.3/sec)        467.5K ( 3829.8B/sec)     234.3B/req
2xx:    1817 (88.9%) 3xx:      52 ( 2.5%) 4xx:     2 ( 0.1%) 5xx:   172 ( 8.4%)
R ( 30s):     476 reqs (  15.9/sec)        116.9K ( 3990.2B/sec)     251.5B/req
2xx:     424 (89.1%) 3xx:       6 ( 1.3%) 4xx:     0 ( 0.0%) 5xx:    46 ( 9.7%)

 REQS REQ/S    KB KB/S URL
  144  4.80   0.7  0.0*/fog/service/servicemodule-active.php
   94  3.13   4.1  0.1 /fog/management/index.php
   66  2.20   2.4  0.1 /fog/status/bandwidth.php
   49  1.63  80.3  2.7 /fog/management/other/ssl/srvpublic.crt
   43  1.43   1.5  0.0 /fog/service/Printers.php
   14  0.48   0.1  0.0 /fog/service/greenfog.php
   13  0.43  27.5  0.9 /fog/service/printerlisting.php
   13  0.43   0.1  0.0 /fog/service/snapins.checkin.php
   11  0.37   0.1  0.0 /fog/service/hostname.php
   10  0.36   0.0  0.0 /fog/service/jobs.php
   10  0.36   0.1  0.0 /fog/service/getversion.php
    6  0.21   0.0  0.0 /fog/index.php
    3  0.10   0.0  0.0 *

looks like most requests are being sent to servicemodule-active.php ?

[RipAU]

Hi, Guys.

As a bit of a hopeful move I updated to the latest SVN update (FOG Version: 7330)
I’ve found the web interface near unusable because of the load that is now being put on the server(Huge number of httpd processes are now being spawned taking up most of the CPU and quite allot of mysqld processes as well)

I’ll roll back to a working version of the SVN with a snapshot. Happy to try to update and test if this helps

I’ve taken several screenshots that hopefully help.

just to clarify this is with Fedora 22 Server, PHP7 from Remi Repository with FOG version 7330 on an VMware ESXi server.
Previously it was working perfectly

Cheers.

[Sebastian Roth]

@RipAU You are seeing a lot of requests to srvpublic.crt which makes me think that this might be just a peek load while a lot of the clients redo the certificate stuff for secure communication to the FOG server.

Not sure if this was asked already but do you know which version you updated from that was perfectly working for you?

[RipAU]

@Sebastian-Roth

Yep just did a revert back to the previous version then. It is Fog Version: 6207
The CPU usage is a little higher than I’d like (fluctuates from 20% to 65% depending on the amount of traffic to the server, which is one of the reasons I was going to update) but otherwise has been working perfectly with no real issues.
Server load generally seems fine as well - load average: 0.88, 1.74, 1.55

[Wayne Workman]

@RipAU While the load is high, go into MySQL and issue this command:

SHOW FULL PROCESSLIST

and give us the output.

Maybe it’s time for you to tell us about the ESXi hosting machine, and the resources you have assigned to FOG.

Specifically, the host’s core speed and processor type. and how many cores you assigned to FOG, and how much RAM.

[RipAU]

@Wayne-Workman
Yep no worries.

The server is ESXi 6.
At this stage it has 1vCPU has been assigned with 2GB of RAM.
Server CPUS are Intel Xenon CPU E5-2630 @ 2.4ghz

Fog Server at this stage only has about 300 hosts (as soon as everything is smoothed out I was going to migrate the rest of our hosts over and this will be about 1100)

I can revert to the updated fog version of the snapshot after It was updated and run that command.
I’ll do that now.

Cheers,

[Wayne Workman]

@RipAU Oh… well there’s your problem. Fog sucks with one core.

give it at least two.

But don’t just bump it to two. Now you need to tear the whole thing down and start over - because your using a 32 bit operating system.

give it 2, and then re-install the OS.

[RipAU]

@Wayne-Workman

I can easily up the CPU and RAM - The server is the 64bit version of Fedora.

 4.2.7-200.fc22.x86_64 #1 SMP Thu Dec 10 03:28:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux