Climbing Server Load

Joseph Hales · Jan 27, 2015, 8:37 PM

since updating to 2929 my server load has been steadily climbing throughout the day. I upgraded early yesterday and by this morning the system was unresponsive to tftp and web access and console access was slow. I rebooted and was back to normal but at this point after 6.5 hours up time server load average is 32 to 33 and pages are slow to respond in the web interface.

Joseph Hales · Jan 27, 2015, 8:55 PM

Apache log last couple of entries

[Tue Jan 27 14:15:21 2015] [error] [client 10.41.50.243] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 7 bytes) in /var/www/fog/lib/fog/FOGController.class.php on line 60, referer: [url]http://10.200.10.150/fog/management/index.php?node=host[/url]
[Tue Jan 27 14:33:14 2015] [error] [client 10.200.32.7] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 44 bytes) in /var/www/fog/lib/fog/FOGController.class.php on line 60, referer: [url]http://fog/fog/management/index.php?node=host[/url]
[Tue Jan 27 14:38:07 2015] [error] [client 10.111.50.31] File does not exist: /var/www/favicon.ico
[Tue Jan 27 14:38:07 2015] [error] [client 10.111.50.31] File does not exist: /var/www/favicon.ico
[Tue Jan 27 14:43:29 2015] [error] [client 10.111.50.31] PHP Warning: trim() expects parameter 1 to be string, array given in /var/www/fog/lib/fog/MACAddress.class.php on line 16, referer: [url]http://10.200.10.150/fog/management/index.php?node=host&sub=add[/url]
[Tue Jan 27 14:43:29 2015] [error] [client 10.111.50.31] PHP Warning: trim() expects parameter 1 to be string, array given in /var/www/fog/lib/fog/MACAddress.class.php on line 16, referer: [url]http://10.200.10.150/fog/management/index.php?node=host&sub=add[/url]
[Tue Jan 27 14:46:29 2015] [error] [client 10.109.48.34] PHP Warning: trim() expects parameter 1 to be string, array given in /var/www/fog/lib/fog/MACAddress.class.php on line 16, referer: [url]http://10.200.10.150/fog/management/index.php?node=host&sub=add[/url]
[Tue Jan 27 14:46:29 2015] [error] [client 10.109.48.34] PHP Warning: trim() expects parameter 1 to be string, array given in /var/www/fog/lib/fog/MACAddress.class.php on line 16, referer: [url]http://10.200.10.150/fog/management/index.php?node=host&sub=add[/url]
[Tue Jan 27 14:47:07 2015] [error] [client 10.112.50.63] File does not exist: /var/www/favicon.ico
[Tue Jan 27 14:47:07 2015] [error] [client 10.112.50.63] File does not exist: /var/www/favicon.ico
[Tue Jan 27 14:50:57 2015] [error] [client 10.109.48.34] PHP Warning: trim() expects parameter 1 to be string, array given in /var/www/fog/lib/fog/MACAddress.class.php on line 16, referer: [url]http://10.200.10.150/fog/management/index.php?node=host&sub=add[/url]
[Tue Jan 27 14:50:57 2015] [error] [client 10.109.48.34] PHP Warning: trim() expects parameter 1 to be string, array given in /var/www/fog/lib/fog/MACAddress.class.php on line 16, referer: [url]http://10.200.10.150/fog/management/index.php?node=host&sub=add[/url]

Joseph Hales · Jan 27, 2015, 9:00 PM

Running Ubuntu 12.04 lts
SVN 2929

fog install log follows
…#######:. …,#,… .::##::.
.:###### .:;####:…;#;…
…##… …##;,;##::::.##…
,# …##…##:::## …::
## .::###,##. . ##.::#.:######::.
…##:::###::…#. … .#…#. #…#:::.
…:####:… …##…##::## … #
# . …##:,;##;:::#: … ##…
.# . .:;####;::::.##:::;#:…
# …:;###…

###########################################

FOG

Free Computer Imaging Solution

[url]http://www.fogproject.org/[/url]

Developers:

Chuck Syperski

Jian Zhang

Peter Gilchrist

Tom Elliott

GNU GPL Version 3

###########################################

Version: 1.3.0 Installer/Updater

Found FOG Settings from previous install at: /opt/fog/.fogsettings
Performing upgrade using these settings…

Starting Debian / Ubuntu / Kubuntu / Edubuntu Installtion.

#####################################################################

FOG now has everything it needs to setup your server, but please
understand that this script will overwrite any setting you may
have setup for services like DHCP, apache, pxe, tftp, and NFS.

It is not recommended that you install this on a production system
as this script modifies many of your system settings.

This script should be run by the root user on Redhat or with sudo on Ubuntu.

** Notice ** Redhat users will need to disable SELinux and iptables in
order to use FOG
Please see our wiki for more information at [url]http://www.fogproject.org/wiki[/url]

Here are the settings FOG will use:
Base Linux: Debian
Detected Linux Distribution: Ubuntu
Installation Type: Normal Server
Server IP Address: 10.200.10.150
DHCP router Address:
DHCP DNS Address: 10.200.10.33
Interface: bond0
Using FOG DHCP: 0
Internationalization: 1
Image Storage Location: /images
Donate: 0

     DHCP will NOT be setup but you must setup your
     current DHCP server to use FOG for PXE services.

     On a Linux DHCP server you must set:
         next-server

     On a Windows DHCP server you must set:
         option 066 & 067

       Option 066 is the IP of the FOG Server: (e.g. 10.200.10.150)
       Option 067 is the undionly.kpxe file: (e.g. undionly.kpxe)

Are you sure you wish to continue (Y/N)
Installation Started…

Installing required packages, if this fails
make sure you have an active internet connection.

Preparing apt-get
Packages to be installed: apache2 php5 php5-json php5-gd php5-cli php5-mysql php5-curl mysql-server mysql-client tftpd-hpa tftp-hpa nfs-kernel-server vsftpd net-tools wget xinetd sysv-rc-conf tar gzip build-essential cpp gcc g++ m4 htmldoc lftp openssh-server php-gettext php5-mcrypt language-pack-it language-pack-en language-pack-es language-pack-zh-hans
Skipping package: apache2 (Already installed)
Skipping package: php5 (Already installed)
Skipping package: php5-common (Already installed)
Skipping package: php5-gd (Already installed)
Skipping package: php5-cli (Already installed)
Skipping package: php5-mysql (Already installed)
Skipping package: php5-curl (Already installed)
Skipping package: mysql-server (Already installed)
Skipping package: mysql-client (Already installed)
Skipping package: tftpd-hpa (Already installed)
Skipping package: tftp-hpa (Already installed)
Skipping package: nfs-kernel-server (Already installed)
Skipping package: vsftpd (Already installed)
Skipping package: net-tools (Already installed)
Skipping package: wget (Already installed)
Skipping package: xinetd (Already installed)
Skipping package: sysv-rc-conf (Already installed)
Skipping package: tar (Already installed)
Skipping package: gzip (Already installed)
Skipping package: build-essential (Already installed)
Skipping package: cpp (Already installed)
Skipping package: gcc (Already installed)
Skipping package: g++ (Already installed)
Skipping package: m4 (Already installed)
Skipping package: htmldoc (Already installed)
Skipping package: lftp (Already installed)
Skipping package: openssh-server (Already installed)
Skipping package: php-gettext (Already installed)
Skipping package: php5-mcrypt (Already installed)
Skipping package: language-pack-it (Already installed)
Skipping package: language-pack-en (Already installed)
Skipping package: language-pack-es (Already installed)
Skipping package: language-pack-zh-hans (Already installed)

Confirming package installation.

Checking package: apache2…OK
Checking package: php5…OK
Checking package: php5-json…OK
Checking package: php5-gd…OK
Checking package: php5-cli…OK
Checking package: php5-mysql…OK
Checking package: php5-curl…OK
Checking package: mysql-server…OK
Checking package: mysql-client…OK
Checking package: tftpd-hpa…OK
Checking package: tftp-hpa…OK
Checking package: nfs-kernel-server…OK
Checking package: vsftpd…OK
Checking package: net-tools…OK
Checking package: wget…OK
Checking package: xinetd…OK
Checking package: sysv-rc-conf…OK
Checking package: tar…OK
Checking package: gzip…OK
Checking package: build-essential…OK
Checking package: cpp…OK
Checking package: gcc…OK
Checking package: g++…OK
Checking package: m4…OK
Checking package: htmldoc…OK
Checking package: lftp…OK
Checking package: openssh-server…OK
Checking package: php-gettext…OK
Checking package: php5-mcrypt…OK
Checking package: language-pack-it…OK
Checking package: language-pack-en…OK
Checking package: language-pack-es…OK
Checking package: language-pack-zh-hans…OK

Configuring services.

Setting up and starting MySql…OK
Backing up user reports…OK
Setting up and starting Apache Web Server…OK
Changing permissions on apache log files…OK
Downloading kernels and inits…OK
Copying back any custom hook files.
Copying back any custom report files.
You still need to install/update your database schema.
This can be done by opening a web browser and going to:

[url]http://10.200.10.150/fog/management[/url]

Press [Enter] key when database is updated/installed.

Setting up storage…OK
Setting up and starting DHCP Server…Skipped
Setting up and starting TFTP and PXE Servers…OK
Setting up and starting VSFTP Server…OK
Setting up sudo settings…OK
Setting up FOG Snapins…OK
Setting up and building UDPCast…OK
Installing init scripts…OK
Setting up FOG Services…OK
Setting up FOG Utils…OK
Starting FOG Multicast Management Server…OK
Starting FOG Image Replicator Server…OK
Starting FOG Task Scheduler Server…OK
Starting FOG Snapin Replicator Server…OK
Configuring Fresh Clam…OK
Setting up and starting NFS Server…OK

Setup complete!

You can now login to the FOG Management Portal using
the information listed below. The login information
is only if this is the first install.

This can be done by opening a web browser and going to:

  [url]http://10.200.10.150/fog/management[/url]

  Default User:
         Username: fog
         Password: password

Wolfbane8653 · Jan 27, 2015, 9:01 PM

[SIZE=13px][FONT=Ubuntu][COLOR=#555555]Latest svn version is 2933

Please upgrade. Tom patched a few mistakes and this could be related. [/COLOR][/FONT][/SIZE]

Junkhacker · Jan 27, 2015, 9:01 PM

[url]http://fogproject.org/forum/threads/session-cleaner-takes-long-and-slows-host.11839/#post-37904[/url]

edited for bad url

Wolfbane8653 · Jan 27, 2015, 9:06 PM

I believe Junk is refering to the PHP Session. cleanup portions.
[url]http://fogproject.org/wiki/index.php/Fog_Tweaks#PHP_Session_Purging[/url]

Joseph Hales · Jan 27, 2015, 10:03 PM

I applied the cleanup edit and latest trunk 2933 and rebooted it seems stable I will keep an eye on it I did notice the bandwidth transmit is not showing anything but receive is any idea why it would stop tracking?

Tom Elliott · Jan 27, 2015, 10:12 PM

Hey also try persistent db for more responsive GUI access

Joseph Hales · Jan 28, 2015, 1:48 PM

15 hours uptime and server load is still below 2 looks like we have a winner I will try the presistant DB access hack next after the last step do i need to restart the DB again or only after step 3?
[SIZE=4][B][FONT=sans-serif][COLOR=#000000]Persistent DB Connection[/COLOR][/FONT][/B][/SIZE]

[LIST]
[]Use a [B]Persistent DB Connection[/B]. This uses the already established connection to get the data, rather than reopening the socket or TCP link
[LIST=1]
[]Open this file [B]/var/www/fog/lib/fog/Config.class.php[/B]
[]Go to line:
[FONT=monospace]define(‘DATABASE_HOST’, ‘localhost’);[/FONT]
[]Change to:
[FONT=monospace]define(‘DATABASE_HOST’, ‘p:127.0.0.1’);[/FONT]
[]Restart mysql:
[FONT=monospace]service mysql restart[/FONT]
[]Open this file [B]/opt/fog/.fogsettings[/B] (so updates/upgrades to overwrite the settings above)
[]Go to line:
[FONT=monospace]snmysqlhost=“”;[/FONT]
[]Change to:
[FONT=monospace]snmysqlhost=“p:127.0.0.1”;[/FONT]
[/LIST]
[*]If you DB is on another server, use p:[ipaddress of DB server].
[/LIST]

Tom Elliott · Jan 28, 2015, 1:53 PM

why would you have to restart mysql at all?