Posts made by Adam Taylor

NOTE, we did run into a issue you all need to be aware of. There is a table called “groups”. Apparently that is now a protected name in MySQL 8 and you HAVE to put ticks around the groups table name for commands or it will throw an SQL syntax error. Took me a while to figure out that one!

OMG…why is the default not INNODB?

I cleared out the invalid times and…10% CPU instead of 70+% for mysqld now…wow…

The system is super happy now but we have some testing to do to make sure we stay good but as I said…OMG that is MUCH better!!!

I found this: https://forums.fogproject.org/topic/14254/high-cpu-usage/23?page=2

Working on it now.

We had done tunning on the old box but a bunch of our tune options are no longer available in MySQL 8. The query_cache_size variable is what what made it work on the old box well enough…that is no longer an option in 8 however.

I’m Don’s boss and I am the original person who set this up back in the .2X days…LONG time ago…LOL I have been out of the loop for a while and man…LOTS of changes since I last looked at it.

In a nut shell, the DB is just slowing to a unusable crawl once all the clients start checking in and the fpm processes just stall because of it.

I sware when I set it up the engine was INNODB on a very old box…but somewhere it changed back to MyISAM…now we know the issue…MyISAM isn’t capable of what we are asking of it…

So george1421…how we get it over to INNODB becuase I don’t have a clue TBH. DBs make my head hurt…lol

4168 == sanity!

Never get past about 30% now.

Adam

From this point on, I’m only going to post what is asked of me to post because i don’t want to flood the thread

I just found it odd that there was that many requests for missing files.

Adam

These are some of the access_httpd.log files are not there:

access(“/var/www/html/fog/lib/plugins/ldap/reg-task/Template.class.php”, F_OK) = -1 ENOENT (No such file or directory)
access(“/var/www/html/fog/lib/plugins/ldap/service/Template.class.php”, F_OK) = -1 ENOENT (No such file or directory)

Something is looking for alot of files that are not there… There was 9727 requests for files that don’t exist.

Running strace on an apache process give me this:\

Process 28758 attached - interrupt to quit
^CProcess 28758 detached
% time seconds usecs/call calls errors syscall

---

28.32 0.002985 1 2262 poll
18.47 0.001947 0 26340 getdents
18.10 0.001908 4 505 brk
13.05 0.001375 0 32536 90 lstat
7.50 0.000790 0 31140 399 stat
4.72 0.000497 0 11704 open
2.57 0.000271 0 9684 8663 access
2.25 0.000237 0 11723 close
1.48 0.000156 0 11107 33 read
0.73 0.000077 0 10781 lseek
0.73 0.000077 0 771 sendto
0.72 0.000076 0 9307 fstat
0.30 0.000032 0 893 munmap
0.28 0.000030 2 16 write
0.28 0.000029 2 16 writev
0.27 0.000028 0 2246 recvfrom
0.23 0.000024 0 893 mmap
0.00 0.000000 0 15 rt_sigaction
0.00 0.000000 0 15 rt_sigprocmask
0.00 0.000000 0 16 pwrite
0.00 0.000000 0 77 setitimer
0.00 0.000000 0 16 accept
0.00 0.000000 0 16 shutdown
0.00 0.000000 0 16 getsockname
0.00 0.000000 0 32 semop
0.00 0.000000 0 79 fcntl
0.00 0.000000 0 15 flock
0.00 0.000000 0 60 getcwd
0.00 0.000000 0 31 chdir
0.00 0.000000 0 15 getuid
0.00 0.000000 0 16 epoll_wait

---

100.00 0.010539 162343 9185 total

This was done for 30 seconds.

The err is access make me look at why access is uphappy. See next post.

Yes, this was just after updating.

httpd -V

Server version: Apache/2.2.15 (Unix)
Server built: Aug 18 2015 02:00:22
Server’s Module Magic Number: 20051115:25
Server loaded: APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)

Server-status

404 not found

mysqladmin -u root -p -i 1 processlist;

±------±-----------±-------------------±----±--------±-----±------±-----------------+
| Id | User | Host | db | Command | Time | State | Info |
±------±-----------±-------------------±----±--------±-----±------±-----------------+
| 11 | root | localhost:51961 | fog | Sleep | 75 | | |
| 14 | root | localhost:51964 | fog | Sleep | 548 | | |
| 15 | root | localhost:51965 | fog | Sleep | 393 | | |
| 2853 | fogstorage | classroomfog:52337 | fog | Sleep | 231 | | |
| 3019 | fogstorage | 10.32.8.220:36559 | fog | Sleep | 182 | | |
| 57280 | root | localhost:53739 | fog | Sleep | 2 | | |
| 57307 | root | localhost:53780 | fog | Sleep | 0 | | |
| 57328 | root | localhost:53833 | fog | Sleep | 0 | | |
| 57330 | root | localhost:53839 | fog | Sleep | 1 | | |
| 57334 | root | localhost:53843 | fog | Sleep | 2 | | |
| 57338 | root | localhost:53847 | fog | Sleep | 1 | | |
| 57339 | root | localhost:53848 | fog | Sleep | 1 | | |
| 57350 | root | localhost:53859 | fog | Sleep | 1 | | |
| 57364 | root | localhost:53881 | fog | Sleep | 0 | | |
| 57366 | root | localhost:53883 | fog | Sleep | 2 | | |
| 57367 | root | localhost:53885 | fog | Sleep | 2 | | |
| 57368 | root | localhost:53886 | fog | Sleep | 0 | | |
| 57376 | root | localhost:53894 | fog | Sleep | 0 | | |
| 57381 | root | localhost:53908 | fog | Sleep | 0 | | |
| 57383 | root | localhost:53910 | fog | Sleep | 0 | | |
| 57387 | root | localhost:53914 | fog | Sleep | 1 | | |
| 57395 | root | localhost:53925 | fog | Sleep | 0 | | |
| 57398 | root | localhost:53966 | fog | Sleep | 2 | | |
| 57399 | root | localhost:53967 | fog | Sleep | 1 | | |
| 57400 | root | localhost:53968 | fog | Sleep | 0 | | |
| 57401 | root | localhost:53969 | fog | Sleep | 2 | | |
| 57402 | root | localhost:53970 | fog | Sleep | 2 | | |
| 57405 | root | localhost:53980 | fog | Sleep | 0 | | |
| 57406 | root | localhost | | Query | 0 | | show processlist |
| 57407 | root | localhost:53983 | fog | Sleep | 1 | | |
| 57408 | root | localhost:53984 | fog | Sleep | 1 | | |
| 57409 | root | localhost:53985 | fog | Sleep | 1 | | |
| 57410 | root | localhost:53986 | fog | Sleep | 0 | | |
| 57411 | root | localhost:53987 | fog | Sleep | 1 | | |
±------±-----------±-------------------±----±--------±-----±------±-----------------+

Top

top - 15:06:50 up 1 day, 5:26, 2 users, load average: 38.21, 23.90, 20.41
Tasks: 258 total, 42 running, 215 sleeping, 0 stopped, 1 zombie
Cpu(s): 89.1%us, 10.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 3899360k total, 1538960k used, 2360400k free, 82128k buffers
Swap: 4194300k total, 145492k used, 4048808k free, 320464k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2027 mysql 20 0 2404m 79m 3300 S 23.9 2.1 334:03.04 mysqld
14615 apache 20 0 343m 20m 3768 R 9.2 0.5 0:05.08 httpd
14454 apache 20 0 343m 21m 3764 R 8.8 0.6 0:05.61 httpd
14457 apache 20 0 343m 20m 3772 R 8.8 0.5 0:05.60 httpd
14475 apache 20 0 343m 20m 3768 R 8.8 0.5 0:05.23 httpd
14602 apache 20 0 343m 20m 3828 R 8.8 0.5 0:04.97 httpd
14617 apache 20 0 343m 20m 3768 R 8.8 0.5 0:05.04 httpd
14216 apache 20 0 343m 21m 3768 R 8.5 0.6 0:27.19 httpd
14384 apache 20 0 343m 20m 3768 R 8.5 0.5 0:10.49 httpd
14435 apache 20 0 343m 20m 3764 R 8.5 0.5 0:05.88 httpd
14459 apache 20 0 331m 10m 3764 S 8.5 0.3 0:05.70 httpd
14460 apache 20 0 343m 20m 3704 R 8.5 0.5 0:05.41 httpd
14472 apache 20 0 343m 20m 3768 R 8.5 0.5 0:05.33 httpd
14473 apache 20 0 343m 20m 3764 R 8.5 0.5 0:05.35 httpd
14613 apache 20 0 343m 21m 3768 R 8.5 0.6 0:04.74 httpd
14672 apache 20 0 343m 21m 3764 R 8.5 0.6 0:03.01 httpd
14707 apache 20 0 343m 20m 3552 R 8.5 0.5 0:00.61 httpd
14708 apache 20 0 343m 20m 3552 R 8.5 0.5 0:00.62 httpd
14408 apache 20 0 336m 14m 3768 S 8.2 0.4 0:08.21 httpd
14696 apache 20 0 336m 14m 3764 S 8.2 0.4 0:00.82 httpd
14166 apache 20 0 343m 20m 3772 R 7.9 0.5 0:30.77 httpd
14438 apache 20 0 343m 20m 3768 R 7.5 0.5 0:05.64 httpd
14443 apache 20 0 343m 20m 3768 R 7.5 0.5 0:05.60 httpd
14476 apache 20 0 343m 20m 3764 R 7.5 0.5 0:05.38 httpd
14704 apache 20 0 343m 20m 3760 R 7.5 0.6 0:00.71 httpd
14368 apache 20 0 331m 10m 3768 S 7.2 0.3 0:10.54 httpd
14455 apache 20 0 343m 20m 3768 R 7.2 0.5 0:05.52 httpd
14356 apache 20 0 343m 21m 3768 R 6.9 0.6 0:10.81 httpd
14411 apache 20 0 336m 14m 3768 S 6.9 0.4 0:07.78 httpd
14720 apache 20 0 338m 16m 3552 R 6.9 0.4 0:00.52 httpd
14481 apache 20 0 333m 11m 3768 S 6.5 0.3 0:05.50 httpd
14717 apache 20 0 343m 20m 3552 R 6.5 0.5 0:00.55 httpd
14718 apache 20 0 343m 20m 3552 R 6.5 0.5 0:00.55 httpd
14471 apache 20 0 338m 16m 3768 R 6.2 0.4 0:05.34 httpd
14715 apache 20 0 343m 20m 3704 R 6.2 0.5 0:00.52 httpd
14719 apache 20 0 343m 20m 3544 R 6.2 0.5 0:00.50 httpd
14721 apache 20 0 336m 14m 3552 R 6.2 0.4 0:00.51 httpd
14722 apache 20 0 338m 16m 3552 R 6.2 0.4 0:00.52 httpd
14442 apache 20 0 333m 11m 3768 R 5.9 0.3 0:05.58 httpd
14485 apache 20 0 332m 11m 3768 R 5.9 0.3 0:05.33 httpd
14665 apache 20 0 334m 12m 3768 R 5.9 0.3 0:03.34 httpd
14710 apache 20 0 337m 15m 3760 R 5.9 0.4 0:00.63 httpd
14703 apache 20 0 331m 9988 3552 S 5.6 0.3 0:00.69 httpd
14365 apache 20 0 337m 16m 3768 S 5.2 0.4 0:10.44 httpd
14478 apache 20 0 332m 10m 3768 R 5.2 0.3 0:05.41 httpd
14603 apache 20 0 331m 10m 3768 S 5.2 0.3 0:04.88 httpd
14716 apache 20 0 337m 15m 3760 S 5.2 0.4 0:00.48 httpd
14428 apache 20 0 331m 10m 3768 R 4.9 0.3 0:06.22 httpd
14477 apache 20 0 331m 10m 3768 S 4.9 0.3 0:05.22 httpd
14486 apache 20 0 333m 11m 3768 R 4.9 0.3 0:05.25 httpd
14456 apache 20 0 331m 10m 3768 R 4.6 0.3 0:05.40 httpd

Question.

Would the old fog client be causing this? We noticed the new MSI fog client but we are still running the old one.

Adam

Can you ask him what OS, apache version, mysql version he is running for a can compare?

Thanks,

Adam

I am still seeing 100% usage for me to after the last update with mysql using~30% and httpd using the rest with LOTS of processes.

Adam

Also, that 100% is always ~30% for mysql and the rest is chewed up with httpd processes.

Adam

The system is still getting hammered and all four cores are at 100% after a complete reboot.

Adam

I changed the DB connect from persistant to non persistant to test and see if the connections would then go away.

The connections go up much faster then before (what i expected since no reuse) but once again…they never go down.

That’s all i know to look at/try at the moment. Do appreciate your help in this!

Thanks!

Adam

3306 is all localhost connections.

80 is all different clients from around the campus.

I added the following to the MySQL.class.php file to see how often it was creating a SQL connection:

public function connect()
try
if (!$this->link)
file_put_contents(‘php://stderr’, print_r(DATABASE_HOST, TRUE))

It seems to be getting called 7-10 times per second currently.

Every single connection is p:127.0.0.1 so every connection is persistant…and they are not getting closed would be my guess.

Thanks!

Adam

Right now i have 33525 mysql connections when i run “show status like ‘Con%’;” in mysql cli. And no…that was not a mistype The connections are just not going away.

Total registered hosts are 1892 of which about 1800 have the fog client.

Running lsof, I am a around 20-30 established connections at any one given time with about the same listening. All while the CPU is around 100%. Every once and a while it might spike to around 180 or so but then drop back down within 30 seconds or so.

FOG_HOST_LOOKUP was checked and I just unchecked it to see if that helps. So far I am not seeing a difference after a total system reboot.

I had to restart the mysqld process so that I could get to that setting as the web gui would not do anything. In that 2 minutes from a reboot, I am back upto 297 mysql connections.

BTW, I am running this on RedHat Enterprise 6.

Thanks!

Adam

And i forgot.

The message only happens every once and a while in the error_log file.