High CPU Usage

Arrowtron · Feb 25, 2020, 4:08 AM

FOG Ver: 1.5.7.2
Ubuntu: 19.04
Master Server on VMWare

Registered Hosts on the server: 1098 (maybe 200-400 could be online in this list)

Getting consistent high CPU as shown in the screenshot below

Restarting the server, CPU usage isn’t as bad, eventually goes to 100% the longer it is on
Took a snapshot, upgraded to the latest dev-branch a couple weeks back (i think ~1.5.8.1), CPU usage was still hovering quite high, reverted back
Ran MySQLTuner, results below
Ran Database Maintenance Commands on https://wiki.fogproject.org/wiki/index.php/Troubleshoot_MySQL
I haven’t changed any polling/refresh rates for hosts, I’ve seen on these forums that has helped for others
https://forums.fogproject.org/topic/12392/fog-server-high-cpu/23?_=1582602629372 - I’ve tinkered with the settings below but no drop in CPU

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 35
pm.min_spare_servers = 5
pm.start_servers = 5

I’ve been thinking I should do a complete export of database and start on a new server (perhaps 18.04), import the database. However, the problem might come back if it is something to do with the data

Any other suggestions?

Note: I have removed security suggestions from the below output

 >>  MySQLTuner 1.7.19 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from Debian maintenance account.
[OK] Currently running supported MySQL version 5.7.28-0ubuntu0.19.04.2
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log(0B)
[OK] Log file /var/log/mysql/error.log is readable.
[!!] Log file /var/log/mysql/error.log is empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[OK] /var/log/mysql/error.log doesn't contain any warning.
[OK] /var/log/mysql/error.log doesn't contain any error.
[--] 0 start(s) detected in /var/log/mysql/error.log
[--] 0 shutdown(s) detected in /var/log/mysql/error.log

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in InnoDB tables: 16.0K (Tables: 1)
[--] Data in MyISAM tables: 7.4M (Tables: 54)
[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 18d 0h 53m 4s (1B q [715.502 qps], 25M conn, TX: 989G, RX: 240G)
[--] Reads / Writes: 98% / 2%
[--] Binary logging is disabled
[--] Physical Memory     : 3.8G
[--] Max MySQL memory    : 2.7G
[--] Other process memory: 0B
[--] Total buffers: 192.0M global + 17.1M per thread (151 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 1.1G (29.54% of installed RAM)
[OK] Maximum possible memory usage: 2.7G (70.22% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/1B)
[OK] Highest usage of available connections: 37% (57/151)
[OK] Aborted connections: 0.00%  (275/25414143)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 1B selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (17 temp sorts / 297M sorts)
[!!] Joins performed without indexes: 5219
[!!] Temporary tables created on disk: 97% (25M on disk / 26M total)
[OK] Thread cache hit rate: 98% (386K created / 25M connections)
[OK] Table cache hit rate: 95% (1K open / 1K opened)
[OK] table_definition_cache(1400) is upper than number of tables(334)
[OK] Open file limit used: 22% (1K/5K)
[OK] Table locks acquired immediately: 99% (1B immediate / 1B locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 72B
[--] Sys schema is installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 28.1% (4M used / 16M cache)
[OK] Key buffer size / total MyISAM indexes: 16.0M/4.4M
[OK] Read Key buffer hit rate: 100.0% (13B cached / 3K reads)
[!!] Write Key buffer hit rate: 3.8% (123M cached / 4M writes)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 128.0M/16.0K
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (584565167 hits/ 584565370 total)
[!!] InnoDB Write Log efficiency: 0% (2 hits/ 0 total)
[OK] InnoDB log waits: 0.00% (0 waits / 2 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Restrict Host for 'fogstorage'@% to fogstorage@SpecificDNSorIp
    UPDATE mysql.user SET host ='SpecificDNSorIp' WHERE user='fogstorage' AND host ='%'; FLUSH PRIVILEGES;
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
             (specially the conclusions at the bottom of the page).
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_limit (> 1M, or use smaller result sets)
    join_buffer_size (> 256.0K, or always use indexes with JOINs)
    tmp_table_size (> 16M)
    max_heap_table_size (> 16M)
    innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

george1421 · Feb 25, 2020, 12:38 PM

Some of the testing a few of the big fog installs have done show that switching mysql over from isam to innodb format help with a heavily loaded mysql instance

If you have the ram, with 400 clients on at one time you might want to take your pm.max_children to 50 from 35. For larger installs (500 fog clients and up) some have switched over php-fpm from dynamic clients to static clients so there isn’t the stand up time for new php-fpm children if the server gets hit hard with requests.

Look at the fog client check in time. If its default, its probably 5 minutes. Increase that to 10 or 15 minutes. Understand that increase will slow the the time between issuing a snapin deployment and the client seeing the request. It will have no impact on imaging.

Sebastian Roth · Feb 25, 2020, 7:33 AM

@Arrowtron Overall from what you wrote I would imagine this is happening due to the clients checking in “too often” in your setup. See there are many variables that influence this, some you can alter very easily but others that are just hard facts (like your network speed and load - the longer one request takes the more need to be handled in parallel).

Restarting the server, CPU usage isn’t as bad, eventually goes to 100% the longer it is on

Sure client connections are all dropped and come back over time.

Took a snapshot, upgraded to the latest dev-branch a couple weeks back (i think ~1.5.8.1), CPU usage was still hovering quite high, reverted back

Hmm? FOG 1.5.8.1 is only available since yesterday. You probably were using a different version then.

Ran MySQLTuner, results below

Sorry, I am not an expert on this. Others might give you hints on this. @george1421 @EduardoTSeoane Sure thing, FOG does not do a great job in optimizing the DB backend as of now. This is something we need to pay more attention to in the future.

Ran Database Maintenance Commands on https://wiki.fogproject.org/wiki/index.php/Troubleshoot_MySQL

Good thing to do although it didn’t help.

I haven’t changed any polling/refresh rates for hosts, I’ve seen on these forums that has helped for others

This is the first thing I’d suggest you do to actually find out if it’s the clients causing the load. At least double the checkin time (FOG web UI settings) and wait for half an hour so every client has pulled and adjusted to the new setting.

https://forums.fogproject.org/topic/12392/fog-server-high-cpu/23?_=1582602629372 - I’ve tinkered with the settings below but no drop in CPU

That might be a hint on this being the database backend causing it and not the webserver. Though we will see.

george1421 · Feb 25, 2020, 12:38 PM

Some of the testing a few of the big fog installs have done show that switching mysql over from isam to innodb format help with a heavily loaded mysql instance

If you have the ram, with 400 clients on at one time you might want to take your pm.max_children to 50 from 35. For larger installs (500 fog clients and up) some have switched over php-fpm from dynamic clients to static clients so there isn’t the stand up time for new php-fpm children if the server gets hit hard with requests.

Look at the fog client check in time. If its default, its probably 5 minutes. Increase that to 10 or 15 minutes. Understand that increase will slow the the time between issuing a snapin deployment and the client seeing the request. It will have no impact on imaging.

george1421 · Feb 25, 2020, 2:55 PM

@Sebastian-Roth The more I look into this the more I’m going to blame mysql and the ISAM table format for the OP’s issue. ISAM uses table locking for updates to the data, where INNODB uses row level locking for updates. SO if you have 10 systems updating the fog history table that is in ISAM format, one system will have update access while 9 will be placed in a blocked status until the first system finishes its update action. When the first finishes, then the second in line will get its chance and so on. I think with FOG 1.6.x stream we should setup the innodb as the default table engine. Understand we NEED to do a lot more bench marking on this issue, but I think with the help of the OP and other large FOG deployments we can get a better understanding of what it will take to make FOG go faster…

Quazz · Feb 25, 2020, 3:12 PM

@george1421 Relevant github issue for further discussion and thoughts on MyISAM -> InnoDB specifically: https://github.com/FOGProject/fogproject/issues/370

george1421 · Feb 26, 2020, 2:07 AM

This post is deleted!

Arrowtron · Feb 26, 2020, 4:22 AM

@Sebastian-Roth
Thanks for your reply, I have changed the client checkin time from 60 to 600 (10mins) and has made a huge difference (picture below)

Client Maxsize: 204800000
Grace Timeout: 60

george1421 · Feb 26, 2020, 12:39 PM

@Arrowtron I have some ideas of what we can do here if you are willing to help the FOG Project team.

Arrowtron · Feb 27, 2020, 1:01 AM

@george1421 said in High CPU Usage:

@Arrowtron I have some ideas of what we can do here if you are willing to help the FOG Project team.

Yes can do, let me know what information you need

george1421 · Feb 26, 2020, 7:11 PM

@Arrowtron Well, what I would like to do is make it worse to make it better. The meaning is to reset as much of the fixes you’ve done to this point to put it back to what way FOG was installed. This includes any adjustments suggested by the mysql tuner script. We need to have it as close to as installed as possible. Also reset the fog client check in time.

Run it in the as installed configuration for a few hours during your peak business time. Use top and sort by processor usage. Grab a screen shot of the top screen.

Shutdown apache

Lets make sure we have a good backup of the mysql server before doing anything. From the fog server linux command prompt key in mysqldump -u root -p --all-databases > all-databases-backup.sql

login to the mysql command line tool with mysql -u root -p fog You will need to use the password you created for the root database user when you installed fog. If you don’t know the password that could be a problem. The older version of FOG had a black password for the mysql root user. Understand I’m using mysql to mean both mysql and mariadb. The command line tool is the same.

At db prompt paste in this command.

SELECT TABLE_NAME,ENGINE
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'fog' and ENGINE = 'MyISAM';

You should see an output that looks like this

+------------------------+--------+
| TABLE_NAME             | ENGINE |
+------------------------+--------+
| LDAPServers            | MyISAM |
| clientUpdates          | MyISAM |
| dirCleaner             | MyISAM |
| globalSettings         | MyISAM |
| greenFog               | MyISAM |
| groupMembers           | MyISAM |
| groups                 | MyISAM |
| history                | MyISAM |
| hookEvents             | MyISAM |
| hostAutoLogOut         | MyISAM |
| hostMAC                | MyISAM |
| hostScreenSettings     | MyISAM |
| hosts                  | MyISAM |
| imageGroupAssoc        | MyISAM |
| imagePartitionTypes    | MyISAM |
| imageTypes             | MyISAM |
| images                 | MyISAM |
| imagingLog             | MyISAM |
| inventory              | MyISAM |
| ipxeTable              | MyISAM |
| keySequence            | MyISAM |
| location               | MyISAM |
| locationAssoc          | MyISAM |
| moduleStatusByHost     | MyISAM |
| modules                | MyISAM |
| multicastSessions      | MyISAM |
| multicastSessionsAssoc | MyISAM |
| nfsFailures            | MyISAM |
| nfsGroupMembers        | MyISAM |
| nfsGroups              | MyISAM |
| notifyEvents           | MyISAM |
| os                     | MyISAM |
| oui                    | MyISAM |
| plugins                | MyISAM |
| powerManagement        | MyISAM |
| printerAssoc           | MyISAM |
| printers               | MyISAM |
| pxeMenu                | MyISAM |
| scheduledTasks         | MyISAM |
| schemaVersion          | MyISAM |
| snapinAssoc            | MyISAM |
| snapinGroupAssoc       | MyISAM |
| snapinJobs             | MyISAM |
| snapinTasks            | MyISAM |
| snapins                | MyISAM |
| supportedOS            | MyISAM |
| taskLog                | MyISAM |
| taskStates             | MyISAM |
| taskTypes              | MyISAM |
| tasks                  | MyISAM |
| userCleanup            | MyISAM |
| userTracking           | MyISAM |
| users                  | MyISAM |
| virus                  | MyISAM |
| wolbroadcast           | MyISAM |
+------------------------+--------+
55 rows in set (0.00 sec)

If your tables show innodb then we have to stop because your database is not in the default state.

If your database files say MyISAM then proceed and paste in these lines. You can do them in one block paste in. Just make sure the last line is entered too.

ALTER TABLE LDAPServers ENGINE=InnoDB;
ALTER TABLE clientUpdates ENGINE=InnoDB;
ALTER TABLE dirCleaner ENGINE=InnoDB;
ALTER TABLE globalSettings ENGINE=InnoDB;
ALTER TABLE greenFog ENGINE=InnoDB;
ALTER TABLE groupMembers ENGINE=InnoDB;
ALTER TABLE groups ENGINE=InnoDB;
ALTER TABLE history ENGINE=InnoDB;
ALTER TABLE hookEvents ENGINE=InnoDB;
ALTER TABLE hostAutoLogOut ENGINE=InnoDB;
ALTER TABLE hostMAC ENGINE=InnoDB;
ALTER TABLE hostScreenSettings ENGINE=InnoDB;
ALTER TABLE hosts ENGINE=InnoDB;
ALTER TABLE imageGroupAssoc ENGINE=InnoDB;
ALTER TABLE imagePartitionTypes ENGINE=InnoDB;
ALTER TABLE imageTypes ENGINE=InnoDB;
ALTER TABLE images ENGINE=InnoDB;
ALTER TABLE imagingLog ENGINE=InnoDB;
ALTER TABLE inventory ENGINE=InnoDB;
ALTER TABLE ipxeTable ENGINE=InnoDB;
ALTER TABLE keySequence ENGINE=InnoDB;
ALTER TABLE location ENGINE=InnoDB;
ALTER TABLE locationAssoc ENGINE=InnoDB;
ALTER TABLE moduleStatusByHost ENGINE=InnoDB;
ALTER TABLE modules ENGINE=InnoDB;
ALTER TABLE multicastSessions ENGINE=InnoDB;
ALTER TABLE multicastSessionsAssoc ENGINE=InnoDB;
ALTER TABLE nfsFailures ENGINE=InnoDB;
ALTER TABLE nfsGroupMembers ENGINE=InnoDB;
ALTER TABLE nfsGroups ENGINE=InnoDB;
ALTER TABLE notifyEvents ENGINE=InnoDB;
ALTER TABLE os ENGINE=InnoDB;
ALTER TABLE oui ENGINE=InnoDB;
ALTER TABLE plugins ENGINE=InnoDB;
ALTER TABLE powerManagement ENGINE=InnoDB;
ALTER TABLE printerAssoc ENGINE=InnoDB;
ALTER TABLE printers ENGINE=InnoDB;
ALTER TABLE pxeMenu ENGINE=InnoDB;
ALTER TABLE scheduledTasks ENGINE=InnoDB;
ALTER TABLE schemaVersion ENGINE=InnoDB;
ALTER TABLE snapinAssoc ENGINE=InnoDB;
ALTER TABLE snapinGroupAssoc ENGINE=InnoDB;
ALTER TABLE snapinJobs ENGINE=InnoDB;
ALTER TABLE snapinTasks ENGINE=InnoDB;
ALTER TABLE snapins ENGINE=InnoDB;
ALTER TABLE supportedOS ENGINE=InnoDB;
ALTER TABLE taskLog ENGINE=InnoDB;
ALTER TABLE taskStates ENGINE=InnoDB;
ALTER TABLE taskTypes ENGINE=InnoDB;
ALTER TABLE tasks ENGINE=InnoDB;
ALTER TABLE userCleanup ENGINE=InnoDB;
ALTER TABLE userTracking ENGINE=InnoDB;
ALTER TABLE users ENGINE=InnoDB;
ALTER TABLE virus ENGINE=InnoDB;
ALTER TABLE wolbroadcast ENGINE=InnoDB;

Once that is done run the same query again to make sure all of the tables now say innodb

SELECT TABLE_NAME,ENGINE
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'fog';

If they are all innodb then you can exit the mysql program if I missed one then run the ALTER TABLE command on those missing tables. Please not any table I missed.

At this point reboot the FOG server.

When the fog server comes back up make sure the web ui works as expected. Let the FOG server run for a few hours then again capture a snapshot for the settings using top. Post the results here.

This next test is to let the FOG server run for a week using the innodb format then rerun the mysql tuner script. DON’T apply the settings, but post the results here. Let us look at the recommendations before you apply them to the FOG server’s database.

In the end what we need to know is the change in performance by simply changing the database engine from isam to innodb format. Also to see the recommendations provided by mysqltuner. I have some default settings for the innodb that I would start out with, but they may not be right for all situation.

Let me say we do need help from FOG Admins who have either unique target hardware or large installations that can’t be simulated easily. With your help I feel strongly that we can improve FOG’s performance and move everyone’s FOG install in a positive direction.

george1421 · Mar 3, 2020, 2:08 AM

@Arrowtron How is this testing going? Do you see measured improvements in performance?

Arrowtron · Mar 3, 2020, 2:08 AM

@george1421 Sorry didn’t get to you sooner, few work things came up

I have changed the client polling back to 60 seconds and left for a day, screenshot taken at peak time with roughly ~600 PC’s online (with FOG service installed)

Made a VM snapshot at this point* - can revert back if needed

I ran into a few issues with the following tables, they wouldn’t change to InnoDB engine, seems like a corrupt row in that table

I have restarted the server and appears to be loading the web/dashboard - however very slow

I’ll keep this running and let you know if things improve, although those tables that haven’t been changed seem like important tables to change.
Let me know how I should proceed

Arrowtron · Mar 10, 2020, 10:38 PM

@george1421 I haven’t heard back from you, here are the results after 7 days
There are roughly 597 PC’s online with FOG service Installed
As stated from previous reply, there were a few tables that didn’t change to MyISAM
The client polling is 60 seconds

 >>  MySQLTuner 1.7.19 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from Debian maintenance account.
[OK] Currently running supported MySQL version 5.7.28-0ubuntu0.19.04.2
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ----------------------------------------------                                                                                                                                                                                                                                             --------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log(0B)
[OK] Log file /var/log/mysql/error.log is readable.
[!!] Log file /var/log/mysql/error.log is empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[OK] /var/log/mysql/error.log doesn't contain any warning.
[OK] /var/log/mysql/error.log doesn't contain any error.
[--] 0 start(s) detected in /var/log/mysql/error.log
[--] 0 shutdown(s) detected in /var/log/mysql/error.log

-------- Storage Engine Statistics ---------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +My                                                                                                                                                                                                                                             ISAM +PERFORMANCE_SCHEMA
[--] Data in InnoDB tables: 14.7M (Tables: 50)
[--] Data in MyISAM tables: 1002.6K (Tables: 5)
[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics ------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- CVE Security Recommendations ------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics ---------------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] Up for: 7d 20h 30m 0s (494M q [728.806 qps], 11M conn, TX: 439G, RX: 106G)
[--] Reads / Writes: 98% / 2%
[--] Binary logging is disabled
[--] Physical Memory     : 3.8G
[--] Max MySQL memory    : 2.7G
[--] Other process memory: 0B
[--] Total buffers: 192.0M global + 17.1M per thread (151 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 1.1G (29.54% of installed RAM)
[OK] Maximum possible memory usage: 2.7G (70.22% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory                                                                                                                                                                                                                                              available
[OK] Slow queries: 0% (0/494M)
[OK] Highest usage of available connections: 37% (57/151)
[OK] Aborted connections: 0.00%  (8/11267058)
[!!] name resolution is active : a reverse name resolution is made for each new                                                                                                                                                                                                                                              connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 454M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (9 temp sorts / 139M sorts)
[OK] No joins without indexes
[!!] Temporary tables created on disk: 97% (11M on disk / 11M total)
[OK] Thread cache hit rate: 98% (209K created / 11M connections)
[OK] Table cache hit rate: 96% (977 open / 1K opened)
[OK] table_definition_cache(1400) is upper than number of tables(334)
[OK] Open file limit used: 2% (144/5K)
[OK] Table locks acquired immediately: 99% (101M immediate / 101M locks)

-------- Performance schema ----------------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] Memory used by P_S: 72B
[--] Sys schema is installed.

-------- ThreadPool Metrics ----------------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] ThreadPool stat is disabled.

-------- MyISAM Metrics --------------------------------------------------------                                                                                                                                                                                                                                             --------------------
[!!] Key buffer used: 18.7% (3M used / 16M cache)
[OK] Key buffer size / total MyISAM indexes: 16.0M/238.0K
[OK] Read Key buffer hit rate: 100.0% (235M cached / 76 reads)
[!!] Write Key buffer hit rate: 83.1% (154 cached / 128 writes)

-------- InnoDB Metrics --------------------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 128.0M/14.7M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.                                                                                                                                                                                                                                             0M should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb                                                                                                                                                                                                                                             _buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (9450445390 hits/ 9450446365 total)
[!!] InnoDB Write Log efficiency: 59.83% (3030357 hits/ 5065232 total)
[OK] InnoDB log waits: 0.00% (0 waits / 2034875 writes)

-------- AriaDB Metrics --------------------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] AriaDB is disabled.

-------- TokuDB Metrics --------------------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics --------------------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] XtraDB is disabled.

-------- Galera Metrics --------------------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] Galera is disabled.

-------- Replication Metrics ---------------------------------------------------                                                                                                                                                                                                                                             --------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server

-------- Recommendations -------------------------------------------------------                                                                                                                                                                                                                                             --------------------
General recommendations:
    Set up a Secure Password for root@localhost user: SET PASSWORD FOR 'root'@'S                                                                                                                                                                                                                                             pecificDNSorIp' = PASSWORD('secure_password');
    Restrict Host for 'fogstorage'@% to fogstorage@SpecificDNSorIp
    UPDATE mysql.user SET host ='SpecificDNSorIp' WHERE user='fogstorage' AND ho                                                                                                                                                                                                                                             st ='%'; FLUSH PRIVILEGES;
    Configure your accounts with ip or subnets only, then update your configurat                                                                                                                                                                                                                                             ion with skip-name-resolve=1
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read t                                                                                                                                                                                                                                             his: https://bit.ly/2TcGgtU
Variables to adjust:
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_limit (> 1M, or use smaller result sets)
    tmp_table_size (> 16M)
    max_heap_table_size (> 16M)
    innodb_log_file_size should be (=16M) if possible, so InnoDB total log files                                                                                                                                                                                                                                              size equals to 25% of buffer pool size.

george1421 · Mar 11, 2020, 6:09 AM

@Arrowtron I’m sorry this dropped off my radar. We need to get the errors converting the tables fixed.

Will you log into the sql server console with mysql -u root or if mysql root user has a password with mysql -u root -p

At the mysql server command prompt key in SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session and post the output.

Also look in the /etc/my.cnf file for sql server and look for a line with a keyword of sql_mode = post the results of that option.

If its not in that file you can find all of the config files with this command /usr/sbin/mysqld --verbose --help | grep -A 1 "Default options"

ref: https://thehikashop.com/blog/8-fix-incorrect-date-value-0000-00-00-in-mysql
ref: https://www.sitepoint.com/quick-tip-how-to-permanently-change-sql-mode-in-mysql/

Arrowtron · Mar 12, 2020, 8:27 AM

@george1421

mysql> SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session;
+-------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| global                                                                                                                                    | session                                                                                                                                   |
+-------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+-------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

/etc/my.cnf doesn’t exist

/usr/sbin/mysqld --verbose --help | grep -A 1 “Default options”

Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf

nano /etc/mysql/my.cnf

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

~/.my.cnf doesn’t exist

george1421 · Mar 11, 2020, 7:12 PM

@Arrowtron said in High CPU Usage:

/etc/mysql/conf.d/

will you confirm there are no files in this location?

This is the problem why those date codes are being rejected.

ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

Arrowtron · Mar 12, 2020, 4:09 AM

@george1421 said in High CPU Usage:

/etc/mysql/conf.d/

2 files in that location

mysqldump.cnf

[mysqldump]
quick
quote-names
max_allowed_packet      = 16M

mysql.cnf

[mysql]

Sebastian Roth · Mar 12, 2020, 8:45 AM

@george1421 said in High CPU Usage:

STRICT_TRANS_TABLES

Interesting find. Hmmm, just did a quick test. This is not the case on Debian 9 (MariaDB 10.1.44). But Ubuntu (20.04 - pre release testing) with MySQL 8.0.19 does have the same sql_mode that as we see from the OP’s Ubuntu 18.04 install. I will see if I can figure out why they have it in Ubuntu.

george1421 · Mar 12, 2020, 2:47 PM

@Sebastian-Roth said in High CPU Usage:

I will see if I can figure out why they have it in Ubuntu.

It looks like its the defaults compiled into the sql server because the OP doesn’t have that value specifically set. So now we need to come up with the correct line to exclude that value from the sql server options. Once we get that turned off we can then convert the rest of the tables to innodb format. I do think it is a great find too because that value would have stopped the creation of the fog tables if we defaulted to the innodb engine.

Sebastian Roth · Mar 12, 2020, 2:58 PM

@george1421 said in High CPU Usage:

So now we need to come up with the correct line to exclude that value from the sql server options.

Definitely worth a try though I am not sure how we will handle this “for the mass”. Sure we can mess with DB settings but I am afraid of doing this and causing a lot more issues than we had before. But we’ll see. Probably we’ll find a god solution to that somehow.

High CPU Usage

149

12.0k

17.3k

155.1k