youdidwhatwithtsql.com » MySQL

MySQL Storage engine benchmarking

Rhys — Mon, 07 Nov 2011 20:49:37 +0000

Here’s a stored procedure I use to perform some simple benchmarking of inserts for MySQL. It takes three parameters; p_table_type which should be set to the storage engine you wish to benchmark i.e. ‘MyISAM’, ‘InnoDB’, p_inserts; set this to the number of inserts to perform. p_autocommit; set the autocommit variable (relevant to InnoDB only) to 0 or 1.

^?View Code MYSQL

DELIMITER $$
USE `test`$$
DROP PROCEDURE IF EXISTS `table_engine_test`$$
        CREATE DEFINER=`root`@`%`
        PROCEDURE `table_engine_test`(IN p_table_type VARCHAR(20),
                                      IN p_inserts    INT,
                                      IN p_autocommit TINYINT)
                SQL SECURITY INVOKER
        BEGIN
 
                DECLARE sql_string VARCHAR(300);
 
                # Set session autocommit
                SET SESSION autocommit = p_autocommit;
 
                # TABLE TO hold session times
                CREATE TABLE IF NOT EXISTS test_session
                             (
                                          Id         INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
                                          table_type VARCHAR(20) NOT NULL                   ,
                                          inserts INT NOT NULL,
                                          autocommit TINYINT NOT NULL,
                                          started    DATETIME NULL                          ,
                                          finished   DATETIME NULL
                             );
 
                CREATE TABLE IF NOT EXISTS test_session_inserts
                             (
                                          id              INTEGER NOT NULL PRIMARY KEY AUTO_INCREMENT,
                                          test_session_id INTEGER NOT NULL                           ,
                                          started         DATETIME NOT NULL                          ,
                                          finished        DATETIME NOT NULL
                             );
 
                # clean up ANY existing test TABLE
                DROP TABLE IF EXISTS test_table_type;
 
                SET sql_string = CONCAT('CREATE TABLE test_table_type
					(
						id INTEGER NOT NULL PRIMARY KEY AUTO_INCREMENT,
						random INTEGER,
						text1 VARCHAR(10) NOT NULL,
						text2 VARCHAR(10) NOT NULL
					)ENGINE = ', p_table_type);
                # PREPARE SQL AND EXECUTE
                SET @q = sql_string;
                PREPARE stmt FROM @q;
                EXECUTE stmt;
                DEALLOCATE PREPARE stmt;
                # START the test session
                INSERT
                INTO   test_session
                       (
                              table_type,
                              inserts,
                              autocommit,
                              started
                       )
                       VALUES
                       (
                              p_table_type,
                              p_inserts,
                              p_autocommit,
                              NOW()
                       );
 
                SET @id               = LAST_INSERT_ID();
                SET @i                = 0;
                while_loop: WHILE (@i < p_inserts) DO
                # Generate VALUES for insert
                SET @random  = RAND();
                SET @text1   = SUBSTRING(MD5(RAND()), -10);
                SET @text2   = SUBSTRING(MD5(RAND()), -10);
                SET @started = NOW();
                # INSERT the test row
                INSERT
                INTO   test_table_type
                       (
                              random,
                              text1 ,
                              text2
                       )
                       VALUES
                       (
                              @random,
                              @text1 ,
                              @text2
                       );
 
                # record INSERT times
                        INSERT
                        INTO   test_session_inserts
                               (
                                      test_session_id,
                                      started        ,
                                      finished
                               )
                               VALUES
                               (
                                      @id     ,
                                      @started,
                                      NOW()
                               );
 
                        # increment counter
                        SET @i = @i + 1;
                END WHILE while_loop;
                # Finish the session
                UPDATE test_session
                SET    finished = NOW()
                WHERE  id       = @id;
 
END$$
DELIMITER ;

Run your tests like so…

^?View Code MYSQL

CALL table_engine_test('MyISAM', 10000, 0);
CALL table_engine_test('InnoDB', 10000, 0);
CALL table_engine_test('InnoDB', 10000, 1);
CALL table_engine_test('MyISAM', 100000, 0);
CALL table_engine_test('InnoDB', 100000, 0);
CALL table_engine_test('InnoDB', 100000, 1);

The test_session contains some summary information about each of the tests;

^?View Code MYSQL

SELECT *
FROM test_session;

Calculate the time taken for each test with the following query;

^?View Code MYSQL

SELECT *, TIMEDIFF(finished, started) AS seconds
FROM test_session;

Happy Benchmarking!

Post from: youdidwhatwithtsql.com

MySQL Storage engine benchmarking

Synchronize Mysql slave tables with mk-table-sync

Rhys — Tue, 21 Jun 2011 19:40:56 +0000

I’ve been meaning to check out Maatkit for a while now. Today I had a reason to as one of our MySQL slaves got out of sync with the master. I’d heard about mk-table-sync, a tool that synchronizes tables, so I thought I’d give it a shot.

As it turns out it’s this easy;

mk-table-sync --execute h=slave_server,u=username,p=secret,D=database,t=table_to_sync --sync-to-master

I was a little worried about how this would pan out as I assumed it would change data on the slave. Cleverly, the Maatkit developers have thought of this;

"When synchronizing a server that is a replication slave with the –replicate or –sync-to-master methods, it always makes the changes on the replication master, never the replication slave directly." ref

So in theory this should alleviate some of those replication pains. Happy days!

Post from: youdidwhatwithtsql.com

Synchronize Mysql slave tables with mk-table-sync

Backing up the structure of MySQL databases

Rhys — Tue, 25 Jan 2011 21:00:20 +0000

Today I wanted a quick and easy way to generate a backup of the structure of all MySQL databases in one easy hit. Here’s a couple of ways you can do this with the tools you’re likely to find everywhere.

Firstly, we can use the following query to generate a list of mysqldump commands. The backups generated here pretty much just contain tables, triggers and functions so customise the command to your needs. Just change the output directory from ./home/rhys.

^?View Code MYSQL

SELECT CONCAT('mysqldump -d --routines -h mysql_host -u username -pSecret ', SCHEMA_NAME, ' > "/home/rhys/Desktop/', SCHEMA_NAME, '.sql"')
FROM INFORMATION_SCHEMA.SCHEMATA
WHERE SCHEMA_NAME NOT IN ('mysql', 'information_schema', 'performance_schema');

These commands can be copied and then pasted into a terminal to be executed.

Here’s a pure bash way of achieving the same thing. The script is called on the command line with the MySQL host, user name and password supplied as parameters. Backups are generated in the current users home directory in the format db_name.sql.

^?View Code BASH

#!/bin/bash
# MySQL details passed on command line
HOST=$1
USER=$2;
PWD=$3;
 
# Query to get database excluding a few system ones
QUERY="SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME NOT IN ('mysql','information_schema','performance_schema')";
# Run the query and get the results
results=`mysql -h $HOST -u $USER -p$PWD -N -e "$QUERY"`;
 
# Loop through each row
for db in $results
do
	bkp="mysqldump -d --routines -h $HOST -u $USER -p$PWD $db > '$HOME/$db.sql'";
	echo "Backing up $db...";
	eval $bkp;
done

Save this to a file called backup_mysql.sh and don;t forget to do a chmod + x to make the script executable. The backup can be executed from the command line with;

^?View Code BASH

./backup_mysql.sh localhost username password

Post from: youdidwhatwithtsql.com

Backing up the structure of MySQL databases

Add column headers to a MySQL Outfile

Rhys — Tue, 14 Dec 2010 21:11:30 +0000

Unfortunately the MySQL SELECT INTO OUTFILE command doesn’t support an option to output the headers of the result set you are exporting. A feature request has been open for over 2 years to sort this with no apparent activity.

A few people have had the idea of using a UNION ALL to include the headers but this becomes tedious with large queries. Tools like sqlyog are handy but can painfully slow for large result sets due to RBAR methods.

For large result sets I like to first run;

^?View Code MYSQL

# Create temp table
CREATE TEMPORARY TABLE tmp_MyTable
ENGINE = MEMORY
SELECT col1, col2, col3, col4 # etc
FROM myTable
INNER JOIN myTable2
	ON myTable.Id = myTable2.Id;
 
# Get the column info
DESCRIBE tmp_MyTable;

This will output the result set we want to export into a temporary table. We can then get the column headers using the DESCRIBE command. Then I export my result set to a file using the OUTFILE command.

^?View Code MYSQL

SELECT *
INTO OUTFILE '/home/user/export.csv'
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
FROM tmp_MyTable;

Now it’s roll your sleeves up time with the Linux command-line and sed.

^?View Code BASH

sed -i '1iOne,Two,Three,Four,Five' /home/user/export.csv

This command will add a single header line, to the file previously exported, and should end up looking something like below…

One,Two,Three,Four,Five
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5

Post from: youdidwhatwithtsql.com

Add column headers to a MySQL Outfile

Parse MySQL Slow Logs with mysqlsla

Rhys — Tue, 10 Aug 2010 19:36:21 +0000

Here’s a bash script that you can use to parse multiple MySQL Slow Query Log files, in one sweep, into something much more understandable. The script uses the handy utility mysqlsla so make sure this is in your path.

mysqlsla parses, filters, analyzes and sorts MySQL slow, general, binary and microslow patched logs in order to create a customizable report of the queries and their meta-property values. Since these reports are customizable, they can be used for human consumption or be fed into other scripts to further analyze the queries. For example, to profile with mk-query-profiler (a script from Baron Schwartz’s Maatkit) every unique SELECT statement using database foo from a slow log: source

Place all your slow logs into a directory. Change the sl_dir variable to point at this directory. When you execute the script it will create a directory, within your slow logs directory, called reports. This will contain the reports produced by mysqlsla.

^?View Code BASH

#!/bin/bash
 
# Script to process multiple mysql slow logs
# using mysqlsla http://hackmysql.com/mysqlsla
 
# Directory containing slow logs
sl_dir="/home/rhys/Desktop/slow_logs";
 
cd "$sl_dir";
#slow_logs=$(ls "$sl_dir");
 
# Folder for reports
if [ ! -d "$sl_dir"/reports ]; then
                mkdir "$sl_dir"/reports;
fi
 
# process each slow log file
for file in "$sl_dir"/*
do
                echo "Processing file: $file";
                filename=$(basename "$file")
                mysqlsla -lt slow "$file" &gt; "reports/$filename.rpt";
                echo "Finished processing file: $file";
done

The reports produced are much easier to work with than the raw mysql logs so this should be a good time saver when optimising those queries!

Post from: youdidwhatwithtsql.com

Parse MySQL Slow Logs with mysqlsla

Can’t reopen table: ‘t1′

Rhys — Mon, 19 Jul 2010 21:24:08 +0000

I’m quite often jumping between MySQL and SQL Server so remembering the quirks and limitations of each system can be difficult. With MySQL, if you attempt to reference a temporary table more than once in the same query, you will encounter the following error;

Error Code : 1137
Can't reopen table: 't1’

The following provides an example of this…

^?View Code MYSQL

USE test;
 
CREATE TEMPORARY TABLE test
(
                Id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY
);
 
SELECT *
FROM test AS t1
INNER JOIN test AS t2
     ON t1.Id = t2.Id;

It’s not just self-joins that have this issue UNIONS do as well;

^?View Code MYSQL

SELECT *
FROM test AS t1
UNION ALL
SELECT *
FROM test AS t2;

There’s a thread over on Stackoverflow discussing this problem. Here’s a solution I commonly use to get around the problem;

^?View Code MYSQL

USE test;
 
# Create temp table
CREATE TEMPORARY TABLE test
(
                Id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY
);
 
# Insert some test data
INSERT INTO test (Id) VALUES (1),(2),(3),(4),(5);
 
# Clone the table. This will do structure & indices but no data.
CREATE TEMPORARY TABLE test2 LIKE test;
 
# Insert the data into the new table
INSERT INTO test2
SELECT Id
FROM test;
 
# Now our queries will work if we use the tables clone
SELECT *
FROM test AS t1
INNER JOIN test2 AS t2
      ON t1.Id = t2.Id;
 
SELECT *
FROM test AS t1
UNION ALL
SELECT *
FROM test2 AS t2;
 
# Clean up
DROP TEMPORARY TABLE IF EXISTS test;
DROP TEMPORARY TABLE IF EXISTS test2;

Post from: youdidwhatwithtsql.com

Can’t reopen table: ‘t1′

Rename MySQL Stored Procedures

Rhys — Wed, 14 Jul 2010 20:47:05 +0000

I’ve previously blogged about the limitations of MySQL Alter Procedure Syntax and I came across a thread on the MySQL forums with a possible solution. I thought it might be handy to wrap this up into a stored procedure akin to SQL Server’s sp_rename.

This procedure will allow you to easily rename MySQL Stored Procedures in any database. Please be aware that this does update the MySQL system tables and has only had minimal testing. As with all tips you find on the Internet please use with caution!

^?View Code MYSQL

DELIMITER $$
 
USE `mysql`$$
 
DROP PROCEDURE IF EXISTS `mysp_rename_proc`$$
 
CREATE DEFINER=`root`@`%` PROCEDURE `mysp_rename_proc`(IN p_proc_name VARCHAR(64), IN p_new_name VARCHAR(64), IN p_db VARCHAR(64))
    MODIFIES SQL DATA
    SQL SECURITY INVOKER
    COMMENT 'Use to rename stored procedures.'
BEGIN
    proc: BEGIN
	# A few tests to see if the input is sensible
	IF CHARACTER_LENGTH(IFNULL(p_proc_name, '')) = 0 OR CHARACTER_LENGTH(IFNULL(p_new_name, '')) = 0
		OR CHARACTER_LENGTH(IFNULL(p_db, '')) = 0 THEN
	BEGIN
		SELECT 'Error: One of more of the input parameters is zero in length.' AS Error;
		LEAVE proc; 
	END;
	ELSEIF (SELECT COUNT(*)
		 FROM mysql.proc
		 WHERE `name` = p_proc_name
		 AND `type` = 'PROCEDURE'
		 AND db = p_db) = 0 THEN
	BEGIN
		SELECT 'Error: The procedure specified in p_proc_name does not exist in this database.' AS Error;
		LEAVE proc;
	END;
	ELSEIF (SELECT COUNT(*)
		 FROM mysql.proc
		 WHERE `name` = p_new_name
		 AND `type` = 'PROCEDURE'
		 AND db = p_db) = 1 THEN
	BEGIN
		SELECT 'Error: Unable to rename the procedure specified in p_proc_name as it already exists in this database.' AS Error;
		LEAVE proc;
	END;
	END IF;
 
	# Rename the proc
	UPDATE `mysql`.`proc`
	SET `name` = p_new_name,
	specific_name = p_new_name
	WHERE db = p_db 
	AND `name` = p_proc_name
	AND `type` = 'PROCEDURE';
 
	# Update any associated privileges
	UPDATE `mysql`.`procs_priv`
	SET Routine_name = p_new_name
	WHERE db = p_db 
	AND Routine_name = p_proc_name
	AND Routine_type = 'PROCEDURE';
 
	# Check update rowcount to see if privileges need to be flushed
	IF(SELECT ROW_COUNT()) > 0 THEN
	BEGIN
		FLUSH PRIVILEGES;
	END;
	END IF;
 
    END proc;
    END$$
 
DELIMITER ;

Usage is as follows;

^?View Code MYSQL

CALL mysp_rename_proc('usp_proc', 'usp_new_proc_name', 'database_proc_exists_in');

As the thread poster mentions the Stored Procedure is callable by it’s old name as well as the new one until you reconnect. There doesn’t seem to be any suitable FLUSH command to resolve this.

If you liked this you may like;

MySQL Clone of SP_SpaceUsed

MySQL Clone of SP_MsForEachTable

Post from: youdidwhatwithtsql.com

Rename MySQL Stored Procedures

Purge MySQL Binary Logs

Rhys — Mon, 12 Jul 2010 10:58:03 +0000

From time-to-time you may need to manually purge binary logs on your MySQL slaves to free up a bit of disk space. We can achieve this by using the PURGE BINARY LOGS command from the MySQL command line client. MySQL advises the following procedure when purging these logs

To safely purge binary log files, follow this procedure:

On each slave server, use SHOW SLAVE STATUS to check which log file it is reading.

Obtain a listing of the binary log files on the master server with SHOW BINARY LOGS.

Determine the earliest log file among all the slaves. This is the target file. If all the slaves are up to date, this is the last log file on the list.

Make a backup of all the log files you are about to delete. (This step is optional, but always advisable.)

Purge all log files up to but not including the target file. source

The below example will purge all binary logs older than the one called backup-master.001271.

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2640654
Server version: 5.1.34-log SUSE MySQL RPM

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> PURGE BINARY LOGS TO 'backup-master.001271'

Provided your slave is running this command is reasonably safe as MySQL will prevent you from purging any log files that are currently being read. If the slave thread is not running then you do have to make sure you are not purging a needed file. If you purge a needed file the slave will break when it is restarted. You can also use the expire_logs_days variable to automatically control when binary logs are purged.

Post from: youdidwhatwithtsql.com

Purge MySQL Binary Logs

For RANGE partitions each partition must be defined

Rhys — Fri, 09 Jul 2010 12:08:31 +0000

If you encounter the following error when trying to create a partitioned table in MySQL

Error Code : 1492
For RANGE partitions each partition must be defined

Assuming you have defined your partitions then you probably have a syntax error. Take the following incorrect example.

^?View Code MYSQL

CREATE TABLE People
(
	PersonId INTEGER NOT NULL AUTO_INCREMENT,
	FirstName VARCHAR(50) NOT NULL,
	LastName VARCHAR(50) NOT NULL,
	DateOfBirth DATE NOT NULL,
	Telephone VARCHAR(30) NULL,
	Email VARCHAR(200) NULL,
	GroupId SMALLINT NOT NULL,
	PRIMARY KEY (PersonId, GroupId)
)
PARTITION BY RANGE (GroupId)
PARTITION p0 VALUES LESS THAN (100),
PARTITION p1 VALUES LESS THAN (200),
PARTITION p2 VALUES LESS THAN (300),
PARTITION p3 VALUES LESS THAN (400),
PARTITION p4 VALUES LESS THAN (500),
PARTITION p5 VALUES LESS THAN (600),
PARTITION p6 VALUES LESS THAN (700);

The above statement is not syntactically correct but the error thrown in this case is not particularly helpful. All that is missing here is a couple of braces around the partition range definitions. The below DDL statement is correct.

^?View Code MYSQL

CREATE TABLE People
(
	PersonId INTEGER NOT NULL AUTO_INCREMENT,
	FirstName VARCHAR(50) NOT NULL,
	LastName VARCHAR(50) NOT NULL,
	DateOfBirth DATE NOT NULL,
	Telephone VARCHAR(30) NULL,
	Email VARCHAR(200) NULL,
	GroupId SMALLINT NOT NULL,
	PRIMARY KEY (PersonId, GroupId)
)
PARTITION BY RANGE (GroupId)
(
	PARTITION p0 VALUES LESS THAN (100),
	PARTITION p1 VALUES LESS THAN (200),
	PARTITION p2 VALUES LESS THAN (300),
	PARTITION p3 VALUES LESS THAN (400),
	PARTITION p4 VALUES LESS THAN (500),
	PARTITION p5 VALUES LESS THAN (600),
	PARTITION p6 VALUES LESS THAN (700)
);

Post from: youdidwhatwithtsql.com

For RANGE partitions each partition must be defined

Error dropping database (can’t rmdir ‘./database’, errno: 39)

Rhys — Wed, 23 Jun 2010 21:12:55 +0000

Just a very quick post today! If you encounter this error when attempting to drop a MySQL database;

Error dropping database (can't rmdir './database', errno: 39)

Then you probably have some rogue files in the folder where the database files are located. If you cd into this directory you will be able to view these files. In my case I had a tar archive containing the files the database was restored from. Just delete any files in this directory and you’ll be able to complete the drop.

Post from: youdidwhatwithtsql.com

Error dropping database (can’t rmdir ‘./database’, errno: 39)