youdidwhatwithtsql.com » MySQL

MySQL Storage engine benchmarking

Rhys — Mon, 07 Nov 2011 20:49:37 +0000

Here’s a stored procedure I use to perform some simple benchmarking of inserts for MySQL. It takes three parameters; p_table_type which should be set to the storage engine you wish to benchmark i.e. ‘MyISAM’, ‘InnoDB’, p_inserts; set this to the number of inserts to perform. p_autocommit; set the autocommit variable (relevant to InnoDB only) to 0 or 1.

^?View Code MYSQL

DELIMITER $$
USE `test`$$
DROP PROCEDURE IF EXISTS `table_engine_test`$$
        CREATE DEFINER=`root`@`%`
        PROCEDURE `table_engine_test`(IN p_table_type VARCHAR(20),
                                      IN p_inserts    INT,
                                      IN p_autocommit TINYINT)
                SQL SECURITY INVOKER
        BEGIN
 
                DECLARE sql_string VARCHAR(300);
 
                # Set session autocommit
                SET SESSION autocommit = p_autocommit;
 
                # TABLE TO hold session times
                CREATE TABLE IF NOT EXISTS test_session
                             (
                                          Id         INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
                                          table_type VARCHAR(20) NOT NULL                   ,
                                          inserts INT NOT NULL,
                                          autocommit TINYINT NOT NULL,
                                          started    DATETIME NULL                          ,
                                          finished   DATETIME NULL
                             );
 
                CREATE TABLE IF NOT EXISTS test_session_inserts
                             (
                                          id              INTEGER NOT NULL PRIMARY KEY AUTO_INCREMENT,
                                          test_session_id INTEGER NOT NULL                           ,
                                          started         DATETIME NOT NULL                          ,
                                          finished        DATETIME NOT NULL
                             );
 
                # clean up ANY existing test TABLE
                DROP TABLE IF EXISTS test_table_type;
 
                SET sql_string = CONCAT('CREATE TABLE test_table_type
					(
						id INTEGER NOT NULL PRIMARY KEY AUTO_INCREMENT,
						random INTEGER,
						text1 VARCHAR(10) NOT NULL,
						text2 VARCHAR(10) NOT NULL
					)ENGINE = ', p_table_type);
                # PREPARE SQL AND EXECUTE
                SET @q = sql_string;
                PREPARE stmt FROM @q;
                EXECUTE stmt;
                DEALLOCATE PREPARE stmt;
                # START the test session
                INSERT
                INTO   test_session
                       (
                              table_type,
                              inserts,
                              autocommit,
                              started
                       )
                       VALUES
                       (
                              p_table_type,
                              p_inserts,
                              p_autocommit,
                              NOW()
                       );
 
                SET @id               = LAST_INSERT_ID();
                SET @i                = 0;
                while_loop: WHILE (@i < p_inserts) DO
                # Generate VALUES for insert
                SET @random  = RAND();
                SET @text1   = SUBSTRING(MD5(RAND()), -10);
                SET @text2   = SUBSTRING(MD5(RAND()), -10);
                SET @started = NOW();
                # INSERT the test row
                INSERT
                INTO   test_table_type
                       (
                              random,
                              text1 ,
                              text2
                       )
                       VALUES
                       (
                              @random,
                              @text1 ,
                              @text2
                       );
 
                # record INSERT times
                        INSERT
                        INTO   test_session_inserts
                               (
                                      test_session_id,
                                      started        ,
                                      finished
                               )
                               VALUES
                               (
                                      @id     ,
                                      @started,
                                      NOW()
                               );
 
                        # increment counter
                        SET @i = @i + 1;
                END WHILE while_loop;
                # Finish the session
                UPDATE test_session
                SET    finished = NOW()
                WHERE  id       = @id;
 
END$$
DELIMITER ;

Run your tests like so…

^?View Code MYSQL

CALL table_engine_test('MyISAM', 10000, 0);
CALL table_engine_test('InnoDB', 10000, 0);
CALL table_engine_test('InnoDB', 10000, 1);
CALL table_engine_test('MyISAM', 100000, 0);
CALL table_engine_test('InnoDB', 100000, 0);
CALL table_engine_test('InnoDB', 100000, 1);

The test_session contains some summary information about each of the tests;

^?View Code MYSQL

SELECT *
FROM test_session;

Calculate the time taken for each test with the following query;

^?View Code MYSQL

SELECT *, TIMEDIFF(finished, started) AS seconds
FROM test_session;

Happy Benchmarking!

Post from: youdidwhatwithtsql.com

MySQL Storage engine benchmarking

Synchronize Mysql slave tables with mk-table-sync

Rhys — Tue, 21 Jun 2011 19:40:56 +0000

I’ve been meaning to check out Maatkit for a while now. Today I had a reason to as one of our MySQL slaves got out of sync with the master. I’d heard about mk-table-sync, a tool that synchronizes tables, so I thought I’d give it a shot.

As it turns out it’s this easy;

mk-table-sync --execute h=slave_server,u=username,p=secret,D=database,t=table_to_sync --sync-to-master

I was a little worried about how this would pan out as I assumed it would change data on the slave. Cleverly, the Maatkit developers have thought of this;

"When synchronizing a server that is a replication slave with the –replicate or –sync-to-master methods, it always makes the changes on the replication master, never the replication slave directly." ref

So in theory this should alleviate some of those replication pains. Happy days!

Post from: youdidwhatwithtsql.com

Synchronize Mysql slave tables with mk-table-sync

Backing up the structure of MySQL databases

Rhys — Tue, 25 Jan 2011 21:00:20 +0000

Today I wanted a quick and easy way to generate a backup of the structure of all MySQL databases in one easy hit. Here’s a couple of ways you can do this with the tools you’re likely to find everywhere.

Firstly, we can use the following query to generate a list of mysqldump commands. The backups generated here pretty much just contain tables, triggers and functions so customise the command to your needs. Just change the output directory from ./home/rhys.

^?View Code MYSQL

SELECT CONCAT('mysqldump -d --routines -h mysql_host -u username -pSecret ', SCHEMA_NAME, ' > "/home/rhys/Desktop/', SCHEMA_NAME, '.sql"')
FROM INFORMATION_SCHEMA.SCHEMATA
WHERE SCHEMA_NAME NOT IN ('mysql', 'information_schema', 'performance_schema');

These commands can be copied and then pasted into a terminal to be executed.

Here’s a pure bash way of achieving the same thing. The script is called on the command line with the MySQL host, user name and password supplied as parameters. Backups are generated in the current users home directory in the format db_name.sql.

^?View Code BASH

#!/bin/bash
# MySQL details passed on command line
HOST=$1
USER=$2;
PWD=$3;
 
# Query to get database excluding a few system ones
QUERY="SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME NOT IN ('mysql','information_schema','performance_schema')";
# Run the query and get the results
results=`mysql -h $HOST -u $USER -p$PWD -N -e "$QUERY"`;
 
# Loop through each row
for db in $results
do
	bkp="mysqldump -d --routines -h $HOST -u $USER -p$PWD $db > '$HOME/$db.sql'";
	echo "Backing up $db...";
	eval $bkp;
done

Save this to a file called backup_mysql.sh and don;t forget to do a chmod + x to make the script executable. The backup can be executed from the command line with;

^?View Code BASH

./backup_mysql.sh localhost username password

Post from: youdidwhatwithtsql.com

Backing up the structure of MySQL databases

Parse MySQL Slow Logs with mysqlsla

Rhys — Tue, 10 Aug 2010 19:36:21 +0000

Here’s a bash script that you can use to parse multiple MySQL Slow Query Log files, in one sweep, into something much more understandable. The script uses the handy utility mysqlsla so make sure this is in your path.

mysqlsla parses, filters, analyzes and sorts MySQL slow, general, binary and microslow patched logs in order to create a customizable report of the queries and their meta-property values. Since these reports are customizable, they can be used for human consumption or be fed into other scripts to further analyze the queries. For example, to profile with mk-query-profiler (a script from Baron Schwartz’s Maatkit) every unique SELECT statement using database foo from a slow log: source

Place all your slow logs into a directory. Change the sl_dir variable to point at this directory. When you execute the script it will create a directory, within your slow logs directory, called reports. This will contain the reports produced by mysqlsla.

^?View Code BASH

#!/bin/bash
 
# Script to process multiple mysql slow logs
# using mysqlsla http://hackmysql.com/mysqlsla
 
# Directory containing slow logs
sl_dir="/home/rhys/Desktop/slow_logs";
 
cd "$sl_dir";
#slow_logs=$(ls "$sl_dir");
 
# Folder for reports
if [ ! -d "$sl_dir"/reports ]; then
                mkdir "$sl_dir"/reports;
fi
 
# process each slow log file
for file in "$sl_dir"/*
do
                echo "Processing file: $file";
                filename=$(basename "$file")
                mysqlsla -lt slow "$file" &gt; "reports/$filename.rpt";
                echo "Finished processing file: $file";
done

The reports produced are much easier to work with than the raw mysql logs so this should be a good time saver when optimising those queries!

Post from: youdidwhatwithtsql.com

Parse MySQL Slow Logs with mysqlsla

For RANGE partitions each partition must be defined

Rhys — Fri, 09 Jul 2010 12:08:31 +0000

If you encounter the following error when trying to create a partitioned table in MySQL

Error Code : 1492
For RANGE partitions each partition must be defined

Assuming you have defined your partitions then you probably have a syntax error. Take the following incorrect example.

^?View Code MYSQL

CREATE TABLE People
(
	PersonId INTEGER NOT NULL AUTO_INCREMENT,
	FirstName VARCHAR(50) NOT NULL,
	LastName VARCHAR(50) NOT NULL,
	DateOfBirth DATE NOT NULL,
	Telephone VARCHAR(30) NULL,
	Email VARCHAR(200) NULL,
	GroupId SMALLINT NOT NULL,
	PRIMARY KEY (PersonId, GroupId)
)
PARTITION BY RANGE (GroupId)
PARTITION p0 VALUES LESS THAN (100),
PARTITION p1 VALUES LESS THAN (200),
PARTITION p2 VALUES LESS THAN (300),
PARTITION p3 VALUES LESS THAN (400),
PARTITION p4 VALUES LESS THAN (500),
PARTITION p5 VALUES LESS THAN (600),
PARTITION p6 VALUES LESS THAN (700);

The above statement is not syntactically correct but the error thrown in this case is not particularly helpful. All that is missing here is a couple of braces around the partition range definitions. The below DDL statement is correct.

^?View Code MYSQL

CREATE TABLE People
(
	PersonId INTEGER NOT NULL AUTO_INCREMENT,
	FirstName VARCHAR(50) NOT NULL,
	LastName VARCHAR(50) NOT NULL,
	DateOfBirth DATE NOT NULL,
	Telephone VARCHAR(30) NULL,
	Email VARCHAR(200) NULL,
	GroupId SMALLINT NOT NULL,
	PRIMARY KEY (PersonId, GroupId)
)
PARTITION BY RANGE (GroupId)
(
	PARTITION p0 VALUES LESS THAN (100),
	PARTITION p1 VALUES LESS THAN (200),
	PARTITION p2 VALUES LESS THAN (300),
	PARTITION p3 VALUES LESS THAN (400),
	PARTITION p4 VALUES LESS THAN (500),
	PARTITION p5 VALUES LESS THAN (600),
	PARTITION p6 VALUES LESS THAN (700)
);

Post from: youdidwhatwithtsql.com

For RANGE partitions each partition must be defined

Error dropping database (can’t rmdir ‘./database’, errno: 39)

Rhys — Wed, 23 Jun 2010 21:12:55 +0000

Just a very quick post today! If you encounter this error when attempting to drop a MySQL database;

Error dropping database (can't rmdir './database', errno: 39)

Then you probably have some rogue files in the folder where the database files are located. If you cd into this directory you will be able to view these files. In my case I had a tar archive containing the files the database was restored from. Just delete any files in this directory and you’ll be able to complete the drop.

Post from: youdidwhatwithtsql.com

Error dropping database (can’t rmdir ‘./database’, errno: 39)

Unsigned Integer Arithmetic in SQL

Rhys — Sat, 12 Jun 2010 16:45:27 +0000

Not the sexiest blog title in the world but I thought I’d knock up a little post on the behaviour of MySQL and SQL Server with integer subtraction. How would you expect a database system to behave with positive and negative data types? Microsoft SQL Server doesn’t really have unsigned data types. All integer types can be positive and negative with the exception of TINYINT. MySQL implements the concept of signedness so we can specify that TINYINT ranges from –128 to 127 or 0 to 255.

What would you expect SQL Server to do with this?

^?View Code TSQL

DECLARE @num1 TINYINT = 50, @num2 TINYINT = 75;
 
SELECT @num1 - @num2;

Well it errors. Good RDBMS!

Msg 8115, Level 16, State 2, Line 3
Arithmetic overflow error converting expression to data type tinyint.

What does MySQL do? Lets see.

^?View Code MYSQL

USE test;
 
CREATE TABLE nums
(
	num1 INTEGER UNSIGNED NOT NULL,
	num2 INTEGER UNSIGNED NOT NULL
);
 
INSERT INTO nums
(
	num1,
	num2
)
VALUES
(
	50,
	75
);
 
SELECT num1 - num2
FROM nums;

Oh dear not a pretty behaviour! Bad RDBMS! A colleague said to me “you need to set sql_mode to traditional”. Mmmmmm, I thought I was running in traditional mode already. As it turns out we need to set the NO_UNSIGNED_SUBTRACTION option.

^?View Code MYSQL

SET @@SESSION.sql_mode = 'TRADITIONAL,NO_UNSIGNED_SUBTRACTION';
 
SELECT num1 - num2
FROM nums;

Got to love MySQL for keeping you on your toes!

Post from: youdidwhatwithtsql.com

Unsigned Integer Arithmetic in SQL

MySQL ALTER PROCEDURE Syntax

Rhys — Thu, 03 Jun 2010 11:15:37 +0000

I usually use SQLYog to write any stored procedures for MySQL. Whenever you alter a procedure the editor essentially generates SQL to drop and then recreate it.

^?View Code MYSQL

DELIMITER $$
 
USE `db`$$
 
DROP PROCEDURE IF EXISTS `my_proc`$$
 
CREATE DEFINER=`root`@`%` PROCEDURE `my proc`()
    MODIFIES SQL DATA
    SQL SECURITY INVOKER
    COMMENT 'Just an example proc.'
BEGIN
       # // proc defintition
END$$
 
DELIMITER ;

I’ve always wondered why it did this as opposed to using the ALTER PROCEDURE Syntax. I attempted to use this syntax but couldn’t get it to parse whatever I did. As usual the manual explains all;

This statement can be used to change the characteristics of a stored procedure. More than one change may be specified in an ALTER PROCEDURE statement. However, you cannot change the parameters or body of a stored procedure using this statement; to make such changes, you must drop and re-create the procedure using DROP PROCEDURE and CREATE PROCEDURE. source

Okay! The alter procedure statement is virtually useless to us then! The current method of dropping and then recreating routines to modify them is not atomic and can cause problems on production systems. The stored procedure is temporarily unavailable after the first drop statement so this might freak out any other thread calling it at that moment.

I admit this isn’t a massive problem, and I assume it wouldn’t be that difficult to implement, but it’s something you’d expect DBMS vendors to put in place for stored procedures. I enjoy working with MySQL but some days you can get irrated by a number of these little issues!

Post from: youdidwhatwithtsql.com

MySQL ALTER PROCEDURE Syntax

MySQL client ran out of memory

Rhys — Fri, 21 May 2010 11:38:45 +0000

I’ve been building utilities with PHP and MySQL command-line tools to clone databases. I ran into an issue when exporting data from multi-gigabyte tables using the mysql client program.

mysql: Out of memory (Needed 4179968 bytes)
ERROR 2008 (HY000) at line 1: MySQL client ran out of memory

The fix for this is easy; just use the –quick option in your call to the mysql command. This prevents the client from caching the resultset forcing it to write each row as it is received.

mysql -q -h localhost -P 3306 -u user -psecret -D db -e 'SELECT * FROM table' > '/tmp/table.csv'

Post from: youdidwhatwithtsql.com

MySQL client ran out of memory

Scripting out database objects with PHP

Rhys — Sun, 25 Apr 2010 14:22:44 +0000

I’ve recently needed to script out the create sql for various MySQL database objects. No Powershell or SMO to help with this so I’ve quickly rolled a PHP script to get this done.

This will script out all tables, views, triggers, stored procedures and functions from the specified database. One file per object in your /tmp directory (you’ll need to change this if you’re running on Windows). Just change the $source_host, $source_db, $source_user and $source_pwd variables to point at the database you want to script out.

^?View Code PHP


 
// set source and connection variables
$source_host = "localhost";
$source_db = "xxxx";
$source_user = "xxxx";
$source_pwd = "xxxx";
 
///////////////////////////////////////////////
// A few helper functions
///////////////////////////////////////////////
function getMySQLConnection($host, $database, $user, $pwd)
{
    $conn = mysql_connect($host, $user, $pwd) or die(mysql_error());
    mysql_select_db($database, $conn) or die(mysql_error());
    return $conn;
}
 
// returns a list of tables
function getTables($connection, $database)
{
    $tables = array();
    $result = mysql_query("SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA = '".$database."'", $connection) or die(mysql_errno());
    while($row = mysql_fetch_row($result))
    {
        array_push($tables, $row[0]);
    }
    return $tables;
}
 
// returns a list of views
function getViews($connection, $database)
{
    $views = array();
    $result = mysql_query("SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'VIEW' AND TABLE_SCHEMA = '".$database."'", $connection) or die(mysql_errno());
    while($row = mysql_fetch_row($result))
    {
        array_push($views, $row[0]);
    }
    return $views;
}
 
// returns a list of procs & functions
function getRoutines($connection, $database)
{
    $routines = array();
    $result = mysql_query("SELECT ROUTINE_NAME FROM INFORMATION_SCHEMA.ROUTINES WHERE TABLE_SCHEMA = '".$database."'", $connection) or die(mysql_errno());
    while($row = mysql_fetch_row($result))
    {
        array_push($routines, $row[0]);
    }
    return $routines;
}
 
// returns a list of triggers
function getTriggers($connection, $database)
{
    $triggers = array();
    $result = mysql_query("SELECT TRIGGER_NAME FROM INFORMATION_SCHEMA.TRIGGERS WHERE TABLE_SCHEMA = '".$database."'", $connection) or die(mysql_errno());
    while($row = mysql_fetch_row($triggers))
    {
        array_push($triggers, $row[0]);
    }
    return $triggers;
}
 
// writes text files
function writeTextFile($contents, $filename)
{
    $writer = fopen($filename, 'w') or die("Unable to open file");
    fwrite($writer, $contents);
    fclose($writer);
}
 
// Gets the sql used to create table
function getTableCreate($connection, $table)
{
    $sql = "SHOW CREATE TABLE `".$table."`";
    $result = mysql_query($sql, $connection) or die(mysql_error());
    $row = mysql_fetch_row($result);
    return $row[1];
}
 
// returns the sql used to create a routine
function getRoutineCreate($connection, $routine)
{
    $sql = "SHOW CREATE PROCEDURE `".$routine."`";
    $result = mysql_query($sql, $connection) or die(mysql_error());
    $row = mysql_fetch_row($result);
    return $row[1];
}
 
// returns the sql used to create a trigger
function getTriggerCreate($connection, $trigger)
{
    $sql = "SHOW CREATE TRIGGER `".$trigger."`";
    $result = mysql_query($sql, $connection) or die(mysql_error());
    $row = mysql_fetch_row($result);
    return $row[1];
}
 
/////////////////////////////////////////////////////////
// EOF FUNCTIONS
/////////////////////////////////////////////////////////
 
$source_connection = getMySQLConnection($source_host, $source_db, $source_user, $source_pwd);
 
$tables = getTables($source_connection, $source_db);
 
// Get the create sql for each table
// writing each one to a text file
foreach ($tables as $table)
{
    $create_sql = getTableCreate($source_connection, $table);
    $filename = "/tmp/".$table.".sql";
    writeTextFile($create_sql, $filename);
    echo "Generated create table sql for ".$table."\n";
}
 
$views = getViews($source_connection, $source_db);
 
// Create the sql for each view
foreach($views as $view)
{
    $create_sql = getTableCreate($source_connection, $view);
    $filename = "/tmp/".$view.".sql";
    writeTextFile($create_sql, $filename);
    echo "Generated create view sql for ".$view."\n";
}
 
$routines = getRoutines($source_connection, $source_db);
 
// get the create sql for each routine
foreach($routines as $routine)
{
    $create_sql = getRoutineCreate($source_connection, $routine);
    $filename = "/tmp/".$routine.".sql";
    writeTextFile($create_sql, $filename);
    echo "Generated create routine sql for ".$routine."\n";
}
 
$triggers = getTriggers($source_connection, $source_db);
 
// get the create sql for each trigger
foreach($triggers as $trigger)
{
    $create_sql = getTriggerCreate($source_connection, $trigger);
    $filename = "/tmp/trg_".$trigger.".sql";
    writeTextFile($create_sql, $filename);
    echo "Generated create trigger sql for ".$trigger."\n";
}
 
?>

Post from: youdidwhatwithtsql.com

Scripting out database objects with PHP