Check Windows Memory Usage with NSClient++

Warn if memory usage reaches 85%, critical if 90%.

define service {
        use                     generic-service
        host_name               windoze
        service_description     Memory Usage
        check_command           check_nt!MEMUSE!-w 85 -c 90
        servicegroups           windows
}

Check Windows Disk Usage with NSClient++

Warn if the disk is 85% full, critical at 90%. It seems there’s no way to check all disks at once. You have to setup a check for each one separately.

define service {
        use                     generic-service
        host_name               windoze
        service_description     C:\ Drive Space
        check_command           check_nt!USEDDISKSPACE!-l c -w 85 -c 90
        servicegroups           windows
}

Check Windows Service is running with NSClient++

This will alert if the MSSQLSERVER service is not running.

define service {
        use                     generic-service
        host_name               windoze
        service_description     SQL Server Engine
        check_command           check_nt!SERVICESTATE!-d SHOWALL -l MSSQLSERVER
        servicegroups           sql-server
}

Check a Windows Executable is running with NSClient++

This will check that the process sqlservr.exe is running.

define service {
        use                     generic-service
        host_name               windoze
        service_description     Process Check Test
        check_command           check_nt!PROCSTATE!-d SHOWALL -l sqlservr.exe=started
        servicegroups           windows
}

Check SQL Server CPU Time with NSClient++

This check accesses Windows performance counters. Documentation on this wasn’t great. Consult the list of counters in perfmon.exe for what’s available. Two backslahes are needed.

define service {
        use                     generic-service
        host_name               windoze
        service_description     Perf CPU Counter Test
        check_command           check_nt!COUNTER!-l "\\Process(sqlservr)\\% Processor Time","CPU Time for sqlservr is %.f.0 %%","Processor Time"
        servicegroups           windows
}

 Check SQL Server Memory Working Set with NSClient++

Virtually the same setup as the previous check. Just a different counter is accessed to return the memory used by SQL Server.

define service {
        use                     generic-service
        host_name               windoze
        service_description     Perf Mem Counter Test
        check_command           check_nt!COUNTER! -d SHOWALL -l "\\Process(sqlservr)\\Working Set","Mem Working Set for sqlservr is %.f.0","Working Set"
        servicegroups           windows
}

Check SQL Server User Connections with NSClient++

Another performance monitor counter check. This one returns the number of user connections to the SQL Server instance.

define service {
        use                     generic-service
        host_name               windoze
        service_description     SQL Server User Connections
        check_command           check_nt!COUNTER!-d SHOWALL -l "\\SQLServer:General Statistics\\User Connections","Number of Users: %.f",Users
        servicegroups           sql-server
}

Post from: youdidwhatwithtsql.com

Monitoring Windows with NSClient++

define timeperiod {
	name			england-wales-holidays
	timeperiod_name		england-wales-holidays
	alias			England & Wales Holidays

	january 1		00:00-00:00	; New Years Day
	march 29		00:00-00:00	; Good Friday (Variable) 29 March good for 2013
	april 1			00:00-00:00	; Easter Monday (Variable) 1 April good for 2013
	monday 1 may		00:00-00:00	; May Day Bank Holiday (1st Monday in month)
	monday -1 may		00:00-00:00	; Spring Bank Holiday (Last Monday in month)
	monday -1 august	00:00-00:00	; Late Summer Bank Holiday
	december 25		00:00-00:00	; Public Holiday rolls forward if this falls on a Weekend (Good until 2016)
	december 26		00:00-00:00	; Public Holiday rolls forward if this falls on a Weekend (Good until 2015)
}

I never realised Scotland had such a variable system for public Holidays so have fun setting that up! Some of the holiday dates here are variable check here for details.

Post from: youdidwhatwithtsql.com

Nagios timeperiod for Bank Holidays

SELECT *
FROM INFORMATION_SCHEMA.TABLES 
GROUP BY TABLE_SCHEMA;

This will generate the following error;

Msg 8120, Level 16, State 1, Line 1
Column 'INFORMATION_SCHEMA.TABLES.TABLE_CATALOG' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause.

While in MySQL this query would return a resultset. I believe this was implemented for performance reasons as it would clearly be more efficient grouping on one column than multiple ones. Obviously you introduce a degree of uncertainty as to what will be presented in your final resultset. That may be tolerable for some use cases but if you want the more standard functionality here’s what you have to do;

First check your sql_mode with the following;

SHOW VARIABLES LIKE '%sql_mode%';

This will return the value for your server which you’ll want to take a note of in case you want to revert changes. To change the default behaviour we need to add ONLY_FULL_GROUP_BY to the sql_mode;

SET sql_mode := CONCAT(@@sql_mode,',ONLY_FULL_GROUP_BY');

Run this query again;

SELECT *
FROM INFORMATION_SCHEMA.TABLES 
GROUP BY TABLE_SCHEMA;

and we will get the following error;

Query : SELECT * FROM INFORMATION_SCHEMA.TABLES  GROUP BY TABLE_SCHEMA
Error Code : 1055
'TABLES.TABLE_CATALOG' isn't in GROUP BY

This will require MySQL queries to use the standard GROUP BY implementation. This would be a breaking change for some queries so be very careful in production systems. Of course, to make this change permanent you’ll need to add it to your my.cnf.

Post from: youdidwhatwithtsql.com

MySQL Group By

UNK could not list MySQL's open files

After a little debugging we tracked this down to a problem caused by running multiple instances of mysqld. This is something the script author (Baron Schwartz) mentions in his script “# TODO: We could auto-check every running instance, not just one.”. I’ve replaced the following line

local PROC_ID=$(_pidof mysqld | head -n1)

With…

local PROC_ID=`ps aux | grep mysqld | grep -v mysqld_safe | grep $OPT_PORT | awk '{print $2}'`;

Note, this uses the MySQL port number supplied to the script to identify the correct MySQL instance. Probably a cleaner way to do this so I’ll think about updating this post with a better way in the future. Usual disclaimer applies; not properly tested, buyer beware, keep out of reach of children, may contain nuts etc.

Post from: youdidwhatwithtsql.com

pmp-check-mysql-deleted-files plugin issue

rpm -qa --queryformat '%{NAME} %{VERSION} %{INSTALLTIME:date}\n' --last;

Output will look something like below…

xclip-0.11-11.1.3                             Wed 07 Mar 2012 18:15:13 GMT
splunk-4.3-115073                             Thu 26 Jan 2012 15:51:36 GMT
glibc-devel-32bit-2.14.1-14.18.1              Thu 26 Jan 2012 15:40:04 GMT
glibc-devel-2.14.1-14.18.1                    Thu 26 Jan 2012 15:40:03 GMT
glibc-locale-2.14.1-14.18.1                   Thu 26 Jan 2012 15:39:58 GMT
busybox-1.18.3-5.1.3                          Thu 26 Jan 2012 15:39:38 GMT
glibc-2.14.1-14.18.1                          Thu 26 Jan 2012 15:39:35 GMT
glibc-32bit-2.14.1-14.18.1                    Thu 26 Jan 2012 15:39:32 GMT
MozillaFirefox-translations-common-9.0.1-1.1  Thu 12 Jan 2012 14:44:46 GMT
mozilla-nss-devel-3.13.1-5.1                  Thu 12 Jan 2012 14:44:25 GMT
MozillaFirefox-9.0.1-1.1                      Thu 12 Jan 2012 14:44:17 GMT
libsoftokn3-3.13.1-5.1                        Thu 12 Jan 2012 14:44:03 GMT
mozilla-nss-3.13.1-5.1                        Thu 12 Jan 2012 14:44:02 GMT
MozillaFirefox-branding-openSUSE-5.0-15.1     Thu 12 Jan 2012 14:44:00 GMT
libfreebl3-3.13.1-5.1                         Thu 12 Jan 2012 14:43:59 GMT
mozilla-nss-certs-3.13.1-5.1                  Thu 12 Jan 2012 14:43:57 GMT
mysql-workbench-5.1.16-2.3.5                  Tue 01 Nov 2011 14:28:47 GMT
mozilla-nspr-devel-4.8.9-2.1                  Mon 31 Oct 2011 13:58:58 GMT
mozilla-nspr-4.8.9-2.1                        Mon 31 Oct 2011 13:58:49 GMT
MySQL-server-5.5.15-1.linux2.6                Tue 30 Aug 2011 13:23:44 BST

I’m auditing a bunch of servers and handling a lot of output in a console windows can be a pain. That’s where xclip can come in. Just pipe the output to xcip and everything will get copied to your hosts clipboard to paste into your reports;

rpm -qa --queryformat '%{NAME} %{VERSION} %{INSTALLTIME:date}\n' --last | xclip;

Post from: youdidwhatwithtsql.com

Quick Linux Tip: rpm query & xclip

One such situation arose recently with Service Principal Names. SPNs are used predominately with impersonation and delegation. There’s a good explanation on SPNs here.

The account SQL Server runs under requires the “Read/Write servicePrincipalName” permission. If for some reason this is removed, and you restart SQL Server, client using the SPN will no longer function.

Luckily this can be easily checked with Quest’s Powershell cmdlets for Active Directory. Once these are installed you can simply do…

Get-QADPermission -Identity Domain\SqlServiceAcc | Where-Object {$_.RightsDisplay -eq "Read/Writ
e servicePrincipalName" -and $_.AccessControlType -eq "Allow"} | SELECT * | Format-List;

The output should look something like below. You may want to hold off rebooting your SQL Server if the ReadProperty/WriteProperty values are missing.

NativeAce         : System.DirectoryServices.ActiveDirectoryAccessRule
TargetObject      : Domain\SqlServiceAcc
Account           : NT AUTHORITY\SELF
TransitiveAccount : NT AUTHORITY\SELF
AccountName       : NT AUTHORITY\SELF
AccessControlType : Allow
Rights            : ReadProperty, WriteProperty
RightsDisplay     : Read/Write servicePrincipalName
Source            : NotInherited
ExtendedRight     :
ValidatedWrite    :
Property          : CN=Service-Principal-Name,CN=Schema,CN=Configuration,DC=domain,DC=co,DC=uk
PropertySet       :
ApplyTo           : ThisObjectOnly
ApplyToDisplay    : This object only
ApplyToType       :
ChildType         :

Post from: youdidwhatwithtsql.com

Check the SQL Server Service Account Can Write the SPN

DELIMITER $$
 
DROP PROCEDURE IF EXISTS `db_maintenance`$$
 
CREATE
    DEFINER = 'root'@'%'
    PROCEDURE db_maintenance
    (
		IN p_mode TINYINT,
		IN p_database VARCHAR(128)
    )
    LANGUAGE SQL
    SQL SECURITY INVOKER
    BEGIN
 
		##################################################
		# Author: Rhys Campbell                          #
		# Created: 2012-03-02                            #
		# Description: Performs Analyze or Optimize      #
		# actions on all tables in the provided db.      #
		##################################################
 
		DECLARE done TINYINT;
		DECLARE my_table VARCHAR(128);
 
		# Table cursor
		DECLARE table_cursor CURSOR FOR SELECT t.TABLE_NAME
						FROM INFORMATION_SCHEMA.TABLES t
						WHERE t.TABLE_SCHEMA = p_database
						AND NOT EXISTS (SELECT *
								FROM db_maintenance_table_excludes t											WHERE t2.database_name = t.TABLE_SCHEMA
														AND t2.table_name = t.TABLE_NAME)
										AND t.TABLE_TYPE = 'BASE TABLE';
 
		DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
 
		SET done = 0;
 
		OPEN table_cursor;										
 
		table_loop: LOOP
 
		FETCH table_cursor INTO my_table;
 
		# Leave the loop if we're done
		IF (done = 1) THEN
			LEAVE table_loop;
		END IF;
 
		# Now lets do the table maintenance
		IF(p_mode = 1) THEN # Optimize
 
			SET @q = CONCAT('OPTIMIZE TABLE ', p_database, '.', my_table);
			PREPARE stmt FROM @q;
			EXECUTE stmt;
 
		ELSEIF (p_mode = 2) THEN # Analyze
 
			SET @q = CONCAT('ANALYZE TABLE ', p_database, '.', my_table);
			PREPARE stmt FROM @q;
			EXECUTE stmt;
 
		END IF;
 
		END LOOP table_loop;
 
		# Clean up
		CLOSE table_cursor;
 
    END$$
 
DELIMITER ;

Tables are retrieved from INFORMATION_SCHEMA and I use the below table to hold any tables I want to exclude from automatic maintenance.

CREATE TABLE db_maintenance_table_excludes
(
	database_name VARCHAR(128),
	table_name VARCHAR(128),
	PRIMARY KEY
	(
		database_name,
		table_name
	)
);

Usage is as follows;

# Run OPTIMIZE TABLE
CALL `db_maintenance`(1, 'db_name');
# Run ANALYZE TABLE
CALL `db_maintenance`(2, 'db_name');

Post from: youdidwhatwithtsql.com

MySQL Database Maintenance Stored Procedure

The original solution consisted of a simple trigger and an archive table something like below;

CREATE TABLE dbo.SomeTable
(
	id INTEGER IDENTITY(1,1) NOT NULL PRIMARY KEY CLUSTERED,
	created DATETIME NOT NULL,
	url VARCHAR(100) NOT NULL,
	html NVARCHAR(MAX) NOT NULL
);

CREATE TRIGGER trg_InsRecord
   ON dbo.SomeTable
   AFTER INSERT
AS 
BEGIN
	-- SET NOCOUNT ON added to prevent extra result sets from
	-- interfering with SELECT statements.
	SET NOCOUNT ON;
 
    INSERT INTO dbo.SomeArchiveTable
    (
		id,
		created,
		url,
		html
	)
	SELECT id,
		   created,
		   url,
		   html
	FROM inserted;
 
END
GO

CREATE TABLE dbo.SomeArchiveTable
(
	id INTEGER NOT NULL PRIMARY KEY CLUSTERED,
	created DATETIME NOT NULL,
	url VARCHAR(100) NOT NULL,
	html NVARCHAR(MAX) NOT NULL
);

Probably a setup many DBAs have seen before. In the dbo.SomeTable we keep just a few hours of data while in dbo.SomeArchiveTable seven days of data was kept. The SQL Agent job to purge data ran nightly and was just a simple delete statement;

DELETE FROM dbo.SomeArchiveTable WHERE created <= DATEADD(dd, -14, GETDATE());

This was fine while the table was small (yes created was indexed) but over time, as the business grew, the amount of data increased from a few gigabytes to hundreds of gigabytes. The job was taking anywhere between 2 and 5 hours, locking the entire database, and causing client timeouts.  My SQL Server probably felt like this;

A temporary solution was to delete the data in small batches to prevent lock escalation.

DECLARE @rc INTEGER = -1;
 
WHILE(@rc != 0)
BEGIN
 
	-- Delete 4999 rows to prevent lock escalation
	-- http://msdn.microsoft.com/en-us/library/ms184286.aspx
	DELETE TOP (4999) 
	FROM dbo.SomeArchiveTable 
	WHERE created <= DATEADD(dd, -7, GETDATE())
 
	SET @rc = @@ROWCOUNT;
 
END

This solved the horrendous blocking issue but our SQL Server still wasn’t happy with several hours of heaving Disk I/O each night.

I’ve read a lot of great blogs about SQL Server Partitioning and thought it could be of use here. Essentially these blogs demonstrate how to use partitioning to make dealing with large amounts of data easier.

Partitioning is an Enterprise only feature and we run standard in production. Do we upgrade to Enterprise and pay for a Microsoft executives bonus?

With a little TSQL trickery we can stick with standard edition but get partition-like benefits.

Say we split our archive table into seven new tables, one for each day of the week, and change our trigger to insert an appropriate table according to the day.

CREATE TABLE [dbo].[SomeArchiveTable_Sunday](
	[id] [int] NOT NULL,
	[created] [datetime] NOT NULL,
	[url] [varchar](100) NOT NULL,
	[html] [nvarchar](max) NOT NULL,
PRIMARY KEY CLUSTERED 
(
	[id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
 
GO
 
CREATE TABLE [dbo].[SomeArchiveTable_Monday](
	[id] [int] NOT NULL,
	[created] [datetime] NOT NULL,
	[url] [varchar](100) NOT NULL,
	[html] [nvarchar](max) NOT NULL,
PRIMARY KEY CLUSTERED 
(
	[id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
 
GO
 
CREATE TABLE [dbo].[SomeArchiveTable_Tuesday](
	[id] [int] NOT NULL,
	[created] [datetime] NOT NULL,
	[url] [varchar](100) NOT NULL,
	[html] [nvarchar](max) NOT NULL,
PRIMARY KEY CLUSTERED 
(
	[id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
 
GO

… and so on for each day of the week.

Then we need to alter our trigger to insert data into the correct table. In this case it is determined by the current day returned via the DATEPART function.

ALTER TRIGGER [dbo].[trg_InsRecord]
   ON [dbo].[SomeTable]
   AFTER INSERT
AS 
BEGIN
	-- SET NOCOUNT ON added to prevent extra result sets from
	-- interfering with SELECT statements.
	SET NOCOUNT ON;
 
	-- What day is it?
	DECLARE @day TINYINT = DATEPART(dw, GETDATE());
 
	IF(@day = 1)
	BEGIN
		INSERT INTO dbo.SomeArchiveTable_Sunday
		(
			id,
			created,
			url,
			html
		)
		SELECT id,
			   created,
			   url,
			   html
		FROM inserted;
	END
	ELSE IF (@day = 2)
	BEGIN
		INSERT INTO dbo.SomeArchiveTable_Monday
		(
			id,
			created,
			url,
			html
		)
		SELECT id,
			   created,
			   url,
			   html
		FROM inserted;	
	END
	ELSE IF (@day = 3)
	BEGIN
		INSERT INTO dbo.SomeArchiveTable_Tuesday
		(
			id,
			created,
			url,
			html
		)
		SELECT id,
			   created,
			   url,
			   html
		FROM inserted;	
	END	
	ELSE IF (@day = 4)
	BEGIN
		INSERT INTO dbo.SomeArchiveTable_Wednesday
		(
			id,
			created,
			url,
			html
		)
		SELECT id,
			   created,
			   url,
			   html
		FROM inserted;	
	END	
	ELSE IF (@day = 5)
	BEGIN
		INSERT INTO dbo.SomeArchiveTable_Thursday
		(
			id,
			created,
			url,
			html
		)
		SELECT id,
			   created,
			   url,
			   html
		FROM inserted;	
	END
	ELSE IF (@day = 6)
	BEGIN
		INSERT INTO dbo.SomeArchiveTable_Friday
		(
			id,
			created,
			url,
			html
		)
		SELECT id,
			   created,
			   url,
			   html
		FROM inserted;	
	END
	ELSE IF (@day = 7)
	BEGIN
		INSERT INTO dbo.SomeArchiveTable_Saturday
		(
			id,
			created,
			url,
			html
		)
		SELECT id,
			   created,
			   url,
			   html
		FROM inserted;	
	END
END

So how do we purge data? Simple, just run something like this a little before midnight each day…

	-- What day is it?
	DECLARE @day TINYINT = DATEPART(dw, GETDATE());
 
	IF(@day = 1) -- Sunday - truncate Mondays table
	BEGIN
		TRUNCATE TABLE dbo.SomeArchiveTable_Monday;
	END
	ELSE IF (@day = 2) -- Monday -- truncate Tuesdays table
	BEGIN
		TRUNCATE TABLE dbo.SomeArchiveTable_Tuesday;
	END
	ELSE IF (@day = 3)
	BEGIN
		TRUNCATE TABLE dbo.SomeArchiveTable_Wednesday;
	END -- and so on for each day of the week...

Truncating a table takes milliseconds so our SQL Server is now much happier. Disk I/O is massively reduced. There’s also a couple of additional benefits from this approach; each table is smaller so re-indexing is much quicker, and usually a table being re-indexed is not being inserted into so no blocking occurs.

Post from: youdidwhatwithtsql.com

Purging data & Partitioning for Paupers

# Load SMO extension
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo") | Out-Null;
 
# Servers to check
$sqlservers = @("server1", "server2", "server3");
foreach($server in $sqlservers)
{
	$srv = New-Object "Microsoft.SqlServer.Management.Smo.Server" $server;
	# Get mirrored databases
	$databases = $srv.Databases | Where-Object {$_.IsMirroringEnabled -eq $true}; 
	Write-Host $server;
	Write-Host "==================================";
	$databases | Select-Object -Property Name, MirroringStatus | Format-Table -AutoSize;
}

This will output something looking like below…

server1
====================================

Name                 MirroringStatus
----                 ---------------
db1         		Synchronized
db2                    	Synchronized
db3              	Synchronized
db4			Synchronized
db5    			Synchronized
server2
=====================================

Name                 MirroringStatus
----                 ---------------
db1         		Synchronized
db2                    	Synchronized
db3              	Synchronized
db4			Synchronized
db5    			Synchronized
server3
=====================================

Name                 MirroringStatus
----                 ---------------
db1         		Synchronized
db2                    	Synchronized
db3              	Synchronized
db4			Synchronized
db5    			Synchronized

Post from: youdidwhatwithtsql.com

Check Mirroring Status with Powershell

function Get-ServerErrors
{
	# Server to check & hours back. Will only support default sql instances
	# Could add a third param for instance and modify script where appropriate if needed
	param ($server, [int]$hours);
 
	[datetime]$after = $(Get-Date).AddHours(-$hours);
 
	# Windows Event Log (Application & System) Errors & Warnings
 
	Write-Host "Application Event Log Errors from $server after $after";
	Write-Host "========================================================================";
	Get-EventLog -ComputerName $server -LogName "Application" -EntryType "Error" -After $after | Format-List;
	Write-Host "Press any key to continue or ctrl + c to quit";
	$r = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown");
 
	Write-Host "System Event Log Errors from $server after $after";
	Write-Host "========================================================================";
	Get-EventLog -ComputerName $server -LogName "System" -EntryType "Error" -After $after | Format-List;
	Write-Host "Press any key to continue or ctrl + c to quit";
	$r = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown");
 
	Write-Host "Application Event Log Warnings from $server after $after";
	Write-Host "========================================================================";
	Get-EventLog -ComputerName $server -LogName "Application" -EntryType "Warning" -After $after | Format-List;
	Write-Host "Press any key to continue or ctrl + c to quit";
	$r = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown");	
 
	Write-Host "System Event Log Warnings from $server after $after";
	Write-Host "========================================================================";
	Get-EventLog -ComputerName $server -LogName "System" -EntryType "Warning" -After $after | Format-List;
	Write-Host "Press any key to continue or ctrl + c to quit";
	$r = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown");	
 
	# SQL Server Error Log
	[reflection.assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo") ;
	$sql_server = new-object ("Microsoft.SqlServer.Management.Smo.Server") $server;
	$sql_server.ReadErrorLog() | Where-Object {$_.Text -like "Error*" -and $_.LogDate -ge $after};
}

Usage is as follows;

Get-ServerErrors <server name> <hours back to check>;

Post from: youdidwhatwithtsql.com

Get-ServerErrors Powershell Function