Comparing accented strings with TSQL

Rhys — Mon, 02 Aug 2010 20:38:47 +0000

Today a colleague was running into some difficulties matching football team names containing accented characters. For example Olympique Alès and Olympique Ales were not matching when he wanted them to. The issue here is all to do with collations. We can use the Latin1_General_CI_AI collation in our queries to force the comparison to ignore accents (AI stands for Accent Insensitive).

^?View Code TSQL

DECLARE @string1 VARCHAR(30), @string2 VARCHAR(30)
 
SET @string1 = 'Olympique Alès'; -- With accented e
SET @string2 = 'Olympique Ales'; -- Without accented e
 
IF @string1 = @string2
BEGIN
	PRINT 'Strings do not match!'; -- This will not print
END
 
IF @string1 COLLATE Latin1_General_CI_AI = @string2 COLLATE Latin1_General_CI_AI
BEGIN
	PRINT 'Strings match if we use COLLATE Latin1_General_CI_AI!'; -- This will print
END

Here’s how it works with GROUP BY.

^?View Code TSQL

CREATE TABLE dbo.Places
(
	Id INTEGER IDENTITY(1,1) NOT NULL PRIMARY KEY,
	Place VARCHAR(30) NOT NULL
);
 
-- Insert some test data
INSERT INTO dbo.Places 
(
	Place
)
VALUES
(
	'Olympique Alès'
),
(
	'Olympique Ales'
),
(
	'Gazélec Ajaccio'
),
(
	'Gazelec Ajaccio'
);
 
 
SELECT Place, COUNT(*)
FROM dbo.Places
GROUP BY Place;

^?View Code TSQL

SELECT Place COLLATE Latin1_General_CI_AI, COUNT(*)
FROM dbo.Places
GROUP BY Place COLLATE Latin1_General_CI_AI;

You can also use this with JOINS.

^?View Code TSQL

-- No COLLATE
SELECT *
FROM dbo.Places t1
INNER JOIN dbo.Places t2
	ON t1.Place = t2.Place;

^?View Code TSQL

-- With COLLATE
SELECT *
FROM dbo.Places t1
INNER JOIN dbo.Places t2
	ON t1.Place COLLATE Latin1_General_CI_AI = t2.Place COLLATE Latin1_General_CI_AI;

So that’s how you can force strings to match if they contain accented characters. I was using this for a quick data mapping task but be wary of using the COLLATE clause, in any applications or processes, where performance may become an issue.

If you use the COLLATE clause then this will mean the database engine cannot use any index on the referenced column. You could replace the accented characters before inserting them into your database but here’s a solution I prefer using persisted computed columns.

^?View Code TSQL

-- Alter Places table. Note we specify the collation
ALTER TABLE dbo.Places ADD CleanPlaceName AS Place COLLATE Latin1_General_CI_AI PERSISTED;
-- Index this column!
CREATE INDEX idx_CleanPlaceName ON dbo.Places (CleanPlaceName);
 
-- Note accents in CleanPlaceName are preserved
SELECT *
FROM dbo.Places
 
-- No COLLATE needed!
SELECT CleanPlaceName, COUNT(*)
FROM dbo.Places
GROUP BY CleanPlaceName;

^?View Code TSQL

SELECT *
FROM dbo.Places t1
INNER JOIN dbo.Places t2
	ON t1.CleanPlaceName = t2.CleanPlaceName;

This solution removes the need to include the COLLATE clause in your queries and keeps the possibility of using indices open!

Post from: youdidwhatwithtsql.com

Comparing accented strings with TSQL

Cannot resolve the collation conflict

Rhys — Fri, 12 Jun 2009 17:57:53 +0000

I do a fair bit of work with Linked Servers and cross-database queries and sometimes come across the following error when joining between databases with different collations;

Msg 468, Level 16, State 9, Line 1

Cannot resolve the collation conflict between 'Latin1_General_CI_AS' and 'SQL_Latin1_General_Pref_CP850_CI_AS'; in the equal to operation.

To replicate this error run the below TSQL to create two databases with tables and data.

^?View Code TSQL

-- Create first database
CREATE DATABASE database1 COLLATE Latin1_General_CI_AS;
GO
-- Create second database
CREATE DATABASE database2 COLLATE SQL_Latin1_General_Pref_CP850_CI_AS;
GO
 
USE database1;
GO
-- Create Customer table in database1
CREATE TABLE Customer
(
	CustID INTEGER NOT NULL IDENTITY(1, 1) PRIMARY KEY CLUSTERED,
	FirstName VARCHAR(30) NOT NULL,
	LastName VARCHAR(30) NOT NULL,
	DOB DATETIME NOT NULL,
	StartDate DATETIME NOT NULL DEFAULT GETDATE()
); 
GO
USE database2;
GO
-- Create Customer table in database2
CREATE TABLE Customer
(
	CustID INTEGER NOT NULL IDENTITY(1, 1) PRIMARY KEY CLUSTERED,
	FirstName VARCHAR(30) NOT NULL,
	LastName VARCHAR(30) NOT NULL,
	DOB DATETIME NOT NULL,
	StartDate DATETIME NOT NULL DEFAULT GETDATE()
); 
GO
 
USE database1;
GO
-- Insert test data
INSERT INTO dbo.Customer
(
	FirstName,
	LastName,
	DOB
)
SELECT 'Joe', 'Bloggs', '1975-01-01 00:00:00'
UNION ALL
SELECT'Dave', 'Smith', '1977-10-11 00:00:00'
UNION ALL
SELECT'Fred', 'Bloggs', '1965-11-28 00:00:00'
UNION ALL
SELECT'Sue', 'Smith', '1974-06-17 00:00:00'
UNION ALL
SELECT 'Steve', 'Smith', '1981-07-07 00:00:00';
GO
USE database2;
GO
-- Insert test data
INSERT INTO dbo.Customer
(
	FirstName,
	LastName,
	DOB
)
SELECT 'Joe', 'Bloggs', '1975-01-01 00:00:00'
UNION ALL
SELECT'Dave', 'Smith', '1977-10-11 00:00:00'
UNION ALL
SELECT'Fred', 'Bloggs', '1965-11-28 00:00:00'
UNION ALL
SELECT'Sue', 'Smith', '1974-06-17 00:00:00'
UNION ALL
SELECT 'Steve', 'Smith', '1981-07-07 00:00:00';
GO

Run the following query to observe the collation conflict.

^?View Code TSQL

-- Cross-database query causing the collation conflict
SELECT *
FROM dbo.Customer c1
INNER JOIN database1.dbo.Customer AS c2
	ON c1.FirstName = c2.FirstName
AND c1.LastName = c2.LastName;

You could fix this by changing the collation in one of the databases, i.e.

^?View Code TSQL

-- Set database2 to the same collation as database1
ALTER DATABASE database2 COLLATE Latin1_General_CI_AS;
-- Change VARCHAR columns on our existing tables
ALTER TABLE Customer ALTER COLUMN FirstName VARCHAR(30) COLLATE Latin1_General_CI_AS;
ALTER TABLE Customer ALTER COLUMN LastName VARCHAR(30) COLLATE Latin1_General_CI_AS;

Sometimes you may not want, or be able, to change a database in this way. In these situations you can add COLLATE DATABASE_DEFAULT to the JOINS or expressions in your query.

^?View Code TSQL

-- Using COLLATE DATABASE_DEFAULT
SELECT *
FROM dbo.Customer c1
INNER JOIN database1.dbo.Customer AS c2
	ON c1.FirstName = c2.FirstName COLLATE DATABASE_DEFAULT
AND c1.LastName = c2.LastName COLLATE DATABASE_DEFAULT;

Post from: youdidwhatwithtsql.com

Cannot resolve the collation conflict

youdidwhatwithtsql.com » collations

Comparing accented strings with TSQL

Cannot resolve the collation conflict