I recently attempted to use the ADO.NET Destination to load files into MySQL without luck. I tweeted about this and Todd McDermid (blog | twitter) kindly pointed me at a blog post about writing data to MySQL with SSIS. This works well but I am having difficulties with data conversions. The author of the blog post, Matt Mason, posted a follow up article with the various options of SSIS / MySQL interaction. All of these involve pushing data into MySQL but I though why not get it to pull data. Here’s an outline of this approach.

First create the following table in a MySQL database. This is a clone of the HumanResources.Employees table from the AdventureWorks sample database. We will be doing a simple extract-truncate-load of the table into MySQL from SQL Server.

# Create MySQL replica table of the HumanResources.Employee from the AdventureWorks database
CREATE TABLE `HumanResourcesEmployee`
(
	`EmployeeID` INT PRIMARY KEY NOT NULL,
	`NationalIDNumber` VARCHAR(15) NOT NULL,
	`ContactID` INT NOT NULL,
	`LoginID` VARCHAR(256) NOT NULL,
	`ManagerID` INT NULL,
	`Title` VARCHAR(50) NOT NULL,
	`BirthDate` DATETIME NOT NULL,
	`MaritalStatus` CHAR(1) NOT NULL,
	`Gender` CHAR(1) NOT NULL,
	`HireDate` DATETIME NOT NULL,
	`SalariedFlag` VARCHAR(5),
	`VacationHours` SMALLINT,
	`SickLeaveHours` SMALLINT,
	`CurrentFlag` VARCHAR(5),
	`rowguid` VARCHAR(40),
	`ModifiedDate` DATETIME
);

Next create a new Integration Services project in BIDS. Add a Data Flow Task to the project and call it "Extract Employees from SQL Server"

This will export the contents of the HumanResources.Employees table and write it to a flat file. Edit the task and add an OLE Source and configure it as illustrated below. The OLEDB connection manager should point at your SQL Server instance hosting the AdventureWorks database.

Next add a Flat File Destination and name it "Write Employees File" then connect the OLEDB Source to it. Configure the connection manager as illustrated below. The important things to note here are the fact I’m writing the file to E:\Employees.txt and it is pipe delimited. You may need to alter these according to your setup.

The dataflow should now look something like below.

Go back to the Control Flow pane and add an Execute SQL Task to the canvas. Call this task "TRUNCATE TABLE HumanResourcesEmployee". Edit the task and configure it like below. You need to add a ADO.NET connection manager which references an ODBC connection to your MySQL database. You’ll need the MySQL ODBC Driver for this.

My ODBC connection looks like below.

Add another Execute SQL Task and call it "Load new Employees file into MySQL". Connect up the tasks in sequence.

Edit "Load new Employees file into MySQL", add the ADO.NET connection to MySQL and enter the below SQL. You will need to change the file path and delimiter character if you have changed them.

LOAD DATA LOCAL INFILE 'E:\\Employees.txt' # Column order MUST match the table 
INTO TABLE HumanResourcesEmployee
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\r\n' # Windows line terminator
IGNORE 1 LINES; # Ignore column header line

I originally attempted to enclose this in a store procedure as this provides more flexibility but MySQL gave the following error.

LOAD DATA is not allowed in stored procedures

MySQL prevents you from using LOAD DATA INFILE inside stored procedures (my major beef with MySQL is what you can and cannot do inside stored procedures) so we have no choice but to enter it in the SQLStatement pane. Execute the package and if all goes well the Employees.txt file will be loaded into MySQL.

I’ve not yet used this in production so use with caution, but it’s simple to setup and fast. One thing to note for this load is that MySQL has replaced backslashes with hyphens in the LoginId column so it would be sensible to check all data conversions.

Post from: youdidwhatwithtsql.com

SSIS: Loading files into MySQL

The package executes daily so I’m lucky I caught this in time. I Googled around and found this thread which advised a reinstall of SSIS. Unfortunately I didn’t have this luxury so I decided to do some digging. I found the location of the Execute Process Task dll file in the Choose Toolbox Items dialog.

Since the error was complaining about the task being unregistered I thought I’d try registering the dll in the GAC. First I executed the below command;

C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322>gacutil /i "C:\Program Files\Micro
oft SQL Server\90\dts\tasks\Microsoft.SqlServer.ExecProcTask.dll"

Microsoft (R) .NET Global Assembly Cache Utility.  Version 1.1.4322.573
Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

Failure adding assembly to the cache: Unknown Error

C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322>

This failed because I was using a different version of the gacutil tool. As it turns out you have to use the appropriate version of gacutil for the dll. A quick 350MB download of the .NET Framework 2.0 Software Development Kit and I was ready to roll. I reattempted the above command with the correct version.

C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin>gacutil /i "C:\Program F
iles\Microsoft SQL Server\90\dts\tasks\Microsoft.SqlServer.ExecProcTask.dll"
Microsoft (R) .NET Global Assembly Cache Utility.  Version 2.0.50727.42
Copyright (c) Microsoft Corporation.  All rights reserved.

Assembly successfully added to the cache

C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin>

After this was registered I was able to edit the task with no problems. I couldn’t find any issues with any other task types so I was happy I didn’t go down the reinstall route.

Post from: youdidwhatwithtsql.com

SSIS Execute Process Task not registered for use on this Computer

• The Script Task seems to buckle for XML data types giving an unsupported conversion error. (Use the Person.Contact table in the AdventureWorks database to view this issue).
• The package does not seem to cope with a large number of rows. Execution appears to freeze for datasets containing somewhat over 1,000 rows. (Use the Person.Address table in the AdventureWorks database to view this issue). I’m wondering if this is due to size limitation with the System.Object type.

Here’s quick run through of how I resolved the issues. The basic idea was to do all of the data access in VB.Net rather than using an Execute SQL Task to store a resultset in a System.Object variable.

Open BIDS and create a new Integration Services Project.

First add a string variable to the project as illustrated below.

Set the EvaluateAsExpression property to true and enter the following expression.

 (DT_WSTR,4)YEAR(GETDATE()) + "_"
    + RIGHT("0" + (DT_WSTR,2)MONTH(GETDATE()), 2) + "_"
    + RIGHT("0" + (DT_WSTR,2)DAY( GETDATE()), 2) + "_"
    + RIGHT("0" + (DT_WSTR,2)DATEPART("hh", GETDATE()), 2) + "_" + RIGHT("0" + (DT_WSTR,2)DATEPART("mi", GETDATE()), 2) + "_" + RIGHT("0" + (DT_WSTR,2)DATEPART("ss", GETDATE()), 2) + ".txt"

This will provide us with a datetime stamped filename looking similar to 2010_02_20_15_40_36.txt.

Next add another string variable called query and enter a statement to execute a stored procedure that returns a resultset. Any one will do as the package should be agnostic to the results.

I’m using the below stored procedure running against the AdventureWorks sample database.

USE [AdventureWorks]
GO
/****** Object:  StoredProcedure [dbo].[usp_test]    Script Date: 02/20/2010 15:48:49 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author:		Rhys Campbell
-- Create date: 2010-02-20
-- Description:	Test proc
-- =============================================
ALTER PROCEDURE [dbo].[usp_test]
AS
BEGIN
	-- SET NOCOUNT ON added to prevent extra result sets from
	-- interfering with SELECT statements.
	SET NOCOUNT ON;
 
    	-- Insert statements for procedure here
	SELECT *
	FROM Person.Contact;
 
END

Next add an ADO.NET Connection Manager to the project pointing this at the database containing your stored procedure. Rename the connection manager to DBConnection as we will be referencing this by name in code later.

Drop a Script Task onto the designer and edit its properties. Add the filename and query variables to the ReadOnlyVariables property.

Set the script language to Microsoft Visual Basic 2008 and then click "Edit Script". Paste the below code into the editor.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime

 _
 _
Partial Public Class ScriptMain
	Inherits Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase

	Enum ScriptResults
		Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success
		Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
	End Enum

    Public Sub Main()

        ' Get the datetime stamped filename
        Dim filename As String = Dts.Variables("filename").Value.ToString
        ' Get the user desktop directory. We'll write the output file to here
        ' You probably want to use some other location for production purposes
        Dim userProfileDir As String = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory).ToString

        ' Get the Connection string. Must be called "DBConnection"
        Dim connStr As String = Me.Dts.Connections.Item("DBConnection").ConnectionString.ToString

        ' Get the query set in the ssis variables
        Dim query As String = Dts.Variables("query").Value.ToString

        ' Setup DB access stuff
        Dim con As SqlClient.SqlConnection = New SqlClient.SqlConnection(connStr)
        Dim comm As SqlClient.SqlCommand = New SqlClient.SqlCommand(query, con)
        Dim reader As SqlClient.SqlDataReader

        ' Open the connection
        con.Open()

        ' Get results
        reader = comm.ExecuteReader()

        ' temp variable to hold lines
        Dim line As String = ""

        ' Get the column names from the resultset
        For index As Integer = 0 To (reader.FieldCount - 1)
            line &= reader.GetName(index) & "|"
        Next

        ' remove last pipe
        line = line.Substring(0, line.Length - 1)

        ' write the column headers to the text file
        appendToTextFile(userProfileDir & "\" & filename, line)

        ' Write each row in the resultset to the text file
        While reader.Read()

            line = ""
            ' Add each column value to line
            For index As Integer = 0 To (reader.FieldCount - 1)
                ' Removing any pipes and newline to avoid screwing up our file
                line &= reader.GetValue(index).ToString.Replace("|", "").Replace(ControlChars.CrLf, vbNullString) & "|"
            Next

            ' remove last pipe and add a new line
            line = line.Substring(0, line.Length - 1)

            ' write the line to the output file
            appendToTextFile(userProfileDir & "\" & filename, line)

        End While

        ' clean up
        con.Close()
        comm = Nothing
        reader = Nothing
        con = Nothing

        Dts.TaskResult = ScriptResults.Success

    End Sub
    Public Sub appendToTextFile(ByVal file As String, ByVal line As String)

        ' Create a writer object that appends to a text file if it exists
        Dim objWriter As New System.IO.StreamWriter(file, True)
        objWriter.WriteLine(line)
        objWriter.Close()
        objWriter = Nothing

    End Sub

End Class

Now execute the package and check your desktop for the output file. My output file looked like below.

Now alter your stored procedure so it returns a different resultset. Here’s another one I used in the AdventureWorks database.

USE [AdventureWorks]
GO
/****** Object:  StoredProcedure [dbo].[usp_test]    Script Date: 02/20/2010 19:20:55 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author:		Rhys Campbell
-- Create date: 2010-02-20
-- Description:	Test proc
-- =============================================
ALTER PROCEDURE [dbo].[usp_test]
AS
BEGIN
	-- SET NOCOUNT ON added to prevent extra result sets from
	-- interfering with SELECT statements.
	SET NOCOUNT ON;
 
    -- Insert statements for procedure here
	SELECT *
	FROM Person.Address;
 
END

Run the package again and you should find another output file on your desktop.

I’ve not thoroughly tested this code but it appears my initial issues have been resolved. This method does take significantly longer to write the output file than the traditional data flow method. This shouldn’t be a huge issue for the purposes I’m thinking of using it in so I should be soon deploying this into a production environment.

Post from: youdidwhatwithtsql.com

SSIS: Make your output files dynamic part 2

I wanted to build a little more flexibility into my packages. Wouldn’t it be great if we could simply modify a stored procedure to include additional columns, and these changes would be reflected in the output files, with no further work required? Here’s an outline of my first steps towards achieving this.

• First create a stored procedure, that returns a resultset, in a test database. Anyone will do. Here’s one I created in the AdventureWorks test database.

USE [AdventureWorks]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[usp_test] AS
BEGIN
 
	SELECT *
	FROM Production.Product;
 
END

• Open up BIDS and create a new integration services project.
• Add an object variable like below called resultset.

• Add an Execute SQL Task to the designer and configure it to execute the procedure you created earlier. Configure it similarly to below, ensuring Result Set is set to Full result set.

• Go to the Result Set tab and add a mapping to the resultset object variable. Configure this exactly as shown below.

• Add a Script Task to the designer and connect the Execute SQL Task to it. Edit the Script Task and enter the resultset variable in the ReadOnlyVariable box.

• Click “Edit Script” and paste the below code into VSTA. The only part of this code you should need to change is the path where the text file is written to.

' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic 2008.
' The ScriptMain is the entry point class of the script.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime
Imports System.Data.OleDb
Imports System.IO

 _
 _
Partial Public Class ScriptMain
    Inherits Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase

    Enum ScriptResults
        Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success
        Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
    End Enum

    Public Sub Main()
        '
        ' Add your code here
        '
        Dim filename As String = Format(Date.Now, "yyyy-MM-dd_hh_mm_ss") & ".csv"
        Dim fileContents As String = ""

        Dim oledb As OleDbDataAdapter = New OleDbDataAdapter()
        Dim table As DataTable = New DataTable()
        Dim rs As System.Object = Dts.Variables("resultset").Value

        oledb.Fill(table, rs)

        ' Get the column names
        For Each col In table.Columns
            fileContents &= col.ColumnName & "|"
        Next

        ' remove last pipe
        fileContents = fileContents.Substring(0, fileContents.Length - 1)
        fileContents &= Environment.NewLine

        ' For each row in the dataset
        Dim i As Integer
        For Each row As DataRow In table.Rows
            ' For each column in the row
            For i = 1 To table.Columns.Count
                fileContents &= row(i - 1).ToString() & "|"
            Next

            ' Remove last pipe and add a newline to the end of each row
            fileContents = fileContents.Substring(0, fileContents.Length - 1)
            fileContents &= Environment.NewLine

        Next

        ' write data to the text file
        ' Change the path to something appropriate
        SaveTextToFile(fileContents, "C:\Users\Rhys\Desktop\" & filename)

        Dts.TaskResult = ScriptResults.Success
    End Sub

    ' Function from http://www.freevbcode.com/ShowCode.Asp?ID=4492
    Public Function SaveTextToFile(ByVal strData As String, _
     ByVal FullPath As String, _
       Optional ByVal ErrInfo As String = "") As Boolean

        Dim bAns As Boolean = False
        Dim objReader As StreamWriter
        Try

            objReader = New StreamWriter(FullPath)
            objReader.Write(strData)
            objReader.Close()
            bAns = True
        Catch Ex As Exception
            ErrInfo = Ex.Message

        End Try
        Return bAns
    End Function

End Class

• Save the script and return to the designer. The final package should look something like below.

• Execute the package and then check the location where your output file should be written to. Here’s what the file should look like if you’re using my usp_test stored procedure in the AdventureWorks database.

• Now comes the fun part! Modify the usp_test procedure to select from a completely different table. For example;

USE [AdventureWorks]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[usp_test] AS
BEGIN
 
	SELECT *
	FROM HumanResources.Employee;
 
END

• Ordinarily this change would break an SSIS package. Execute the package again and check the output file.

• Let’s try another version of usp_test.

USE [AdventureWorks]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[usp_test] AS
BEGIN
 
	SELECT 'I can change my output files simply by changing this stored procedure!', GETDATE();
 
END

• Run the package and check the new output file.

OK, so we now have an SSIS package, producing output files, that copes with complete modifications to stored procedures with no other changes needed. This isn’t yet production ready as there are a few issues I need to resolve.

• The Script Task seems to buckle for XML data types giving an unsupported conversion error. (Use the Person.Contact table in the AdventureWorks database to view this issue).
• The package does not seem to cope with a large number of rows. Execution appears to freeze for datasets containing somewhat over 1,000 rows. (Use the Person.Address table in the AdventureWorks database to view this issue). I’m wondering if this is due to size limitation with the System.Object type.

I’ll update this post one I get around to resolving these issues.

UPDATE – See Part 2 to see how I resolved these issues.

Post from: youdidwhatwithtsql.com

SSIS: Make your output files dynamic

For me, any day of the week, it should always be a proc. Why on earth would you want to enter raw queries into this task? I would argue against this even for basic message logging. Enclosing your TSQL in stored procedures gives you much more flexibility. It’s so much easier to make minor modifications to a proc. Would you rather have to go through the rigmarole of opening and deploying SSIS packages. Sure, I know you’ll have to open the package if you introduce, or remove, new columns. But you’re pretty much free to modify your WHERE without issue. Lets not forget this also allows your system to be modified by those who don’t know SSIS. Perhaps you’ll be able to enjoy your holiday a little better next year.

I’m even tempted to open a Microsoft Connect item to fix this ‘bug’. Perhaps one day I will be lucky enough to see this task available in my toolbox.

Post from: youdidwhatwithtsql.com

Execute SQ… Procedure Task

First you’ll need to download the source code as there’s no installation package. Then you’ll need to compile the project to produce the dll. I used Visual Studio 2008 but I guess Visual C# Express Edition will do it.

Double click the file called Defiant.SqlServer.PowerShellScriptTask.csproj from the source code you downloaded. This will open the project in your IDE. Build the project and a bin\debug  directory containing a dll will be created in the project folder.

We are only interested in the file called Defiant.SqServer.PowerShellScriptTask.dll. We will need to register this file in the GAC and copy it to a place where SSIS can find it. Save the commands below, with appropriate modifications, to a batch file.

cd\
c:
cd C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin
gacutil /uf "Defiant.SqlServer.PowerShellScriptTask";
gacutil /if "C:\Users\Rhys\Desktop\Source\Defaint.SqlServer.PowerShellScriptTask\bin\Debug\Defiant.SqlServer.PowerShellScriptTask.dll";
copy "C:\Users\Rhys\Desktop\Source\Defaint.SqlServer.PowerShellScriptTask\bin\Debug\Defiant.SqlServer.PowerShellScriptTask.dll"; "C:\Program Files\Microsoft SQL Server\100\DTS\Tasks";

You may need to modify; the path on line 3, this is the location of gacutil. line5, this is the full path of Defiant.SqlServer.PowerShellScriptTask.dll. line 6 the full path of Defiant.SqlServer.PowerShellScriptTask.dll and the path to DTS\Tasks for the instance on SQL Server you want to copy it to. For some reason I couldn’t get this to work on 2005 but it worked fine for 2008. Restart the SSIS service using services.msc then we are ready to fire up BIDS.

When inside BIDS the Powershell Script Task will not appear in the toolbox automatically. Right click on the toolbox and select Choose Items. In the SSIS Control Flow Items tab scroll down to Powershell Script Task and check the box next to it.

Now the task will appear in the toolbox. Drop the task onto the designer to begin working with it.

Right click the task and choose edit. Anyone who has worked with the VB.Net \ C# Script Task should work this out straight away.

Let’s try something simple first. Enter the below line of Powershell in the Script window;

Write-Host "This is a test!";

Click OK and then run the package. Below is the output I received.

SSIS package "Package.dtsx" starting.
Information: 0x0 at PowerShell Script Task, PowerShellScriptTask: This is a test!
Information: 0x0 at PowerShell Script Task, PowerShellScriptTask:
SSIS package "Package.dtsx" finished: Success.

Great, it works! Now lets try working with some variables. Add two variables like below (substituting your name for mine);

Then edit the Powershell Script Task and add these variables as below.

Add the below Powershell to the Script window;

$ssis.Variables["User::write"] = $ssis.Variables["User::readOnly"];

This code just assigns the value of the readOnly variable to write. Next drop a VB.net Script task onto the designer and connect the Powershell Script Task to it. We’re going to add code here to display a message box with the contents of the write variable. This will show that our Powershell script task has successfully read from, and written to, our SSIS variables.

Click edit script and and the below VB.Net code.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime

 _
 _
Partial Public Class ScriptMain
	Inherits Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase

	Enum ScriptResults
		Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success
		Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
	End Enum

	Public Sub Main()
        Dts.Variables("write").Value = Dts.Variables("readOnly").Value.ToString
        MsgBox("Hello " & Dts.Variables("write").Value.ToString)
        Dts.TaskResult = ScriptResults.Success
    End Sub

End Class

Your final package should look something like below.

Execute the package and, if all goes well, you should see the below message box.

So that’s the basics of working with the Powershell Script Task. I’ve got over my deep hatred of VB.net, since working with it in SSIS so much, but Powershell is a welcome addition to my box of tricks! I’m looking forward to deploying this in future projects. It’s a shame the project seems inactive but this is really something I’d like to see Microsoft provide out-of-the-box in SSIS.

Post from: youdidwhatwithtsql.com

Powershell Script Task for SSIS

First lets create a simple SSIS package that the Powershell script will call. Open BIDS and create a new project. Add a string variable to the project called sql_date like below.

We’re going to use this variable to get the date as SQL Server recognises it so we know it’s the same as the system date. During my testing I discovered that SQL Server would take a few seconds pickup a system date change. Obviously this could distort your results so we need to ensure these are the same.

Next add an Execute SQL Task, from the toolbox, onto the designer. Call it "Get SQL Server Date". Add a connection to the SQL Server your package will be executing against and change the ResultSet property to "Single Row". Add the following TSQL to SQLStatement.

SELECT CONVERT(VARCHAR, GETDATE(), 103);

Next add a Script Task onto the designer and call it MsgBoxDate. Right click the task, click edit, add sql_date to ReadOnlyVariables on the script tab.

Click "Design Script" and add the below code into the ide.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime

Public Class ScriptMain

    Public Sub Main()
        Dim sql_date As String = Dts.Variables("sql_date").Value.ToString
        Dim msg As String = "SQL Server Date = " & sql_date & Environment.NewLine & "System Date = " & Format(Date.Now, "dd/MM/yyyy")
        MsgBox(msg)
        Dts.TaskResult = Dts.Results.Success
    End Sub

End Class

Connect the tasks together with a precedence constraint. The package should look something like below.

Return to the SSIS Designer, click File > Save Copy of Package.dtsx As. Save this package in the file system and remember the path as you’ll need it later. Below is the Powershell script that will change the current day, by one day, and then execute the package we just made. The while loop will verify that SQL Server has picked up the system date change before executing the package. Change the settings for $sqlserver and the package path for dtexec as appropriate. The $days variable should be changed to the number of days you want to run the package for. In this example it’s just ten. The script will set your systems date back once it has successfully completed..

# SQL server and database settings
$sqlserver = "localhost\sql2005";
$database = "master";
# Record the current date so we can set it back at the end of the script
# You may need to change this depending on your regional settings
$date = Get-Date -Format "dd/MM/yyyy";
 
# Connection string used for verifying sql server has registered the date change
$connection_string = ("Data Source=$sqlserver;Initial Catalog=$database;Integrated Security=SSPI")
$days = 10;
 
# Setup the database connection
$conn = New-Object System.Data.SqlClient.SqlConnection($connection_string);
$conn.Open();
$cmd = $conn.CreateCommand()
$cmd.CommandText = "SELECT DATEPART(dd, GETDATE())";
 
 
for($i = 0; $i -lt $days; $i++)
{
	# Change the date by one day
	Set-Date (Get-Date).AddDays(1);
	# SQL Server can take a few seconds to register the change
	# so we need to loop until the day matches before 
	# executing our ssis package
	$reader = $cmd.ExecuteReader();
	$reader.Read();
	$sql_day = $reader.GetValue(0);
	$reader.Close();
	while(($sql_day) -ne [DateTime]::Now.Day)
	{
		Start-Sleep -Seconds 2;
		# Check the sql day again
		$reader = $cmd.ExecuteReader();
		$reader.Read();
		$sql_day = $reader.GetValue(0);
		$reader.Close();
	}
	# Execute the package for this date
	dtexec /f "C:\Users\Rhys\Desktop\Package.dtsx"
}
 
# Clean up
$reader.Close();
$conn.Close();
 
# Put the date back with the current time
$time = [DateTime]::Now.TimeOfDay;
Set-Date "$date $time";

The script will work through each day displaying a message box for each iteration.

Obviously this particular package needs human intervention to get it to complete. Your real-life packages wouldn’t have this limitation so this script could be used to test over a large date range. Hopefully this script will allow me to get out of the office a bit earlier when I need to test my packages!

Post from: youdidwhatwithtsql.com

Automated Date Range Testing of SSIS Packages

The TSQL (SQL Server 2008) below will create a table called BankHolidays and populate the table with public holidays up to the end of 2010. The Bank Holiday dates have been taken from the DirectGov site and are for England & Wales only. Depending on your requirements you may want to consider dates, like Christmas, that fall on a weekend. These technically aren’t Bank Holidays so haven’t been included here, but you may wish to add them if you don’t want to run on these days.

-- Create a table to contain Bank Holidays
CREATE TABLE dbo.BankHolidays
(
	BankHoliday DATE NOT NULL PRIMARY KEY CLUSTERED,
	ActiVe BIT NOT NULL DEFAULT 1
);
GO
 
INSERT INTO dbo.BankHolidays
(
	BankHoliday
)
VALUES
(
	'2009-12-25'
),
(
	'2009-12-28'
),
(
	'2010-01-01'
),
(
	'2010-04-02'
),
(
	'2010-04-05'
),
(
	'2010-05-03'
),
(
	'2010-05-31'
),
(
	'2010-08-30'
),
(
	'2010-12-27'
),
(
	'2010-12-28'
);
GO

Note the Active flag in the table. This is just in case we need to deactivate a Bank Holiday for some some reason. Added flexibility is always useful.

Launch BIDS and create a new SSIS project. The first thing we need to do is add a variable called BankHoliday. We will use this in determining whether the day is a Bank Holiday or not. Add an Int32 variable in the SSIS variables window, as illustrated below.

Add an OLE DB connection that points at the database containing the BankHolidays table. Drop an Execute SQL task from the toolbox onto the design canvas. Right click the task and click edit to configure the task properties. Add the OLE DB connection manager to Connection and the below TSQL to SQLStatement.

SELECT COUNT(*)
FROM dbo.BankHolidays
WHERE BankHoliday = CONVERT(DATE, GETDATE())
AND Active = 1;

ResultSet should be changed to “Single row”.

UPDATE: (Thanks to Dr Drew in the comments for pointing this omission out.)

Click on the Result Set tab and map the variable BankHoliday as shown below. This variable mapping will be used to determine if the execution date is a bank holiday.

Now drop two Script Task components onto the design canvas and connect them from the Execute SQL Task. Call one “Display “Not Bank Holiday” message” and the other “Display “It’s a Bank Holiday” message”. This should look something like this.

Next we need to edit the constraints connecting our Script tasks to implement the logic to determine if it is a Bank Holiday or not. Right click the constraint connecting the script task called “Display “Not Bank Holiday” message” and choose Edit. Change “Evaluation Operation” to ‘Expression and Constraint’ and enter the following expression into the appropriate text box; @BankHoliday == 0. Finally click the radio button labelled “Logical OR”. This should look like this…

Click OK to save your changes. Now we need to edit the constraint connected to the Script Task called “Display “It’s a Bank Holiday” message”. The setup here is exactly the same except for the Expression which should be entered as @BankHoliday == 1.

By now your design canvas should look something like below.

Finally we’re just going to add a little code to each script task to display a message box. In the script task called “Display “Not Bank Holiday” message” click edit, go to the script tab, and click the “Design Script button”. Add the below code.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime

Public Class ScriptMain

    Public Sub Main()
        MsgBox("Hi, today is not a Bank Holiday, get back to work!")
        Dts.TaskResult = Dts.Results.Success
    End Sub

End Class

In the script task called “Display “It’s a Bank Holiday” message” add the below code.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime
 
Public Class ScriptMain
 
	Public Sub Main()
        MsgBox("Hi, today is a Bank Holiday, put your feet up!")
		Dts.TaskResult = Dts.Results.Success
	End Sub
 
End Class

Now we should be ready to execute the package. Click run and you should see the following message (assuming it’s not a bank holiday!).

Now, either wait until a bank holiday occurs, or change the date on your computer to one contained in the BankHolidays table. I changed mine to 2009-12-28 and here’s what I saw when I executed the package.

Post from: youdidwhatwithtsql.com

SSIS: Don’t run the process on a bank Holiday

' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime
Imports System.Diagnostics.EventLog
Imports System.Diagnostics.EventLogEntryType

Public Class ScriptMain

	' The execution engine calls this method when the task executes.
	' To access the object model, use the Dts object. Connections, variables, events,
	' and logging features are available as static members of the Dts class.
	' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
	'
	' To open Code and Text Editor Help, press F1.
	' To open Object Browser, press Ctrl+Alt+J.

    Public Sub Main()

        Dim eventLog As New Diagnostics.EventLog

        ' If the SSIS event log doesn't exist on the target machine then create it
        If Not eventLog.SourceExists("SSIS") Then
            eventLog.CreateEventSource("SSIS", "Application")
        End If

        ' Create an instance of the EventLog class
        eventLog = New Diagnostics.EventLog()
        ' Specify the source as SSIS
        eventLog.Source = "SSIS"
        ' Add an event log message
        eventLog.WriteEntry("This is a message from a Script Task!", Diagnostics.EventLogEntryType.Information)

        Dts.TaskResult = Dts.Results.Success
    End Sub

End Class

Execute the task and you should find the following message in the event viewer.

Post from: youdidwhatwithtsql.com

SSIS: Writing to the Windows Event Log

The script uses two variables, inFiles and archiveFiles. The variable inFiles contains the folder we want to move files from, and archiveFiles is the destination.

Configure your variables in BIDS like below.

Add a script task to the designer.

Right click the script task, choose ‘Edit’ then ‘Script’. Add the inFiles and archiveFiles variables to the ReadOnlyVariables textbox.

Click the “Design Script” button and paste the below code into the VSA editor.

' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime

Public Class ScriptMain

    ' The execution engine calls this method when the task executes.
    ' To access the object model, use the Dts object. Connections, variables, events,
    ' and logging features are available as static members of the Dts class.
    ' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
    '
    ' To open Code and Text Editor Help, press F1.
    ' To open Object Browser, press Ctrl+Alt+J.

    Public Sub Main()
        '
        ' Add your code here
        '
        Dim dir As System.IO.Directory
        ' fetch the folder locations from SSIS variables
        Dim inFiles As String = Dts.Variables("inFiles").Value.ToString
        Dim archiveFiles As String = Dts.Variables("archiveFiles").Value.ToString

        ' Get the system date and time separators for removing from the folder name we'll create later
        Dim dateSeparator As String = System.Globalization.DateTimeFormatInfo.CurrentInfo.DateSeparator
        Dim timeSeparator As String = System.Globalization.DateTimeFormatInfo.CurrentInfo.TimeSeparator

        ' Used to create a datetime stamped folder removing separators
        Dim dt As String = DateTime.Now.ToString.Replace(" ", "").Replace(dateSeparator, "").Replace(timeSeparator, "").Replace("/", "")

        ' Check folder paths end with a "\"
        If inFiles.EndsWith("\") = False Then
            inFiles &= "\"
        End If

        If archiveFiles.EndsWith("\") = False Then
            archiveFiles &= "\"
        End If

        archiveFiles &= dt

        ' Create datetime stamp archive folder if it doesn't already exist
        If dir.Exists(archiveFiles) = False Then
            dir.CreateDirectory(archiveFiles)
        End If

        ' Move all files to archive
        For Each archFile As String In dir.GetFiles(inFiles)
            dir.Move(archFile, archiveFiles & "\" & archFile.Substring(archFile.LastIndexOf("\") + 1, archFile.Length - archFile.LastIndexOf("\") - 1))
        Next

        Dts.TaskResult = Dts.Results.Success

    End Sub

End Class

Place some files into the location your inFiles variable points to. Then right click the script task and choose “Execute Task”.

Once the package has executed successfully the files will be moved to archiveFiles from inFiles.

Post from: youdidwhatwithtsql.com

SSIS: Archiving files with VB.Net