SELECT *
FROM OPENROWSET
('MSDASQL',
'Driver={Microsoft Text Driver (*.txt; *.csv)}; 
DefaultDir=C:\Users\Rhys\Desktop\csv;Extended properties=''ColNameHeader=True;Format=Delimited;''',
'SELECT * FROM file1.csv');

The DefaultDir property points to a directory containing standard comma separated files. Now the text files can be treated just like tables, note the SELECT query above in red text. There can be any number of files in the same directory and they can be queried simply by changing the appropriate filename, i.e. file1.csv to file2.txt.

If you experience the below error you need to enable the feature on your SQL Server instance.

Msg 15281, Level 16, State 1, Line 1
SQL Server blocked access to STATEMENT 'OpenRowset/OpenDatasource' of component 'Ad Hoc Distributed Queries' because this component is turned off as part of the security configuration for this server. A system administrator can enable the use of 'Ad Hoc Distributed Queries' by using sp_configure. For more information about enabling 'Ad Hoc Distributed Queries', see "Surface Area Configuration" in SQL Server Books Online. 

Run the below T-SQL to enable.

EXEC sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;

Now this is handy but some people, myself included, prefer to use Pipe-Delimited csv files. Microsoft seem insistent on calling pipes the vertical-bar. If you attempt the above, with pipe-delimited files, here’s what you get!

Not something we can work with very easily! By modifying a registry entry we can quickly resolve this. Usual warnings about editing your registry apply, i.e. backup and know what you’re doing.

Using regedit, navigate to the following key;

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Jet\4.0\Engines\Text

Edit the value for the Format key;

Change the value from CSVDelimited to Delimited(|). To reverse this change just set the value back to CSVDelimited when you need to.

Run the same query again and you’ll see our data is how we need it to be!

Read more about the Text Data Source Driver if you need to query files with other types of delimiters.

Post from: youdidwhatwithtsql.com

TSQL: Query Pipe-Delimited text files with OPENROWSET

# csv file to convert
$csv = "C:\Users\Rhys\Desktop\csv\file1.csv";
# xml file to create
$xml = "C:\Users\Rhys\Desktop\csv\file1.xml";
 
Import-Csv -Path $csv | Export-Clixml -Path $xml;

This will produce xml looking something like this.

<?XML:NAMESPACE PREFIX = [default] http://schemas.microsoft.com/powershell/2004/04 NS = "http://schemas.microsoft.com/powershell/2004/04" /><?XML:NAMESPACE PREFIX = [default] http://schemas.microsoft.com/powershell/2004/04 NS = "http://schemas.microsoft.com/powershell/2004/04" /><objs xmlns="http://schemas.microsoft.com/powershell/2004/04" version="1.1.0.1">
<obj refid="0">
<tn refid="0">
<t>System.Management.Automation.PSCustomObject</t>
<t>System.Object</t>
</tn>
<ms>
<s n="FirstName">Rhys</s>
<s n="LastName">Campbell</s>
<s n="Age">29</s>
</ms>
</obj>
<obj refid="1">
<tnref refid="0"></tnref>
<ms>
<s n="FirstName">Joe</s>
<s n="LastName">Bloggs</s>
<s n="Age">40</s>
</ms>
</obj>
<obj refid="2">
<tnref refid="0"></tnref>
<ms>
<s n="FirstName">Steve</s>
<s n="LastName">Smith</s>
<s n="Age">35</s>
</ms>
</obj>
</objs>

With the inclusion of the Get-ChildItem cmdlet we can merge multiple csv files into a single xml file with just a few lines of code.

# Folder of csv files
$csvFiles = Get-ChildItem -Path "C:\Users\Rhys\Desktop\csv\*" -Include *.csv;
 
# Process each csv file. Need to be the same structure
foreach($file in $csvFiles)
{
	$csvContent += Import-Csv -Path $file;
}
 
# Export the imported data as one xml file
$csvContent | Export-Clixml -Path C:\Users\Rhys\Desktop\csv\merged.xml;

Turning this back into a csv file is a simple Powershell one-liner!

# Turn this back into csv
Import-Clixml -Path C:\Users\Rhys\Desktop\csv\merged.xml | Export-Csv -Path C:\Users\Rhys\Desktop\csv\BackToCsv.csv -NoTypeInformation;

Check out my other data related Powershell posts; splitting csv files with Powershell, merging csv files with Powershell and trimming whitespace with Powershell.

Post from: youdidwhatwithtsql.com

Converting CSV FileS to XML with Powershell

While this is easy to do using SSIS and a bit of T-SQL it is a little tedious, and more time consuming, that it needs to be for adhoc jobs. Again Powershell came to the rescue! Here’s an example.

First I created a csv called bigCsvfile.csv, with 1, 000 records of data and placed it into my user profile directory; C:\Users\Rhys on my laptop.

This Powershell script does require a sequential integer id for each record to work correctly as it uses this to divide up the records. Now assuming you have a csv file, called bigCsvFile.csv, in the correct location then the only thing you should need to change is the $split variable value. This is now many files you would like to split the original file into.

# Author: Rhys Campbell
# 2009-09-22
# Splitting up csv files
 
# Path to csv file. Must contain a sequential unique integer id column 
$csvFile = "$Env:USERPROFILE\bigcsvFile.csv";
# Slit into how many files?
$split = 10;
 
# Get the csv file content
$content = Import-Csv $csvFile;
 
# So we start from Id = 1 in the csv file
$start = 1;
$end = 0;
 
# calc records per file
$records_per_file = [int][Math]::Ceiling($content.Count / $split);
 
for($i = 1; $i -le $split; $i++)
{
	# Set the end value for selecting records
	$end += $records_per_file;
	# Need to cast to int or we get an alphabetic comparison when we want a numeric one
	$content | Where-Object {[int]$_.Id -ge $start -and [int]$_.Id -le $end} | Export-Csv -Path "$Env:USERPROFILE\file$i.csv" -NoTypeInformation;
	# Update start value for selecting records
	$start = $end + 1;
}

Once the script has executed successfully it will create the split csv files in your user profile folder. The new files are named sequentially, i.e. file1.csv, file2.csv, file3.csv etc.

Depending on the number of records, and the number of files split to, the records may not be the same in each file. Provided the number of records versus file split is reasonable it will be pretty close. Now with a Powershell script it takes just a few moments to perform those previously tedious tasks!

Post from: youdidwhatwithtsql.com

Splitting csv files with Powershell