$xml = New-Object XML
$url = "http://www.tfl.gov.uk/tfl/syndication/feeds/stations.kml"
$csvFile = "$env:UserProfile\Desktop\tfl_tubes.csv";
 
# Empty file if it already exists
Set-Content -Path $csvFile $null;
# Add headers to file
Add-Content -Path $csvFile "Station|Address|Coordinates";
 
# Load the xml
$xml.Load($url);
$stations = $xml.kml.Document.Placemark;
 
foreach($station in $stations)
{
	$name = $station.name;
	$description = $station.description;
	$coordinates = $station.Point.coordinates
	# This data needs cleaning a bit
	$name = $name.Trim();
	$description = $description.Trim();
	$coordinates = $coordinates.Trim();
	# Ad line to the csv file
	Add-Content -Path $csvFile "$name|$description|$coordinates";
}

After running the script check your desktop for a file called tfl_tubes.csv which should look something like below.

Post from: youdidwhatwithtsql.com

Get TFL Tube data with Powershell

The site is being officially launched by Sir Tim Berners Lee tomorrow, but you can access the site now. The US has their own version data.gov but the British site boasts nearly three times as many datasets. Take a peek at the available datasets. It always fantastic looking through the great number of apps that people create around freely available data. Why not take a look and start your own project?

Those interested in London specific data can check out the London DataStore. People in sunny San Francisco have datasf and Chicago also have a dataset section on their website. Read more about this.

Post from: youdidwhatwithtsql.com

British Government launches free data site

# csv file to convert
$csv = "C:\Users\Rhys\Desktop\csv\file1.csv";
# xml file to create
$xml = "C:\Users\Rhys\Desktop\csv\file1.xml";
 
Import-Csv -Path $csv | Export-Clixml -Path $xml;

This will produce xml looking something like this.

<?XML:NAMESPACE PREFIX = [default] http://schemas.microsoft.com/powershell/2004/04 NS = "http://schemas.microsoft.com/powershell/2004/04" /><?XML:NAMESPACE PREFIX = [default] http://schemas.microsoft.com/powershell/2004/04 NS = "http://schemas.microsoft.com/powershell/2004/04" /><objs xmlns="http://schemas.microsoft.com/powershell/2004/04" version="1.1.0.1">
<obj refid="0">
<tn refid="0">
<t>System.Management.Automation.PSCustomObject</t>
<t>System.Object</t>
</tn>
<ms>
<s n="FirstName">Rhys</s>
<s n="LastName">Campbell</s>
<s n="Age">29</s>
</ms>
</obj>
<obj refid="1">
<tnref refid="0"></tnref>
<ms>
<s n="FirstName">Joe</s>
<s n="LastName">Bloggs</s>
<s n="Age">40</s>
</ms>
</obj>
<obj refid="2">
<tnref refid="0"></tnref>
<ms>
<s n="FirstName">Steve</s>
<s n="LastName">Smith</s>
<s n="Age">35</s>
</ms>
</obj>
</objs>

With the inclusion of the Get-ChildItem cmdlet we can merge multiple csv files into a single xml file with just a few lines of code.

# Folder of csv files
$csvFiles = Get-ChildItem -Path "C:\Users\Rhys\Desktop\csv\*" -Include *.csv;
 
# Process each csv file. Need to be the same structure
foreach($file in $csvFiles)
{
	$csvContent += Import-Csv -Path $file;
}
 
# Export the imported data as one xml file
$csvContent | Export-Clixml -Path C:\Users\Rhys\Desktop\csv\merged.xml;

Turning this back into a csv file is a simple Powershell one-liner!

# Turn this back into csv
Import-Clixml -Path C:\Users\Rhys\Desktop\csv\merged.xml | Export-Csv -Path C:\Users\Rhys\Desktop\csv\BackToCsv.csv -NoTypeInformation;

Check out my other data related Powershell posts; splitting csv files with Powershell, merging csv files with Powershell and trimming whitespace with Powershell.

Post from: youdidwhatwithtsql.com

Converting CSV FileS to XML with Powershell

While this is easy to do using SSIS and a bit of T-SQL it is a little tedious, and more time consuming, that it needs to be for adhoc jobs. Again Powershell came to the rescue! Here’s an example.

First I created a csv called bigCsvfile.csv, with 1, 000 records of data and placed it into my user profile directory; C:\Users\Rhys on my laptop.

This Powershell script does require a sequential integer id for each record to work correctly as it uses this to divide up the records. Now assuming you have a csv file, called bigCsvFile.csv, in the correct location then the only thing you should need to change is the $split variable value. This is now many files you would like to split the original file into.

# Author: Rhys Campbell
# 2009-09-22
# Splitting up csv files
 
# Path to csv file. Must contain a sequential unique integer id column 
$csvFile = "$Env:USERPROFILE\bigcsvFile.csv";
# Slit into how many files?
$split = 10;
 
# Get the csv file content
$content = Import-Csv $csvFile;
 
# So we start from Id = 1 in the csv file
$start = 1;
$end = 0;
 
# calc records per file
$records_per_file = [int][Math]::Ceiling($content.Count / $split);
 
for($i = 1; $i -le $split; $i++)
{
	# Set the end value for selecting records
	$end += $records_per_file;
	# Need to cast to int or we get an alphabetic comparison when we want a numeric one
	$content | Where-Object {[int]$_.Id -ge $start -and [int]$_.Id -le $end} | Export-Csv -Path "$Env:USERPROFILE\file$i.csv" -NoTypeInformation;
	# Update start value for selecting records
	$start = $end + 1;
}

Once the script has executed successfully it will create the split csv files in your user profile folder. The new files are named sequentially, i.e. file1.csv, file2.csv, file3.csv etc.

Depending on the number of records, and the number of files split to, the records may not be the same in each file. Provided the number of records versus file split is reasonable it will be pretty close. Now with a Powershell script it takes just a few moments to perform those previously tedious tasks!

Post from: youdidwhatwithtsql.com

Splitting csv files with Powershell

In this example I have two csv files in a directory

These just contain some simple name and age data.

This simple script will produce a third file, merging the contents of file1.csv and file2.csv.

# Author: Rhys Campbell
# 2009-08-22
# Merging identical csv files
 
# Directory containing csv files, include *.*
$directory = "C:\Users\Rhys\Desktop\csv\*.*";
# Get the csv files
$csvFiles = Get-ChildItem -Path $directory -Filter *.csv;
 
# Updated 01/03/2010. Thanks to comment from Chris.
# Resolves error Method invocation failed because [System.Management.Automation.PSObject] doesn't contain a method named 'op_Addition'.
#$content = $null;
$content = @();
 
# Process each file
foreach($csv in $csvFiles)
{
	$content += Import-Csv $csv;
}
 
# Write a datetime stamped csv file
$datetime = Get-Date -Format "yyyyMMddhhmmss";
$content | Export-Csv -Path "C:\Users\Rhys\Desktop\csv\merged_$datetime.csv" -NoTypeInformation;

If all goes as planned a new file will be created.

This file will contain data from both csv files.

This is obviously a fairly trivial example but it’s a massive timesaver when you have many such files to merge. Word of warning, if you’ve got very big files, you may want to change the script to use Add-Content, to flush each csv file to disk in the foreach loop, to avoid munching up all your RAM.

Post from: youdidwhatwithtsql.com

Merging CSV Files with Powershell