youdidwhatwithtsql.com » Data

Get TFL Tube data with Powershell

Rhys — Sat, 27 Feb 2010 19:18:24 +0000

The London Datastore has loads of datasets available that we can use for free. One of the datasets available is a list of TFL Station Locations. The station location feed is a geo-coded KML feed of most of London Underground, DLR and London Overground stations. Here’s Powershell script that will extract this data from a url and write it to a pipe-delimited file ready for import into the database of your choice.

^?View Code POWERSHELL

$xml = New-Object XML
$url = "http://www.tfl.gov.uk/tfl/syndication/feeds/stations.kml"
$csvFile = "$env:UserProfile\Desktop\tfl_tubes.csv";
 
# Empty file if it already exists
Set-Content -Path $csvFile $null;
# Add headers to file
Add-Content -Path $csvFile "Station|Address|Coordinates";
 
# Load the xml
$xml.Load($url);
$stations = $xml.kml.Document.Placemark;
 
foreach($station in $stations)
{
	$name = $station.name;
	$description = $station.description;
	$coordinates = $station.Point.coordinates
	# This data needs cleaning a bit
	$name = $name.Trim();
	$description = $description.Trim();
	$coordinates = $coordinates.Trim();
	# Ad line to the csv file
	Add-Content -Path $csvFile "$name|$description|$coordinates";
}

After running the script check your desktop for a file called tfl_tubes.csv which should look something like below.

Post from: youdidwhatwithtsql.com

Get TFL Tube data with Powershell

British Government launches free data site

Rhys — Wed, 20 Jan 2010 21:46:14 +0000

Just a quick post about the British Governments launch of data.gov.uk offering free access to a large number of datasets. It’s good to see this data, that we as taxpayers have funded, available for free use. Hats off to The Guardian’s long running free our data campaign.

The site is being officially launched by Sir Tim Berners Lee tomorrow, but you can access the site now. The US has their own version data.gov but the British site boasts nearly three times as many datasets. Take a peek at the available datasets. It always fantastic looking through the great number of apps that people create around freely available data. Why not take a look and start your own project?

Those interested in London specific data can check out the London DataStore. People in sunny San Francisco have datasf and Chicago also have a dataset section on their website. Read more about this.

Post from: youdidwhatwithtsql.com

British Government launches free data site

Primary Keys do not always have to be ‘Id’!

Rhys — Sun, 03 Jan 2010 15:41:50 +0000

We’ve all more than likely spotted tables in databases with no primary keys. But does a primary key always have to be defined something like…

^?View Code TSQL

ALTER TABLE MyTable ADD Id INTEGER NOT NULL IDENTITY(1,1) PRIMARY KEY CLUSTERED;

I think we have ORM to blame for this widespread practice of using auto-incrementing integers as PK’s.

In the ORM, these additional restrictions are placed on primary keys:

Primary keys should be anonymous integer or numeric identifiers.

source http://en.wikipedia.org/wiki/Unique_key

I’m not saying this isn’t generally sensible. Probably most of the primary keys I have ever defined have been of this type. This is the best choice if the key will be used regularly in table joins. But this need not always be the case. Lets take an example. The table below contains UK Postcodes and location information.

What’s my beef with this? Well, lets assume that this table is solely used for for location lookups by postcode. What is the purpose of Id? Will it ever be used in any joins to other tables? Very unlikely in my opinion. Why not change the Postcode column to be the primary key?

I saw this exact situation in a live system a few years ago. The Id column was redundant but the Postcode column wasn’t even indexed. Not good in a table with 2.4+ Million records. Careful selection of your Primary Keys not only result in fewer redundant columns but will also help with your indexing strategy.

Primary Keys, when selected like this, also protect us from screwing up our data. The Postcode table above contained over 300K duplicated postcodes. I’ve seen this cause developers to eliminate duplicates with SQL SELECT DISTINCT. Lets fix our data, not our compose our queries around it causing performance hits!

For further reading see this article about the proper selection of a Primary Key.

Post from: youdidwhatwithtsql.com

Primary Keys do not always have to be ‘Id’!

London Postcode District Data

Rhys — Fri, 03 Apr 2009 12:11:48 +0000

The requirement to work with postcodes, and specifically London postcode districts, has followed me around during my professional life. Below is a TSQL script to create a table for the postal districts in London. Enjoy!

^?View Code TSQL

CREATE TABLE london_postcodes (postcode VARCHAR(4) NOT NULL PRIMARY KEY CLUSTERED)
GO
 
SET NOCOUNT ON;
SET XACT_ABORT ON;
GO
BEGIN TRANSACTION;
 
INSERT INTO [dbo].[london_postcodes]([postcode])
SELECT N'E1' UNION ALL
SELECT N'E10' UNION ALL
SELECT N'E11' UNION ALL
SELECT N'E12' UNION ALL
SELECT N'E13' UNION ALL
SELECT N'E14' UNION ALL
SELECT N'E15' UNION ALL
SELECT N'E16' UNION ALL
SELECT N'E17' UNION ALL
SELECT N'E18' UNION ALL
SELECT N'E2' UNION ALL
SELECT N'E3' UNION ALL
SELECT N'E4' UNION ALL
SELECT N'E5' UNION ALL
SELECT N'E6' UNION ALL
SELECT N'E7' UNION ALL
SELECT N'E8' UNION ALL
SELECT N'E9' UNION ALL
SELECT N'EC1' UNION ALL
SELECT N'EC2' UNION ALL
SELECT N'EC3' UNION ALL
SELECT N'EC4' UNION ALL
SELECT N'N1' UNION ALL
SELECT N'N10' UNION ALL
SELECT N'N11' UNION ALL
SELECT N'N12' UNION ALL
SELECT N'N13' UNION ALL
SELECT N'N14' UNION ALL
SELECT N'N15' UNION ALL
SELECT N'N16' UNION ALL
SELECT N'N17' UNION ALL
SELECT N'N18' UNION ALL
SELECT N'N19' UNION ALL
SELECT N'N2' UNION ALL
SELECT N'N20' UNION ALL
SELECT N'N21' UNION ALL
SELECT N'N22' UNION ALL
SELECT N'N3' UNION ALL
SELECT N'N4' UNION ALL
SELECT N'N5' UNION ALL
SELECT N'N6' UNION ALL
SELECT N'N7' UNION ALL
SELECT N'N8' UNION ALL
SELECT N'N9' UNION ALL
SELECT N'NW1' UNION ALL
SELECT N'NW10' UNION ALL
SELECT N'NW11' UNION ALL
SELECT N'NW2' UNION ALL
SELECT N'NW3' UNION ALL
SELECT N'NW4'
 
COMMIT;
RAISERROR (N'[dbo].[london_postcodes]: Insert Batch: 1.....Done!', 10, 1) WITH NOWAIT;
GO
 
BEGIN TRANSACTION;
 
INSERT INTO [dbo].[london_postcodes]([postcode])
SELECT N'NW5' UNION ALL
SELECT N'NW6' UNION ALL
SELECT N'NW7' UNION ALL
SELECT N'NW8' UNION ALL
SELECT N'NW9' UNION ALL
SELECT N'SE1' UNION ALL
SELECT N'SE10' UNION ALL
SELECT N'SE11' UNION ALL
SELECT N'SE12' UNION ALL
SELECT N'SE13' UNION ALL
SELECT N'SE14' UNION ALL
SELECT N'SE15' UNION ALL
SELECT N'SE16' UNION ALL
SELECT N'SE17' UNION ALL
SELECT N'SE18' UNION ALL
SELECT N'SE19' UNION ALL
SELECT N'SE2' UNION ALL
SELECT N'SE20' UNION ALL
SELECT N'SE21' UNION ALL
SELECT N'SE22' UNION ALL
SELECT N'SE23' UNION ALL
SELECT N'SE24' UNION ALL
SELECT N'SE25' UNION ALL
SELECT N'SE26' UNION ALL
SELECT N'SE27' UNION ALL
SELECT N'SE28' UNION ALL
SELECT N'SE3' UNION ALL
SELECT N'SE4' UNION ALL
SELECT N'SE5' UNION ALL
SELECT N'SE6' UNION ALL
SELECT N'SE7' UNION ALL
SELECT N'SE8' UNION ALL
SELECT N'SE9' UNION ALL
SELECT N'SW1' UNION ALL
SELECT N'SW10' UNION ALL
SELECT N'SW11' UNION ALL
SELECT N'SW12' UNION ALL
SELECT N'SW13' UNION ALL
SELECT N'SW14' UNION ALL
SELECT N'SW15' UNION ALL
SELECT N'SW16' UNION ALL
SELECT N'SW17' UNION ALL
SELECT N'SW18' UNION ALL
SELECT N'SW19' UNION ALL
SELECT N'SW2' UNION ALL
SELECT N'SW20' UNION ALL
SELECT N'SW3' UNION ALL
SELECT N'SW4' UNION ALL
SELECT N'SW5' UNION ALL
SELECT N'SW6'
 
COMMIT;
RAISERROR (N'[dbo].[london_postcodes]: Insert Batch: 2.....Done!', 10, 1) WITH NOWAIT;
GO
BEGIN TRANSACTION;
 
INSERT INTO [dbo].[london_postcodes]([postcode])
SELECT N'SW7' UNION ALL
SELECT N'SW8' UNION ALL
SELECT N'SW9' UNION ALL
SELECT N'W1' UNION ALL
SELECT N'W10' UNION ALL
SELECT N'W11' UNION ALL
SELECT N'W12' UNION ALL
SELECT N'W13' UNION ALL
SELECT N'W14' UNION ALL
SELECT N'W2' UNION ALL
SELECT N'W3' UNION ALL
SELECT N'W4' UNION ALL
SELECT N'W5' UNION ALL
SELECT N'W6' UNION ALL
SELECT N'W7' UNION ALL
SELECT N'W8' UNION ALL
SELECT N'W9' UNION ALL
SELECT N'WC1' UNION ALL
SELECT N'WC2'
 
COMMIT;
RAISERROR (N'[dbo].[london_postcodes]: Insert Batch: 3.....Done!', 10, 1) WITH NOWAIT;
GO

Post from: youdidwhatwithtsql.com

London Postcode District Data