Re.Mark

My Life As A Blog

Archive for the ‘Python’ Category

Dynamic Football Stats

with one comment

A couple of days ago I noticed that The Guardian had made data about the England vs USA game that took place last week available.  I downloaded the data (which is in a Google Apps spreadsheet) and saved each sheet as a CSV file.

Originally, I intended to read the data with IronPython.  Reading CSV data with Python is very simple – there’s a built in CSV module.  However, this module is written in C, which means it’s not available in IronPython – see here for more info.  There is aproject called IronClad that allows Python modules written in C to be used from IronPython.  At the moment, it’s built against .NET 2, which means that I could get it to work in .NET 2, but I had plans to use .NET 4 and the dynamic support in C#.   Time for another approach.

Using the CsvReader class,it’s easy to access the data in a CSV file.  I started with the Player Summaries sheet.  To make this dynamic (and, therefore, useful for each of these sheets and, potentially, other as yet unknown sheets) I created a class to hold each row of data.  Here it is:


public class DynamicDataObject : DynamicObject
{
    private readonly Dictionary<string, dynamic> data;

    public DynamicDataObject(Dictionary<string, dynamic> data)
    {
        this.data = data;
    }

    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        result = data[binder.Name];
        return (result != null);
    }
}

By inheriting from DynamicObject, it will be possible to call this class dynamically – meaning that I can use the names of the data fields as defined properties on the class.  Next I created a DataReader class that reads the data from the CSV file and stores it as an IEnumerable<DynamicDataObject>.  Here’s that class:


public class DataReader : IEnumerable<DynamicDataObject>
{
    private readonly List<DynamicDataObject> dataList;

    public DataReader(string filename)
    {
        this.dataList = new List<DynamicDataObject>();
        using (StreamReader streamReader = new StreamReader(filename))
        {
            using (CsvReader reader = new CsvReader(streamReader, true))
            {
                string[] headers = reader.GetFieldHeaders();
                Dictionary<string, string> cleanHeaders = CleanHeaders(headers);
                while (reader.ReadNextRecord())
                {
                    Dictionary<string, dynamic> data = new Dictionary<string, dynamic>();
                    foreach (string header in headers)
                    {
                        int result;
                        dynamic value;
                        if (int.TryParse(reader[header],  out result))
                        {
                            value = result;
                        }
                        else
                        {
                            value = reader[header];
                        }
                        data.Add(cleanHeaders[header], value);
                    }
                    this.dataList.Add(new DynamicDataObject(data));
                }
            }
        }
    }

    private Dictionary<string, string> CleanHeaders(string[] headers)
    {
        Dictionary<string, string> result = new Dictionary<string, string>();
        foreach (string header in headers)
        {
            string cleanheader = header.Replace(' ', '_');
            cleanheader = cleanheader.Split('(')[0];
            result.Add(header, cleanheader);
        }
        return result;
    }

    #region IEnumerable<DynamicDataObject> Members

    public IEnumerator<DynamicDataObject> GetEnumerator()
    {
        return this.dataList.GetEnumerator();
    }

    #endregion

    #region IEnumerable Members

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return this.dataList.GetEnumerator();
    }

    #endregion
}

There’s a couple of things worth pointing out.  The first is that I’ve cleaned up the field names so that they can be used in code (by replacing spaces with underscores and removing anything in brackets).  The second is that if a value is an integer, I’m storing it as an integer.  I’m storing these values of type dynamic, which will come in handy when we want to query the data.

Speaking of querying that data, I wanted to use LINQ.  Here’s some simple code I wrote in a console application to try it out:


static void Main(string[] args)
{
    DataReader reader = new DataReader(@"C:\Users\Mark\Downloads\Eng-USA Data\Player Summaries.csv");

    var result = from dynamic player in reader
                 where player.Goals > 0
                 select player;

    foreach (dynamic player in result)
    {
        Console.WriteLine(player.Player_Name + " - " + player.Goals + " goals");
    }

    Console.WriteLine("Press any key to exit...");
    Console.ReadKey();
}

And here’s the output:

image

By using the dynamic support in C#, the LINQ query just works and I can reference properties dynamically without having to create a class specifically for each sheet of data.  It’s important in the LINQ query to declare the player of type dynamic – otherwise C# will revert to its statically typed ways and inform you that the Goals property doesn’t exist, which, given that it only exists at runtime, is correct.  Now I can analyse the data easily.  Doesn’t change the result though…

Written by remark

June 17, 2010 at 12:41 pm

Posted in .NET, c#, Python

Finding Lyrics

with 2 comments

I was looking some lyrics up online this week, so I wondered how hard to would be to write a simple application to find lyrics to your favourite song.  Or to your least favourite song.  Or, in fact, to any arbritrary song.  Via programmableweb, I found the API to lyricsfly, which looked easy to use.  Another IronPython console app beckoned.

Keeping it simple, I decided to use optparse to parse the command-line options and urllib to make the http calls.  This way the program can be called with the user_id that lyricsfly requires (head to their website and you can get a temporary weekly key to try this out) along with the artist name and song title.  What I decided not to do at this stage was to process the resulting XML.  Or handle any errors.  Or handle cases where the user_id, artist or title is not supplied.  But, although rudimentary, it works.  Here’s the code:


from System import Console
import urllib
from optparse import OptionParser

print "Starting"

parser = OptionParser()
parser.add_option("-i", "--user_id",
                  action="store", type="string", dest="user_id",
                  help="The user id for the Lyrics Fly service")
parser.add_option("-a", "--artist",
                  action="store", type="string", dest="artist",
                  help="Artist name")
parser.add_option("-t", "--title",
                  action="store", type="string", dest="title",
                  help="Song title")                  

(options, args) = parser.parse_args()

print "Parsed options"

if (options.user_id):
    user_id = options.user_id
 
if (options.artist):
    artist = options.artist
    
if (options.title):
    title = options.title  
    
print "Getting Lyrics for " + artist + " - " + title 

query = urllib.urlencode([("i", user_id), ("a", artist), ("t", title)])   

url = "http://api.lyricsfly.com/api/api.php?" + query
print url

data = urllib.urlopen(url)
print data.read()

print
print "Press any key to exit.."
Console.ReadKey()

Written by remark

June 4, 2010 at 4:51 pm

Posted in .NET, Python

IronPython Console App

leave a comment »

Having installed the CTP of IronPython tooling for Visual Studio, I thought I’d better try it out and write a simple app.  A Console Application seemed like a good choice – and I haven’t done much with getting data over the internet in Python, so I combined the two.  Here’s the resulting simple (and, in its current form, not especially useful) application:


import urllib2
from System import Console

print "Starting..."
data = urllib2.urlopen("http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/front_page/rss.xml").read()
print data

Console.WriteLine()
Console.ReadKey()

Written by remark

May 28, 2010 at 4:33 pm

Posted in .NET, Python, Tools

IronPython meet Visual Studio

leave a comment »

On IronPython URLs, I just spotted that a CTP of IronPython tooling for Visual Studio 2010 is available.  Now you can create IronPython projects in Visual Studio like this:

image

It’s an early preview, but you still get intellisense, a REPL, debugging and a lot more besides – there’s more detail here.

Written by remark

May 1, 2010 at 6:20 pm

Posted in .NET, Python

Dynamic Configuration with IronPython and .NET 4

leave a comment »

A while back I posted a simple way of using IronPython to configure a .NET application.  The I spotted Herman’s question about whether the application would react to a change in the Configuration.py file.  The way I wrote the original sample the configuration is read in once.  But it’s fairly simple to modify the code to react to changes in Configuration.py.  Here’s the modified Program class:


class Program
{
	static void Main(string[] args)
	{
		FileSystemWatcher watcher = new FileSystemWatcher(AppDomain.CurrentDomain.BaseDirectory, "*.py");
		watcher.EnableRaisingEvents = true;
		watcher.NotifyFilter = NotifyFilters.LastWrite;
		DateTime modified = DateTime.Now;

		Console.TreatControlCAsInput = true;
		
		dynamic configuration = ConfigureFromIronPython();
		DisplayConfiguration(configuration);

		watcher.Changed += (sender, eventArgs) =>
		{
			if (DateTime.Now - modified > TimeSpan.FromMilliseconds(100))
			{
				modified = DateTime.Now;
				configuration = ConfigureFromIronPython();
				DisplayConfiguration(configuration);
			}
		};


		Console.WriteLine("Press any key to exit...");
		Console.ReadKey();

		watcher.EnableRaisingEvents = false;
		watcher.Dispose();

		Console.WriteLine("Exiting...");
	}

	private static dynamic ConfigureFromIronPython()
	{
		dynamic configuration = new ExpandoObject();
		ScriptEngine engine = Python.CreateEngine();
		ScriptScope scope = engine.CreateScope();
		scope.SetVariable("configuration", configuration);
		engine.ExecuteFile("Configuration.py", scope);
		return configuration;
	}

	private static void DisplayConfiguration(dynamic configuration)
	{
		for (int i = 0; i < configuration.Count; i++)
		{
			Console.WriteLine(configuration.Text);
		}
	}
}

By adding a FileSystemWatcher, we can react to changes in Configuration.py and update the application configuration.  (If you’re wondering the check of when the last modification occurred is to prevent handling the same change twice which can occur when using the FileSystemWatcher.)  The only other change is a minor refactoring to extract a method that displays the configuration.

Written by remark

April 30, 2010 at 8:44 pm

Posted in .NET, c#, Python

Batteries Included

leave a comment »

Sometimes you’ll hear the Python standard library referred to as “batteries included” – a little more info hereIronPython can also use these included batteries.  As an example, today I needed to list the files in a folder.  So, a simple Python script seemed like a good way of doing that (I’m sure there are better and more ingenious ways.)  Here it is:


import os
import os.path
import sys
from optparse import OptionParser

def list_files(path, indent=0):
    for filename in sorted(os.listdir(path)):
        print " " * indent + filename
        full_path = os.path.join(path, filename)
        if (options.recursive) and (os.path.isdir(full_path)):
            list_files(full_path, indent + 2)
 
parser = OptionParser()
parser.add_option("-d", "--directory",
                  action="store", type="string", dest="path",
                  help="The directory to list.")
parser.add_option("-r", "--recursive",
                  action="store_true", default=False,
                  help="Whether to list subdirectories.")
parser.add_option("-f", "--output_file",
                  action="store", type="string", dest="output_file",
                  help="Directory contents will be listed to this file if specified.")

(options, args) = parser.parse_args()

if (options.path):
    path = options.path
else:
    path = sys.path[0]

out = sys.stdout
if (options.output_file):
    output_file = open(options.output_file, 'w')
    sys.stdout = output_file

list_files(path)

if (options.output_file):
    output_file.flush()
    output_file.close()
    sys.stdout = out

As you can see, the script takes advantage of optparse to process the command line arguments, sys (to get the current folder and to get access to and redirect the output of the script), os (to list the contents of a folder) and os.path (to test if a given path is a folder.)  And, being IronPython, you also get a second set of batteries in the form of the .NET framework.

Written by remark

March 23, 2010 at 3:00 pm

Posted in .NET, Python

Iron Python at QCon

leave a comment »

I spent yesterday on the Microsoft stand at QCon 2010.  I took a few Iron Python samples with me to show to those who are interested.  I wanted to be able to show three things:  .NET runs Python, Python extends .NET and Python runs .NET.

.NET runs Python

To show that .NET can run Python I used the Text Processing sample I’ve blogged about before.  I’ve subsequently added optparse to it so that it can be driven from the command line.  The point of this sample is that it uses standard Python libraries, the whole application is written in Python (there’s a little XAML to describe the UI) and runs on the DLR courtesy of IronPython.

Python extends .NET

For a simple demonstration of extending a .NET application with Python, I took the sample application described here.  This application allows the user to write Python (at runtime) that interacts with the application.

Python runs .NET

The last sample was an adaptation of the code here that reads a Twitter feed.  Rather than use Twitter (with all the shortened urls and abbreviations) I decided to use an RSS feed from the BBC to create an Iron Python newsreader.  The code is remarkably simple:

import clr
clr.AddReference('System.Speech')
clr.AddReference('System.Xml')

from System.Speech.Synthesis import SpeechSynthesizer
from System.Xml import XmlDocument, XmlTextReader

xmlDoc = XmlDocument()
xmlDoc.Load("http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/front_page/rss.xml")
spk = SpeechSynthesizer()

itemsNode = xmlDoc.DocumentElement.SelectNodes("channel/item")
for item in itemsNode:
  print item.SelectSingleNode("title").InnerText
  news = "<?xml version='1.0'?><speak version='1.0' xml:lang='en-GB'><break />" + item.SelectSingleNode("description").InnerText + "</speak>"
  spk.SpeakSsml(news)
 

This is Python using standard .NET libraries to show how a Python programmer has the .NET framework available to them through Iron Python.

Gestalt

The final thing I talked about is Gestalt, which allows you to run Python (and Ruby) in the browser.  It does this by using the DLR, which is part of Silverlight – this is all encapsulated in javascript.

Written by remark

March 12, 2010 at 7:21 pm

Posted in .NET, Events, Microsoft, Python