Politics
Ireland | Politics
Submitted by johnh on May 22, 2009 - 23:43.
...elapsed since we told people about KildareStreet, and so I'm going to tell you some stuff about how it's been going. Partly because a few people have asked and partly because we're quite keen to get some of the smug out of our systems :)
Traffic
9,000 people have pitched up and looked through the first genuinely-searchable version of the Oireachtas debates. They arrived in slightly more than 12,000 visits and have generated 55,000 page views.
There was an initial peak as a large number (too large to mention everybody here, but you know who you are) of Irish bloggers and Twitter users passed our new site around in the first 48 hours. Then it tailed off quite a lot. Then a fortnight ago Google results started turning up in large numbers for lots of relevant topics and Googlebot is currently retrieving about 3,700 KildareStreet pages every day and taking an average of 0.2 seconds each to do so. Those pages are not counted in the total I just mentioned. Since Google showed up it has added an extra 400% to the Google-less total of the week before.
We currently have a Pagerank of zero. I'm expecting that when it climbs off the floor next month we'll get still more traffic -- though we're already in the top ten for most queries I idly try at the moment.
Oh, and of the top ten search queries, eight are variations on 'KildareStreet.com'. The other two relate to Garda recruitment.
So in the past ten days traffic is now averaging higher than during our initial three-day boost and this week we've been doing an average of 3,000 pages every 24 hours.
And yes, I know it's a truly dismal metric in terms of objectivity, but about a week ago we overtook oireachtas.ie in Alexa's site rankings for Ireland.
Who's looking?
94% of the site's traffic originates in (the Republic of) Ireland.
Eircom's the largest ISP source of traffic, which doesn't surprise us since it's the largest ISP in the country. I'm a little surprised, however, that the number 2 spot was taken on day two by the Houses of the Oireachtas itself, and it's remained there ever since.
Joiners
397 people have signed up for email alerts for either search keywords or named individuals. We're currently sending about 130 email alerts every day. By way of comparison the numbers for TheyWorkForYou (which, to be fair, covers a country with fifteen times more people in it and has been running for about seven years now) are about 70,000 and about 20,000 respectively. The ratio is similar, though...
I'll tell you that several identifiable members of the Dáil and Seanad are among the signers-up. I won't tell you who they are or what they're tracking.
Effects so far
The piece which appeared on May 10th in the Sunday Business Post triggered a flurry of additional press items and got me talking about it on several radio stations. For future web site launchers, I cannot begin to tell you how much more significant the result was when we got two column inches in Metro than it was for anything else. You can see some clippings over here.
In our first week there was no new Oireachtas business.
In the second week they came back to work and the official site was generally being (fully) updated by about 2300 the day after each sitting day, but often containing major errors.
In the third week I got annoyed about this and called them out about the errors on KildareStreet's sideblog.
Since the day after that rant the official site has been finished for the day, on average, without errors and six hours more quickly. If we'd had no effect but that, I'd consider this worthwhile.
Finally
I did not know what would happen after we pushed this thing out of the door. We just knew it needed doing, and that if anybody else was ever going to do it, they would have by now.
In fact KildareStreet has become, in this small pond, a much bigger fish than I ever expected and very much more quickly than anybody anticipated.
I'm out of interesting info for now, and I've got five years of deliberately-ill-formatted TD and Senator expenses to finish parsing, so I'll end here.
Meanwhile: cheers, everybody.
read more | 2 comments | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on April 28, 2009 - 21:31.
A substantial part of the parser was reusable enough to make short work of the next major milestone for KildareStreet.com, although the standard of source material published by the Oireachtas was an order of magnitude worse than the Dáil's.
Je vous présente: Seanad Éireann -- every word uttered since January 20th, 2004.
That's another 1.5 million words or so.
And yes, this means you can sign up for email alerts and whatnot for Senators too, and search through the record for things uttered in the Seanad. All the same good stuff, but twice as much. Well, one and a half times.
read more | 3 comments | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on April 22, 2009 - 16:54.
Since 20th January 2004, the Dáil has been in session for 493 days (including today).
Up until this morning TDs made 249,489 speeches, and asked 160,503 questions which received written responses.
On average, TDs are producing 507 speeches and 326 written answers per sitting day.
Sadly you won't have had access to information like this before, since the Houses of the Oireachtas publishes the Dáil record primarily on an illegible website ridden with elementary display errors.
What we needed, it was decided (by myself and independently also by some other people, though I didn't discover their existence until last month), was a port to Ireland of the spiffing UK site TheyWorkForYou.com.
Then we could (gasp!) search the parliamentary record. Get RSS feeds for TDs. Set up email or RSS alerts for people or search terms. Maybe even be able to read the bloody thing without wanting to kill oneself after about twenty seconds of stupidly-and-permanently-underlined horror.
MySociety helpfully publishes the source code which powers TheyWorkForYou, and I installed it back in October. Some readers may recall me showing that site, containing nothing but a list of TDs in the right place, back at Barcamp Cork 2.
The trick, however, is to get all the actual debates and questions into the site's database. You need to build a parser program to convert all that data from one format into another -- in Ireland's case, from the raw XML published by the Houses of the Oireachtas which is primarily intended as a print publishing format for the official record, but which also powers debates.oireachtas.ie.
The parser project languished for a good while until MySociety got another contact (this time from Gavin Sheridan) and forwarded him to me. Gavin, basically, annoyed me into making more progress.
I started in earnest again about six weeks ago and have worked on almost nothing since then.
The end result
As of this afternoon, then, we're pleased to announce that Ireland's local version of TheyWorkForYou is called KildareStreet.com, and that it's now available in public beta.
It still contains bugs, so be careful about jumping to early conclusions :)
At KildareStreet.com you can:
- Read a dramatically-more-legible version of the Dáil Record going back to January 2004,
- Search that record using a fabulous search engine which I didn't write - you can restrict searches to speeches or written questions, or by speaker, or by date or date range,
- Sign up for email alerts for when a search query you're interested changes, or whenever a TD of your choosing says something or asks a question which generated a written reply, and
- Subscribe to RSS feeds for individual TDs or for search queries.
The site will be updated the day after each sitting day shortly after the Oireachtas publishes that day's first report version in XML. RSS feeds and emails are generated shortly after that.
Coming soon
In the next few weeks you will also be able to inspect the past five and a half years of TD expenses, and the register of members' interests.
There's a programming API in the code base which we have not yet properly localised, and is therefore not yet available. I shall sort that out as quickly as possible.
Once we've done all of that, I'll get started on the Seanad.
Thanks due
None of this would be possible without the code for TheyWorkForYou having been released under a permissive license. This is the second non-UK installation of this code base (after Australia) and there's also a similar project in New Zealand. So thanks to MySociety, and to Matthew Somerville in particular for answering a large number of stupid questions while I was putting this together.
The Houses Commission employs people who could, when they got wind of it, have made this project more difficult or even impossible. They did not do so, and I and you owe them our thanks for that. Leo Bollins and Tom Malone have helpfully answered questions about their publishing formats and not been freaked out when I sent them bug reports about their source material.
Gavin Sheridan poked and chivvied this project into existing now instead of later, is sourcing ancillary data like member interests and expenses, and will be helping to keep the site ticking over as a moderator. For the first of these above all, this site etc without whom blah blah blah. Cheers, Gav :)
Justin Mason contributed actual code to start parsing written answers and has offered to assist in making the data import process follow at least some sensible test-driven methods which my shonky parser does not even begin to follow yet. He is the only actual code contributor to KildareStreet other than myself and code is, frankly, everything. The parser as it now is isn't using his stuff (yet) but we owe him a debt of gratitude also.
Simon McGarr is quietly providing assistance in the background, the fruits of which will become apparent in due course. Thanks to you.
Finally, and more than all the above, I live with a very tolerant and patient wife. Sabrina Dent, not entirely unknown in these parts, has been working for money and covering everything in our house in that manner while I've been doing all this for no cash. She thinks it's a worthwhile project and if she didn't, KildareStreet absolutely would not exist. Hire her!
Anyway. KildareStreet. It's now in testing. Have at it. Send me bug reports. Enjoy.
And above all? I think it's time we raised our game generally. Blogs are fine, sure, but actual apps that help people are required. This is my first contribution. How about you?
read more | 17 comments | delicious | digg | reddit
Ireland | Politics
Submitted by johnh on May 25, 2007 - 12:17.
Because you don't need to travel to Dublin to sit in a room and watch television </only joking, guys>.
Chat with others as the results start coming in, and/or post/watch the progress at Kildarestreet.com.
See you in there. Some well-configured users can click this link and join in right now. Everyone else - server is irc.freenode.net, channel #kildarestreet.
read more | delicious | digg | reddit
Ireland | Politics
Submitted by johnh on April 30, 2007 - 14:00.
I'm only bringing this up because it's relevant to what follows, but I maintain that Twitter still sucks (it was down all night again last night, by the way and its user-hostile and flaky nature just bit another respected ex-colleague).
That said, and moving right along to the real point of this post, there's a lot to be said for the idea that maintaining a blog could be simpler. Today I'm going to share with you the real reason I don't post here as often as I should.
It's because I post elsewhere. Some years ago I unilaterally that evolt.org's predominantly-social IRC channel could do with its own blog. That decision was based predominantly on having tech-envy of two illustrious predecessors: The Daily Chump, powered by a Python bot penned by its members; but above all 2lmc's spool. 2lmc was once a house in Islington containing several twentysomething perl programmers (including my former business partner who's now doing something complicated at Flickr). The spool, then, was partly software written by people I know -- but mostly it was also a very funny blog written by people I know (it's very much less active these days since the house was scattered to the four corners of the tech universe).
Unable to source the code at the time to run 2lmc's bot, I settled on the Chumps' engine. And so spool.evolt.org was born.
Someone posts a link, and it either sits there for others to follow at their leisure, or the rest of the group leaves comments on it. These could be more related links, dumb pictures, or (most usually) snark. (Apparently the kids are calling this a 'tumblelog' now because they don't know it's been around forever and -- at the risk of reigniting a debate with James yet again, it seems everything has to be invented all over again indefinitely by the less-geeky ones who didn't see it the first time.)
So?
This format's bluddy grate. It's social (it's written by everyone in the group or any ad-hoc subset thereof). It's quick and dirty, like all good tech is. But -- and here's the Twitter reference -- it's insanely easy to write. I can make a new post by copy-and-pasting something off my browser's address bar and pressing Enter.
Compare that to writing this. I need to attempt to cram everything into some sort of logical order despite sleep deprivation, after logging in and typing it all into a web form, then checking and unchecking a bunch of stuff and submitting and rechecking. And then editing out the inevitable typo or three. Blogging software these days is great as a simple content management system. But for actual blogging, for throwing a bunch of stuff up or covering something in real time? These IRC things are so much better.
Finally the promised election reference
In 2005 I was running a then-rather-well-known political blog before and during the UK general election campaign. In the last week I set up a second blog to cover as-it-happens election news and invited the world and her dog to join in on election night.
The channel was packed on election night, and twenty or more people were posting results in real time as they arrived for the sixty-odd people in IRC and another several thousand hitting Refresh watching the IRC blog on the web.
In other words, it's been tested in action during an election and it was a) successful and b) inclusive, but most of all c) great fun.
So here's today's new project: KildareStreet.com. News and commentary as it's posted elsewhere, sifted by whichever group of people ends up coalescing around it. Plus as much snark as you like because, let's face it, no matter how much heat is generated in the next 23 days we can be pretty sure there won't be any light.
Come on down and try it for yourself. #kildarestreet on Freenode.net -- the IRC bot doing all the work has been christened 'ceanncomhairle'.
After the election I imagine I'll keep it going to cover whatever comes next, but right now I've got no idea how it'll turn out. More interesting that way.
read more | delicious | digg | reddit
Code | Ireland | Politics | Vocal Voter
Submitted by johnh on October 31, 2006 - 17:06.
For those of you who are as frustrated as I am at the relatively slow pace of progress at Vovo, an update of sorts.
Last week I unilaterally decided to postpone the collection of more indices to electoral registers until after the site is launched -- which sounds remarkably counterintuitive since that was supposed to be how I'm resolving addresses to constituencies.
Simon McGarr and I spoke at some length at BarCamp Ireland last month about a different approach: perhaps, we thought, the famed map hack plus the OSI's DigiBoundary would get us very close very quickly?
Sadly not without dropping at least seven thousand euro. Because obviously it's a frickin' state secret which places are in which constituencies. So that idea died about two weeks back.
Anyway. I've since had another idea, and it's working. I spent some of Friday night writing code which will take another couple of weeks to finish doing its thing. So far, this new process has added another half a million people to my home-served private prototype.
If (as herself says), God willing and the creek don't rise, and nothing else goes wrong, there's a good chance I'll have a full national address-to-TD lookup working in four weeks.
That'll be the hard bit done, and we can get to sorting the writing to them part and launch the bugger -- and all in under a year from the first promise. Barely under a year, I grant you, but still.
read more | 1 comment | delicious | digg | reddit
Politics
Submitted by johnh on May 15, 2006 - 18:07.
ABC hairdo Brian Ross seems to have rediscovered what his job is, finally.
Bush administration people are going through ABC News's land line and cellphone records to trace calls made from inside the Federal government, hoping to find out who's leaking all this inconvenient information about the rampant lawbreaking so many of them are up to.
Tip: UK reporters use no-name "pay as you go" cell phones for this stuff. Smart people assume that someone's going to want to listen in. No point in making it easy for them.
read more | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on March 20, 2006 - 14:08.
The development site now covers an extra half million people compared to yesterday, since I've finished merging the Dublin Corporation's data set.
Top marks to Dublin City, by the way, for having the best data in the country. Streets which are divided between electoral areas are clearly marked as such (Cork County, by way of contrast, just lists them once in each area with no information about which parts are which). EAs which straddle constituency boundaries are divided accordingly with pseudocodes.
Yay Dublin, in short.
read more | 2 comments | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on February 23, 2006 - 12:49.
...have residents who can determine their local eleced representatives on the Vovo dev site.
Because I've added Galway City and Galway County to the database this morning.
read more | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on February 17, 2006 - 23:56.
The Vocal Voter prototype now also covers Leitrim.
Adding a second county required a whole bunch of UI work which counties three through 26 won't now require.
read more | delicious | digg | reddit
