Vocal Voter
Ireland | Politics | Vocal Voter
Submitted by johnh on April 28, 2009 - 21:31.
A substantial part of the parser was reusable enough to make short work of the next major milestone for KildareStreet.com, although the standard of source material published by the Oireachtas was an order of magnitude worse than the Dáil's.
Je vous présente: Seanad Éireann -- every word uttered since January 20th, 2004.
That's another 1.5 million words or so.
And yes, this means you can sign up for email alerts and whatnot for Senators too, and search through the record for things uttered in the Seanad. All the same good stuff, but twice as much. Well, one and a half times.
read more | 3 comments | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on April 22, 2009 - 16:54.
Since 20th January 2004, the Dáil has been in session for 493 days (including today).
Up until this morning TDs made 249,489 speeches, and asked 160,503 questions which received written responses.
On average, TDs are producing 507 speeches and 326 written answers per sitting day.
Sadly you won't have had access to information like this before, since the Houses of the Oireachtas publishes the Dáil record primarily on an illegible website ridden with elementary display errors.
What we needed, it was decided (by myself and independently also by some other people, though I didn't discover their existence until last month), was a port to Ireland of the spiffing UK site TheyWorkForYou.com.
Then we could (gasp!) search the parliamentary record. Get RSS feeds for TDs. Set up email or RSS alerts for people or search terms. Maybe even be able to read the bloody thing without wanting to kill oneself after about twenty seconds of stupidly-and-permanently-underlined horror.
MySociety helpfully publishes the source code which powers TheyWorkForYou, and I installed it back in October. Some readers may recall me showing that site, containing nothing but a list of TDs in the right place, back at Barcamp Cork 2.
The trick, however, is to get all the actual debates and questions into the site's database. You need to build a parser program to convert all that data from one format into another -- in Ireland's case, from the raw XML published by the Houses of the Oireachtas which is primarily intended as a print publishing format for the official record, but which also powers debates.oireachtas.ie.
The parser project languished for a good while until MySociety got another contact (this time from Gavin Sheridan) and forwarded him to me. Gavin, basically, annoyed me into making more progress.
I started in earnest again about six weeks ago and have worked on almost nothing since then.
The end result
As of this afternoon, then, we're pleased to announce that Ireland's local version of TheyWorkForYou is called KildareStreet.com, and that it's now available in public beta.
It still contains bugs, so be careful about jumping to early conclusions :)
At KildareStreet.com you can:
- Read a dramatically-more-legible version of the Dáil Record going back to January 2004,
- Search that record using a fabulous search engine which I didn't write - you can restrict searches to speeches or written questions, or by speaker, or by date or date range,
- Sign up for email alerts for when a search query you're interested changes, or whenever a TD of your choosing says something or asks a question which generated a written reply, and
- Subscribe to RSS feeds for individual TDs or for search queries.
The site will be updated the day after each sitting day shortly after the Oireachtas publishes that day's first report version in XML. RSS feeds and emails are generated shortly after that.
Coming soon
In the next few weeks you will also be able to inspect the past five and a half years of TD expenses, and the register of members' interests.
There's a programming API in the code base which we have not yet properly localised, and is therefore not yet available. I shall sort that out as quickly as possible.
Once we've done all of that, I'll get started on the Seanad.
Thanks due
None of this would be possible without the code for TheyWorkForYou having been released under a permissive license. This is the second non-UK installation of this code base (after Australia) and there's also a similar project in New Zealand. So thanks to MySociety, and to Matthew Somerville in particular for answering a large number of stupid questions while I was putting this together.
The Houses Commission employs people who could, when they got wind of it, have made this project more difficult or even impossible. They did not do so, and I and you owe them our thanks for that. Leo Bollins and Tom Malone have helpfully answered questions about their publishing formats and not been freaked out when I sent them bug reports about their source material.
Gavin Sheridan poked and chivvied this project into existing now instead of later, is sourcing ancillary data like member interests and expenses, and will be helping to keep the site ticking over as a moderator. For the first of these above all, this site etc without whom blah blah blah. Cheers, Gav :)
Justin Mason contributed actual code to start parsing written answers and has offered to assist in making the data import process follow at least some sensible test-driven methods which my shonky parser does not even begin to follow yet. He is the only actual code contributor to KildareStreet other than myself and code is, frankly, everything. The parser as it now is isn't using his stuff (yet) but we owe him a debt of gratitude also.
Simon McGarr is quietly providing assistance in the background, the fruits of which will become apparent in due course. Thanks to you.
Finally, and more than all the above, I live with a very tolerant and patient wife. Sabrina Dent, not entirely unknown in these parts, has been working for money and covering everything in our house in that manner while I've been doing all this for no cash. She thinks it's a worthwhile project and if she didn't, KildareStreet absolutely would not exist. Hire her!
Anyway. KildareStreet. It's now in testing. Have at it. Send me bug reports. Enjoy.
And above all? I think it's time we raised our game generally. Blogs are fine, sure, but actual apps that help people are required. This is my first contribution. How about you?
read more | 17 comments | delicious | digg | reddit
Ireland | Vocal Voter
Submitted by johnh on November 21, 2006 - 14:58.
As promised, another Vovo update. I'm now sitting on a truly enormous database of addresses in Ireland. The database-minded among you will spot some horrific query times (and, for that matter, horrific queries) in what follows, and you'd be right in thinking that this database is pretty raw and needs some work. Right now I've got a seven-figure number of rows in one table.
Still... here's some screenshots of my terminal window to whet your appetites in the meantime.
Tipperary is counted twice, since you asked.
Handy info for figuring out who's a councillor of what.
Since all the addresses in this Cork street are in the same electoral division, we can replace 11 database records with just one, and still correctly identify the electoral representatives for the whole street.
read more | delicious | digg | reddit
Code | Ireland | Politics | Vocal Voter
Submitted by johnh on October 31, 2006 - 17:06.
For those of you who are as frustrated as I am at the relatively slow pace of progress at Vovo, an update of sorts.
Last week I unilaterally decided to postpone the collection of more indices to electoral registers until after the site is launched -- which sounds remarkably counterintuitive since that was supposed to be how I'm resolving addresses to constituencies.
Simon McGarr and I spoke at some length at BarCamp Ireland last month about a different approach: perhaps, we thought, the famed map hack plus the OSI's DigiBoundary would get us very close very quickly?
Sadly not without dropping at least seven thousand euro. Because obviously it's a frickin' state secret which places are in which constituencies. So that idea died about two weeks back.
Anyway. I've since had another idea, and it's working. I spent some of Friday night writing code which will take another couple of weeks to finish doing its thing. So far, this new process has added another half a million people to my home-served private prototype.
If (as herself says), God willing and the creek don't rise, and nothing else goes wrong, there's a good chance I'll have a full national address-to-TD lookup working in four weeks.
That'll be the hard bit done, and we can get to sorting the writing to them part and launch the bugger -- and all in under a year from the first promise. Barely under a year, I grant you, but still.
read more | 1 comment | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on March 20, 2006 - 14:08.
The development site now covers an extra half million people compared to yesterday, since I've finished merging the Dublin Corporation's data set.
Top marks to Dublin City, by the way, for having the best data in the country. Streets which are divided between electoral areas are clearly marked as such (Cork County, by way of contrast, just lists them once in each area with no information about which parts are which). EAs which straddle constituency boundaries are divided accordingly with pseudocodes.
Yay Dublin, in short.
read more | 2 comments | delicious | digg | reddit
Vocal Voter
Submitted by johnh on February 28, 2006 - 15:45.
read more | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on February 23, 2006 - 12:49.
...have residents who can determine their local eleced representatives on the Vovo dev site.
Because I've added Galway City and Galway County to the database this morning.
read more | delicious | digg | reddit
General | Ireland | Vocal Voter | Webcasting: EU
Submitted by johnh on February 20, 2006 - 09:25.
...but then again I can only parse about 10-15% of it, not speaking the language as I don't. Still, this expat got a kewlness buzz from it when it showed up in my referer log.
An tImeall immortalises yours truly in the Irish language. I can read the bit about me having done something around Xmas, and having recently moved from England (Shasana) to Cork (Corcaigh).
Not sure I like the look of that John Bull clause though :)
(Incidentally, I've been here since August, and I'd really rather people didn't refer to me as English, what with my Scots family and my Irish passport and my sincerely-held belief that multicultural London and overwhelmingly-right-wing-and-backward England are best mentally seperated. I'm obviously in no position to tell whether that's being said, being Gaelige-ly challenged, but just in case...)
I've left a comment/appeal there which I'm happy to repeat here: IMHO VocalVoter must be bilingual and I'll need more volunteer help to make that happen.
read more | 2 comments | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on February 17, 2006 - 23:56.
The Vocal Voter prototype now also covers Leitrim.
Adding a second county required a whole bunch of UI work which counties three through 26 won't now require.
read more | delicious | digg | reddit
Ireland | Politics | Vocal Voter
Submitted by johnh on February 14, 2006 - 23:56.
It's not finished, but then again I have often found that posting something here is the next best thing to never telling anyone.
It contains only streets and townlands in County Mayo.
It allows you to find out who your TDs are. It does not yet have the whole faxing-people thing going on.
Clear? OK.
I've knocked up a prototype. It's at http://dev.vocalvoter.com . If you live in Mayo, you may find it useful :)
If you want to help us scrape up the data we don't yet have, join the list and then get into the wiki to find out more.
If you care about the details, it's Drupal 4.7. The Ajaxy live search thing is generated by my new Street.module's implementation of Drupal's new autocompletion hook.
And I'd just like to say, for the record, that figuring out fecking MySQL's cackhanded UTF8 handling cost me nine hours of my life last night that I will never see again.
Still. Pretty, isn't it?
read more | delicious | digg | reddit
