REST-assured on Cent OS

To get REST-assured running on my CentOS VM with the sqlite3 gem I needed to install sqlite3 manually and export the LD_LIBRARY_PATH. Like this.

sudo wget http://www.sqlite.org/sqlite-autoconf-3070400.tar.gz
sudo tar xzvf sqlite-autoconf-3070400.tar.gz
cd sqlite-autoconf-3070400
sudo ./configure
make install
export LD_LIBRARY_PATH=/usr/local/lib

Query DOM with Xpath in Chrome

Go to Developer Tools, hit ‘Escape’ on the Elements tab, then type :-

document.evaluate( 'count(//p)', document, null, XPathResult.UNORDERED_NODE_ITERATOR_TYPE, null );

Responsive design @ bbc.co.uk

I spoke to an internal audience last week about a responsive design project my team has been working on at the BBC.

The slides, which explain the motivation behind the project, are available as a PDF, or you can view them on slideshare. The video of our prototype that accompanied the presentation is available on YouTube.

Design Sans Frontières

Imagine you didn’t understand the difference between a hardware accelerated CSS3 animation, a JQuery tween or a declarative SVG transition. How would you be able to decide on how your UI moved gracefully from one state to another on a dozen different devices?

Or imagine you couldn’t mentally measure an interface feature in kilobytes. How would be able to decide on the fastest experience for a person on a slow connection?

Imagine if you never read the data model documentation and had to guess the presence, function, relationship and structure of each object. How would you decide the composition of an interface?

Imagine if you had to create tactile interface without ever touching it. How would you comprehend what gestures didn’t confuse?

Imagine you didn’t spend Sunday evening reading through the W3 specifications and browser vendors nightly READMEs. How would you understand what’s going to happen next?

Design is a manifestation of experience, and the experience is ever more complex.

Happy path design pattern

The happy path design pattern involves pushing down improbable but important worst-case scenario execution paths to the deepest part of your application so they are only executed in the exceptional circumstances at which they occur.

I wrote this because I’m unaware of it being written anywhere else.

Worst-case coding

Developers are trained to code cautiously. Errors and exceptions are predicted and trapped, databases are read before insertions and updates happen, we check caches before we attempt to purge their items. Our code is littered with conditional constructs to determine whether or not we should to do something.

The problem also extends to business analysts. Functional specifications are full of edge cases, ‘a user can’t do this if they’ve already done such and such.’

Your code probably has a lot of this going on :-

if (thing.exists === false)
	thing.create
else
	thing.update
end

Each execution cycle is pessimistic, assuming the worst will happen (that the thing exists and we will try to create it again rather than update it) and so you are, understandably, trying to prevent that and avoid your boss/colleagues/clients from thinking you are bad at your job (by building things that don’t result in error states).

This approach doesn’t always scale.

In many circumstances the chosen path is overwhelmingly going to be one or the other. Lets call this the happy path, the other paths being edge cases, 1/1000 events that will break your system but hardly ever happen.

Here’s an example of our simple code sample presented as a registration system where a user object is queried with a unique id (Eg, an email address) and then created if it doesn’t exist, otherwise updated.

if (user.exists(uid) === false)
	user.create(uid)
else
	user.update(uid, telephone)
end

How many times a day does somebody attempt to register with the same email address as someone else or bypass your systems security to update a user that doesn’t exist? Both these requirements are essential but may be edge cases in some systems.

In this case we make a minimum of two calls to the user object for every registration/update - a read (exists) and a create or update.

Twice doesn’t sound too bad, and it’s often not of course. But let’s say the user object is talking to a web service in a different data centre, each function (exists, create, update) maps to a HTTP method (GET, PUT, POST) on stateless, RESTful API. Let’s also say your site is popular to the tune of 1000’s of requests a second at certain times of the week.

You’ve just doubled the network traffic and operations on that database because of an edge case.

Happy path

Lets look at this problem again. We are wrapping edge cases around our optimal path. Is there a better way we can solve this? What would our code look like if we assumed the best-case scenario usually happens?

In the following example we pass the error handling down to the deepest application layer and the handle the exceptional case if and when it occurs. It works nicely when you know that one path is far more likely to happen the the other.

// assume something will blow up underneath if bad, Eg. mysql
// duplicate primary key error) and then catch that exception
try
	user.create(uid)
	catch(e)
		user.update(uid, telephone)

The above example is crude but came about from high-transaction survey software that I was helping design and build at work.

We found while load testing we could tear out many cautious calls (mostly reads, naturally) to sub-systems like memcached that would throw graceful exceptions when, for example, we tried to create something that was already there. Likewise our HTTP API would send back 409 code when tried to update something that had gone stale.

It’s a simple trick that had a significant impact on our performance.

It’s interesting to me as a developer because knowingly causing lower level errors/exections is counter-intuative to what we’ve been trained to do.

Who Is The Secret Footballer?

The identity of the Guardian’s secret footballer is keeping me awake at night.

Summary

• Not foreign, most likely English due to his articles on Capello and foreigners.
• Probably in his 30s, but not past it. (not Sol Campbell, Danny Mills, Keown).
• Current in a Premiership squad. (not Craig Bellamy)
• Happily married. (not Peter Crouch or Frank Lampard)
• An outfield player. (not David James)
• Played for several Premiership clubs (unlikely to be Phil Neville, Owen Hargreaves, Clarke Carlisle, Carragher, Kevin Nolan etc.)
• A long stint a one club under a single manager (4+ seasons?) at at least one of them (not Jason Roberts or other journeymen etc.)
• Possibly past involvement in international football (unlikely to be Matthew Taylor, Kevin Davies)
• Played a match the weekend 2nd - 3rd April (so not Joe Cole, Woodgate, Gerrard, Andy Johnson, Heskey etc)
• Strong farther figure (not Joey Barton!)

I think this leaves three possible candidates - Scott Parker, Danny Murphy, Kevin Phillips.

They’ve all had potential to form ‘lifelong friendships’ with an unnamed Scandinavian - Phillips (Sørensen), Scott Parker (Claus Jensen), Danny Murphy (Hyypiä or Riisee). They’ve all played under good and bad managers, most squads have had a small French contingent in the last decade, and gambling seems rife across the industry so that doesn’t give me any specific clues.

Many people think Kevin Davies, but hasn’t played at enough clubs for my liking, so I’m striking off my list - the secret footballer has regularly been involved in transfer negotiations whereas Kevin seems content at Bolton.

I can’t narrow it down further with any certainty.

Murphy is the front runner because of his past media work but that might be too obvious and his 29th January column railed against pundits (he occasionally works for ITV).

A couple of tweets make me think it is Murphy though.

• “Which 3 will be relegated from the Prem this season? My friend is adament Villa will go but they don’t see the type!” - I wouldn’t expect this from a player in the middle of a relegation battle, as Parker is at West Ham.
• “Congratulations to Birmingham. What do you think the difference was? Except one goal…” - Referring to Birmingham in this way would be odd for Phillips.

For my own sanity I’m assuming he’s not given out any false information and that it is a single person, not a group of player, nor a Guardian hack as some have suggested.

So, I think it’s Danny Murphy for now.

Weekly Notes

9 April

• I once won a free-kick playing at Old Trafford and a Manchester United player brought the ball over to the spot where it was to be taken from and handed it to the referee. [I take this to mean he's not played for Manchester United. also definitely an outfield player]
• In a nutshell, that is the “justification” for shouting in the referee’s face, waving your finger at him, lambasting him every time he gives something against you and jogging past him and “getting in his ear”. I don’t overly like it and I’m uncomfortable doing it. [Only an outfield player would have regular opportunity to berate a referee]
• My wife told me that last week … [Currently married]
• I still grew up to be a footballer and, may I say, a fairly well-adjusted one, and that is because of my upbringing and the values that were instilled in me. [strong father figure]

2 April

• I have made some lifelong friends, like the Scandinavian I’ve known for so long that I forgot he wasn’t English. [Long-term EPL Scandinavians]
• A group of French players I played with at one club who were not interested in communicating with anybody else. [Could be French speaking, so covers much of North/West Africa]
• And there was a striker who refused to move unless the pass went exactly where he wanted. [Could be one of many primma donna forwards - Berbatov, Anelka]

19 March

Played at a club with a culture of gambling.

• I’ve known one player who operates on what I would describe as very close to a professional level when it comes to gambling.
• And then there is the complete opposite type of gambler. I’ve actually roomed with one of these and it isn’t pleasant … taking calls from bookmakers all over the world at all hours of the day.

12 March

• I saw a talented young kid reduced to tears in the changing room … thinking back, I can definitely guess [Several references throughout various columns to the younger generation.]
• It isn’t because we’re a superior breed – even I wouldn’t take on that argument with half the England team conspiring against me. [this isn't a very well worded sentence, I'm not sure if we can conclude he's an ex-international]

5 March

This article hints that he’s got an agent with a good reputation.

• When I hang up my boots, I will have opportunity to move on to things I have always wanted to do before I undergo the hip and knee replacements my doctor promised a few years ago. [Obviously an older player]
• I have been in contract negotiations where the Bosman ruling hung over proceedings. [Has been around a few clubs with several contracts]

26 February

Nothing specific in this column but the examples he gives examples of laws he thinks FIFA should introduce, “the situation that arises when a defender shepherds the ball out of play for a goal-kick, preventing his opponent from getting the ball despite making no attempt to play it”, which makes me think it can’t be a defensive player.

Also, this :-

• A friend of mine recently told me his team lacked any clear direction or ambition and, as a result, problems came about which had not been addressed. He said things got so bad that a group of players even began to raise the possibility that it could be a deliberate ploy by the manager to get himself sacked. [99% sure this is a friend of Liverpool]

19 February

• I’ve played for great managers, and I’ve also played for one or two where I would happily have faked my own death if it meant not working with them a minute longer. [Well travelled]
• At one club I played for, it was because of a shared belief that our tactics were flawed, that this was making us look like poor players and, in turn, lose matches. [RE: managers loosing the dressing room]
• I remember at one club being fined for going out to a pub with a couple of friends while injured.
• He was referring to whether or not I escorted a young lady home, despite the fact that he knew I had a long-term girlfriend … I was young back then.
• The first time this happened I had been playing for the manager in question for a long time [had a long stint at least one club] … a man who I looked up to as a god of the game from the way he handled the players, the fans and the media. [But it's not Harry Redknapp] … and it showed him up as a man who, at the time, was troubled. [Houlier?]

CVs

These are my thoughts, not those of my employer

I’m reading quite a lot of CVs at the moment. Here’s what I think is important.

Keep it to about two pages when printed. I can’t stress enough the improbability of an unfocussed eleven pager making it past the initial CV filtering process. I do actually print CVs out and scribble on them rather than reading off the screen.

Unless you are a very recent graduate it’s not overly important to me if you got a B+ in your Geography GSCE in 1992. Tell me (briefly) if you got ten straight A’s or were awarded a scholarship, but please don’t list every exam result you’ve ever had.

List your jobs in reverse chronological order. It’s weird how some CVs start with what they were doing in 1995. Fine if you invented the cure for the common cold and won the 1995 Nobel Prize in Medicine. Not fine for less significant events.

I’m no good at spelling, and worse at grammar, so I don’t care overly if the CV has the odd hard to parse sentence, but make an effort. Spellcheck it at least.

Include a URL to some code you wrote. You are applying for a programming job.

Writing this is pretty meaningless :-

• PHP – 7 years
• Ruby on Rails – 4 years
• JavaScript (JQuery, Prototype) - 6 years
• CSS & HTML – 7 years
• XML – 7 years
• XSLT - 4 years
• ActionScript – 8 years
• SEO – 4 years
• …

Am I to think that you have over 92 man-years of experience in your short career or that you’ve stuck a lot of buzzwords in your CV to fish for work? Separate out what you think you really are good at, probably things you’ve been doing on a daily basis for few years. Then mention the few other relevant things you’ve got decent exposure to. The clue as to what to leave in/out will be in the job ad.

On that note, doing this is even worse (taken from an otherwise great CV) :-

PHP OOP, Zend Framework, MVC, Drupal, XML, XSD, XSL, DOM, Xpath, Smarty, Pear, XMLRPC, SOAP, Lucene, Sphinx, SOLR, Cassandra, Thrift, MySQL (foreign keys, optimisation, indexing), ORM (Active Record), Properl, Doctrine, Caching, Memcache, OAuth, Apache auth (basic, digest) EC2, S3, MongoDB, TDD, RAD, Agile, PHPUnit, Xdebug, LAMP, Apache (Linux config, mod_rewrite, system admin, testing, stats - Munin), Tomcat, SVN, Git, Eclipse, Aptana, JEE, Spring Framework, MVC, Hibernate, Tomcat, JUnit, Quartz, Velocity, SOAP, Maven, Ant, Netbeans, Eclipse, Query, Prototype, Ajax, Comet, Mootools, DOM, Behaviour scripting, HTML 5 (canvas), CSS, Mail (SMTP, IMAP, POP3), DNS, DHCP, SSH, Telnet, FTP, Database Server (Sybase, MySQL, PostgreSQL, LiteSQL), Apache, TOMCAT, PHP Server, Ruby on Rails configuration, Memacached, SVN Server …

It’s recruitment spam. Don’t do it.

Also, don’t cut bits and pieces out of the job description and insert them in your personal statement. I’ll spot it.

A four page CV mentioned the word PHP no less than twenty-nine times.

Some people think I need every technical acronym expanding. RDBMS becomes, “relational database management system (RDBMS)”. It’s not necessary.

Irrelevant stuff. Bartender, ice-cream van driver, gardener, please don’t mention jobs that aren’t related to the one advertised. Some people put the fact they have a clean driving license on their CV. The job doesn’t involved driving, or cars.

I don’t really read the personal interests section. I think I might be sociopathic.

Illustrated London News on Broadcasting History

I found an old copy of the Illustrated London News on my mother-in-laws bookshelf. More specifically it’s the special edition published to celebrate George V’s Silver Jubilee in 1935.

The 50-or-so-page book is largely a review of the key events and scientific, cultural and technical achievements falling with in the monarchs reign, which, with hindsight, was only to last a further 7 months (he died in early 1936 from assorted smoking related diseases).

The early 1920s saw attempts to popularise radio broadcasting so included in the book are several references to (upper-class) members of society experiencing the medium.

Here’s my favourites…

1913: A family calibrating their various time pieces using the Parisian 11:30pm broadcast.

1923: Several overdressed folk sat on the London to Liverpool Express were entertained by radio broadcasts from London and Birmingham. The caption reads as if the train carriage was a successful proof of concept rather than an everyday occurrence.

1923: It was noted that a live band in one location could simultaneously play at many private parties. This was later to inspire the work of a Mr Peter Tong of Kent.

1933: Broadcasting House is finished and the 10-year old BBC moves in.

1935: The announcement of a ‘London television station’ was illustrated by an early model of a home tv set (aka. the Baird receiver). Note the comically small screen, I think displaying Adrian Chiles’ face, and some kind of mechanical apparatus built in to the armchair that appears to be associated with the function of the gentleman’s telly (possibly the receiver).

Have you?

• dev checked it
• cached it
• unit tested it
• profiled it
• acceptance tested with whoever wanted it
• load tested it (reads & writes)
• monitored it
• stress tested it for 12 hours
• volume tested (with 5m users)
• documented it

Wget & robots.txt

Wget obeys robots.txt files. And that’s why I’ve spend the last 45 minutes trying to figure out why it didn’t download an Apache directory of images. It says so in the first paragraph of the man page. One day I will learn.

wget -e robots=off -r http://foo...

Just saying.

Tasks, Variables and Participants: Experiments on bbc.co.uk

My team has spend the last 10 months writing software to run mass-participation experiments on bbc.co.uk. It’s been a fun project to work not least because it’s the first time we’ve tried to model the domain in which we are working and then write the code, features, tests, APIs etc. around it.

The most useful thing I’ve found about this approach is the common vocabulary that lets developers, non-technical production staff and academics converse in a much more fluent way that if we were each speaking in our own language. It forces the engineering teams to understand the problem in the terms of the client, then write their schemas, their models and their APIs around this language. I feel this leads to a more coherent, explainable system.

I thought it would be interested to explain the domain that is driving our software. I make apologies now for any inexpert curiosities below, I’m not a scientist and our software represents only a generalized model of an experiment. Comments and corrections are welcome.

Academic Sponsors

Each experiment is devised by a qualified academic, someone with an expert understanding of the scientific field under study.

The academic will ensure the scientific integrity of the experiment, oversee the ethical approval process and develop the mathematical models needed to analyze the data collected. If the experiment succeeds in producing new knowledge they may also report their findings through peer-reviewed journals.

Experiments (and Research Questions)

As a starting point for the design of an experiment the sponsor will typically phrase their work as an investigation to a research question.

For example, in our Child of our Time personality test the question formulated was to help further understand the correlations between demographic, life-style and personality traits. Or, in other words:

“Do our personalities shape our lives or do our lives shape our personalities?”.

With this question at the forefront of the academic’s mind they can begin to think about what sort of data they need to collect and measure for to aid this research.

Variables

Wikipedia defines an experiment as, “a method of investigating causal relationships among variables”. Variables represent the raw unit of data within the experiment, the empirical measurable things.

Here’s our variables:

If you run an experiment to test the speed of sound under different environmental conditions your variables might represent things like altitude, air pressure, temperature and time of day. In a sociological study the variables might represent demographic information about a person, age, ethnicity, salary etc.

Our software doesn’t distinguish between the various types of variables (dependent, independent, background …). The variable type seemed to depend very much of the perspective of the observing scientist and more important to the models they create than how that data is collected.

Is important to note that our variables aren’t just domain-agnostic key-value pairs, they have a slightly more interesting internal structure.

credit: socialresearchmethods.net

Values are the internal representations of variables, in contrast to the public facing attributes. For example, the public-facing attribute of a response might be ‘yes’ and ‘no’ whereas the internal values for analysis might be 1 or 0. The attributes are legible to humans, the values legible to computers. Because of this it usually makes more sense if values are numeric.

If the prompt ‘Do you like chocolate?’ has four attributes ‘detest’, ‘dislike’, ‘like’, ‘love’ then the internal values for each of these might be ‘0′, ‘1′, ‘2′, ‘3′ (3 = love, 0 = detest). For the purpose of analysis the internal values might often represent a ratio rather than fixed intervals, so ‘detest’ has a internal value of ‘-10′, and ‘dislike’, ‘like’, ‘love’ have values of ‘-3′, ‘0′, and ‘5′ respectively. Separating values from attributes makes it easier to represent these sort of relationships in data.

The relationship between values is known as the level of measurement, which determines the type of statistical analysis one can perform on the data once collected. i.e. data representing time needs different analysis to data representing a ranked scale.

Some variables are specialized, especially where we think they will appear again and again in multiple experiments.

Distance for example is a specialized variable with two constraining properties, min and max. Where this happens we’ve tended to pay special attention to the UI. In this instance the height of a person (height is a type of distance) is represented as a slider, which conveniently converts between metric and imperial as the handle is dragged horizontally.

We have similar specializations for things like weight, data collected against a Likert-scale, as well as several common data-types (boolean, enumerated, alphanumeric), each one enforces a standard interaction pattern across experiments.

Variables can also exist in composite form, share a co-dependence on one another and have validation criteria assigned to them. I expect as the system evolves and projects grow in scope our understanding of variables will grow deeper.

Tasks

A task is a container for a set of variables to be collected.

A task can be presented as a test, a game, a set of questions or, really, anything capable of collecting the variables required by the experiment.

Say an experiment needs to answer a question about wind speeds in the UK the task is then standing in a middle of a field with an anemometer (if it happens to have an http client built in).

If another experiment, as in Brain Test Britain, wants to the determine effectiveness of Brain Training then the tasks could be a series of puzzles to benchmark a person’s mental agility over a period of time.

So, a task is just a means to generate data for our experiment.

Here’s out experiment variables split in to three tasks.

Splitting the variables in to tasks has lots of benefits.

We can ask half you users to complete task 1 and the other half task 2 as found in classic A/B testing, or designate task 3 as the placebo group.

We can mandate that task 2 can be completed only once whereas tasks 1 and 3 can be completed multiple times by the same person.

We can designate periods of time that must occur between taking the tasks (ie. daily, three times a week, every other Tuesday) or we can decide how to assign (or prevent) certain tasks from people based on their given age, gender or some previous variables we hold against them. For example, the ethical considerations might require us to prevent under 18’s from taking part.

We can also randomize the order of tasks or serve them up a in fixed sequence.

Tasks, if you haven’t guessed yet, form the structure of the experiment, they control the flow of how we collect variables.

Participants (not Users)

I keep saying ‘users’, but people who go to websites are called users. People who take part in experiments (as the human subjects under study) are more usually called participants, so this is what we refer to them as. Presently a participant is an alias for a person with a BBC iD account.

Modeling the data this way allows us to inherit the benefits of this external service. A BBC iD user has a date-of-birth (i.e. an age at the point of participation), a country of residence and perhaps later things like friends and relations. All these things may be of interest to a scientist during their research.

The three experiments so far have all been sociological based, which lend themselves well to this vocabulary. As our understanding of different experiments grows perhaps there might be other types of user.

Variable Sets and Traits

In the last couple of months we’ve realized that we needed to do something with the variables the participant enters in to the site. There’s not a lot of reward in just filling out forms, so people want feedback.

If tasks are sets of variables that control how the data is collected we need a second set containing the same variables that control how the variable are analysed. We call this second group variable sets.

Here’s our variables again:

Lets say variables D, H, and K belong to a variable set. And lets say D represents height, H represents weight, and K the participant’s age. We could call this variable set, ‘body mass index‘, or BMI for short.

Note that these variables don’t need to be collected at the same time, they can span tasks. Age can collected at the point of registration, upon consenting to the experiment, and the other variables might be collected at some future point in time or conditionally based on their past activity.

We can think of the variables in this set as inputs to a function. In the case of our BMI, the function might calculate your age/weight/height ratio. The output of this function can be called, in very, very general terms, a trait, something that the participant can be said to exhibit give the measures they have divulged.

As far as is possible the technical system does not interpret the trait as being positive or negative. If a participant has a BMI value of less than 16.5 then we present that figure to the front-end (editorial) system that can chose to interpret the finding as it wishes and present them to the participant.

This is an important distinction. Our system does not know if a participant is morbidly obese or if they are severely emaciated, nor how to present that is a sensitive way to the participant, all these things are editorial judgements. Our software’s responsibility is to accurately calculate the BMI from a given set of inputs, not to have return things like, “needs to diet”, “needs to see relationship councillor”.

In the personality experiment the variable sets were based around the Big Five traits. Once calculated, the participant’s traits were mapped to videos of Robert Winston explaining what they meant.

Fin.

When I find some time I’ll write some more notes on the technical components, the APIs and how the software evolves over the coming months.

Quote from Bill Watterson (lesson for Quentin Blake et al.)

Actually, I wasn’t against all merchandising when I started the strip, but each product I considered seemed to violate the spirit of the strip, contradict its message, and take me away from the work I loved. If my syndicate had let it go at that, the decision would have taken maybe 30 seconds of my life.

Bill Watterson, interview, also here.

Heartbeat

pip-4-months.mp3, with artwork & composer credits.

Bee-Sting of Lomie

This quote from a Simon Kuper book made me laugh a lot but I’ve no idea of it’s veracity,

The general director agreed to an interview (for free) and the next day I found him in his office. It is basic and battered and located in the basement of the Omnisports Stadium, just a few doors down from the room where he kept 120 pygmies from the Cameroonian rainforests locked up last summer. Milla [a Cameroonian star at the 1990 World Cup] had invited the pygmies to play a few games at the Omnisports, to raise money for their health and education, but he imprisoned them there, issued them with guards (one of whom wore a Saddam Hussein T shirt) and seldom fed them. A tournament spokesman explained to Reuters: “They play better if they don’t eat too much”. As for the imprisonment: “You don’t know the pygmies. They are extremely difficult to keep in control”. The Omnisports cook concurred: “These pygmies can eat at any time of the day and night and never have enough”. The little hunters themselves were too frightened to comment.

Their tournament was a disaster. Team names included Bee-Sting of Lomie and the aptly named Ants of Salapoumbe, but only 50 fans bought tickets, and most of these came strictly to shout abuse at the pygmies.

From Football Against The Enemy - Simon Kuper

Baby

Due sometime in Feb. Woot!

Quite a good picture, definitely has a head, a heart, hands etc.

More Timeplotting

I made another timeplot similar to the Wimbledon Singles thing I produced earlier this month.

This one shows the final day of England’s win at Lords.

It reminded me of Tufte’s Sparklines, so I plugged the data file in to timetric, a Cambridge based company who will plot your data to a pretty graph if you give them a half-sensible URI.

Out popped this:

The Sparkline shows England’s odds of winning which went from ‘likely’ to ‘very likely’ shortly after 11am when Flintoff took the first wicket, indicated by the sudden drop in odds from 1.43 to 1.13 as the line starts. The visitors spent the next hour battling back to only to find the game effectively over when Swann took the second, as seen in the cliff-like drop in odds to 1.01 towards the centre of the sparkline.

You can visit the (bigger) graphs at my timetric page.

I think next time I’ll find a sport with multiple participants to see what lots of interspersed lines over the x-axis look like. The British Open golf would have been a neat example of this but I wasn’t in front of a computer yesterday, alas.

Sauce Labs ‘Not Found’

Saucelabs is a cunning idea. Running Selenium RC (the Selenium HTTP API) off some EC2 instances means I don’t need to build and maintain my own functional test boxes.

It cost me about $0.02 to run my first simple test and when I have a 50 tests running each night of various complexity I guess is going to cost perhaps $3 or $4 a day, which isn’t a lot for the hassle of avoiding purchase/maintenance of a little farm of selenium boxes. It’s peanuts compared to developer time, who I’d rather pay to write tests than maintain infrastructure.

The only problem I found was an unhelpful error message sent from the RC server when my Selenium client supplied it bad credentials.

Not Found
com.thoughtworks.selenium.HttpCommandProcessor.getCommandResponse(HttpCommandProcessor.java:124)

The problem in my case was twofold. Firstly I’d cut and pasted the Saucelabs access-key incorrectly causing an authentication error (my fault) and secondly, for some reason, I quoted the browser string in my Ant properties file with single quotes which caused ResourcesBundle to read and pass the complete string (quotes intact) to the RC server, ie.

selenum.server=saucelabs
selenum.port=4444
selenium.browser='{"username": "bbc_labuk", "access-key": "xxx", "os": ... }'
...

Why did I do that? My fault again I suppose. Or maybe ResourceBundle should strip quoted properties?

Either way, when Saucelabs RC server doesn’t like the smell of your browser string it will give you a ‘Not Found’ error and it confused me for the best part of an afternoon - ‘Not Found’ being very synonymous with a certain other class of HTTP error.

Updated.

John from Saucelabs has been in touch. Error messages is something they are ‘investing more attention’ too in the near future, so stay tuned!

Timeplotting Betfair data

Betting odds are a great (if not the greatest) indicator of future truths.

Most online betting companies operate live markets that are left open to new bets as the event is taking place. For example, last weekend you could still bet on Andy Roddick to win Wimbledon right up until the final point of the final game, the odds growing smaller and smaller as his chance to win grew ever more impossible.

This data about the truth can tell a fascinating retrospective story about the market. If there’s a lot of market movement during the game, with odds fluctuating between the eventual victor and loser, the event could be deemed more exciting as the collective wisdom couldn’t make up their minds as to who was going to win, the outcome only being discovered in the dying moments of the match.

In the dullest matches the odds flat-line, showing little movement in any direction or very quickly favouring one team over the other.

Similarly, points of high drama (a sending off in rugby, a tie break in tennis…) tend to swing the markets rapidly in one direction or another for a short period of time as the crowd herd towards a particular outcome.

So, if we have data that can determine competitiveness and amount of drama in a sporting event then these are at least two of the prerequisites that determine whether something is worth watching on the various online catch-up services. Furthermore, now that iPlayer can link to time segments within a show, the betting data could be also be used to provide indexes to key moments in a match.

I thought it would be fun to log the live market movement of the 2009 Women’s singles final every few seconds to see what story it would tell. The image below uses timeplot from the MIT SIMILE project, showing each player’s odds along the y-axis and time along x-axis,

demo here (requires html canvas support)

The red line shows Venus the clear favourite (with lower odds on the y-axis) right up until the first set tie break at 3pm after which the odds were reversed as the match gradually slipped away from her over the next half-an-hour. It was an exciting match for an hour or so.

Looking at the corresponding BBC Sport live text reports there were two moment of drama. The first around 30 minutes in to the match can be seen obout a third of the way along the timeplot, where Serena’s odds jump sharply up for a couple of minutes. Here’s the BBC Sport notes from around that time,

14:38 Venus *4-4 Serena Venus ramps up the power on her return, jumping out to a 30-0 lead, and peppering the baseline with some ferocious groundstrokes, she earns two break points. Serena’s second serve kicks up viciously to force the error, before lil sis comes galloping to the net. Venus misses by inches with the pass and Serena comes through with two aces on the trot.

It seems to describe the first important moment in the match, Serena nearly losing her serve.

The other obvious change in the time series, at 3pm, where the market swings rapidly between the two possible outcomes is again described by the BBC,

15:02 Venus 6-7 (3-7) Serena Serena nudges ahead, a rocketing forehand making Venus net for 3-1. HawkEye challenge on the next point, but Venus’s backhand brushes the baseline. A crunching off-forehand means Serena swaps sides at 4-2. At 5-2, Serena comes up with a brutal combination of groundstrokes, finding the angle to wrong-foot Venus - leaving her sister on the ground. Serena doesn’t see that, she has turned around and is pumping her fist. She misses the first set point but then produces a stunning, stunning backhand lob to claim it. More fist-pumping. Brilliant stuff from the younger Williams.

After that the odds diverge to the extent where the match is effectively over 15 minutes before the last point.

It would be neat if I could link the timeline up to the time stamps in archived Wimbledon final on iPlayer so that people could dip in and watch the clips of these two important moments - bbc.co.uk/i/p003n7bt/?t=25m30s - but it doesn’t seem the feature is enabled for that video.

You can get the time series data yourself, as well as download the betfairfree project that generated this data.

Random (enough) strings on OS X

I needed some big random strings.

cat /dev/random | hexdump | tr -d ' \n'

Take your pick,

f8af39f196671bedac2a252a400219f905a40aea18a7c8460ef37e7a1993b51130219fa
06da0efd4e57c77f3ef898b93a08905b70219fb07451adff722635f0e9a91393f9046a9
b0219fc0a05f41d1225cc5544b5440926bb56dd50219fd0233f831154277b7c2f2ea6c2
e354f5690219fe0ee82dc88de5647dc1f68377af4f570890219ff0bcc39ff21e0108c85
535f4082aed3463021a000c3451ca6d31d6bb4eced539502fa6295021a01023432a6f94
dfb09eb0e2ad431ba399a1021a02088df34efac7cf34b3505d7a0db85e194021a030dfc
348f9f4ca7ac4cec11186c981da86021a04078bd504f169b45fc5ecadbf5c4bcb9cf021
a050136134d82552d6e66ed98df588a0a458021a060e6c545ee29682cec0220ef2d509c
9a13021a070e5530fd5f5111dc5a3c5caac7d7707a8021a0800aa56c58be02cd787ff67
a49da53adac021a09099cb0e2c7bd6c371b2583a346795128e021a0a0cb8e0c94a3fa89
e224d4ef4605c9dc9e021a0b0bcb328a854c94085b122db28fed33056021a0c0ba33066
3abd4e8c93dfb844638c8e39c021a0d02bdd93d26b01b12e864ee479d204d97a021a0e0
b34d88531075b4dd4743516d41e8a120021a0f072d129dde427facf3007c837202c8d25
021a10057d9eec29ffeaaaaad30e445b501cf44021a110372eb24a3fd950e2a4fcc2ad4
8a5f36a021a120937e4c3a5f8a1c2361c9686dd84a72f1021a1305eba3541beb6aaf543
ade2896466f0cf021a140eb357c1630dac7822de7cf314970e1a9021a150873934b7e02
cc986e6042f75778f7250021a16095f8936a531133dcea3ea1f74dec2286021a170c5d9
4f7a1ad2bd535a29019bd539b22a021a180f808b28cf6ef5998b1835e12997131d6021a
190ea14d048bc3dcc5602ca8ee034b6d219021a1a097870e6061d8d03f7de6a84858cd7
af9021a1b0f91a388d379dd71bfb2927468d643358021a1c0f49219d7b3f24d3389bd76
1f0742bce7021a1d022158a751e9f56e42be5621951bff4d9021a1e0471b33bcbe3bae8
63bec16954ae2bf28021a1f0d71a9e8dccf8b4fc3e7b6f235c2ba06e021a2007a443a07
7309130d2eaceafafd159b83021a210ff4db4075c4ba736b7aa318b009d2383021a220b

Journey to Australia

The plan: http://tinyurl.com/cmlcp5.

Took a 42 hour trip from Dorking, Surrey to Moina, Tasmania, only survived by a kip in the Hong Kong airport nap room and food and lie down the Virgin lounge in Sydney domestic terminal. 42 is a long time. Don’t ever do this.

Wondered up a hill and amongst the wallabies and wombats in Narawntapu on the Northern coast.

Saw devils, and koalas at the Trowunna Wildlife Park.

More walking, this time up Marion’s Lookout on Cradle Mountain, truly deserving of it’s World Heritage status.

We had a quick tour of a Hobart vineyard and ate some fish and chips by the harbor then went to Clare and Mark’s wedding. The taxi driver got lost on the way but the bride was late anyway.

Then up Wineglass Bay to walk around more eye-wateringly beautiful national park.

Then to Melbourne to see many, many comedians at the festival. Personal fav’s, Hannah Gadsby and Al Madrigal.

The ground floor of the Ian Potter Centre has a fantastic indigenous gallery.

We drove from Melbourne to Sydney along the coast. Lots of surfers.

Lots more national parks to nip off the road and visit. Lake Elizabeth, Twelve Apostles marine park, Booderee.

We hired a little motor boat and spent the afternoon sailing around a lake in Mallacoota.

We saw flocks of cockatoos, parrots, pelicans, and kangaroos, an echidna about to be squashed by a megatonne truck.

We had to divert to Canberra to swap the hire car. Canberra reminded me of a Dan Dare megalopolis, or maybe a little bit like some mid-30’s German architected city.

The colours in night sky was amazing. Thousands of stars in dark amber galatic clusters.

Kiama was a really cute little seaside resort. We squished a fly as we drove along some more.

Finally arrived in Syndey, tripped across the habour to Manly & watched the Jerry Springer in the Opera House. We shared our hotel with the cast. It was odd eating breakfast next to Jesus, he had eggs and bacon.

The Sydney Aquarium was good for a couple of hours.

Actually, you probably just want to look at my wifes flickr album.