wget -e robots=off -r http://foo...

Just saying.

The most useful thing I’ve found about this approach is the common vocabulary that lets developers, non-technical production staff and academics converse in a much more fluent way that if we were each speaking in our own language. It forces the engineering teams to understand the problem in the terms of the client, then write their schemas, their models and their APIs around this language. I feel this leads to a more coherent, explainable system.

I thought it would be interested to explain the domain that is driving our software. I make apologies now for any inexpert curiosities below, I’m not a scientist and our software represents only a generalized model of an experiment. Comments and corrections are welcome.

Academic Sponsors

Each experiment is devised by a qualified academic, someone with an expert understanding of the scientific field under study.

The academic will ensure the scientific integrity of the experiment, oversee the ethical approval process and develop the mathematical models needed to analyze the data collected. If the experiment succeeds in producing new knowledge they may also report their findings through peer-reviewed journals.

Experiments (and Research Questions)

As a starting point for the design of an experiment the sponsor will typically phrase their work as an investigation to a research question.

For example, in our Child of our Time personality test the question formulated was to help further understand the correlations between demographic, life-style and personality traits. Or, in other words:

“Do our personalities shape our lives or do our lives shape our personalities?”.

With this question at the forefront of the academic’s mind they can begin to think about what sort of data they need to collect and measure for to aid this research.

Variables

Wikipedia defines an experiment as, “a method of investigating causal relationships among variables”. Variables represent the raw unit of data within the experiment, the empirical measurable things.

Here’s our variables:

If you run an experiment to test the speed of sound under different environmental conditions your variables might represent things like altitude, air pressure, temperature and time of day. In a sociological study the variables might represent demographic information about a person, age, ethnicity, salary etc.

Our software doesn’t distinguish between the various types of variables (dependent, independent, background …). The variable type seemed to depend very much of the perspective of the observing scientist and more important to the models they create than how that data is collected.

Is important to note that our variables aren’t just domain-agnostic key-value pairs, they have a slightly more interesting internal structure.

credit: socialresearchmethods.net

Values are the internal representations of variables, in contrast to the public facing attributes. For example, the public-facing attribute of a response might be ‘yes’ and ‘no’ whereas the internal values for analysis might be 1 or 0. The attributes are legible to humans, the values legible to computers. Because of this it usually makes more sense if values are numeric.

If the prompt ‘Do you like chocolate?’ has four attributes ‘detest’, ‘dislike’, ‘like’, ‘love’ then the internal values for each of these might be ‘0′, ‘1′, ‘2′, ‘3′ (3 = love, 0 = detest). For the purpose of analysis the internal values might often represent a ratio rather than fixed intervals, so ‘detest’ has a internal value of ‘-10′, and ‘dislike’, ‘like’, ‘love’ have values of ‘-3′, ‘0′, and ‘5′ respectively. Separating values from attributes makes it easier to represent these sort of relationships in data.

The relationship between values is known as the level of measurement, which determines the type of statistical analysis one can perform on the data once collected. i.e. data representing time needs different analysis to data representing a ranked scale.

Some variables are specialized, especially where we think they will appear again and again in multiple experiments.

Distance for example is a specialized variable with two constraining properties, min and max. Where this happens we’ve tended to pay special attention to the UI. In this instance the height of a person (height is a type of distance) is represented as a slider, which conveniently converts between metric and imperial as the handle is dragged horizontally.

We have similar specializations for things like weight, data collected against a Likert-scale, as well as several common data-types (boolean, enumerated, alphanumeric), each one enforces a standard interaction pattern across experiments.

Variables can also exist in composite form, share a co-dependence on one another and have validation criteria assigned to them. I expect as the system evolves and projects grow in scope our understanding of variables will grow deeper.

Tasks

A task is a container for a set of variables to be collected.

A task can be presented as a test, a game, a set of questions or, really, anything capable of collecting the variables required by the experiment.

Say an experiment needs to answer a question about wind speeds in the UK the task is then standing in a middle of a field with an anemometer (if it happens to have an http client built in).

If another experiment, as in Brain Test Britain, wants to the determine effectiveness of Brain Training then the tasks could be a series of puzzles to benchmark a person’s mental agility over a period of time.

So, a task is just a means to generate data for our experiment.

Here’s out experiment variables split in to three tasks.

Splitting the variables in to tasks has lots of benefits.

We can ask half you users to complete task 1 and the other half task 2 as found in classic A/B testing, or designate task 3 as the placebo group.

We can mandate that task 2 can be completed only once whereas tasks 1 and 3 can be completed multiple times by the same person.

We can designate periods of time that must occur between taking the tasks (ie. daily, three times a week, every other Tuesday) or we can decide how to assign (or prevent) certain tasks from people based on their given age, gender or some previous variables we hold against them. For example, the ethical considerations might require us to prevent under 18’s from taking part.

We can also randomize the order of tasks or serve them up a in fixed sequence.

Tasks, if you haven’t guessed yet, form the structure of the experiment, they control the flow of how we collect variables.

Participants (not Users)

I keep saying ‘users’, but people who go to websites are called users. People who take part in experiments (as the human subjects under study) are more usually called participants, so this is what we refer to them as. Presently a participant is an alias for a person with a BBC iD account.

Modeling the data this way allows us to inherit the benefits of this external service. A BBC iD user has a date-of-birth (i.e. an age at the point of participation), a country of residence and perhaps later things like friends and relations. All these things may be of interest to a scientist during their research.

The three experiments so far have all been sociological based, which lend themselves well to this vocabulary. As our understanding of different experiments grows perhaps there might be other types of user.

Variable Sets and Traits

In the last couple of months we’ve realized that we needed to do something with the variables the participant enters in to the site. There’s not a lot of reward in just filling out forms, so people want feedback.

If tasks are sets of variables that control how the data is collected we need a second set containing the same variables that control how the variable are analysed. We call this second group variable sets.

Here’s our variables again:

Lets say variables D, H, and K belong to a variable set. And lets say D represents height, H represents weight, and K the participant’s age. We could call this variable set, ‘body mass index‘, or BMI for short.

Note that these variables don’t need to be collected at the same time, they can span tasks. Age can collected at the point of registration, upon consenting to the experiment, and the other variables might be collected at some future point in time or conditionally based on their past activity.

We can think of the variables in this set as inputs to a function. In the case of our BMI, the function might calculate your age/weight/height ratio. The output of this function can be called, in very, very general terms, a trait, something that the participant can be said to exhibit give the measures they have divulged.

As far as is possible the technical system does not interpret the trait as being positive or negative. If a participant has a BMI value of less than 16.5 then we present that figure to the front-end (editorial) system that can chose to interpret the finding as it wishes and present them to the participant.

This is an important distinction. Our system does not know if a participant is morbidly obese or if they are severely emaciated, nor how to present that is a sensitive way to the participant, all these things are editorial judgements. Our software’s responsibility is to accurately calculate the BMI from a given set of inputs, not to have return things like, “needs to diet”, “needs to see relationship councillor”.

In the personality experiment the variable sets were based around the Big Five traits. Once calculated, the participant’s traits were mapped to videos of Robert Winston explaining what they meant.

Fin.

When I find some time I’ll write some more notes on the technical components, the APIs and how the software evolves over the coming months.

Actually, I wasn’t against all merchandising when I started the strip, but each product I considered seemed to violate the spirit of the strip, contradict its message, and take me away from the work I loved. If my syndicate had let it go at that, the decision would have taken maybe 30 seconds of my life.

Bill Watterson, interview, also here.

The general director agreed to an interview (for free) and the next day I found him in his office. It is basic and battered and located in the basement of the Omnisports Stadium, just a few doors down from the room where he kept 120 pygmies from the Cameroonian rainforests locked up last summer. Milla [a Cameroonian star at the 1990 World Cup] had invited the pygmies to play a few games at the Omnisports, to raise money for their health and education, but he imprisoned them there, issued them with guards (one of whom wore a Saddam Hussein T shirt) and seldom fed them. A tournament spokesman explained to Reuters: “They play better if they don’t eat too much”. As for the imprisonment: “You don’t know the pygmies. They are extremely difficult to keep in control”. The Omnisports cook concurred: “These pygmies can eat at any time of the day and night and never have enough”. The little hunters themselves were too frightened to comment.

Their tournament was a disaster. Team names included Bee-Sting of Lomie and the aptly named Ants of Salapoumbe, but only 50 fans bought tickets, and most of these came strictly to shout abuse at the pygmies.

From Football Against The Enemy - Simon Kuper

Due sometime in Feb. Woot!

Quite a good picture, definitely has a head, a heart, hands etc.

This one shows the final day of England’s win at Lords.

It reminded me of Tufte’s Sparklines, so I plugged the data file in to timetric, a Cambridge based company who will plot your data to a pretty graph if you give them a half-sensible URI.

Out popped this:

The Sparkline shows England’s odds of winning which went from ‘likely’ to ‘very likely’ shortly after 11am when Flintoff took the first wicket, indicated by the sudden drop in odds from 1.43 to 1.13 as the line starts. The visitors spent the next hour battling back to only to find the game effectively over when Swann took the second, as seen in the cliff-like drop in odds to 1.01 towards the centre of the sparkline.

You can visit the (bigger) graphs at my timetric page.

I think next time I’ll find a sport with multiple participants to see what lots of interspersed lines over the x-axis look like. The British Open golf would have been a neat example of this but I wasn’t in front of a computer yesterday, alas.

It cost me about $0.02 to run my first simple test and when I have a 50 tests running each night of various complexity I guess is going to cost perhaps $3 or $4 a day, which isn’t a lot for the hassle of avoiding purchase/maintenance of a little farm of selenium boxes. It’s peanuts compared to developer time, who I’d rather pay to write tests than maintain infrastructure.

The only problem I found was an unhelpful error message sent from the RC server when my Selenium client supplied it bad credentials.

Not Found
com.thoughtworks.selenium.HttpCommandProcessor.getCommandResponse(HttpCommandProcessor.java:124)

The problem in my case was twofold. Firstly I’d cut and pasted the Saucelabs access-key incorrectly causing an authentication error (my fault) and secondly, for some reason, I quoted the browser string in my Ant properties file with single quotes which caused ResourcesBundle to read and pass the complete string (quotes intact) to the RC server, ie.

selenum.server=saucelabs
selenum.port=4444
selenium.browser='{"username": "bbc_labuk", "access-key": "xxx", "os": ... }'
...

Why did I do that? My fault again I suppose. Or maybe ResourceBundle should strip quoted properties?

Either way, when Saucelabs RC server doesn’t like the smell of your browser string it will give you a ‘Not Found’ error and it confused me for the best part of an afternoon - ‘Not Found’ being very synonymous with a certain other class of HTTP error.

Updated.

John from Saucelabs has been in touch. Error messages is something they are ‘investing more attention’ too in the near future, so stay tuned!

Most online betting companies operate live markets that are left open to new bets as the event is taking place. For example, last weekend you could still bet on Andy Roddick to win Wimbledon right up until the final point of the final game, the odds growing smaller and smaller as his chance to win grew ever more impossible.

This data about the truth can tell a fascinating retrospective story about the market. If there’s a lot of market movement during the game, with odds fluctuating between the eventual victor and loser, the event could be deemed more exciting as the collective wisdom couldn’t make up their minds as to who was going to win, the outcome only being discovered in the dying moments of the match.

In the dullest matches the odds flat-line, showing little movement in any direction or very quickly favouring one team over the other.

Similarly, points of high drama (a sending off in rugby, a tie break in tennis…) tend to swing the markets rapidly in one direction or another for a short period of time as the crowd herd towards a particular outcome.

So, if we have data that can determine competitiveness and amount of drama in a sporting event then these are at least two of the prerequisites that determine whether something is worth watching on the various online catch-up services. Furthermore, now that iPlayer can link to time segments within a show, the betting data could be also be used to provide indexes to key moments in a match.

I thought it would be fun to log the live market movement of the 2009 Women’s singles final every few seconds to see what story it would tell. The image below uses timeplot from the MIT SIMILE project, showing each player’s odds along the y-axis and time along x-axis,

demo here (requires html canvas support)

The red line shows Venus the clear favourite (with lower odds on the y-axis) right up until the first set tie break at 3pm after which the odds were reversed as the match gradually slipped away from her over the next half-an-hour. It was an exciting match for an hour or so.

Looking at the corresponding BBC Sport live text reports there were two moment of drama. The first around 30 minutes in to the match can be seen obout a third of the way along the timeplot, where Serena’s odds jump sharply up for a couple of minutes. Here’s the BBC Sport notes from around that time,

14:38 Venus *4-4 Serena Venus ramps up the power on her return, jumping out to a 30-0 lead, and peppering the baseline with some ferocious groundstrokes, she earns two break points. Serena’s second serve kicks up viciously to force the error, before lil sis comes galloping to the net. Venus misses by inches with the pass and Serena comes through with two aces on the trot.

It seems to describe the first important moment in the match, Serena nearly losing her serve.

The other obvious change in the time series, at 3pm, where the market swings rapidly between the two possible outcomes is again described by the BBC,

15:02 Venus 6-7 (3-7) Serena Serena nudges ahead, a rocketing forehand making Venus net for 3-1. HawkEye challenge on the next point, but Venus’s backhand brushes the baseline. A crunching off-forehand means Serena swaps sides at 4-2. At 5-2, Serena comes up with a brutal combination of groundstrokes, finding the angle to wrong-foot Venus - leaving her sister on the ground. Serena doesn’t see that, she has turned around and is pumping her fist. She misses the first set point but then produces a stunning, stunning backhand lob to claim it. More fist-pumping. Brilliant stuff from the younger Williams.

After that the odds diverge to the extent where the match is effectively over 15 minutes before the last point.

It would be neat if I could link the timeline up to the time stamps in archived Wimbledon final on iPlayer so that people could dip in and watch the clips of these two important moments - bbc.co.uk/i/p003n7bt/?t=25m30s - but it doesn’t seem the feature is enabled for that video.

You can get the time series data yourself, as well as download the betfairfree project that generated this data.

cat /dev/random | hexdump | tr -d ' \n'

Take your pick,

f8af39f196671bedac2a252a400219f905a40aea18a7c8460ef37e7a1993b51130219fa
06da0efd4e57c77f3ef898b93a08905b70219fb07451adff722635f0e9a91393f9046a9
b0219fc0a05f41d1225cc5544b5440926bb56dd50219fd0233f831154277b7c2f2ea6c2
e354f5690219fe0ee82dc88de5647dc1f68377af4f570890219ff0bcc39ff21e0108c85
535f4082aed3463021a000c3451ca6d31d6bb4eced539502fa6295021a01023432a6f94
dfb09eb0e2ad431ba399a1021a02088df34efac7cf34b3505d7a0db85e194021a030dfc
348f9f4ca7ac4cec11186c981da86021a04078bd504f169b45fc5ecadbf5c4bcb9cf021
a050136134d82552d6e66ed98df588a0a458021a060e6c545ee29682cec0220ef2d509c
9a13021a070e5530fd5f5111dc5a3c5caac7d7707a8021a0800aa56c58be02cd787ff67
a49da53adac021a09099cb0e2c7bd6c371b2583a346795128e021a0a0cb8e0c94a3fa89
e224d4ef4605c9dc9e021a0b0bcb328a854c94085b122db28fed33056021a0c0ba33066
3abd4e8c93dfb844638c8e39c021a0d02bdd93d26b01b12e864ee479d204d97a021a0e0
b34d88531075b4dd4743516d41e8a120021a0f072d129dde427facf3007c837202c8d25
021a10057d9eec29ffeaaaaad30e445b501cf44021a110372eb24a3fd950e2a4fcc2ad4
8a5f36a021a120937e4c3a5f8a1c2361c9686dd84a72f1021a1305eba3541beb6aaf543
ade2896466f0cf021a140eb357c1630dac7822de7cf314970e1a9021a150873934b7e02
cc986e6042f75778f7250021a16095f8936a531133dcea3ea1f74dec2286021a170c5d9
4f7a1ad2bd535a29019bd539b22a021a180f808b28cf6ef5998b1835e12997131d6021a
190ea14d048bc3dcc5602ca8ee034b6d219021a1a097870e6061d8d03f7de6a84858cd7
af9021a1b0f91a388d379dd71bfb2927468d643358021a1c0f49219d7b3f24d3389bd76
1f0742bce7021a1d022158a751e9f56e42be5621951bff4d9021a1e0471b33bcbe3bae8
63bec16954ae2bf28021a1f0d71a9e8dccf8b4fc3e7b6f235c2ba06e021a2007a443a07
7309130d2eaceafafd159b83021a210ff4db4075c4ba736b7aa318b009d2383021a220b