Info Clarity

Making sense of big numbers

As the New York Times says so clearly: "The human mind isn’t very well equipped to make sense of big numbers. We don’t deal with a trillion of anything in our daily lives, and so when we come across such a big number, it is hard to distinguish it from any other big number. Millions, billions, a trillion — they all start to sound the same."

So when trying to communicate the annual cost of the Iraq war, the New York Times article compared it to things we could understand, like healthcare, schooling and cancer research. After all: if you don't spend the money on war, you can certainly spend it somewhere else: "The way to come to grips with $1.2 trillion is to forget about the number itself and think instead about what you could buy with the money. When you do that, a trillion stops sounding anything like millions or billions."

This is just great. Suddenly the big numbers mean something. The only other place I deal with big numbers is at work though. So perhaps next time I have to communicate the benefits of a project at work, I'll compare it to the things that money could otherwise be spent on: like shorter lines at the checkout (more checkout staff), cheaper veg and such like.

I remember seeing this when The Times first published it in early 2007 and loving it. Glad I finally got around to posting it on the blog!

Understanding the root causes

When you are not happy about something there are probably a dozen or more potential causes. Its usually easy to jump to an idea which of these to blame. But do we really know it is the main cause? And have we even thought about the other causes to see how much each of them is to blame?

I had this problem in a big way at work. A project wasn't working. Something big. All sorts of systems and processes and people weren't quite playing together nicely and the collective outcome was bad, bad, bad. Each team involved had an idea of what the problem was. Some blamed a system. Some blamed processes that weren't being followed. But the fascinating thing was than nobody involved could come up with a complete list of what all the root causes could be, and how much each was contributing to the poor performance. So we developed a guide (obscured because its confidential - sorry!):

This shows the three main measures of 'performance': What are the things we can measure to know whether performance is 'good' or 'bad'? It then lists the seven things that can cause performance to be bad. It has to be one of these and all should be investigated to understand poor performance. Finally it branches out in to a tree: If one of the seven things looks like its a cause, it shows you what to check next.

This was used to great effect recently by one of our projects. Things were looking bad. The overall results were terrible. I visited the country in question and they had used the principles from the root cause tree to really understand it. They could tell me straight away that although overall performance was poor, 60% was due to XXX, 20% due to YYY, 10% was ZZZ and the remainder was a bunch of smaller issues that weren't worth investigating. Amazing insight. We were then able to confidently and quickly focus our efforts on the real root cause.

Its not just boring work projects that would benefit from explanation in this way. I thought about what it is that makes me happy and came up with this:

... which immediately made me realise that I had got the balance wrong in daily life!!

I also thought about what makes me happy at work, and came up with this:
A little simpler than the life version, and a lot simpler than the one I did for work. But it gives me something to measure my job against and to judge other jobs by.

Finally I was looking back with frustration at political campaigns I have been involved in. The bits I was interested in is the gathering of data and the use of data to help increase the campaign's effectiveness. I sketched out a diagram with the overall thing I sought to achieve in campaigns at the top (Improved effectiveness and clear reporting on progress) and all of the things that were needed to achieve this. No campaign I have worked on had these elements. But I have also never seen a diagram with all of the elements on it before. So its not really a surprise that it was never achieved if nobody had ever clearly set out what needs to be achieved!

So all in all I have fallen in love with drawing root cause trees as a way to understand complex problems and thought I'd write a little blog post to share it. (Even though there are no numbers or charts involved.)

I guess the moral of this story is, if there is a complex problem out there with lots of things contributing to it, try listing them all and joining them with little arrows. It just may help to bring some clarity.

Tree maps for playing with data

Treemaps are an excellent tool to allow you to play with and really understand a data set. Take, for example, the use for which they were invented: visualising hard drive usage:

The screenshot is from SpaceMonger. I honestly don't see how else you could really understand how your hard drive was being used. Which folders are big? Where are the big files? How much space could I free up if I ...

I have started to use treemaps at work now, courtesy of MacroFocus. I work with data across thousands of products. In the exxample below (dummy data!!!) I show the treemap I look at to understand which products are contributing most to waste. Equally, you could look at stock holding, availability, ...
Manyeyes has a great web-based implementation of treemaps, but the data you upload becomes public. This obviously limits its use when I don't want my data sets floating around the net!!

Mapping Iowa ... in Excel

I popped over to Iowa just before the causus in January to play with some data for one of the presidential candidates. One thing that has always irritated me is how difficult for most people to get access to a decent map of data.

We were looking forward to the causus and how it might turn out, based on campaign data and trends. We were interested in running some scenarios by precinct and looking at the results by county. A table is the usual way to do this, but it is 100 times easier to visualise and understand the data if it is laid out on a map.

Usually I would have used a GIS program, but I was using a laptop where I can't install programs and i had 24 hours in Iowa to visualise some data and leave a tool there for the campaign to use on election night.

Here's what I built ... in Excel.
Not perfect, but fun :)

The excel tool allows you to slot in data of your own and see the map of it. I actually built it to allow EASY customisation by state, so that it could be used for the rest of the primaries. But it both wasn't as easy to use as I thought it was and people on the campaign actually had better things to do than to play with Excel, so it was only used in Iowa. Oh well!

The map layout was inspired by http://www.style.org/iowacaucus/

Visualising your Gmail inbox

Word from information aesthetics on a great tool from some nerds that does something I'm amazed Gmail hasn't done for you: provide visualisation of your emails.

an IMAP-based email analysis project, which generates tables, graphs & visual distributions based on time of day, senders, recipients, mailing lists, & so on.
graphs include distribution of messages by year, month, day, day of week & time of day, distribution of messages by size & the top 40 largest messages, the top senders, recipients & mailing lists subscribed, distributions of senders, recipients & mailing lists over time, & the distribution of thread lengths.

Get it here.

I spent about an hour and couldn't get it to work. That doesn't make it not exciting though! I'm sure a .exe version or an official Gmail version will be around the corner ...

I have 92,705 emails in my Gmail going back as far as 2000, so this would be a pretty amazing tool. [How? By transferring my old Outlook archives using LimitNone's fantastic little application]

Visualising Facebook friends

For sites that have so much information about me, both Facebook and Google do remarkably little to visualise it. Today I found Nexus: a great application for Facebook that lets me visualise my friends.

There are two ways to visualise your friends:

The second is my favourite. You can really see clumps of friends there. For example, here are the clumps that I can spot right away: