I gave a talk at the Big Data Insight Group in London recently and they've just posted my talk online.

We all know there are decisions where you need data to help you make them and there are decisions where data just isn't that important. This morning XKCD did a wonderful job of illustrating it. http://xkcd.com/1036/

Buying a lamp is a creative decision. Turn your eye away from the reviews and go with your heart :)

The same is true of many decisions data folks are asked to help with every day in organisations. We shouldn't be afraid to champion this strategy there, either!

We sat down recently to talk data and insight. Here is what we talked about, plus a little video of me talking about insight at both zeebox and EMI.

http://www.thebigdatainsightgroup.com/site/article/david-boyle-emi-zeebox-data-driven-includes-video

I don't like the term 'scientist' as it makes the role sound unaccessible and elite. Google's Hal Varian said "the sexy job in the next ten years will be statisticians" ... but I don't like that term either. I'd replace 'statisticians' with 'working with data' or something ... and then I believe it!  I think data people have a tendency to overplay the role of the 'statistics' and magic of it and underplay the importance of the 'bringing it to life' and 'helping people understand / make use of it' parts of working with data.

I thought about this because of this cool article in The Guardian about data scientists.

As it points out, "science" is defined as "systematic study of natural or physical phenomena". I guess that's us all. Perhaps I shouldn't shy away from that phrase.

The journalist describes the role well, as "someone who can bridge the raw data and the analysis - and make it accessible. It's a democratising role; by bringing the data to the people, you make the world just a little bit better." Perfect, eh?

One last quote: "the four qualities of a great data scientist are creativity, tenacity, curiosity, and deep technical skills." That list sounds pretty good to me, also. So perhaps I should rename this the 'data scientist' blog and be done :)

I spoke on a panel last night on the subject 'data as the new black gold'. There are three challenges I think this metaphor poses to the data world.







First, that of crude oil. Data is everywhere in organisations, but too often left in it's crude form: gloopy and unusable. The oil industry had to work this out before it could be mainstream. It had to refine oil to a form that works for consumers day-to-day and it had to make it available to consumers in ways that fitted in to their daily life. It's trivial to stop by a petrol station and pick up some oil in a format you can instantly make use of. Data doesn't yet work the same way: it's rare to find an organisation that appropriately refines it and then makes it available to it's people in a way they can access and make use of as part of their day-to-day work.


Second, I think we need to demand higher 'miles per gallon' from our data. Often we gather fantastic raw data, capable of being a really powerful part of decision making ... but then business leaders don't ask interesting questions of it. They don't demand smart analysis and challenge the data to offer insight. It's like demanding that cars offer higher miles per gallon from the oil they are burning.


Finally, I think we need to embrace hybrid technology. In cars that's about oil being only part of the story for how the car gets powered. In data it's about saying that data is only part of the story for how organisations get powered. We need to be honest and bold about the role of skills & judgement alongside data in powering organisations. Too many people believe / pretend that data alone can power organisations to greatness. Everything I've seen tells me that data is necessary but not sufficient: smart people to use the data alongside their expertise is ALWAYS required. The data world should be honest about this and build data and systems around that truth. I've always found that has a much greater impact :)

I've used a lot of word clouds recently. But I think of them as charts really, since they are still pretty faithful to the underlying data. The size of the word is proportional to the number of times that word is in the data set. Simple.

But reading a cool data visualization book I came across this. Really it not based on 'data', but it's interesting his words and their location on the page conveys such a lot of information. Perhaps some good, well placed words can replace the need to chart actual data?

http://creativeroots.org/2011/03/italy-infographic-map/

Simple, easy to read, but really powerful. Nice little sparklines spotted in the papers from the 20 week scan my wife just had. Cool little chart like this should be everywhere!

And by the way, it's a boy!