Amongst many fascinating stories in Malcolm Gladwell's Outliers, chapter eight has one that brings back a strange memory for me. And one that should remind us to always question whether the data we are shown really supports the conclusions we draw from it.
Consider the following table of scores achieved by kids in Baltimore public schools across 1st-5th grade. (The test referred to is the California Achievement Test, but that's not important to the example.)
What conclusions might we draw from this? I think we might reasonably start to think that Baltimore public schools were failing low income pupils. They start off with only a slight disadvantage from their moneyed peers (32 poits), but end school significantly under-performing them (73 points).
You get a very different story if you look just at what happens during the time pupils were in school. Karl Alexander tested pupils at the start and at the end of every school year, enabling him to measure how many points they gained while actually in school. Here are the results:
Now it seems that, if anything, schools are of more benefit to poorer kids. Across the five grades, during the school years, they gaines 189 points, while the wealthy kids gained only 184 points. The difference between the first table and the second lies in how many points the pupils gained or lost during the long summer holidays:
And if we know this, we know that the story isn't about education at all. Its about what happens in the school holidays. Poor kids on average neither gain nor loose points over holidays. But richer kids consistently gain. So by the end of the school period, they outperform their peers.
So the analytical lesson here is to always be careful that the data you are using really supports the conclusions you are drawing from it. Ask yourself: If you cut the data differently, might it tell a completely different story? If so, give it a shot.
And what of the Beano you might ask? I have an odd memory in primary school of my class gathering around the teacher after the long summer break one year. He asked us to each pick one book we'd read over the summer and tell the class about it. One by one the pupils in my class told their classmates about one of the books they had enjoyed. When the teacher pointed to me, I had to ask whether a comic counted, since that was all I'd read over the summer.
Add a comment