Survivorship bias occurs when we draw conclusions only from examples which have passed some sort of selection criteria and systematically discount those which have not.

During World War II, British bombers were suffering a fairly awful attrition rate. Understandably, the RAF were keen to try to improve the survivability of their aircraft. Most of the bombers that limped back to base showed signs of heavy damage around the cockpit and wing tips and so the prevailing opinion was that if these sections of the aircraft were reinforced more planes would survive. Then, along came statistician, Abraham Wald who pointed out that engineers were only considering the planes that had survived being shot at. The planes that had been shot down were being discounted altogether. Wald speculated that the areas that showed the most damage in surviving aircraft were the sections least in need of reinforcement and that instead engineers should focus on those areas of the planes which showed the least damage as these were, most probably, where the non-returning bombers had been hit. He was right:  reinforcing the more vulnerable parts planes meant that survival rates jumped dramatically.

Survivorship bias is hard to spot. When we consider the reasons for our success we take inventory of our actions  and understandably conclude that it what we did that made a difference. It’s exactly this sort of faulty logic that leads to self-made millionaires telling school students not to worry about exams; after all, they left school with nothing but a pocket full of pencil shavings and look at them now! The trouble is, these people are successful despite their lack of formal qualifications not because of it. As I’m sure you’ve noticed, the overwhelming majority of people who do badly at school do not become millionaires.

Recently, I was talking to a group of teachers about John Dunlosky’s review of effective study skills, and it turned out that they had all – without exception – been firm proponents of the least effective methods. According to a survey of students’ study habits, 84% report spending most of their time rereading and highlighting their notes and text books. Dunlosky found that despite their popularity, rereading and highlighting are the most inefficient, ineffective ways to study. There’s even some evidence that highlighting leads to decreased performance. (I can recommend Jake Hunton’s book, Exam Literacy if you want to explore this topic more fully.)

But if these approaches to study are so ineffective, why are they so popular? Well, this can be explained partly because they’re easy, and partly because they result in the comforting belief that we’re familiar with what we’re rereading, and so they seem to lead to a strong illusion of knowledge; students end up feeling confident that they know things they’re unable to recall independently. But despite these attractions you’d still think that by now we’d have generations of students who’d be testifying that their shoddy study habits had led to exam failure. So why don’t teachers routinely warn students against the dangers of highlighting? Survivorship bias.

Teachers were all relatively successful at school; if we hadn’t passed some exams we wouldn’t be teachers. We are academic survivors. We believe that our students can learn a lot from following the paths we trod, but are we, in fact, little better than Jeremy Clarkson spouting his annual advice to students not to worry about failing their exams because he failed his and now he has a villa somewhere nice. Or something. I suspect that most teachers will have engaged in a spot of rereading and highlighting back in the day. (Even if they didn’t inhale!) We look back at our study habits and our successful track record and conclude, it worked for me! But what if it didn’t? What if we’re successful despite our terrible study habits? What if we’re the lucky ones and the majority ended up doing far worse? And what about the path not taken? We can never know how much better we might have done if we’d committed to self-testing and distributed practice.

I’ve argued before that in many ways modern education is in the same parlous state as medieval medicine. It makes sense for a doctor who believes in the four humours theory of disease to prescribe bloodletting even though modern doctors now know that, in most cases, it’s a terrible idea. But such doctor’s were victims of survivorship bias: they systematically ignored patients who didn’t recover and focussed on those who did. There’s certainly no shortage of ‘good ideas’ in education but might we suffer with the same systematic bias?

We decide our ‘good ideas’ work because we fail to test them against better ideas. And even where they are tested, we ignore unwanted results. This is true even in large organisations: the OECD ignore the consistent finding from their own data that enquiry methods are not only much less effective than teacher-directed approaches but are actually negatively correlated with achievement. The EEF have even tried to make a virtue out of discounting contrary evidence in their justifications for reporting setting as having a negative effect. It’s easy to see such things as unscrupulous but it could just as well be an understandable inability to see what is not right there in front of you.

5 ways to avoided being blindsided by survivorship bias

  1. Think critically about your own experiences: just because you are successful does not mean that others will benefit from imitating you.
  2. Look for failures. Try to learn from those experiences that didn’t work, they’ll likely tell you more than those that did.
  3. Garbage in, garbage out. Bad or partial data collection leads to biased data. More here.
  4. Always think about opportunity cost.
  5. If you always do what you’ve always done you’ll always get what you’ve always got. Don’t try new stuff just for the hell of it, but it there’s compelling evidence that an approach might have merit, don’t discount it because it’s different to your current practice.