We don’t know what we don’t know: the uses of humility

In my last post I challenged the widely-held belief that teachers’ judgements are generally sound and suggested instead that we are routinely beset by very predictable but unconscious bias. Two criticisms emerged that I want to address. Firstly, some commenters noted that it’s impossible to prevent teachers making judgments and that, in essence, a large part of the act of teaching is making judgements about pupils’ learning. As such, any attempt to remove subjective bias from teaching is fundamentally flawed. The second, and related question was to ask if there’s anything we can do to improve our judgement or am I just offering a counsel of despair? Both points are valid and deserve some consideration.

Challenging bias is hard work. I have tried to make a habit of challenging my assumptions and recognising my prejudices, but I routinely fail. Of course this could just mean that I’m a particularly hidebound individual with an especially low-level of introspective skill, but, unsurprisingly, I don’t believe that. Like most people, I believe I’m better than average at spotting flaws in my thinking and errors in my judgement. Like most people, I’m guilty of valuing my intuition far more than objective, verifiable data. Like most people, I routinely miss the fact that it’s practically impossible to spot self-deception on your own.

The solution is to put in place systems which force to receive improved feedback on our decision-making. In this post I discussed five techniques we can use to overcome overconfidence; in this post I suggested ways we can take a more objective view of data, and in this post I recommended a checklist for weighing our judgements. These measures will help, but they take time. As David James pointed out, often the speed of decision-making is crucial:

…leaders often have to take lots of decisions quickly, and many will appear relatively unimportant to them, whereas they are hugely important to those who they affect. This latter group would have liked consultation, checklist, pre-post-mortems, but a huge factor (in schools anyway) is the lack of time. Knowing what to act on quickly, and what to take some time over, is a skill learned by experience, but leaders are also very aware of being seen to be prevaricating, rather than being decisive.

And he’s right: being seen as a ditherer is fatal. I’m going to refer to this as the John Wayne effect. Gary Klein says, “Society’s epitome of credibility is John Wayne, who sizes up a situation and says, “Here’s what I’m going to do”— and you follow him.” Daniel Kahneman adds, “There’s a cost to not being John Wayne, since there really is a strong expectation that leaders will be decisive and act quickly. We deeply want to be led by people who know what they’re doing and who don’t have to think about it too much.”

But here’s the rub: intuitive judgements only appear to improve in certain, very specific situations. As I set out in this post, teaching appears to be one of those areas – ‘wicked domains’ – in which many of our intuitions don’t seem to improve over time. So, is there anything we can do to improve our ability to make instant, intuitive judgements? Kahneman and Klein set out two conditions which are essential for the development of such intuitive thinking. They say, “reliably skilled intuitions are likely to develop when the individual operates in a high-validity environment and has an opportunity to learn the rules of that environment.”

Can we make teaching high-validity?

High-validity environments are predictable. This allows expertise to develop as people learn to recognise patterns. The reason chess grandmaster can make incredibly rapid, highly intuitive decisions is because chess is fairly predictable. Teaching is not. Lee Shulman says teaching “is perhaps the most complex, most challenging, and most demanding, subtle, nuanced, and frightening activity that our species has ever invented… The only time a physician could possibly encounter a situation of comparable complexity would be in the emergency room of a hospital during or after a natural disaster.” (p 504) There’s probably little we can co to reduce this complexity and chaos, but still most teachers can become highly skilled at recognising patterns within classrooms which allows them to predict how students will behave and can make their judgement sometimes seem almost magical.

Can we give teachers and leaders more opportunity to learn the ‘rules’ of their environment?
As this study by Hamre et al shows, we get better at managing behaviour and forming relationships, but don’t improve nearly as much at ‘instructional support’. This is what Kahneman & Klein call  ‘fractionated expertise’. In common with most professions, teachers regularly have to deal with situations and tasks that they have not had an opportunity to master.

Instructional support includes things like promoting higher order thinking, giving formative feedback, and using language to promote thinking. Why don’t we get better at these things? Possibly because we get very little feedback on whether the instructional support we offer is any good. Unlike behaviour management where we get instant and highly effective feedback on our mistakes – I cringe at some of the memories – we rarely, if ever, discover the effects of our teaching on learning. What this suggests is that the ‘rules’ of behaviour management are ‘high-validity’ whereas the ‘rules’ of assessing students, giving feedback and teaching difficult concepts is ‘low-validity’. This is precisely why teacher assessments are so ridden with bias: we don’t know what we don’t know.
The ‘fractionation of expertise’ leads to the illusion of validity. But as we become more familiar with our environment we become more confident. As we become more confident in our abilities, the harder it can be for us to learn. We tend to believe that because we are expert in some areas of our jobs we must be expert in all areas. This leads inexorably to overconfidence in dealing with problems in which we have little or no skill.

Pride cometh before a fall

The biggest problem is that people have no way to know where their intuitions came from. They pop, fully formed, into our heads and we are hard-wired to believe what we think is de facto correct. Here’s Kahneman & Klein again:

Trying to convince someone their intuition might not be right is next to impossible. The only way we can have a chance to learn the rules of a low-validity domain is to accept how routinely and regularly we are wrong. This is why I like illusions so much: they are a gateway drug to humility. As the philosopher Simone Weil put it, “the virtue of humility is nothing more nor less than the power of attention.” If we are to learn from our mistakes we must pay attention to them. True experts know when and what they don’t know.
NB I am not naturally humble and find it extremely difficult to overcome my sense of certainty. This is not me being boastful about how ace I am, but if even I can learn to admit I’m wrong, maybe you can too?