On March 3, 1969 the United States Navy established an elite school for the top one percent of its pilots. Its purpose was to teach the lost art of aerial combat and to insure that the handful of men who graduated were the best fighter pilots in the world.
They succeeded.
Today, the Navy calls it Fighter Weapons School. The flyers call it: TOP GUN.
As I’m sure you know, these are opening credits of the 1986 movie starring Tom Cruise and Val Kilmer as pilots graduating the elite Navy fighter school. What you may not know is the background to the real school. The impetus to train pilots in aerial combat came from the appalling attrition rates experience in the Vietnam War where, at it’s worst US pilots were shooting down enemy fighters at the cost of 1 for 1. But winning one dog fight meant you were increasingly likely to win the next. By the time a pilot had survived about 20 aerial encounters they had an almost 100% probability of winning subsequent ones. The question was, how do you ensure a pilot survives for long enough to become expert enough to become likely to win dogfights?
The US Navy’s response was to recruit their best pilots as instructors and their next best pilots as students. The instructors set up mock combats designed to push their students to the limits of their ability and then systematically helped them to identify and eliminate errors and refine their performance. As pilots graduated, they returned to the front to train other pilots and the instructors remained to become ever more expert in training others to become fighter aces. The results were dramatic. By 1972, Navy pilots were shooting down an average of 12.5 enemy planes for every plane they lost. A remarkable improvement.
So, what, if anything, can we learn from this in education? In his new book, Peak, Anders Ericsson suggests the Top Gun model could provide a recipe for implementing the kind of deliberate practice regimen which leads to expertise. The trick, as Ericsson sees it, is to find out what the very best performers do and emulate them. In fields where there are clear, objective measures of success this is relatively straightforward. Identifying the best teachers is trickier.
Of course there’s Doug Lemov’s approach: hunt down teachers whose students produce the best test scores as compared to other teachers in similar schools, watch what they do, see what commonalities there are and design a taxonomy which enables other teachers to isolate, practise and perfect their skills. This is probably as good as we’re likely to get in teaching, but as Dylan Wiliam explains in his new book, Leadership for Teacher Learning, whilst data allows us to say with certainty that some teachers are better than others, for all sorts of complex reasons, it doesn’t allow us to reliably identify who those teachers are. The best measures of teachers’ performance we have are like a scales which gives us an individual’s weight to within +/- 50 pounds. This is enough to tell us that men are, on average, heavier than women, but tells us nothing useful about the weight of an individual. Does this matter? Well, as Wiliam explains, it depends what you do with the information. The very best we can probably manage is to say with a probability of between 0.6 – 0.8 whether a teacher is performing well. This is not good enough for high stakes decisions about pay or employment but it should be sufficient to design effective training.
So, what are the ingredients of a Top Gun model and how might it look in education? Ericsson says the US Navy wasted little time on trying to quantify the expertise of the best pilots. Instead they “just set up a program that mimicked the situations pilots would face in real dogfights and that allowed the pilots to practice their skills over and over again with plenty of feedback and without the usual costs of failure.”
Practically, this is tricky to do in education: we’d have to hire lots of child actors. But, as the costs of failure are so much less for teachers than for fighter pilots maybe we can risk using real live students in actual classrooms for teachers to practice on. After all, this is effectively how most current teacher training models already work. But, once teachers qualify, observation becomes increasingly rare. We assume that meeting minimum qualification plus experience will result in expertise. This is probably wrong.
There’s a lot of evidence that people don’t just get better doing their jobs. In order to improve we need to engage in purposeful, conscious practice. Ericsson trots out many examples of professions where, because of the quality of the practice, people actually get worse over time. Radiologists are a case in point. In most cases, radiologists are sent X-rays to examine without ever finding out the consequences of their diagnoses. They rarely get useful feedback on the judgements. In such cases experiences may result in increasing confidence, but the judgement of experienced radiologists is, if anything, slightly worse than that of colleagues with about 3 years experience.
This echoes the findings of Rivkin, Hanushek & Kain, and of Kraft & Papay in regard to teachers: teachers improve dramatically in the first 3 years of practice, then plateau before start to dip after about 10 years experience. Teaching is an interesting case in that teachers get a mix of excellent and very poor feedback. We get great feedback on aspects of teaching like behaviour management: either children behave or they don’t. But other aspects, like how well students retain information, are badly neglected. Teachers tend to only get feedback on how well children perform within an individual lesson and not on well content from previous lessons has been applied. This can lead teachers to believe that what results in improved short-term performance will also result in better learning, but this belief is contradicted by the evidence. Thankfully, this is a relatively easy problem solve: all we have to do is rethink what we mean by learning.
None of this is really about Top Gun teachers. We don’t actually need to set up an elite teacher training school in which participants are required to play a lot of topless beach volley ball. But if we want to commit to maintaining and improving teachers’ performance we need to consider the following:
- Frequent, low-stakes lesson observations. Ideally, teachers would get regular intensive sessions maybe a week at a time – where observation was followed by feedback and then further practice.
- Much better feedback on learning. This would require teachers to teach lessons which allowed students to demonstrate the retention and application of content covered weeks, months, maybe even years before.
- Guided, purposeful practice. Teachers will make more progress with a mentor or coach to help them focus and pay attention to the skills they are trying to develop.
- A codified body of knowledge (ideally phase and subject specific) which would give teachers a means to objectively assess where they need to make progress, isolate skills and practise – this could result in much better mental representations of what effective teaching looks and feels like. Lemov’s Teach Like a Champion taxonomy might provide a useful starting point in designing such a body.
- All this needs to voluntary. You can’t force someone to practice purposefully.
You might not like the sound of any of this, but what’s the alternative? If we accept that the status quo is a recipe for mediocrity and decline, what would you do instead? If you think everything’s fine, do you have any evidence beyond the anecdotal?
Now I’m not gonna sit here and blow sunshine up your ass, Lieutenant. A good pilot is compelled to evaluate what’s happened, so he can apply what he’s learned. Up there, we gotta push it. That’s our job. – Viper, Top Gun
You know that bit you’ve got about Dylan William? As you’ve read the book, you must be right about what you’re saying it says. I haven’t read the book. I notice that in the blurb you link to, it says:
“Leading education authority Dylan Wiliam explains how formative assessment, when applied properly, helps to create a structured and rigorous learning environment that increases student achievement. He also presents compelling research to give readers a clear picture of:
· Changes in classroom practice that are likely to increase learning
· Differentiated instruction (DI) and response to intervention (RTI)
· Group leadership’s role in ensuring productive collaboration
· Strategies to integrate formative assessment into teacher evaluation
Wiliam also discusses why efforts to change classroom practice have been relatively unsuccessful—and explores specific classroom tactics that do tend to raise student achievement.”
I have read William’s inaugural lecture for when he became a prof at the Institute of Education and his core interest then (and according to this blurb) seems to be about ‘assessment for learning’ and systems of feedback from learners to teachers, teachers talking to teachers, and these processes being active in how teachers teach.
I’m sure Dylan will appreciate you wanting to republish his blurb on my blog. Presumably though he’d be as confused as I am about why you wanted to.
Confused by ‘constructive feedback’, then. It happens.
Yeah, really confused. What part of copying out a blurb is either constructive or feedback?
Hi David,
Fascinating stuff as ever. I wanted to add a small point of information which is that Kraft and Papay’s study showed that teachers do, on average, improve after their 3rd year, albeit more slowly than initially, and that the rate appears to correlate strongly with the quality of behavioural, instructional, cultural and professional development support that they receive.
My second point is one about the nature of teacher learning. I would suggest that there may be some parallel between more expert teachers and their observations of colleagues/mentors with the science of expertise and worked examples. We know (e.g. Kalugya, 2001) that worked examples can suppress learning in experts and that these learners may learn more effectively through problem-based approaches. Granted, for more strategic tasks (Nievelstein et al 2013) showed that both groups can benefit from examples.
This could map to the Developing Great Teaching report which suggests that the greatest gains for pupil outcomes are found for the overall teacher population, in the majority of subjects, when teachers engage in problem-solving type activities alongside an expert facilitator who can strategically guide their activity with judicious use of modelling/example practice. The report also suggested that where teachers focused only on improving a specific practice, in isolation from a classroom environment, or perhaps focused only on performance rather than impact on learning, then this did not seem to equate with improved outcomes for learners. You’d have to refer to Philippa Cordingley for more specific detail on this, however.
Best wishes,
David
Thanks David – point taken about K&P and the institutional climate of PD: in my striving for brevity I necessarily picked and chose. You are, of course, right.
I think the point I’m making about teachers’ development is that there is a lot less genuine expertise within the system than we want to believe. Those identified as ASTs, SLEs etc tend, I think, to be good at engagement and increasing short-term performance – it may be the case that they are actually degrading learning. We probably need a lot more humility in the system.
As far as problem-solving vs worked examples go, again, I take your point. My reading of Ericsson suggests that it’s the quality of mental representations that make the difference and this might shed some light on why experts and novices do better with different kinds of inputs.
After 40 years teaching I am not the least concerned about anybody coming to watch my classes. I would welcome helpful comments. There is a but. The but is that I must have seen the person teaching. If they were what I judged as good all would be well. I am not so keen on some one who is not a good teacher watching me then giving their opinion.
I would also like to explain to them before the lesson what it is about. I have seen the situation where the person observing did not understand what the lesson was about and then ripping it to shreds as it is not what they wanted.
Rant over. No if you tell me how to improve you must show me how telling is not good enough. Actions rather than words.
I think you’re right Philip. The instructors in the Top Gun programme are the very best pilots. The students are the next best: good enough to be difficult about accepting feedback from anyone except someone demonstrably better. I find I get the best results when working with teachers if I show them what I mean by teaching a lesson to their class.
Hi David. Have you seen the paper released recently by the Learning Policy Institute on the relationship between teaching experience and effectiveness? It reaches a different conclusion from K&P.
https://learningpolicyinstitute.org/our-work/publications-resources/does-teaching-experience-increase-teacher-effectiveness-review-research
I hadn’t, thanks
[…] is Mainstream, but it is complicated’ while David Didau gets maverick with his ace ‘Top Gun for Teachers’ […]
Thanks for including the link Thom. For me there are two important aspects to the ‘Does Teaching Experience Increase Teacher Effectiveness?’ report.
The first is that it puts under the spotlight the shortcomings of the methodologies used for some of the research on the effects of teaching experience and warns readers of the upwardly and downwardly biased estimates of the effects of teaching experience on student outcomes resulting from these methodologies.
The second is that when methodologies are employed that reduce these biases different findings are produced which thankfully show that teaching experience is positively associated with student achievement gains throughout a teacher’s career and that while the gains from experience are highest in teachers’ initial years, the gains continue for teachers in the second and often third decades of their careers.
It *might* show that, but I’m not wholly convinced. I explore further here: https://www.learningspy.co.uk/training/developing-intuition-can-trust-gut/
Thanks for the link to your more recent post. May I ask why you didn’t mention the ‘Does Teaching Experience Increase Teacher Effectiveness?’ report in it?
I did. I refer to it as Kini & Podolsky (2016)
Apologies for the oversight on my behalf David.
I am not sure they are wrong about everything else. I haven’t checked the methodologies of the extensively documented examples you quote to support your assertion. But I would add that they are not suggesting that experience automatically confers expertise. They point out that ‘Of course, there is variation in teacher effectiveness at every stage of the teaching career, so not every inexperienced teacher is less effective, and not every experienced teacher is more effective.’
Yes, I know. But they say that it’s obvious experience should lead to greater expertise “given the evidence that professionals in a wide range of contexts improve their performance with experience. For example, a surgeon’s improved performance is associated with increased experience gained at a given hospital. An increase in a software developer’s experience working on the same system is associated with increased productivity. What is common sense in the business world—that employees improve in their productivity, innovation, and ability to satisfy their clients as they gain experience in a specific task, organization, and industry—is not the commonly accepted wisdom in public education.”
This is demonstrably false. The evidence is that in many fields the feedback professionals receive is absent or biased and this leads to greater confidence but not greater expertise. I’m not saying their wrong, just that their reasoning is flawed and based on an erroneous assumption. It my be that teaching is a kinder domain than I give it credit, but I don’t think so.
I appreciate that some studies have reported that professionals get better with experience, others report experience has minimal effect and yet others report a reduction in effectiveness over time. We might say this illustrates the complexity of the subject under investigation.
Nevertheless the credibility of all research rests on effective design and effective analysis of results. It is crucial therefore that we recognise that some of the research on how experience relates to practice in all and any areas of work will be less credible than other research on the same topic. Consequently quoting numerous studies has little value in supporting a reasoned argument if those studies are less credible than a single study. This point is alluded to in the ‘Does Teaching Experience Increase Teacher Effectiveness?’ report which is why the researches take steps to reduce the influence of less effective research design and analysis when carrying out the review.
Moreover the researchers draw our attention to advances in research methods and data systems which now enable researchers to more accurately investigate the relationship between a teacher’s performance and experience. While a detailed study of the research you mention in respect of performance over time in settings other than teaching would be needed to ascertain the credibility of the designs and the analyses, I think it is not unreasonable to expect the credibility of these will vary in much the same way as it does for studies of teaching performance over time. The issues of advances in research methodologies and data systems are also likely to apply.
I believe it to be most unlikely therefore that the findings of the ‘Does Teaching Experience Increase Teacher Effectiveness?’ report are inaccurate or based on flawed reasoning. Most importantly perhaps we should acknowledge the synergies between the recommendations in the report and some of the recommendations advocated by other researchers you mention.
Unfortunately, the ‘Does Teaching Experience Increase Teacher Effectiveness?’ doesn’t actually go into any detail about HOW they use the fixed effects models so it’s very difficult for me – a non mathematician – to assess the credibility of their claims. I do know that there’s an equally dogmatic camp of credible seeming expert statisticians in the opposing camp were are equally certain they’re right. Both camps make some fairly firm suggestions which are not really supported by the research.
As for Fixed Teacher Effects, I recommend Jesse Rothstein’s Do Value Added Models Add Value? which you should find interesting reading. Essentially, JR says that VAMs are circular arguments which fail falsification tests – “5th grade teachers have nearly as large effects on 4th grade gains as on 5th grade grades”.
Briefly, there is a lot of VAM research which is all very similar, making similar assumptions and drawing similar conclusions. This echo chamber can make it look like there is a consensus that VAM works. You probably know that there is an entirely separate Anti-VAM echo chamber – see Audrey Amrein-Beardsley’s Vamboozled – which has a different view entirely.
I think we have to consider the costs of being wrong. If it’s true that teachers will, on the whole, get better with experience, then additional efforts to help teachers develop expertise are unlikely to cause a problem. At worst we might have wasted some time & money. But if it’s true that teachers don’t get better at instruction then the cost of doing nothing is potentially tremendous. I’d rather err on the side of caution. To do otherwise is, in my view, unscrupulous optimism.
Yes, the literature on VAMs, along with that on other measures of teaching effectiveness such as test scores, classroom observations, student evaluations and teacher qualifications, to name a few, is full of contradictory findings. Yet we should neither be surprised nor alarmed by these results not least because of the inherent limitations of each research study but perhaps more importantly because, as Marilyn Cochran-Smith notably remarked, ‘Teaching is unforgivingly complex. It is not simply good or bad, right or wrong, working or failing.’ Put simply there are no absolute measures of teaching effectiveness which means any attempt to articulate the effectiveness of teachers over time must take this into account.
For me the ‘Does Teaching Experience Increase Teacher Effectiveness?’ report shows that teachers working in the USA tend to get better over time and the researchers acknowledge there will be exceptions.
And for me it fails to show that by ignoring the paradoxes associated with fixed effects models. So we’re left with a “who’s got the best maths” argument. Meanwhile, I maintain that the costs of assuming that teachers will, on average improve with experience are outweighed by the benefits of assuming they don’t.
Well there we have it. Same research findings leading to two polemic interpretations.
I will finally add that I think there are significant merits in the findings of the ‘Does Teaching Experience Increase Teacher Effectiveness?’ report. I believe ‘it is important to recognize that, on average, the most effective 20-year teachers are significantly more effective than the most effective first-year teachers’ with the caveat that ‘not every inexperienced teacher is, on average, less effective, and not every experienced teacher is more effective.’
If we think otherwise it may well lead us to question the trustworthiness of advice provided by experienced teachers in the form of conversations, research articles, social networking blogs and published books when compared with their less experienced colleagues.
And I think experience alone should not endow anyone with greater respect or authority.
At least we agree on one thing!
Don’t you think that teaching ability is also greatly improved if teachers are challenged. Doing things such as moving schools, teaching new subjects, or teaching to different levels of ability to what they are used to will surely improve teaching ability.