The other day I read Greg Ashman’s post Why Education is like smoking which talked about the way teachers often generalise from anecdotes in the same way that when smokers are confronted with statistics about the health risks of smoking they might say things like, “Well, my nan smoked 400 cigarettes a day! She may have had bright yellow fingers but she lived to the ripe old age of 130!” Or whatever.
Teachers do this sort of thing all the time. We say things like, “Well, the research may say x, but I find y works so much better for me!” Maybe it does, maybe it doesn’t but ignoring probability is a poor way to make decisions. As I explained here, relying on individual teachers’ intuitions is rarely a good indication of what will be most effective. Of course, no one can say with any certainty what will be the most effective approach to take on Monday period 2 with Philip in 9b. The physicist Wolfgang Pauli put it like this: “Like an ultimate fact without any cause, the individual outcome of a measurement is, however, in general not comprehended by laws. This must necessarily be the case.” But this doesn’t mean that we can’t say what is more or less likely to effective with a majority of children in a range of circumstances.
Then I read Gary Davis’s blog on why giving teachers access to education research could have unforeseen negative consequences. He argues that teachers, like most other human beings, are poorly equipped to understand probability. He gave this example of where we often go wrong:
The average heights of adult men and women in the UK are 5 ft 10 in. and 5 ft 4 in, respectively. A scientist picks a random sample of either men or women from the population. Which of these two options is more likely to be a sample of men?
Sample A consisting of a single person whose height is 5 ft 10 in. or
Sample B consisting of 6 people whose average height is 5 ft 8 in.
If you were tempted to pick Sample A then you’d be in the majority. You’d also be wrong. Probability is counterintuitive and its prediction don’t work at the level of individuals. Predictions and generalisations only really work when applied to large samples. The general rule of thumb is, the large the sample the more likely it is to accurately representative the mean. In the case of height, if we have enough men, although an individual could range anywhere from 21 ½ inches to 8′ 3”, their average height will be 5’10”.
What are the consequences of this sort of mistake in education? Let’s imagine we have a class of 30 4-5 year olds. We need to teach them to read and we can choose from 2 different methods. Your colleague in the next door classroom is a licensed Method 1 trainer and tells you that research shows Method 1 has an 80% success rate. She also shows you some negative press about Method 2 which indicates that it could result in up an average of 3 children in every school failing to learn to read. It’s an easy choice.
Now, let’s say every one in your class learns to read using Method 1 and you conclude it’s a winner. What’s your experience worth? If some snarky Method 2 proponent points out that their preferred method has the better probability of resulting in fluent readers, what will you say? Will you be convinced if they whip out their calculator and show you both probabilities as percentages? Or will your intuition and go with the evidence of your own experience and what ‘feels right’? Most of us like easy answers and we tend to ignore anything too complicated sounding and go with our guts. Even if, on average, we’re wrong more than we’re right.
Our best bet would be to go with ‘best bets’, but it’s really hard to accurately compare different studies. Tables of effect sizes can seem compelling but they conceal as much as they reveal. We could see what the best bets look like by comparing the bell curves produced by collating evidence on competing interventions. The trouble is, these are almost impossible to find unless you have the wherewithal; to create your own. This is something I have suggested the EEF ought to make available. As of yet, they haven’t.
There’s a slight problem with this analysis: comparing teaching methods–indeed the whole Hattie and Yates approach to what works–is bedeviled by the effects of prior teacher training and experience. A 1991 study of Federally-funded initiatives found little evidence that they were implemented consistently or faithfully. In a 2001 study of a Michigan reading initiative, Standerford found that “. . .changes attempted were rarely inconsistent with the teacher’s prior knowledge or beliefs about the best way to teach reading”.
Maybe the gut feeling is the correct one, since for “up to 4” gives an average of 2ish, and 20%, which is the “can’t read” estimate for 30 kids, is 7.6%. Simple proportions reveal almost all in this case.
Wooosh! (That was you missing the point :))
Not exactly! I was commenting on the dangers of putting fancy statistics in the hands of teachers, as most would leave alone most of the stuff. Also “Maybe..” was the wrong word.
I liked the comment that when teachers use some novel, fancy stuff they still hold on to a variation of the “old” stuff. Am I missing this point as well?
I should have been clearer – it’s definitely a bad idea to give teachers ‘fancy statistics’ – I deliberately chose that example so that it was hard to compare – but this doesn’t mean that professionals shouldn’t be given clear guidance on what research tells us about competing methodologies.
My gut feeling was to choose method 2 – it was clearly more successful if only 3 children per school did not learn to read, as opposed to 20% (unless all schools contain only 15 children, of course). Actually, it clearly wasn’t a gut feeling, it was based on the statistics offered (I used to do statistics for my school and also worked on them as a civil servant and did a lot of research on the per 1000 tables of the census for historical research, so perhaps not typical).
‘As I explained here, relying on individual teachers’ intuitions is rarely a good indication of what will be most effective.’
However, as the article itself almost implicitly suggests, this is frequently what we are left with as so little research is even close to absolute truths in education – maybe even hence the trad v prog debate continuing pointlessly for so long. The outcome is all too often characterised by polarised posturing.
Hattie admits to flaws in his maths – should have been knowledge instructed better? Dweck admits to misinterpretation of her work and therefore application – should have explained herself more clearly? EEF use criteria which may make some scientific sense (allegedly) but seem to have removed human interaction and equated monetary value as learning value – should have consulted teachers alot more carefully? VAK theory didn’t take account of a lack of understanding of the brain and neuroscience (ironically) – should have gone to spec savers!
Problem is we don’t know but in between not knowing (for sure) brilliant teachers make intuitive judgements and get brilliant results so, it may not be the best way, but it is a constant referencing and re-referencing from what we have taught ‘well’, to whom and in what context that influences us most.
Lots of food for thought as usual.
I had a similar reaction which was that I’ve never read any research that makes such a clear cut claim that Method A is better than Method B. Teaching is a personal experience. I have often attempted to teach using an approach or a resource that others have found wonderful, only to fall on my face with it. I get the probabilities thing, but that should be overlaid with the personal preference of the professional giving the lesson. If that person is not confident / happy with the thing that the research says works most of the time, then it’s probably not going to work this time.
Seriously? If you like using brain gym we should just accept that and leave you to get on with it?
If Method A is Reading Recovery and Method B is SSP and you say, “Oh, I can’t get on with phonics,” then you have no business teaching. This is a profession.
Bit rude. If you really want to promote a debate, try being less aggressive in your responses. I have no idea what brain gym is. I teach secondary maths (mainly) and in my world educational research rarely gives me answers to the questions I need, e.g. what is the “best” order to structure a scheme of work on fractions, or should I start simultaneous equations using a graph or algebraically. You quote some fields that presumably have clear-cut evidence. The rest of the time teachers need to work together benefiting from their collective years of experience in the classroom. That is what makes us professional.
Could I have a go. Your assumption that research is unclear in all areas is the normal starting position however if you manage to wade through the stuff there are a few well supported comparisons with clear evidence in one direction . (Synthetic phonics vs whole word,. Explicit vs discovery etc) you are right that many interesting questions such as detailed scheme structure aren’t really answerable at the moment.
Out of a department of 20plus I’m the only one who regularly reads anything yet when you discuss an idea nearly everyone already has an opinion and reasons to support despite the likelihood that it’s the first time they have thought about it. It’s not just teachers but rather a predictable human fallacy.
In summary we have an issue around both accessibility and believability in research whuch leads to a crude rejection of the whole idea even when we have clear results.
Wrote this on my phone so excuse the lack of polish.
Fair comment. I will continue to wade! I agree that a significant problem can be teachers deciding that they have found the “best” way and therefore sticking with it. But totally understandable given how hard the job is and how little time there is read research (in the UK, at least)
The ineffiency of reading it is pretty irritating. EEF (think that’s right) try to summarise research but constantly falls into the trap of trying to fix it. I’m always triped up by conclusions which often mix valid inferences with ideas (often) reasonable on what to do about it. I only notice this when I reread the thing later on. That’s without even aknowledging the mathematical analysis, in which I lack both skill and time(and no-one to whom I can refer).
An example would be the DISS study by P Blatchford which concludes that students supported in class by TAs have poorer support as it is usually in lieu of the tutor. But the conclusion mixes this up with lots of ideas on what to do about it. (Many reasonable but some impractical). When attempts are made to summarise research it is often presented in this conclusion style. (I should aknowledge Blatchford follows up his own ideas with more research).
Not sure what can be done but it would be interesting to see what would happen if we kept summarise succinct and on point and then pushed the analysis onto a separate document.
Rude schmood.
I made the exact point in my post that you appear to be making in this comment, namely that research cannot tell you what to do in a given lesson with a particular group of students. That’s way too specific. There is however good evidence on overlearning fluency in maths facts and the efficacy of explicit instruction – this will be useful whether you want to teach fractions, negative numbers or calculus.
My view is that if you ignore this stuff and instead rely on “collective years of experience in the classroom” you may well be doing your students a disservice.
We are saying variations on the same points David and it is slightly frustrating that wasn’t picked up tight at the beginning.
Mhorley is only going over well trodden ground and while repetitive it at least allows us to create the best and most persuasive version of our arguments.
Read first sentence as: we are all saying variations. I know your comment was directed at mhorley and I meant that I understand why you are frustrated at him repeating your own points.
That last comment was a bit poorly constructed overall, apologies. In mitigation I did knock it off while waiting for an x-ray.
Yeah, I’m never at my best immediately after an xray 🙂