Alex Quigley has just responded to my post Two Stars and a Bloody Wish! with the revelation that it works for him and others:
Using a ‘Two Stars and a Wish’ model ironically meant that many teachers were writing more concise comments and spending less time on marking than before. Rather than proving a waste of time as David Didau suggests, it was saving time for many (teachers weren’t beholden to two wishes each time and there was seldom ‘lavish praise’).
Well, good. If using a particular marking structure does actually save teachers time then who am I to criticise? Alex goes on to say that, “though flawed (there is no perfect method for marking), the two stars method at least provides consistency for students and it can better delineate feedback than some of the chunky paragraph responses I have seen given by teachers.”
OK. Although I’d like to suggest that teachers be dissuaded from producing “chunky paragraph responses”. Then he delivers his knock out blow: “Like most approaches in the classroom, it can be used well and used badly.” And, of course, this is true. Or at least, it’s become a truism. I’ve said it myself. In fact, I once watched a presentation on the EEF Toolkit where the presenter ended with the Fun Boy Three/Bananarama classic It Ain’t What You Do It’s The Way That You Do It (that’s what gets results, dontcha know?)
In one sense I have no argument with this point – you really can do anything badly or well, but as I explained here, the distribution of data matters. Doing some things well might not be as good as doing other things badly. For instance, the EEF toolkit helpfully tells us that the average progress you might expect from implementing a Learning Styles intervention is +2 months. Some studies have reported greater gains, some less, but the average has been rounded at +2 months. Compare that with the behemoths like Feedback & Metacognition which both report average gains of +8 months. You can implement a sub-standard feedback intervention and still out perform the very best efforts to use Learning Styles. It’s vital to understand the concept of Opportunity Cost when considering these issues.
The point of all this is that although the data gleaned from these studies might be suspect, it’s a hell of lot better than relying on teachers’ intuition about what works. I can say anything works. Consider Alex’s reflection on why he likes the two stars method:
I gave two stars to the above essay because that helped me recognise and record individual strengths. I also explained what any abstract fancy language meant when I talked to the individuals and the class. It is always essential? Certainly not all the time, but then I only do it when I think it is appropriate or serves a use. Sometimes students get just one star if the boot fits.
For me, it is important that students to reflect upon their work that has been assessed, if it is a substantive piece of work – like an essay in my English class. I want them to take the time to see if there is a pattern of wishes over time. Thinking about one key improvement that can be revisited is fine for students to handle and so the ‘Two Stars and a Wish’ model proves helpful here. I want students to think hard about how they can meaningfully ‘fix’ their wish. [my emphasis]
I’m not saying he’s wrong – how could I know? – but I am saying that he’s relying on his intuitive judgement on what he thinks is most likely to be effective for his students. What so wrong with that? Well, it’s exactly the same process as that used by doctors up until the late 19th century. Until then bloodletting was routine. Doctors intuitively understood that using leeches to relieve patients of bad humours would cure a wide range of illnesses. In the absence of other treatments for conditions like hypertension, bloodletting might sometimes have had a beneficial effect in temporarily reducing blood pressure by reducing blood volume, but since hypertension is often asymptomatic and thus undiagnosable without modern methods, this effect was down to luck. The fact that in the overwhelming majority of cases bloodletting was harmful to patients went undetected as the treatment was never subjected to a proper test. If patients recovered, then it was down to bloodletting, if they died that was just one of those things.
The alarming truth is that none of the marking practices routinely undertaken by teachers in UK schools have ever been subjected to any kind of fair test. Two stars and a wish, what went well/even better if, DIRT, triple impact marking, green pens and all the other marking fads which have come and done obver the past few years have gained traction purely on the basis of teachers’ intuition. The only practice for which there is any kind of empirical support is the finding that giving grdaes seems to act against students reading and acting on written comments.
I sincerely hope that the forthcoming EEF report on marking will reveal the full extent of the paucity of research in this crucial area and that researchers will respond by setting up and funding large-scale RCTs as soon as is practical. Obviously, using two starts and a wish isn’t going to kill anyone, but it might not be helping much either. But more important than this, it might, just might prove to be a massive waste of time. Maybe it’s not. Maybe we’ll run trials and discover that so-called ‘deep marking’ is the longed for magic bullet, but I for one would like to be a little more confident of my facts before encouraging any teacher to invest time in routinely writing extensive comments on students’ work.
Any idea when eef report out? It says late 2015. Will be incredibly interesting (and presumably useful).
It should be out in the next few weeks
Whilst this post makes on the surface a compelling case for the use of evidence-informed practice, some of the other arguments in the post have significant limitations.
First, the post and its predecessor are great examples of cognitive bias. You describe working in a school which has a 2 stars and a wish stating : My working hypothesis was that this is a waste of time. With this as the starting post it’s difficult to see how any kind of balanced analysis could have been undertaken. Whereas, if a well formulated and answerable question had been posed – such as – For Y7 pupils does the use of 2*s and a wish improve pupil outcomes comparison to another specified marking protocol
Second, the question of ‘what works’ is largely redundant – the questions now being are ‘how and why does this work and/or not work, for whom, to what extent, in what respects, in what contexts and over what period?’. In other words, interventions such as marking policies cannot be seen in isolation, they need to be seen in the context of the school, the teachers and the pupils. In other word, it could be that 2*s and a wish – works well with Y6 pupils in a particular primary school – it may not work with Y13 pupils in a sixth form
Third, what matters is the interaction between the context, the interventions and the mechanism that it is designed to trigger – in other words, how does the mechanism influence the choices and decisions being made by the subject of the intervention – in this case both the teachers and the pupils. All these might not be directly observable it is possible to use conjecture to identify what they might be e.g. teachers by using 2*s and a wish may save time and use this time to better prepare lessons.
Fourth,systematic reviews and meta-analyses are at the apex of the evidence pyramid, not RCTS – indeed an individual RCT can be extremely misleading. So we need more than RCTs.
So whilst I accept your argument that there may not be sufficient research evidence to justify 2 stars and a wish – sometimes in evidence-informed a decision has to be made using the best available other evidence – be it school data, pupil and teachers views, and the experience of staff making the decisions.
1. Anyone who thinks they don’t labour under cognitive bias is either a fool or a liar. I’ve written extensively about my own biases and am happy to acknowledge them. What about you?
2. We’re talking about a one-size-fits-all approach to marking in secondary schools. I think that was made clear in the first post. The context which prompts my writing is the the on the ground perspective in schools where teachers are exerted to ever greater efforts.
3. My argument is that using 2*s takes more time and thus prevents teachers from doing things which may be more productive. read the post on opportunity cost.
4. Fair enough. But we ain’t gonna have any worthwhile meta-analyses until we have some good quality RCTs. First things first.
My issue is that as there is NO worthwhile evidence to inform decision making on marking, any sense of certainty is dubious at best. We know that feedback is very powerful, we also know that power can either be very good or very bad. It seems sensible to conclude in the interim that less has to be better.
Dear David
1 I have written about cognitive bias on a number of occasions http://evidencebasededucationalleadership.blogspot.com/2015/04/the-school-research-lead-and-cognitive.html. and the actions that can be taken to offset them.
2 In writing about one-size fits all approaches – you are potentially over-simplifying the argument. 2*s and and a wish may work for some schools, some staff and some pupils, and as such should not be dismissed out of hand – on the basis of no evidence.
3 Again – this goes back to the point I was making about evidence-informed practice and the development of PICO questions – normally you would compare two interventions when making judgments. In this case your comparison is between 2*s and everything else, and as such would be not be recognised as an appropriate approach in evidence-based medicine. http://evidencebasededucationalleadership.blogspot.com/2015/01/the-school-research-lead-and-asking.html
4 I think we can agree about
5 I think your point about there being NO worthwhile evidence to inform marking is probably an over-statement. What do you mean by worthwhile – on what basis have you excluded any existing research evidence on marking.
As for less being better, I can certainly agree with that
1. Thanks for linking to your blog post about cognitive bias and how to avoid it. Can I in return refer you to my book in which I do the same in greater detail: http://www.amazon.co.uk/exec/obidos/ASIN/1845909631/theleaspy-21
2. In the absence of evidence I’m never going to recommend something which is onerous in terms of teacher time. Nor should you.
3. I’m not comparing 2*s to everything else at all. That comparison appears to be taking place in your imagination.
5. Please let me have any details you have about any research on marking, “worthwhile” or otherwise. As far as I’m aware (and according to the EEF team conducting the review of the state of evidence into marking) there isn’t any. I’d be overjoyed to be proved wrong on this point.
Thanks for the reply David. I do want to be clear that I am not encouraging teachers to do anything – I was reflecting upon your critique and applying it to what I saw in my school the past few years. That is the only context I can really comment on relative to two stars, as I’m it takes on a thousand iterations in different schools. As stated in my post, the two stars approach was an old school policy and it had benefits that were not noted in your original post – such as it saving time for some – compared to what was happening before.
I very much hope that I can support the trials that you suggest and helping teachers get involved.
I’d also be interested in your DfE group on marking, what evidence is drawn upon and the insights/recommendations.
Now, where are those leeches…
Ok, so two stars and a wish isn’t perfect but what would you do instead if presented with a student’s mock essay?
Feedback as a key question (e.g. How could you improve……?) which students should respond to (thus promoting some sort of instant metacognition)?
This is what I’ve recommended: https://www.learningspy.co.uk/featured/what-does-feedback-look-like/
Such mechanistic routines mean that the teacher is no longer thinking about the student as a learner and adapting feedback to meet their needs. Why two stars? Why not 6? It also means, in teaching writing, that every time a child/student writes then it is never good enough – dispiriting and mean. Helping students develop the ability to understand what makes effective writing and be able to reread their own writing as a reader and edit for effect and accuracy cannot be reduced to two stars and a wish.
‘Outstanding Formative Assessment – culture and practice’ by Shirley Clarke covers all of this area and may clarify thinking.
I was talking generally – and not responding to your post at all – I didn’t post my reply on your wall – have not read your post – and my comment was intended as a general comment about 2 stars and a wish – sorry if you took offence but I was not talking about you or your ideas – I have no clue about what you do in your school and could not possibly comment about your approaches or ideas. It was just a general comment. I don’t think that I have ever commented on an educationalist’s blog before. As you find my post ‘rude’, then do not worry as I won’t post on such a blog again. No offence meant – have a good weekend.
What if it’s not the way you do it, but what you believe about the way you do it? Medical research example (as we like to reference the med industry) and part of a growing body of research: Patient comes to a doctor with an illness and nothing the doctor knows is helping. The doctor does however know of a research centre trailing new drugs that might help this patient, so he calls them and asks that his patient be included in the trial. The pills arrive and are administered to the patient. The illness and pain subsides, for 3 whole weeks. The doctor is encouraged and want now to test the pills he prescribed so calls the research centre and asks for placebo versions of the pills. The patient becomes ill on these placebos. Convinced at the power of this new drug to do good, he calls the research institute to congratulate them of their success at which point he is informed: “every pill we sent you was a placebo.”
[…] 23rd January – Is it what you do or the way that you do it? […]