Measuring progress is a big deal. I’ve written before about the many and various ways we get assessment wrong but, increasingly, I’m becoming convinced there are some ways we might get it right. As regular readers will know, I’m interested in the potential of comparative judgement (CJ) and have written about it here and here.

Greg Ashman mentions the process obliquely in his new book:

When we measure on an absolute scale using a set of criteria, we introduce the possibility of all students scoring 9 or 10 out of 10, particularly if we have trained them well. However, what is really of instructional value are the differences between essays that score 10. What makes the best essays better than the next best essays? We won’t even know there is a difference if they all score 10.

A way that we can do this is to force a comparison. We can lay the essays out on a large table and start to rank them using our expertise; our concept of quality. Once we have a rough ordering of the essays, we can start to ask: What makes these ones better than those ones?

… a useful check would be to intersperse essays of calibrated quality – perhaps from an external examination or scored by a different school or group of schools – into the ranking. Schools could collaborate to make this work. There are even computer programs now available that help teachers to rank essays by picking the better paper from a pair of papers.

For all our talk about progress in schools, most of our data is garbage. Anything based on individual judgement using rubrics will be unreliable (markers suffer from quite predictable unconscious bias) and invalid (rubrics cannot adequately describe expert performance.) If you want meaningful information about how your students are actually doing then you need to collect data differently.

Like anything else, there are different ways to approach anything and I was really interested to read this description of an English department’s attempt to use CJ to forensically investigate students’ progress. But, at what cost?

The process of reading twenty pairs of essays took about three hours for each judge, with the median time per judgement varying lying around the five minute mark.

This is totally at odds with my experience of CJ. Dr Chris Wheadon suggests that judging should be seen as fundamentally different to marking:

…the transition away from marking takes time. Judging is instinctive, so should be quick and easy. We’ve found that English teachers can make good judgements about GCSE essays in 7 seconds, with a median time of 30 seconds. The process is hugely slowed when you ask judges to take notes – which is extraneous to CJ. The problem with note taking is that teachers tend to slip back into a marking mindset rather than staying in the judging zone.

This is the approach that I’m interested in trialing – can teachers get reliable and valid data about pupils’ progress through a process which takes substantially less time than that required by marking and using rubrics. And this is exactly what we’ve decided to do at Swindon Academy.

Full disclosure: Chris Wheadon of No More Marking has waived a fee to work with us on this trial and in return I have agreed to blog about the experience.

So, here’s the set up. We’re going to start by judging our Year 5 students in English. I met with Year 5 teachers to find out what students had been reading and how assessments were usually structured. This is typical of the sort of thing we found in students’ books:

Screen Shot 2016-01-30 at 14.16.42

We agreed that the task on which they would be assessed would require them to read a short extract from Michael Morpurgo’s version of Beowulf and then write a description in response.

Obviously, we want to enable our students to write as well as possible and so decided the assessment task should not be too unfamiliar. Here is the question we agreed on:

Screen Shot 2016-01-30 at 14.03.31

Once students have completed the assessment, the next stage is for Chris to give teachers some training on how judgements should be made. Primary teachers and Key Stage 3 English teachers will then go through the process of using CJ to build up a, hopefully, accurate picture of exactly where the students are.

One problem we have anticipated is that as teachers we are so used to using rubrics to direct teaching that we may end up looking for those aspects of writing we have taught rather than looking at the writing in front of us. In order to test this out we’re paying the paltry sum of £100 to tap into Chris’s network of expert markers (all of whom in this case will be students completing PhDs in creative writing) to see if their perception of good writing is different from ours.

We’ll have to wait and see…