Student voice: windmills of the mind

Does it matter if students like their teachers? Is it worth knowing if students don’t maths or hate PE? Should students be asked to evaluate the quality of their lessons? It sometimes seems that the clamour of ‘what students want’ drowns out even the presumed demands of ‘what Ofsted want’. Students’ opinions might be interesting but should they be used to judge the effectiveness of teachers? Certainly some school leaders appear to think so.

An anonymous blog on the Labour Teachers site* reveals the extent of the rise of this sort of managerialism:

This year has seen our graded lesson observations replaced with something else entirely. Pupil Learning Interviews. Fresh from the mouth of an ex-Inspector-now-consultant, this new form of monitoring is where it’s at. Here’s how it works:

1. A small number of children are chosen in advance: underperforming boys, PP, school refusers – you get the idea. Obviously being a nice and hard working girl is not good enough for your views to be polled

2. On the day of the interview, two observers arrive at the door and seat themselves within the vicinity of these lucky children

3. The children are watched – not the teacher. In fact, what happens in the actual lesson itself has no relevance at all

4. The children’s books are studied, so they cannot complete any work in the lesson

5. The children are then interviewed either in the lesson, whilst everything is going on or they are asked to step out into the corridor.

After the lesson children are asked questions like, What have you been learning this year? I see here you have been learning about rivers, what can you tell me about that? What do you find hard in this subject? What do you find easy?How does the teacher help you? What sorts of things do you learn about?

In a Schools Week article about pupil interviews, opinion seems to be divided. Christine Blower general secretary of the NUT, is against “snapshots” being used to judge teaching: “Attempts to do so risk turning our schools into exam factories, and they ignore deeper learning and valuable learning experiences.”

But Brian Lightman from the ACSL thinks it’s fine because schools have been doing it for years(!) He is reported as saying, “They are an informative and useful form of school self-evaluation designed to understand the learning experience from the pupils’ viewpoint… The feedback and insight gathered is useful in identifying how pupils can be helped to do as well as possible.” And maybe in the right

And maybe in the right hands such feedback really could be useful in improving how students make progress, but I doubt it will in this case. As Nick Rose says in this blog, “These kinds of data can serve a useful purpose so long as it is not being used to make superficial judgements about teacher performance.” In another post, Rose talks about how students surveys might be used well. He concludes by saying, “As an accountability measure, I think student surveys would be stressful and unfair – but as a coaching tool to get inside your own teaching, I think it has merit.”

There’s no doubt that students routinely perceive differences in their teachers and that they see some teachers as better than others. One of the most widely researched student survey systems, Tripod, reveals that students of teachers with the top 25% of Tripod score, make 4.6 months more progress in mathematics than those in the classrooms taught by teachers with Tripod scores in the lowest quartile. On average, the most effective 25% of teachers are around 130% more effective than the least effective 25%. Clearly students’ perceptions are worth something.

This research from the MET Project shows the sorts of questions students can be usefully asked:

You might have noticed that these items are rather different to the questioned being asked in the school in question. The authors of the MET Project report, Asking Students about Teaching say,

For results to be useful, they must be correct. It’s of no benefit if survey responses reflect misunderstanding of what’s being asked, or simply what students think others want them to say. Worse yet are results attributed to the wrong teacher. Honest feedback requires carefully worded items and assurances of confidentiality. Standardized procedures for how surveys are distributed, proctored, and collected are a must. Accurate attribution requires verification… (p. 11)

I wonder to what extent these considerations were taken into account before deciding to judge teachers on pupil interviews? The report continues:

Consistency builds confidence. Teachers want to know that any assessment of their practice reflects what they typically do in their classrooms, and not some quirk derived from who does the assessing, when the assessment is done, or which students are in the class at the time. They want results to be reliable. (p. 14)

The suggestions set out to ensure consistency and reliability appear to be in complete opposition to the way interviews are conducted in the school above. But then, we only have one teacher’s view – maybe they just need to be a little more trusting? Maybe, but as the report clearly states,

No policy can succeed without the trust and understanding of those who implement it. Although often ill-defined, collaboration is key to effective implementation of new practices that have consequences for students and teachers. Every system referenced in this brief has made stakeholder engagement central to its rollout of student surveys—involving teachers in the review of survey items, administration procedures, and plans for using results in feedback and evaluation. (p. 21)

I suspect considerations of trust and understanding may have been dismissed as unnecessary.

The other big problem with relying on student surveys is the Dr Fox effect. The original experiment was conducted by Donald Naftulin, John Ware, and Frank Donnelly at the University Of Southern California School Of Medicine in 1970. Two speakers gave lectures to 3 different audiences of MDs and PhDs. The first audience listened to a genuine professor and the other audiences both listened to an actor posing as Dr Myron L. Fox. When ‘Dr Fox’ was instructed to speak in a dull monotone, the audience rightly dismissed him as a know-nothing ignoramus, but when he delivered the same meaningless material with enthusiasm and humour, the students rated Dr. Fox as highly as the genuine professor. This lack of correlation between content-coverage and ratings under conditions of high expressiveness is well-known; even very knowledgeable, expert students are fooled by a good speaker, how much more likely are school students to be swayed by ignorant yet charismatic teaching?

As Naftulin, Ware and Donnelly concluded by saying, these results ought to suggest to teachers that “the extent to which his students are satisfied with his teaching, and even the degree to which they feel they have learned, reflects little more than their illusions of having learned.”

All this should convince us that while it may be very useful to survey students on their perceptions of lessons, this should never be used as a crude, high-stakes accountability measure to judge teacher effectiveness.

*The teacher who wrote the Labour Teachers blog is well-known to me and someone I respect and have every reason to believe.