It may sound like common sense to say policy should be based on what we know actually works, but in reality it’s never that simple. There are all sorts of challenges, from doing the initial research to convincing often sceptical stakeholders to rolling out reforms and scaling them up.
Lee Elliot Major, CEO of England’s Sutton Trust, understands these issues all too well. The charity he runs was founded in 1997 to help improve social mobility through education. Calling itself both a think tank and a “do tank”, the trust has funded over 200 programs, commissioned more than 140 research studies and advocates for a more evidence-based approach to education policy.
Sutton is particularly focused on helping smart students from poor backgrounds reach the same kind of opportunities better off students have access to. After running for 18 years, there are now Sutton Trust beneficiaries running for parliament.
Current research areas include earlier school start times and paying financial incentives — early evidence suggests paying students money at the beginning of semester and threatening to take it away if they don’t turn up to class may be useful. Researchers are also looking at whether the Singaporean style of teaching maths can be transplanted to the United Kingdom.
Major is visiting Australia as a guest of the Mitchell Institute for Health and Education Policy and the Victorian Department of Education, meeting with officials in Canberra and Melbourne prior to the launch in a few weeks of a Victorian version of the English Teaching and Learning Toolkit, an evidence-based guide for teachers to what works in education.
“When I talk about the nuances of evidence, I can see the politicians’ eyes glaze over,” he told The Mandarin in Melbourne this week.
“I can understand it because what they need is a very simple message. It’s about public or private schools, it’s about class sizes. It’s about being tough on school standards.”
Yet the evidence suggests these issues favoured in public debates about education are not among the most important questions when it comes to improving student learning outcomes. The biggest influence is teaching quality — the teacher’s knowledge of the content, instructional skills, ability to elicit feedback from children and so on. There tends to be greater variance within individual schools than between them, whether public, Catholic or private.
Unfortunately, teaching quality is more difficult to quantify, meaning political discourse tends to fixate on easily measurable things like class sizes instead. But, says Major, the evidence shows reducing class sizes is not a cost-effective approach — there needs to be a significant reduction in student numbers before there’s a noticeable difference in outcomes.
“I do think there’s a fundamental tension at the heart of modern public policy that professional teaching for example needs to have a sophisticated evidence-based discussion, but it’s got to happen in a world where you have a political system … how the two interact is a really interesting question for me,” he said.
Independent advice for government
It’s not a reason for pessimism. The Sutton Trust has been a key participant in the Education Endowment Foundation, an independent but government-funded charity and one of the seven members of the British government’s “What Works Network” of organisations that provide information to help government make better decisions in social policy. The EEF produced the groundbreaking Teaching and Learning Toolkit.
He believes the value of Sutton and the EEF lies in their ability to be independent yet still work closely with government on policy development. The EEF has been funded over a 15-year timespan, enabling long-term planning and ensuring independence from the parliamentary cycle.
“The independence gives you a principle whereby you can publish the evidence without fear or favour. We’re not at that stage yet, but there will be some things that the EEF trials that the UK government might say ‘actually, that’s slightly different to what our current policy is’. So I think we do need that independence,” he said.
“I think we’re very careful in presenting the evidence and then letting other people think about what the implications are for practice. I suppose we do come up with our interpretation of that evidence overall. We are careful. Partly that’s because we believe teachers are professionals who should try that in their own schools.”
[pullquote] “What we’re wrestling with … is how you take that evidence and actually have intelligently informed practice …” [/pullquote]
The toolkit is a big part of the empowerment of teachers to make their own decisions at a time when schools are being given greater autonomy from education departments. By providing them with information about what the evidence suggests does and does not work, teachers, it’s hoped, will be able to break out of bad practices to the benefit of students. Given education has become big business, with a range of private companies offering competing pedagogical products, having such a toolkit helps schools to sort the useful programs from the fads.
The focus shouldn’t just be on educating teachers while they’re at university, he argues — continuing teacher training when they’re in the job is very important. “Normally there’s a plateau after the first few years of teaching, you don’t see any more gains in terms of student outcomes. Some really interesting evidence we’ve found is that if there is a teaching development program you see improvement in student outcomes for much longer. It would be great to have more evidence on those programs.”
Building the evidence base has benefits for governments beyond educational outcomes, too, by showing where money can be saved or put to better use. One area where the EEF research has led to a reconsideration of spending is around the impacts of teaching assistants.
“Over the past few decades there has been a growth in terms of people hired to help teachers in the classroom,” Major said. “We spent billions of pounds on those teaching assistants. But the research that we summarised for our guide shows that on average they have zero impact on attainment.
“So that prompted a big discussion about what was going on and subsequently we’ve done trials in schools to find out what actually does work. Basically it’s around how they’re trained, how they’re managed by the teachers, how you train teachers to manage the teaching assistants in the classroom. So that’s been a good example of how evidence-based policy can help in terms of expenditure.”
But even when you get hold of evidence about what works, there’s the problem of actually implementing it.
“The problem we found,” Major explained, “was that some schools would use the evidence quite intelligently, and look at the evidence about how to improve their teaching assistants, while others would just sack their teaching assistants. So the danger with evidence is that it doesn’t solve the whole issue; it gives you evidence upon which to make your decisions.
“What we’re wrestling with, and what we’re discussing with the Australian officials here, is how you take that evidence and actually have intelligently informed practice and how you scale it up as well. What we’ve also found is that some schools do this brilliantly, and they tend to be the high-performing schools, what we would love to do is have evidence-based practice in the schools in the most challenging areas.
“But there’s a big question about how, even when you’ve got an example of good practice, how you scale that up and implement it across a system. I think we’re in the very early days of that and we’re discussing that with Australian public servants and it seems they’re wrestling with the same issues.
“We’re researching into the uptake of research!”
The ‘Bananarama principle’
Having a strong regime of school inspections is one of the key ways of ensuring information gets through to those at the coal face. “One of the things I would like to see more is inspectors going to schools and saying ‘what evidence have you got to justify your strategy here?’. The inspectors would come with evidence and say ‘why aren’t you trying this?’, rather than being ideological or based on fads,” Major argued.
One of the strengths of Australia, he says, is that while the federation presents its own co-ordination challenges, the different state education systems allow for a certain amount of experimentation.
“One of the things academics are always wanting is some variation on a model, so I think there’s potential there, though you’d have to make some assumptions about context being roughly similar. Certainly in urban areas you could try different variants of a program. That could be quite good from an evidence base perspective,” Major said.
He thinks Australia could do more to evaluate schools based on educational progress rather than simply achievement. A lot of schools are high performing because they’re the ones that attract the already bright children. You want to know where the children were when they started, says Major, and where they finish up, “so progress measures are the thing you really need to look at”.
[pullquote] “There’s a tendency to get busy doing stuff and not challenge yourself in terms of what the evidence is …” [/pullquote]
Major reckons it’s a good idea to think about funding in terms of what he calls “the Bananarama principle”. While there’s always debate about how much money everyone receives, “it ain’t what you do, it’s the way that you do it, and that’s what gets results”, to quote the British band Bananarama.
“Lots of the teachers are too young to get the reference now, but that is really a key principle,” he said. “I know it sounds obvious, but you go into a school or a department and the tendency is to say ‘let’s do more’, usually they’re run off their feet, and they’re adding extra things to what they’re doing.
“When I go to visit schools my first question is always: what are you doing? Let’s have a look and see what the evidence suggests, because there might be some things you’re doing that are not having much impact. There’s a tendency to get busy doing stuff and not challenge yourself in terms of what the evidence is, because it’s quite hard I think.
“One head teacher got up at the [Gates Foundation] global summit [on education] and said we’ve got this mantra of teach less, teach better. Politically, you can see why politicians wouldn’t like that as a slogan, but actually it does make sense because what they’re doing in that school is spending more time on teacher development, observation, which then improves the teaching, rather than doing long hours of teaching that probably isn’t actually having as much of an impact as you’d like.
“There is the challenge between what works in terms of public service and practice and the political realities.”