‘Some things cannot be measured’: the limits of evidence-based policy


Evidence-based policy is a powerful notion that has entered the public lexicon in recent years. But, like another well-worn idea, agile, it has become increasingly debased as it has grown in popularity.

Evidence-based policy is now invoked to provide a supposedly apolitical, scientific solution to any number of problems, many of which are primarily about ideology and values.

The pursuit of data and evidence can improve our knowledge of what’s happening on the ground and inform our response, but it can’t give us the answers to everything.

It’s important to remember the limits of evidence-based policy, argued Melbourne School of Government foundation director Professor Helen Sullivan at Monday’s Power to Persuade symposium in Melbourne.

“We start focusing on those people who are seduced by the slogans. Not us, it’s those people. And that’s really dangerous.”

Plenty of important social policy questions are difficult to evaluate and cannot be reduced to a set of numbers or a formula.

“There are some things that cannot be measured, and most of the things we do in collaboration cannot be measured,” Sullivan says.

Boiling things down to the data is also expensive and time-consuming. Policy experts need to think about what is appropriate to measure and what’s not.

“It’s really important both to recognise that there are things that can be measured and should be measured and we should spend time doing that — but it’s also important to recognise that sometimes the pursuit of measurement leads you to more and more resources spent on things that tell you less and less,” she thinks.

This trend didn’t come from nowhere.

“We’ve only really become completely obsessed with measuring things over the last thirty years. It’s not coincidental that it emerged alongside the marketisation and managerialisation of our public services,” Sullivan noted.

Relying only on abstracted data can hide as much as it reveals. Sharon Fraser, general manager of central Victorian social policy initiative Go Goldfields, highlighted the problem of policy processes ignoring the lived experience of those it impacts.

It’s often assumed expertise means having letters after your name or having a theoretical framework — in practice this has seen community organisations having to fight to have survivors of family violence, for example, recognised as experts in that field and given formal input into policy development.

A narrow focus on evidence can lead policy experts to play down their own ideological and personal backgrounds.

Sullivan pointed to a tendency to dismiss the concerns of some sectors of the public because the “evidence” shows that they’re wrong. Polls showing low public trust in institutions suggest there’s a problem that needs to be addressed.

“This isn’t just about government, this is about the fact that people don’t trust big institutions. People are frightened, people are anxious,” she explained.

“One of the interesting things that happens then is we start focusing on those people who are seduced by the slogans. Not us, it’s those people. And that’s really dangerous.

“I also have a real problem with the fact that ‘we’ — whoever ‘we’ are — are all about evidence, and ‘they’ — whoever they are — are all about things other than evidence.”