Experimental trials: a BETA way of making policy

By Stephen Easton

Monday November 28, 2016

From a scientific point of view, the standard way public policy is implemented at scale with little testing is extremely reckless. But human behaviour is “predictably irrational” and experimental trials offer policy assurance.

The head of the Commonwealth’s relatively new behavioural economics unit thinks it is “sort of scandalous” that all public policy is not rigorously tested by default before it is rolled out.

Experimental trials — randomised controlled trials wherever possible — are simply part and parcel of the behavioural interventions known as nudges that are fast becoming a standard tool for policymakers across the world.

Behavioural insights expert Michael Hiscox (pictured) wondered aloud why that isn’t the case for all public policy, in a speech on Friday that drew parallels between the pharmaceuticals prescribed by doctors, and the social or economic medicine prescribed by government ministers.

“We don’t have [experimental proof] for hardly any area of policy at the moment, and it’s sort of scandalous,” he said at an event jointly hosted by the Institute for Public Administration Australia (ACT Branch) and the Department of the Prime Minister and Cabinet.

Hiscox contended “the history of policy design tells us that we’re going to make lots of mistakes” and gave three strong examples of policies that sounded good, but actually were clearly proven to produce the opposite of the desired effect.

An attempt by Israeli childcare centres to discourage late parents with fines actually increased late pick-ups in one example, as the parents decided the fine was worth the extra child-minding time. In another case from the United States, young offenders are supposed to be scared straight by talking to hardened convicts, but evaluation showed later the program increased their chances of continued criminal behaviour later in life.

Another popular program that aims to discourage teen pregnancy by getting children to look after dummy babies, again, had the opposite effect. Unfortunately, such is the lack of demand for evidence to back up policy interventions that aim to influence human behaviour that strong evidence of their failure is often still ignored.

“We don’t always know what will work, right? And we don’t want to just go out and [roll out a new policy] at scale, assuming that we know it’s going to work,” said Hiscox, a Harvard University professor who took leave to return to his home country and get the Behavioural Economics Team of the Australian Government (BETA) off to a solid start.

“There’s going to be lots of unanticipated, unintended effects, and we need to test.

“So part of what we do [in behavioural insights] is a rigorous application of randomised controlled trials. Just like in the process of approving the new drugs that we now have and take for granted, we want to create a process like that, that we can use for design and optimising new programs and policies.”

To illustrate his point, Hiscox asked the audience to recall the Thalidomide scandal of the late 1950s and early 1960s. International outrage — and importantly, stricter testing regulations for pharmaceuticals — followed the revelation that a drug marketed to pregnant mothers caused deaths and serious birth defects in thousands of babies.

“I think we’d all agree that nobody would want to go back to the way it was before,” Hiscox said.

“Isn’t it strange then, that we don’t require something similar, or even halfway approximate, when we think about putting out new policies and new programs, in areas like health and education and employment and criminal justice, that can be just as impactful in terms of how we affect health outcomes and wellbeing?

“And yet we are pretty much sitting back and saying that policymakers get to decide, and put something out, at full scale, with no rigorous scientific peer-reviewed testing as to whether or not it’s safe and effective.”

“This is a huge juxtaposition or contradiction, I think, in our positions of how we apply science to the design of large-scale interventions, what we think should be regulated, what we think governments should do, and all of that.

“We exist with this contradiction every day.”

There is no reason to trust politicians and policymakers more than doctors, and few would agree the policy elite have an “understanding of social and economic interactions” that is more advanced than a typical medical scientist’s understanding of chemistry and biology.

Why then, asked Hiscox, do we have a system where policymakers only have to assert their new ideas will “obviously” work based on their chosen logic and simplistic beliefs about human behaviour, perhaps with a dash of political ideology thrown in.

This arrangement “flies in the face of the evidence” that shows governments “repeatedly” design policies that are later proven to be ineffective “and often have terrible unintended negative effects”  in the worst cases, he said. The real danger is the potential for rolling out bad policies at scale, leading to negative outcomes on a national or even “generational” scale.

When you’re dealing with the complex cognitive biases that influence human decision-making and trying to leverage the “predictably irrational behaviour” of citizens to improve policy outcomes, it’s important to prove that a proposed intervention will actually work.

BETA is promoting a much more rigorous, scientific approach to policy design based on more “realistic” models of human behaviour, working through a cross-government model with 17 partners across the APS so far. Hiscox said its first trial looks at more ways to reduce the effect of cognitive biases in recruitment and promotion decisions, and has just been completed with a report to be published soon.

The professor also promised BETA would soon become much more open about all its work.

Renee Leon. All images: RLDI.
Renee Leon. All images: RLDI.

Department of Environment and Energy secretary Gordon de Brouwer said BETA was helping his staff with “anything from cost-effectiveness to streamlined administration, greater compliance with regulation … and encouraging sustainable practices” in his opening address as IPAA ACT president.

Department of Employment secretary Renee Leon also spoke at the event, conceding her department had often rolled out programs on a large scale without being strongly certain about how successful they would be. But, she added, Employment has begun using the behavioural economics approach, to test out ways of encouraging citizens to use a new website.

Leon pointed out ministers are generally not keen to wait patiently for the results of rigorous policy trials before making a decision, so she proposed public servants could proactively test out different ideas and interventions and present a “menu” of options for ministers that are likely to be relevant in the next few years.

About the author
Inline Feedbacks
View all comments

The essential resource for effective
public sector professionals