Social impact bonds: how much evaluation is too much?

By David Donaldson

May 29, 2017

Calls for improved monitoring and evaluation are commonly heard in government — and for good reason, as their frequent absence means government is often flying blind when it comes to the question of whether the money it spends really does anything.

Improved rigour in monitoring and evaluation is one of the most commonly cited benefits of social impact bonds. The presence of private capital in the equation pushes both government and investor to ensure they’re getting the best deal out possible.

But requests for the measurement and reporting of inputs, outputs and processes by social impact bond investors can also create “substantial additional administrative burden” for service providers, according to a paper recently published in the United Kingdom.

“As a result, some third sector stakeholders felt that the degree of micro-management built into the SIB was actually reducing their flexibility to autonomously pursue their social mission,” argue Daniel Edmiston and Alex Nicholls of Oxford University in Journal of Social Policy

Some staff at the four programs they studied “felt that the resources and time that went into these additional forms of performance management and measurement could be better spent on front-line services,” Edmiston and Nicholls write.

Misaligned expectations between organisations created problems, according to one third sector stakeholder they quote:

“We underestimated the amount of management time that was required to run an SIB and we radically under-estimated the kind of information demand there would be from investors … progress reports and forecasts and re-forecasts”

And although it’s often claimed social impact bonds will lead to greater innovation and responsiveness, the addition of a third-party funder can lead to increased red tape, said one interviewee:

“If anything I would say less, there was less flexibility. Because the amount of management structures that are placed on an SIB mean that you have less manoeuvrability, you have to clear everything with this board to get permission to do anything.”

Some service providers reported feeling pressured to secure outcomes to such an extent that they even tried to “insulate” front-line staff from the influence of certain social investors.

Savings may be smaller than expected

Another key claim about the benefits of social impact investing is that it has the potential to save government money over the long term.

This is despite SIBs typically having high up-front costs, as consultants, lawyers and managers are required to set up the intervention structure, work out the terms of the contract and measure outcomes. Such costs, required for social impact bonds, are in addition to the expense of providing the high-intensity services — clients tend to be people requiring significant help, such as at-risk families or homeless people, for example.

The trade off is supposed to be realised in the long terms costs avoided by helping clients to live healthy, productive lives they might not have otherwise attained — helping someone avoid a lifetime in and out of gaol and hospital would result in significant savings for government.

But Edmiston and Nicholls question whether this idea is realistic, noting it is premised on gains made in a short to medium term program enduring. While the economic modelling used to justify the SIB approach tends to assume outcomes at the time of final evaluation will be permanent, this will not eventuate in many cases. Clients interviewed for the study reported feeling abandoned once targets were reached and the program wrapped up, expressing a desire for a transition period.

“I’m surprised the way it’s just been ended so suddenly — it’s meant I’m not doing so well now,” one service user told the academics. Another told them:

“I’m struggling at the moment … I’ve moved so far away from everybody. That hasn’t helped and it’s something I’ve got to sort out. I’ve relapsed and I’m hoping it’s not permanent.”

Social impact bonds vs payment by results

So does the addition of private or civil sector capital make social impact bonds more useful than regular payment by results methods?

The fact that the service provider was not typically risking their own money under a social impact bond model, unlike with a standard payment by results contract, meant that they tended to be less risk averse. The traditional public by results model encourages providers to focus more on cost-cutting than innovations in service delivery, the authors argue.

They write that “the SIB funding model does, overall, appear to have helped overcome some of the limitations of previous PbR schemes that have exhibited an aversion to service experimentation, flexibility and innovation due to the risks associated with prospective financial losses.”

But the authors were sceptical the newer model was unambiguously better than traditional payment by results contracts.

“There is, at present, very little definitive evidence to suggest that services funded through such a mechanism lead to any relative improvement in social outcomes compared to more conventional PbR commissioning models,” argue Edmiston and Nicholls. This is in large part because the evidence is not good enough — it’s difficult to work out the counterfactual around the presence or absence of third party capital.

The authors note that “where there is evidence available, it is rather mixed”.

While risking someone else’s money might give service providers a greater risk appetite, social impact investing introduces its own risks.

In the case of the SIBs considered in the study, the authors argue that “the prospective benefits of service innovation appeared to originate more from the novelty, size and experimental nature of the PbR contract, rather than the presence of private capital and the attendant functions it brought to bear on service operations.

“… the public sector runs the risk of paying increased transaction costs without realising the putative benefits offered through the SIB model.”

“If anything, the presence of private social investment appeared to stifle the flexibility and autonomy of service providers to innovate and deliver services according to their social mission within a PbR contract.”

The administrative requirements necessary to make the provider “investment ready” also narrows the field, leading to small organisations being left out of contention.

Additionally, the mix of intentions among funders — which might include the desire to make money, to do social good, or a mix of the two — presents problems, given that social impact investment effectively transfers regulatory oversight from the state to private organisations. A funder focused solely on profit and operating under a poorly designed contract clearly has the potential to do damage to clients’ lives.

As it stands, the relative benefits of social impact bonds over the regular payment by results model are unclear and further study is needed, the authors conclude.

“Without this and evidenced effects of improved (and sustained) social outcomes, the public sector runs the risk of paying increased transaction costs associated with private social investment without realising the putative benefits offered through the SIB model.”

About the author
Inline Feedbacks
View all comments

The essential resource for effective
public sector professionals