Defining success for post-disaster recovery is harder than it seems

By David Donaldson

Monday February 20, 2017

Being stretched for time is a common problem, but few government agencies are charged with the kind of work faced by New Zealand’s Canterbury Earthquake Recovery Authority.

CERA was established to demolish buildings, rebuild and assist broader recovery following the 2010 and 2011 earthquakes that destroyed much of Christchurch, killing 185 people, damaging infrastructure and more than 100,000 homes. It was disbanded — as planned — in April 2016.

Its expenditure was significant — during these five years, CERA was responsible for spending about $4 billion on a range of recovery programs. The country’s controller and auditor-general, Lyn Provost, says it difficult to determine if it demonstrated value for money because performance measures were “focused on what it was going to do, rather than on what it was trying to achieve.”

It’s hard to know whether you’ve been successful if you don’t know what you’re aiming to do. CERA never established clear outcomes to be pursued. And it wasn’t alone.

A research team from the Australia and New Zealand School of Government looked at 217 disaster events between 1995 and 2015 to identify what was learned from their evaluations.

Lacking articulated outcomes and high quality evaluation is a fairly common problem for disaster response efforts, the researcher’s evidence review paper found. Of the disasters examined in the study, fewer than half had been subject to post-evaluation — and most of these were focused on processes rather than outcomes. The authors argue:

“The lack of evaluations means we do not know whether the community is actually benefiting from interventions. Without examining the appropriateness, effectiveness and efficiency of interventions, we cannot determine whether the programs being implemented are contributing to recovery and delivering outcomes. As such, it is unknown if the current spending on post-disaster recovery is justified and the most efficient use of resources, or whether funding and intervention approaches require a re-think.”

The researchers made another troubling finding, that the practitioners in government themselves don’t have a clear understanding of the field’s scope of responsibility, including what defines ‘post-disaster recovery’, the steps or interventions included the recovery phase of disaster management, and what success should look like. “This is further complicated by no clear end-point for disaster recovery intervention. Without these aspects being clearly defined, it is extremely challenging for practitioners to design and undertake evaluations.”

The ANZSOG team recommends a national monitoring and evaluation framework to address those scope and evaluation issues.

Back in New Zealand, Provost also concludes that having a better performance framework is an important lesson for the future.

NZ earthquake recovery defined by haste

Delaying work when homes and businesses lacked services or were even uninhabitable clearly had the potential to cause major inconveniences in citizens’ lives and damage the economy. Yet the need for pressing action cannot excuse the failure to set up effective financial and management controls from the beginning, argues Provost.

Poor processes increased the chance of public funds being misused. “The early stage of the recovery is when there is the greatest risk and opportunity for fraudulent activities and inappropriate spending of public funds,” the auditor says in a recently released report.

“This meant that CERA’s staff were engaged in challenging tasks, many of which involved transactions of significant value, without being able to refer to signed-off policies and procedures. CERA was slow in establishing delegation policies, which meant that some financial decisions took longer than necessary.” The agency also spent money for which parliament had not provided appropriations.

The rush to start meant “staff had to work in a challenging environment without the usual back-office support and controls that we expect in a public entity”. Some of the early problems persisted — when the agency was shut down five years after its creation, there were still improvements unimplemented.

Although CERA was supposed to lead the multi-agency response, the auditor reiterated previously identified issues with the authority’s across-government coordination, suggesting the omission of goals and a lack of time for senior staff to do strategic thinking damaged its ability to be an effective leading agency.

Outcomes were further confused by the gradual agglomeration of projects taken on. The authority’s original purpose was to be the coordinator, but it ended up assuming responsibility for a lot of service delivery work.

“CERA struggled to prioritise and balance its strategic leadership role with delivering the expanding portfolio of programmes and projects. As a result, its role became less clear, which confused staff, stakeholders, and the community. At times, CERA failed to prioritise its relationship and leadership role with Wellington-based government agencies. The lack of clarity in its role made it more challenging for CERA to influence and co-ordinate the work of the wider public sector,” says the report.

Getting it right next time

The auditor provided some advice for future disaster recovery efforts, firstly on the issue of governance, organisational structure and functions:

  • There needs to be clarity about what a recovery agency should achieve by the end of its lifetime. This should be expressed in a performance framework with realistic targets, and be regularly reported on.
  • Skills and capabilities need to be regularly assessed during the different phases of the recovery so that the recovery agency has the right skills for the tasks at hand. It is important that a recovery agency has strong programme management and commercial skills, particularly in the reconstruction and regeneration phases of recovery.
  • Governance arrangements need to be reviewed for each phase of the recovery and when activities change. This will ensure that the governance arrangements are fit for purpose to deliver the recovery agency’s outputs and outcomes in the most effective and efficient way. Particular attention needs be given to the clarity of role definition between the responsibilities of governance and management at both an organisational and project level.
  • To ensure that these decisions are made at the right level, there needs to be an agreed process for making timely decisions about the recovery. For example, strategic decision-making should be separate from operational decision-making.
  • During a recovery, central agencies need to regularly assess whether the recovery agency is the right vehicle for delivering particular outputs and outcomes. This would help to keep the recovery agency focused on its role and not be distracted by additional responsibilities.
  • Agencies need good communication and engagement with the community. Communication needs to acknowledge delays as well as celebrate progress. This helps manage people’s expectations and build trust and confidence in the recovery agency. Mechanisms need to be put in place that give the communities effective opportunities to participate in the recovery.
  • Tensions need to be prepared for and managed. Establishing a clear and detailed funding agreement, outlining the roles and responsibilities of all parties, and ensuring open discussion at a governance level, will help to ease inter-agency and intra-governmental tensions to ensure that progress is effective and efficient.

And secondly on the issue of managing operations and reporting performance:

  • To manage uncertainty in a disaster recovery, a recovery agency needs to have flexible arrangements for funding and staffing. For example, it might need to use multi-year and category appropriations, and a mixture of fixed-term and short-term staff.
  • A recovery agency needs effective financial and management controls from the start. The early stage of the recovery is when there is the greatest risk and opportunity for fraudulent activities and inappropriate spending of public funds.
  • A recovery agency needs to think ahead about the future phases of the recovery and plan for them at a strategic level. This will allow the agency to be more proactive in anticipating future issues and identifying its needs for the next phase.
  • An effective performance framework needs to link recovery activities to the desired outcomes. Effective performance reporting gives recovery agencies the opportunity to adjust their processes to achieve the desired outcomes. The World Bank has identified that a performance framework (also called a results framework) needs to harmonise and integrate all the strategic pillars and areas of a given reconstruction programme. The framework would establish a streamlined results chain by focusing on key results, and measuring intermediate outcomes (contributions made to an outcome by a specific set of outputs) rather than outputs. Intermediate outcomes need to be carefully designed to capture and to track intended changes as they begin to unfold.

About the author
Inline Feedbacks
View all comments

The essential resource for effective
public sector professionals