‘We need to move away from our focus on quantitative KPIs’

By Stephen Easton

Monday September 28, 2015

Jane Halton
Jane Halton

The Enhanced Commonwealth Reporting Framework is forcing public servants to rebuild a culture of evaluation and develop the skills to measure the real impact of the policies and programs they implement, rather than just basic outputs.

The bureaucracy is not very good at acknowledging when something isn’t working, Finance secretary Jane Halton told a recent seminar organised by her department and the Australasian Evaluation Society. Neither is the government of the day, one might observe, in which case a more meaningful reporting framework is all the more important in the long run.

For those worried that greater transparency and more genuine accountability sounds risky, Halton pointed out the Public Governance, Performance and Accountability Act also demands “a more mature approach to risk”.

“… you could really argue we’ve been spinning our wheels.”

It’s easy to explain where all the dollars go and what they’re spent on, harder to measure the actual impact a given program achieves in the real world. To do so, public servants must use a more diverse set of methodologies including evaluation, benchmarking and peer reviews, the Finance chief says.

“We need to move away from our focus on quantitative KPIs,” she said, explaining there are a staggering 3500 quantitative KPIs under previous performance reporting arrangements.

She reminded those present that parliamentarians are watching, and some want agencies held accountable for non-compliance with the PGPA Act. ACT Senator Katy Gallagher recently criticised the 11% of agencies which missed the August 31 deadline to have new corporate plans in place, in the Joint Committee of Public Accounts and Audit:

“I think in any other area of public life we look very dimly on people that do not observe their legal obligations, particularly those in senior leadership positions who should know better.”

Said Halton: “So [complete the plans] before that next Senate Estimates round is my strong advice.”

She suggested agencies were struggling to get the plans in place due to the difficulty of setting out those measures of real-world impact that they would later report against. A complete picture of performance requires information on the “how” and “why” — qualitative data — as well as the much simpler “what” that is described by quantitative data. Some Commonwealth entities already use “mature” methods of performance measurement that lead to meaningful monitoring and reporting, the Finance head acknowledged, but the standard of the APS as a whole is insufficient.

Despite the thousands of KPIs and reams of reporting being produced under the old system, parliamentarians are still not happy with the level of actual accountability.

“I don’t know how many of you have been grilled for hours and hours and hours in Senate Estimates, but you could really argue we’ve been spinning our wheels. And certainly there are many still-unhappy senators in relation to what we’re actually achieving with our reports.”

The lost art of evaluation

Earlier in the year, Infrastructure and Regional Development secretary Mike Mrdak said evaluation became something of a lost art in the APS around the year 2000, after Finance gave up central responsibility for it.

The PGPA has brought the need for evaluative thinking back in focus, and Halton advised senior bureaucrats to look to international expertise, share expertise with each other, and “plagiarise shamelessly” from successful work in other jurisdictions.

“Embedding best practice across the whole of the Commonwealth will take many years, but the commitment to do better is apparent,” she said.

“I feel a slight sense of déjà vu because I think I gave my first talk on evaluation in the public sector in about 1985, and we’ve come some distance but it has waxed and waned, so in my department we hope that it’s on the up. But to be frank, producing our first corporate plan under the new PGPA framework was challenging.”

Putting together the corporate plan required new “clarity of purpose” and a clear understanding of the department’s current operating environment, according to its secretary.

“This actually led to some very important, incredibly useful conversations in our leadership team, so of itself, it’s galvanised change,” she said.

Halton added that if asked for a self-assessment of her own department’s first effort at a corporate plan, she and her senior team would argue: “We’ve done OK.” She said they would give themselves a “pass” grade with a note: could do better next time.

About the author
1 Comment
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Kevin Riley
6 years ago

Finance Secretary, Jane Halton is right to highlight the concern
about key performance indicators (KPIs). They have tended to be the things we
can count rather than the quality of the service, or the impact that service or
programme has had on an individual, family or group within the community. And
of course, outcomes might only start to be achieved after a number of years of
service or programme delivery.

But just because it is hard, doesn’t mean we shouldn’t try.
And while a focus on performance and KPIs opens agencies up to greater scrutiny,
this is what we expect of our accountability framework.

The Department of Finance has developed very good guidance
on the development of outcome and programme objective statements. When well
developed, outcome and objective statements provide a solid platform for the
identification of appropriate KPIs.

As Jane Halton identifies, a mature approach to risk also
needs to be taken in focusing on performance and KPIs. One reason why many managers
are reluctant to take responsibility for achieving KPIs is that they do not ‘control’
the performance. And this is the nature of outcomes, they are affected by our
own actions as well as a range of externalities (or external uncertainties).

Risk is defined as the effect of uncertainty on objectives. The
starting point for mature service and programme risk management is the
identification of the potential external uncertainties and the clarification
with Ministers and Parliamentarians about the possible effects of changes in
the external environment and how agency managers plan to respond to any of
these uncertain events. The Department of Finance has done much to communicate
the Commonwealth Risk Management Policy which became effective on 1 July 2014.

And this is not just an issue within the public sector.
Broad-based business reporting, or the ‘Six Capitals’ approach to reporting on
outcomes in the for-profit sector has also broadened the reporting of KPIs which
go beyond a simple measure of profit or return on equity – ‘financial capital’.
Focusing on outcomes in relation to ‘human capital’, ‘manufactured capital’, ‘intellectual
capital’, ‘social capital’ and ‘environmental capital’ with KPIs will stretch
many private sector managers.

Strong, meaningful KPIs come from considered, well-developed
and clear programme objectives and outcomes.

In 2001 the International Federation of Accountants Public Sector
Committee published a definition of accountability. That says that “Accountability
is the process whereby public sector entities, and the individuals within them,
are responsible for their decisions and actions, including their stewardship of
public funds and all aspects of performance, and submit themselves to
appropriate external scrutiny.” And it is the all aspects of performance that
appeals to me. This definition shows that accountability is more than good
spending (stewardship). Accountability is for service delivery and achieving
programme objectives and outcomes. And this means looking at the quality and safety
standards of service and the impacts – short, medium and long term – that programmes
are having on individuals, families, groups within the Australian community,
and the nation as a whole.

Until we see meaningful KPIs about aspects of performance,
balanced with a mature assessment of risk to that performance, we will not have
a framework that enhances accountability.