Federal agencies put a lot of effort into measuring effectiveness but “rarely develop indicators of efficiency” these days.
It’s often the job of the Australian National Audit Office (ANAO) to check if such measures are in place and, if so, whether they provide clear and meaningful information about performance in the particular situation.
“The simple response is to say that the agency is being efficient because it has paid an efficiency dividend, but how does the agency executive really know … that services, outputs or outcomes have been maintained or improved over time, despite a reduction in the budget?
“They can know this if they derive efficiency measures, or compare their input-to-output ratio over time, or compare themselves to similar organisations in terms of their inputs to outputs,” explains ANAO executive Lisa Rauter (pictured) in a new better-practice video.“A key factor to remember when considering the use of external benchmarks is that the comparator does not need to be identical.”
“Efficiency is generally relative to some standard, not absolute,” notes the latest edition of the quarterly Audit Insights series from ANAO. “Identifying a suitable reference point, benchmark or comparator is an important step in measuring efficiency.”
The point that efficiency is often “relative rather than absolute” is reinforced by the recent ANAO report on the Australian Commission for Law Enforcement Integrity, it adds.
The ACLEI report is also presented as an example of why it is better to “focus on the final outputs to be produced rather than interim processes” to provide meaningful performance information that supports resource allocation and provides genuine accountability and transparency.
“A key factor to remember when considering the use of external benchmarks is that the comparator does not need to be identical,” notes the latest instalment of Audit Insights.
“This argument is sometimes made by those trying to explain why efficiency measurement should not apply to them.”
The ACLEI essentially did present a version of this argument in their response, but the audit office says it is much like when trying to assess value for money in purchasing; the options are rarely identical products.
“As long as the purchaser knows the differences in functionality, the price/functionality relationship between the two can be determined and this used to assess comparative value for money. This may be particularly useful where the procurement involves a partnership in product development and the price/functionality of the product is likely to vary through the procurement process.
“Similarly, when assessing efficiency the comparator benchmark does not need to be the same, particularly if you know what the differences are and the benchmark can be used to assess reasons for variations through time.”
Looking at the implementation and performance of the Cashless Debit Card trial, on the other hand, the auditors found effectiveness was being measured but “operational and efficiency aspects” were not evaluated.
The auditor-general was not satisfied with the methodology used to estimate savings from Operation Tetris, the strategy to minimise empty office space rented by the Australian Public Service, and found the actual savings were not verified and compared with the estimates — although Finance Minister Mathias Cormann waved the audit report away, claiming the savings were surely underestimated.
ANAO found his department “had not established all the elements of a fit-for purpose performance framework to assess and report on its program of work on property efficiency” either.
“Broad objectives have been set but are not well-defined, and current performance metrics are incomplete and not properly integrated – addressing occupational density but not cost or other relevant indicators.”
When the auditors find there are no appropriate efficiency measures in place — ones that provide “meaningful insights” into the operation of the program or entity — they follow a standard three-step process to remedy that:
- Identify the relevant inputs and outputs for the entity or program being audited. Some examples of inputs include direct staff costs, contractor and consultant costs, funding streams, buildings, cash and other assets, revenue collected and other resources. Examples of outputs include grant dollars approved, paid and acquitted, telephone queries effectively handled, licences issued and complaints handled.
- Identify the outcomes sought by government. Examples might include reduced environmental impact, reduction of long-term unemployment, and the protection of threatened species.
- Assess the relationship between inputs and outputs, or inputs and outcomes to determine the appropriateness of the measure, or set of measures. A common approach to developing efficiency measures is to calculate ratios of inputs to outputs. Calculating ratios of inputs to outcomes provides measures of cost effectiveness.
A lack of direct data does not make this impossible either. Proxy performance indicators based on other data that are “strongly correlated with the activity” can still be developed in many cases.