Federal Indigenous affairs bureaucrats have released a draft of their new evaluation framework, eight months after the Commonwealth committed $40 million over four years to evaluate policies in the portfolio and put a highly regarded university professor in the driving seat.
The draft — comments welcome till October 30 — sets out processes to look more objectively at national policies to support Aboriginal and Torres Strait Islander communities and contribute to Closing the Gap, which have been led by the Department of the Prime Minister and Cabinet for the past few years.”This is intended to align with the role of the Productivity Commission in overseeing the development and implementation of a whole of government evaluation strategy of policies and programs that effect Indigenous Australians,” it explains.
Around the same time as the new evaluation funding was announced, Malcolm Turnbull sought out indigenous health expert Ian Anderson to take over as deputy secretary leading the PM&C indigenous affairs group, which is also the only group within the central department overseen by an associate secretary, Andrew Tongue.
Anderson’s first major task was a review of the Closing the Gap target framework, which focuses attention on particular indicators of disadvantage. A few months into the job he set out some of his thoughts in a public speech at a special event marking 50 years since the referendum that effectively created this area of federal policy.
The framework notes good evaluation is “planned from the start, and provides feedback along the way” (referencing the audit office’s 2014 better practice guide to public sector governance).
“Good evaluation is systematic, defensible, credible and unbiased. It is respectful of diverse voices and world-views.
“Evaluation is distinct from but related to monitoring and performance reviews. Evaluation may use data gathered in monitoring as one source of evidence, while information obtained through monitoring and performance reviews may help inform evaluation priorities.”
The credibility of future evaluations depends on demonstrating their independence. To this end, the framework says a new external advisory committee, membership so far unknown, will “support transparency and ensure the conduct and prioritisation of evaluations is independent and impartial” by overseeing how the new framework is applied, checking the annual evaluation plan and with “ongoing advice, quality assurance and review”.
A “commitment to transparency” is also included. The committee will publish “all high priority evaluations” and reviews of them. Others will be randomly reviewed and summarised in an annual report.
“At the three year mark an independent meta-review of IAG evaluations will be undertaken to assess the extent to which the Framework has achieved its aims for greater capability, integration and use of robust evaluation evidence against the standards described under each of the best practice principles.”
All the actual evaluation reports will be published as well, at least in summary form, including “where ethical confidentiality concerns or commercial in confidence requirements” apply. Indigenous communities that have participated in evaluations will get to see the results too and additional “knowledge translation” efforts are proposed:
“Evaluation findings will be of interest to communities and service providers implementing programs as well as government decision-makers. Evaluation activities under the Framework will be designed to support service providers in gaining feedback about innovative approaches to program implementation and practical strategies for achieving positive outcomes across a range of community settings.”
The draft framework says it aims to:
- generate high quality evidence that is used to inform decision making,
- strengthen Indigenous leadership in evaluation,
- build capability by fostering a collaborative culture of evaluative thinking and continuous learning across the IAG and more broadly across communities and organisations, and
- place collaboration and ethical ways of doing high quality evaluation at the forefront of evaluation practice in order to inform decision making.
Higher quality evaluation that is “ethical, inclusive and focused on improving outcomes” is more likely to have impact, the draft points out. “It aims to pursue consistent standards of evaluation of Indigenous Advancement Strategy (IAS) programs but not impose a ‘one-size-fits-all’ model of evaluation.”
The guide calls for best-practice evaluation to be “integrated into the cycles of policy and community decision-making” in a way that is “collaborative, timely and culturally inclusive.”
“Our approach to evaluation, as outlined in this Framework, reflects a strong commitment to working with Indigenous Australians.
“Our collaborative efforts centre on recognising the strengths of Aboriginal and Torres Strait Islander peoples, communities and cultures.
“Fostering leadership and bringing the diverse perspectives of Indigenous Australians into evaluation processes helps ensure the relevance, credibility and usefulness of evaluation findings. In evaluation, this means we value the involvement of Indigenous Australian evaluators in conducting all forms of evaluation, particularly using participatory methods that grow our mutual understanding.”
Indigenous Advancement Strategy evaluations will look at how well programs meet three criteria:
Do they build on strengths to make a positive contribution to the lives of current and future generations of Indigenous Australians?
Are they designed and delivered in collaboration with Indigenous Australians, ensuring diverse voices are heard and respected?
Do they demonstrate cultural respect towards Indigenous Australians?
Four elements of good evaluation
The draft framework lists four elements of good evaluations — they are robust, relevant, credible and appropriate, which is to say they are “fit for purpose” and done in a timely fashion — and explains in detail how each of these ideals is to be achieved in Indigenous affairs through higher standards.
“Evaluation needs to be integrated into the feedback cycles of policy, program design and evidence-informed decision-making,” explains a chapter on relevance. “Evaluation feedback cycles can provide insights to service providers and communities to enhance the evidence available to support positive change. This can occur at many points in the cycle.”
While not being too prescriptive, the framework aims to set a high standard for the evidence that is used to judge the impact of programs.
“A range of evaluation methodologies can be used to undertake impact evaluation. Evaluations under the Framework will range in scope, scale, and in the kinds of questions they ask. Measuring long-term impact is challenging but important. We need to identify markers of progress that are linked by evidence to the desired outcomes.
“The transferability of evaluation findings are critical to ensure relevant and useful knowledge is generated under the Framework. High quality impact evaluations use appropriate methods and draw upon a range of data sources both qualitative and quantitative.
“Evaluation design should utilise methodologies that produce rigorous evidence and make full use of participatory methods. Use of participatory approaches to evaluation is one example of demonstrating the core values of the Framework in practice.”
Perhaps the moves to take a more academic approach at the federal level will allow for more open discussion of what works, in a portfolio where this year the minister has seen fit to publicly attack researchers in the field, and blast the independent audit office for doing its job instead of helping him attack the opposition.