Who ever heard of an independent evaluation?


There is no elephant in any of the rooms at the convention centre where the Australasian Evaluation Society is staging its annual conference in Canberra this week; it took centre stage in the opening keynote presentation.

Evaluation does not contribute much to the public good because it mainly serves those who pay for it, argued Canadian evaluator Sandra Mathison, in a brutally honest assessment of the profession she has worked in for over 40 years.

Over that time her enthusiasm about its potential for positive impact – she thought it would be a “helping profession” that makes the world a better place – has waned, she told the AES 2017 conference yesterday.

“… I find myself pessimistic, maybe even cynical, about the extent to which evaluation has contributed to the public good, by which I mean the wellbeing of all people, globally, manifested in things such as food security, healthcare, education, clean water, adequate housing.”

Rattling off statistics on a huge range of global and local social problems to prove her point, she said evaulators always think their work “will make things better, lead to better outcomes, solve social and environmental problems” but they really “need to own that claim” and question whether it is true.

 “Those with the money dominate the definition of what matters, what counts as success and how it is to be demonstrated.”

As well as lacking genuine independence, Mathison argues evaluation serves to maintain the status quo. Its narrrow focus on specific policies and programs that ostensibly address certain social problems helped deflect attention from investigating their root causes, she said.

Most evaluators think in a micro context, the conference heard, a legacy of a practice that is too meek and subservient to “other disiciplines, decision makers, policymakers, funding agencies and sometimes beneficiaries”.

As such, evaluation is always “a reaction to the ideas of others” that can’t challenge the assumptions underlying policies and programs, or whether there’s a better way to approach the social issue or challenge they seek to address. Perhaps, in that case, it is perfect for the public service.

She contends that evaluation is always locked into the “sociopolitical zeitgeist” of the times and recalled a “gold rush era” under the influence of the social democratic ideology, until a decline under neoliberalism, which remains the most influential school of political, social and economic thought today.

Now and then

Sandra Mathison

The progressive era was defined by “advocacy for social reform, especially a reduction of income inequality, expanding freedoms and rights, and expressions of collective action and humanitarianism – the idea that the human condition would improve through the application of science, technology and various social organisations,” she contended.

“In its early days, evaluation practice bore the mark of progressivism,” said the University of British Columbia professor. “Our work was often supported by public funding and defined itself as a public good, something that was in the interests of all. Evaluation reflected progressive values, including efficiency, but also social justice and democracy.”

What she called the “gold rush era” – defined by expansion of evaluative models and methods, and clear social purpose – ended in the early ‘80s.

“Evaluators were confident their craft would help to make the world a better place. But evaluation has now laboured under neoliberalism … for decades. And we, in the current state of neoliberalism, see that evaluation increasingly reflects those values, including competition, commodification, and privatisation.”

The most popular political and economic ideas “shape how evaluation is conceptualised, the methods and models used, how and by whom it is funded, and the efficacy in promoting positive social change”. In the professor’s view, the global shift to neoliberalism — which “rejects political partisanship and disregards national boundaries” — has been especially profound.

“This is a time when capitalism has trumped democracy and when the interdependence of capital and government has been solidified. A basic tenent of neoliberalism is the pivotal role [nation-]state governments play in facilitating and fostering the interests of economic elites.”

Program evaluation these days often reflects “a focus on efficiency, effectiveness and short-term measurable outcomes” in spite of “heartfelt” beliefs that it could be democratising, empowering, inclusive, transformational or participatory, Mathison added.

“Evaluation adopts market-speak and measures program success in terms of profit. Programs are in a competition with each other, outcomes are conceptualised in economic terms, and individuals are to blame for failure.

“Evaluators talk about a good program as one that has a good return on investment or gives good value for money, one that decreases costs — whether the context is health, housing or welfare … and when programs don’t work or fail in some way, it’s due to the service provider’s implementation infidelity, or lack of willingness or willpower on the part of beneficiaries.”

Evaluation is hardly ever independent

Robust evaluation of policy implementation is an important way for the public service to make its advice stronger, increasing the chances it will be heeded by government. Ministers love evaluations that show their policies are largely working, like the Orima Research report on the Commonwealth’s cashless welfare debit card trial published on Friday.

This report was always going to be controversial, like the early results from the same project. It shows that according to the government’s mostly quantitative measures of what success looks like, the cashless welfare cards are broadly working as intended.

Critics have questioned the report’s independence, and its methodology, and some have noted its qualitative findings that suggest the cards make a lot of recipients miserable. Regardless of your position on the full and final report and how it is presented by the Minister for Human Services, Alan Tudge, it is a perfect example of the role evaluation has assumed.

Tudge’s report came from a private firm, not the Department of Social Services, for example, which has a policy capability and evaluation branch. Lyn Alderman, the Australasian Evaluation Society president who opened the conference and introduced Mathison, recently became the chief evaluator in charge.

Alderman described evaluation capability as a “durable asset for sound governance” and said “longevity and permanence should be built into our evaluation systems, because societies with more evaluation capacity are better off than those with less”.

She also acknowledged the discipline is “inherently political” and said she had just started “living this reality” in her new role with DSS. “Ask me next year how it’s going,” she quipped.

If we face the facts, it’s clear that evaluation is hardly ever truly independent. “It’s a service that’s provided to those with power and money, and in that relationship, becomes a practice constrained in its capacity to contribute to the public good,” Mathison argued.

Its status as a “commodity” has been exacerbated in the current neoliberal framework, she added.

“Evaluation is a service bought and sold and while many evaluators frame their practice within larger principles of our professional organisations, evaluation and evaluators are nonetheless responsive to those who pay for their services.

“I believe it’s difficult for most practising evaluators to imagine that whoever is commissioning the evaluation does not have the most say in what the evaluation questions, and preferred outcomes, will be.

“Those with the money dominate the definition of what matters, what counts as success and how it is to be demonstrated.”

A way forward: speak truth to the powerless

Mathison isn’t entirely embittered and cynical about the potential for evaluation; she proposed that it could have a lot more positive impact if perhaps its clients were the “beneficiaries” of the programs being evaluated, rather than those holding the purse strings.

“We hear the phrase ‘speaking truth to power’ used to express what we see as our contribution to doing the right thing, contributing to the public good. A valiant but usually futile act,” she said.

“Speaking truth to power is a cliché, used often by leftists and liberals, and it neglects the likelihood that the powerful already know the truth, and choose to ignore it, or modify it, to suit their interests and already well-developed ideologies.

“Unfortunately our reliance on others in a service role, and a longstanding couching of our findings in other understandings, preferences and assumptions, makes speaking truth to power the entirely wrong approach, or at least not a very effective approach.”

Perhaps a better approach would be “speaking to the powerless” instead of the powerful. Mathison admitted she had no clear idea of how to get there, and conceded funding would have to come from somewhere; possibly philanthropy or public funding could work, if the intention was right.

“The powerless may not know the truth or may be confused about it, which helps to make them inactive, unable to pursue their own interests, unable to see their interests are shared with others,” she told the AES delegates.

“We have a proliferation of approaches to participation, emancipation, transformation – indeed I am part of the evaluation community that makes these promises.

“I’m not simply advocating that this type of evaluation is better. But rather for us to more fundamentally consider how evaluation might contribute to the public good, if the poor, the homeless, the diseased, were our clients.”

Mathison noted that “independent evaluations” is not actually a simple idea to put into practice.

External evaluation was often claimed to be independent, but “whether this is true is much debated” – as with Tudge’s Orima report. Any time the evaluation is paid for by someone with a vested interest in the program, its outcomes will be influenced by those interests, she argued.

“I’m suggesting that more independent evaluations, ones done with no monetary interest in the program, although perhaps intellectual or moral interests, can provide information about the value and consequences of programs and interventions.

“And so independent funding of such evaluations allows evaluators the opportunity to step outside the frames of neoliberalism, to look at long and short-term outcomes, and to investigate unplanned and unanticipated outcomes.”

To be clear, she is not talking about just involving the people targeted by the program in the evaluation, more consultation or the co-design we hear about so often these days. She means genuinely working to provide information to the subjects of the policy intervention.

“The powerless need evidence that will allow them to understand ideas like blaming the victim, how privatisation works, what the meaning of union-busting is, how free-market principles work, how these things sustain policies and ideologies that are detrimental to their wellbeing,” Mathison said.

“We need more truth to the powerless, not the powerful. And we need to empower the powerless to speak for themselves, to be able to act in their own best interests. Evaluators might consider how we would do this.”

The speech got one of the loudest rounds of applause this correspondent has ever witnessed at a conference of such an academic nature, and a standing ovation from some members of the audience moments after the final call to arms.

“I believe we can do better. I believe we should try to do better.”