Ethics in program evaluation

By Stephen Easton

Tuesday October 6, 2015

All research using human subjects or information about specific people must respect the enduring ethical principles famously enshrined in the 1947 Nuremberg Code, and that includes policy and program evaluations.

In the Australian Public Service, the PGPA reforms have triggered a need for agencies to brush up on evaluation and quality assurance activities. With that in mind, Canberra Evaluation Forum participants discussed human research ethics in the context of evaluation at their most recent get-together, and heard from two public servants who are knee-deep in the subject.

“Examples of human research include interviews, surveys, focus groups, observations, chat rooms, testing,” explained Gary Kent, who heads up the Australian Institute of Health and Welfare’s governance unit. Kent gave the CEF a general overview of the subject and the work of the AIHW ethics committee.

“You can also talk about obtaining specimens from people, like DNA or blood or something else. All of that is human research. It doesn’t have to involve the physical body. It can even be interviewing someone, because there are ethical issues even in talking to someone. Even in a focus group, issues arise.”

Of course, there are no ethical problems with using aggregate data which genuinely cannot be linked back to individuals.

The current Australian version of the principles set out in Nuremberg is contained in a National Health and Medical Research Council statement, last updated in May. The four key values are: respect, research merit and integrity, justice, and beneficence.

At the same time, other laws like the Privacy Act and those that define and govern public services also demand high standards of ethics and respect for the rights of the individual. Such concepts are pretty clear and well established, but what about the rights of whole communities, wondered Australian National Audit Office senior audit director Richard Lansdowne.

“The one that comes quickly to mind is indigenous communities, and there’s about eight or nine sets of rules and regulations governing how you relate with those communities,” Kent explained.

“[Some regulations] are about how you engage with the indigenous people as a whole. They require, for example, that you have an indigenous expert as a consultant or on your committee [and] you’re encouraged to have a relationship with the local indigenous community structure of some sort. There is an indigenous human research committee of South Australia, for example.”

A practical example

Kyleigh Heggie from the Department of Veterans Affairs Human Research Ethics Committee presented an overview of a recent telehealth trial. The DVA committee, which has been around since 1993 and will merge with the broader Defence committee (ADHREC) in 2017, had to decide if the trial would do more harm than good for its elderly participants.

Ethical reviews are no a tick-and-flick process, Heggie explained.

“A lot of proposals that come by us are probably under par, and our ethics committee has a definite interest in robust, rigorous research; that means it must be ethical,” she said, adding that in a past job, she saw lots of proposals for “pretty cowboy” research that involved criminals in custody and victims of crime.

Telehealth equipment allows health professionals to remotely monitor certain vital signs of patients, and has been shown to assist with the management of certain chronic conditions. But the DVA trial would also involve risks of harm: significant intrusion into the homes and lives of veterans with chronic diseases, as well as digital collection, transmission and storage of their medical information.

One of the key questions the committee looked at was how much merit there was in the proposal. Telehealth has its success stories, but as with most emergent technology, it also has its cheer squad of companies that make money from it whether it works or not.

“Were we being sucked in to this new wave of technology? Was it worth it? This was something the ethics committee were very keen to address,” said Heggie.

The trial has apparently been successful for at least some of the participants, and has been extended by 18 months. The forum heard DVA has had no complaints from the participants or their families. Heggie also confirmed that the evaluation of the telehealth trial and the trial itself needed separate ethics approvals.

“We don’t allow any programs [involving veterans] to operate without ethics approval,” she said. “The evaluation … may have very different approaches to dealing with the participants, so any evaluation has to have independent ethics approval as well.”

When to worry about ethics, and how much to worry

As one CEF participant pointed out, the NHMRC offers guidance on ethics in evaluation and recognises a difference between quality assurance activities and more complex human research. But it makes no clear distinction between the two:

In some circumstances, attempts to clearly separate QA from research are unhelpful. Moreover, QA, evaluation and research exist on a continuum of activity, and work that begins as one form of activity can evolve into another over time.

Importantly, QA and evaluation commonly involve minimal risk, burden or inconvenience to participants, and, while some level of oversight is necessary, Human Research Ethics Committee (HREC) review processes are often not the optimal pathway for review of these activities.

What really matters is that:

  • participants in QA/evaluation are afforded appropriate protections and respect;
  • QA and/or evaluation is undertaken to generate outcomes that are used to assess and/or improve service provision;
  • those who undertake QA and/or evaluation adhere to relevant ethical principles and state, territory and Commonwealth legislation;
  • organisations provide guidance and oversight to ensure activities are conducted ethically including a pathway to address concerns.

DVA “doesn’t see the difference” between the two, according to Heggie.

“This is my opinion, not DVA’s: there’s a fine line between research and evaluation and a lot of companies skirt around it, saying it’s evaluation when it’s possibly research; actually, mainly research,” she said. “And there’s much more structure around ethical research than ethical evaluation.”

The Department of Immigration and Border Protection doesn’t have an ethics committee, explained one delegate, but relies on the ethical guidelines for Australasian Evaluation Society members. DIBP has also availed itself of one of the private firms that can provide independent ethical oversight, including for a recent “large international project” that involved identifiable individuals, the forum heard.

The AIHW does not link evaluation and ethics and if it does undertake evaluation, it’s a separate process the ethics committee isn’t involved in, according to Kent. “We don’t have any concept of ethically approved evaluation at the Institute,” he said.

Former senior public servant Stephen Bartos, fresh from a stint as parliamentary budget officer in New South Wales, suggested ethical approvals could be avoided altogether for some quality assurance work.

“Almost every evaluation involves some research of some sort, but it in many cases requires no ethical clearance at all, even if it involves human subjects,” Bartos contended. “As Gary knows, I did some work for AIHW that involves human subjects and I never got ethical clearance, mainly because the subjects were the members of the AIHW board, so we considered they were big and bold enough to make informed consent for themselves.”

The public administration pundit reminded the forum that ethics committee processes have a downside: they slow everything down. He said that as a university professor, he avoided going through the process whenever he could because it could have delayed the work by about six months.

“Committees are a wonderful mechanism for taking good evaluation ideas down a long corridor and slowly smothering them to death under paperwork,” said Bartos. “It’s absolutely needed in the areas like health research that our speakers have been talking about, but there’s a range of others where probably the more appropriate mechanism is independent advice or assurance.”

About the author
0 Comments
Inline Feedbacks
View all comments
The Mandarin Premium

Insights for policy professionals.

Subscribe for only $5 a week.

Get Premium Today