Opinion: Google searches are no substitute for systematic reviews when it comes to policymaking

By Peter Bragge

Tuesday May 26, 2020

With all public attention on the COVID-19 pandemic, it is easy to forget that Australia suffered traumatic bushfires last summer, and that a royal commission is investigating the fires and will report in August. According to its Terms of Reference, the commission will examine how Australia’s national and state governments can improve the ‘preparedness for, response to, resilience to and recovery from, natural disasters.’

Many would assume that the commission will identify and use all best-available research knowledge from around the world. But this is highly unlikely because royal commissions are not designed in a way that is fit-for-purpose in the 21st century. Specifically, their terms of reference do not mandate the inclusion of knowledge from world-leading research, even though such research has never been more accessible. This design failure provides critical lessons not only for future royal commissions and public inquiries but for public servants developing policy, including for the COVID-19 crisis, and for academics, journalists, and all researchers who want to keep up with the best global thinking in their field.

The risk of not employing research knowledge that could shape policy and practice could be significantly reduced if the royal commission drew upon what are known as systematic reviews. These are a type of literature review that identify, evaluate and summarise the findings and quality of all known research studies on a particular topic. Systematic reviews provide an overall picture of an entire body of research, rather than one that is skewed by accessing only one or two studies in an area. They are the most thorough form of inquiry, because they control for the ‘outlier’ effect of one or two studies that do not align with the weight of the identified research.

Systematic reviews are known as the ‘peak of peaks’ of research knowledge

They became mainstream in the 1990s through the Cochrane Collaboration – an independent organisation originating in Britain but now worldwide — which has published thousands of systematic reviews across all areas of medicine. These and other medical systematic reviews have been critical in driving best practice healthcare around the world. The approach has expanded to business and management, the law, international development, education, environmental conservation, health service delivery and how to tackle the 17 United Nations Sustainable Development Goals.

There are now tens of thousands of systematic reviews spanning all these areas. Researchers who use them can spend much less time navigating the vastly larger volume of up to 80 million individual research studies published since 1665.

Sadly, they are not. Few policymakers, decision-makers and media are using systematic reviews to respond to complex challenges. Instead, they are searching Google, and hoping that something useful will turn up amongst an estimated 6.19 billion web pages.

The vastness of the open web is an understandable temptation for the time poor, and a great way to find a good local eatery. But it’s a terrible way to try and access relevant, credible knowledge, and an enormous risk for those seeking to address hugely difficult problems, such as responding to Australia’s bushfires.

The deep expertise of specialist professionals and academics is critical to solving complex societal challenges. Yet the standard royal commission approach of using a few experts as a proxy for the world’s knowledge is selling short both their expertise and the commission process. If experts called before the bushfire royal commission could be asked to contribute not just their own expertise, but a response to the applicability of systematic review research to Australia, the commission’s thinking could benefit hugely from harnessing the knowledge both of the reviews and of the experts.

For example, our firefighters and other emergency workers are deeply valued both here and overseas, as the size of charitable donations to first responder organisations such as the NSW rural fire service reveals. How can we best help them? A 2016 systematic review of 111 papers examining the psychological impact of disasters on first responders reported that:

  • The psychological impact of disasters on responders varies according to pre-disaster factors (such as specialised training); during-disaster factors (such as time spent on site) and post-disaster factors (such as professional support).
  • First responders who are especially vulnerable to psychological impacts include those working at the epicentre of a disaster; who arrive on the scene earliest; who have the greatest exposure to the disaster and who perform tasks outside of their usual roles.
  • At all stages of a disaster, steps can be taken to minimise risks to responders and increase their resilience. These include preparation for the unpredictability of disasters; reducing foreseeable risks by committing strongly to occupational health and safety; strengthening social bonds through team-building exercises; and providing workshops on resilience and coping strategies.

Systematic reviews are not a panacea. Research findings need to be integrated with an openly debated assessment of alternative options, and discourse between researchers, experts and decision-makers. But systematic reviews of research are a chronically underused asset that can “help to challenge prevailing perceptions and gatekeepers of knowledge”. Royal commissions are particularly well positioned to draw upon, re-package and transmit knowledge to influence policy. Systematic reviews should become part of their toolkit, and in the kit of anyone who wants to base their thinking on the world’s best evidence.

Peter Bragge is Associate Professor and Director of Health Programs BehaviourWorks Australia at the Monash Sustainable Development Institute, and Lead in the Monash-McMaster Social Systems Evidence Collaboration.

About the author
Inline Feedbacks
View all comments
The Mandarin Premium

Insights & analysis that matter to you

Subscribe for only $5 a week


Get Premium Today