Nicholas Gruen: the academy and partners try wellbeing frameworks


Economist Nicholas Gruen looks at problems with various attempts to measure wellbeing and the struggle to get from noble principles to practical outcomes.

This is part three of Nicholas Gruen’s essay series about the difficulty of translating policy into outcomes. Read part one, on wellbeing frameworks, and part two on commonsense hacks government could use to bolster Australians’ wellbeing.

“[T]he sciences we now possess are merely systems for the nice ordering and setting forth of things already invented, not methods of invention or directions for new works.” — Francis Bacon, Novum Organum, 1620.

In the first of these essays I tried to show how little change occurred as the result of the Australian Treasury’s Wellbeing Framework (now abandoned). That’s partly because our policymakers took it far less seriously than those in other countries like the UK and New Zealand. Nevertheless, even those countries’ wellbeing frameworks didn’t focus as productively as they might have done on wellbeing. If they had, they’d have generated sufficient information to allow us to start constructing – both generally and within specific portfolios – schedules of increasingly promising ‘no-regrets’ initiatives as we did with greenhouse emissions abatement curves, as illustrated.

Here I want to argue that what we see up close in policymaking has its close analogue in the academy. Firstly in the academic literature, I’ve found incredibly little discussion of the kinds of considerations illustrated in the chart. I presume there are others, but this is the only article of its kind I can find, and even here it’s notable that the focus is on developing frameworks for generating lists of interventions rather than the process of identifying obviously high-quality projects to which frameworks can be applied in due course when one finds the need and the time to prioritise.

In this part I take a look at a strangely becalmed Australian wellbeing project going on essentially within the academy – though with multiple stakeholders from the wider world. After around nine years, it seems to be just getting going. Yet even now the priority remains establishing a framework rather than searching for the most worthwhile policy interventions. And, even once it does get going, I’m doubtful as to what it will achieve. I’ll illustrate my claim by looking at the corresponding Canadian initiative, which is much further advanced.

Let me summarise my claims in this part and relate them to the general claims introduced in the previous part of the essay. Almost all the discussion occurs at a very high level. As we saw with the Australian Treasury’s wellbeing framework, it’s easy to announce some attractive principles – that policy advice should be framed with regard to a concept of wellbeing that goes beyond the purely economic outcomes captured in GDP. But if I might use the term, if these fine ideas are to ‘trickle down’ to improve our lives, these high-level ideas need to be operationalised. To do so we need to relate the apex principles in a wellbeing framework to concrete policy advice. This would require the framework to change business-as-usual advice and so change policy priorities in some noticeable way, for instance by changing policy advisors’ understanding of the relative value of different programs or changing their priorities in how existing and new programs are developed.

It also calls for investigation of causal questions. We might give more attention to defining and measuring wellbeing, but it’s of little use if we’re not interested in understanding causes, in understanding how policy can improve wellbeing. When we examine the Canadian initiative, it’s hard to see what useable policy insights it has generated. And when we look closely at the way it’s put together, we see something deeply unsettling. The process by which high-level statements about wellbeing are anchored to the nitty gritty of the index of wellbeing is essentially incoherent on numerous levels. The very choices of value that the index promises to clarify – so we can “move beyond” GDP and build an index around what we truly value – are consistently made by default rather than by design. Form dominates substance at virtually every turn.

The Australian and Canadian initiatives are left-of-centre political projects promoting the causes of the environment, community and greater equality over the preoccupation with GDP. Not, as Jerry Seinfeld might say, that there’s anything wrong with that. I’m certainly comfortable with such sentiments. The real problem is that we never get beyond general endorsements of very high-level principles – of the kind you’d hear in a TV studio. The various projects we’ll investigate in this essay never develop the intellectual tools necessary to do so.

The Australian National Development Index (ANDI)

ANDI was an initiative that emerged from the 2020 Summit and commenced in earnest in 2010. This 2013 press release on the Deakin University Website – itself a rebadging of an earlier media release tells us that ANDI is a “new paradigm for democracy”. It goes on:

Deakin Vice-Chancellor, Professor Jane den Hollander said Deakin was delighted to be the invited to “host” ANDI [sic.].

“Deakin has been selected because of its reputation for innovation,” she said.

Something tells me that second sentence is PR – which is to say not to be taken seriously. (More technically it’s bullshit – something said strictly for effect without concern for its truth or otherwise). Anyway, four years from these announcements, a search of Deakin’s website reveals nothing more. Meanwhile, an article on the University of Melbourne’s website from this year claims that “ANDI is hosted at the University’s Melbourne Graduate School of Education”. So much for Deakin University’s “reputation for innovation”. In any event, a University of Melbourne website search turns up no further research and the relevant link on the page takes you back to the ANDI website which has no link to research.

Mike Salvaris, the director of the program is quoted as saying:

[W]hat we are grasping after is a better way of involving ordinary people in a conversation on where we want Australia to be heading. It is about articulating what we want as a whole community, and we can’t do that in a legitimate and inclusive way if we just ask the elites and the politicians.

So here we are, seven years in from ANDI’s launch and we’re still talking about an intent to engage with the community. I have two problems with that. Firstly, though ANDI’s engagement ambitions are grand – involving more than half a million people1 – and I’m all for involving people other than “the elites”, I’ve no idea how you “articulate what we want as a whole community” by doing so. Different people want different things.

Secondly, if they just asked one person – me – how I want the world to be – how I’d describe my ideal of societal wellbeing, I couldn’t say much. I’d say what everyone says. I want a prosperous and fair economy (though the devil’s certainly in the detail of that second word), a healthy and clean environment, and a well functioning society and governance (though I’m not too sure how to secure them).2 But what would I say beyond that? Is ‘the environment’ — whatever that means – more or less important to me than ‘the economy’ – whatever that means? Hard to say unless I know more about this choice I’m being asked to make. (Remember R. G. Collingwood’s comments cited in the previous essay, “knowledge comes only by answering questions, and that these questions must be the right questions and asked in the right order”.) Knowledge of important social facts – like what you, I or we value and how much we value it compared with something else – arises in a context.

I can’t see how anyone, much less ‘the community’ can come to much of a view as to what it values and how much – how it can answer the Spice Girls’ Question, and tell us what it really, really wants – unless it considers, and preferably is forced to choose between concrete alternatives. As I argued regarding ends and means in strategic planning, in some contexts the relationship is one in which ends take precedence over means and in others it’s the reverse. As is evident in virtually any complex engineering process to build something concrete – like designing and building a new model of a car or a major bridge, the relationship between the Big Picture and the details is dialectical, not hierarchical.

Yet ANDI and, as we’ll see, so many other similar projects, is rooted in hierarchical thinking in which the Big Questions dominate the design. Only then can we ‘drill down to the detail’ as the saying goes – perhaps with research assistants – filling things out as already determined at the top or at the previous workshop. In fact one will often gain insight as to how the Big Picture might be framed from the details we encounter doing the small stuff. I know of no other way of answering Big Picture questions like, “How important is the environment compared with the economy?”

Moreover, since building the index is ultimately intended to help policymaking, you’d think such considerations would be integral to developing the index as understanding how a part will be manufactured is integral to designing a production car. Alas, seven years on, we’re still waiting on any serious elaboration of the policy implications of ANDI. Indeed, as we’ll see in the case of ANDI’s sister project the Canadian Index of Wellbeing (CIW), which has been reporting for around six years, policy implications beyond broad generalities remain thin on the ground.

The problem of incommensurability

Asking about the relative importance of ‘the environment’ and ‘the economy’ or any similar question raises the issue of incommensurability. There are two reasonable responses. The least contentious one is taken by the ABS which reports incommensurable domains separately on it’s Measures of Australian Progress website.

Measures of Australia’s Progress at the ABS website

But this leaves a dilemma. Since people are unhappy about the way GDP has come to dominate the discussion of wellbeing; if we want to rescue other aspects of our lives from this invisibility we need some plausible way of bringing them to account in the same frame as GDP. Amatya Sen, whose work on capabilities has been influential in this area, has always been wary of summarising the wealth of data bearing on human wellbeing into a single index. Yet he was persuaded that it was necessary to shift policymakers’ attention from material output to human wellbeing as a real measure of progress.3

If a single number generated by an index of wellbeing is to help measure the relative importance of different social phenomena – crime and underemployment for instance – the relative importance of each of these compared with other factors in wellbeing, it needs some way of bringing these things into relation with each other. Here’s how Lateral Economics tried to solve this problem as we built the Herald/Age Lateral Economics (HALE) Index of wellbeing.

We began with the national accounts from which GDP is derived and then sought to correct them wherever we thought the dollars aggregated in the accounts missed some clearly important aspect of wellbeing. Subject to the basic idea that we wanted to be measuring the same thing – wellbeing – and we wanted to do it in the same basic unit – which you can call wellbeing-adjusted dollars, or utils or well-beans if you like – each issue was then tackled, as best we could, on its merits.

Our anchor in tackling incommensurability was self-reported wellbeing. By asking, as one might in a court of law, how much money you would need to restore someone to their state of wellbeing without some disadvantage, whether it was low relative income, mental illness or long-term unemployment, we built a highly imperfect but informative bridge between things that would otherwise simply stop us in our tracks with their incommensurability.

As you’d expect the result is messy. Because it was done ‘on the merits’ of each case, the methodology and magnitude of our adjustment to GDP for its inadequacies differed greatly in measuring the wellbeing implications of different phenomena such as inequality, unemployment, long-term unemployment, overwork, mental illness, obesity, hospital safety, life expectancy and human capital. But if you’re going to build a unitary index, there’s no alternative but to make the heroic assumptions necessary and then to open up the methodology and the measurements to scrutiny. It’s intended as the beginning of a conversation in which everyone has somewhere coherent to proceed, either to work with our methodology and/or numbers, or to improve it from their perspective and perhaps for others.

Yet as we worked away, I became increasingly bemused by the lack of these kinds of basic considerations in the indexes I was seeing built at the time, from the OECD Better Life Index to the Canadian Index of Wellbeing, to which our attention now turns.

The Canadian Index of Wellbeing and other ‘composite’ indices

The Canadian Index of Wellbeing (CIW) seems very similar to ANDI but it’s been up and running for years, producing a unitary index since 2011 to assess, as it puts it how Canadians are “really doing”. The process of building and maintaining the index has been very involved with a large number of contributors. There were inevitable practical trade-offs involved in choosing indicators. Just a few of the criteria by which indicators were chosen include the ease with which data can be obtained, the ease with which it can be publicly understood, its susceptibility to any number of biases or sampling errors, the extent to which it measures an outcome or output rather than an input, and so on.4

However alongside this conscientiousness something happens which is so remarkable we should pause to reflect on it. The critical matter of deciding what things are more and less important for our wellbeing – or to put it differently, the values of the project itself – are handled by default rather than by design. The choice is made, as choices are, but all in the guise of refusing to choose. As the latest CIW report puts it:

“There are many reasons for regarding one or another indicator as more important in some way or other, but what is missing is a good reason for assigning any particular indicator a weighting greater or less than that of some or all other indicators. The absence of such a reason justifies the equal treatment of all indicators at this time.”

I invite the reader to closely re-read the quoted passage, to ponder its intellectual confusion and dissimulation, its feinting one way – “there are many reasons” – and then another – “what is missing is a good reason”. Rather than inhabiting a world of careful thought, we’re in the weightless world of simulated thought, of PR, right down to the telltale temporising of the last three words.

Unmoored from the strictures of reconciling the Big Picture with the details, or the weighting of different aspects of our wellbeing, the process floats free. The difficulty of making intellectual progress is displaced by the pleasing appearance of the same. The aesthetic appeal of easy symmetry fills the vacuum – accompanied, as one might expect, with a strategic diagram. The CIW comprises eight domains.5 Why eight? (The Bhutanese equivalent has nine, the OECD project has eleven). Who knows? And each of the domains is weighted equally. How many indicators will be used to determine how wellbeing is proceeding in each of the domains? Why, eight! They’re weighted equally too. Certainly it makes for a fetching diagram.

From CIW’s latest annual report.

Meanwhile ANDI is currently intended to have 12 domains. There are no prizes for guessing how many indicators comprise each of the 12 domains: 12.

Virtually all the ‘composite’ indices are plagued by these kinds of issues. They all collapse intellectually, as it were, under the burden of doing what they say was so urgently needed – to reweight our understanding of what matters alongside or instead of economic measures like GDP. The OECD’s Better-Life Index appears better in this respect because it enables the user to weight different domains differently.6

Certainly this delegation to the user is a nimbler way out of the conundrum which ends so ignominiously in choice by non-choice, in the other indexes. But if I’m correct in arguing that progress on comparing different domains of wellbeing depends on some plausible criterion for asserting some commensurability between them, then allowing people to twiddle some dials without the project spelling out the issues and offering some assistance with them, such strictures don’t seem like such an advance. Arguably it sets us back by lulling the user into a complacency about the coherence of what they’re involved in. Wasn’t the whole idea of broader wellbeing indexes supposed to help us understand the choices we’re making?

Moreover some domains are obviously correlated. Housing, income and jobs are clearly correlated in various ways. Shouldn’t the user be informed of this and taken through some examples and strategies for using the discretion they’ve been given to make more sense of the indicators than simply playing their own hunches and preferences. And dig further into the Better Life Index and we find the same old problem. As its website informs us, “Each topic of well-being is measured by one to four indicators. Within each topic, indicators are averaged with equal weights”.

Curiouser and curiouser

In fact the CIW and the way it reports its results is even more problematic than suggested so far. Though the decision – or non-decision – made at the top of the CIW project is to weight all indicators equally, the way the CIW is calculated produces wild disparities in the impact of different indicators based on quite arbitrary decisions about how the indicators are calculated. Within each of its eight domains, each indicator is indexed to 100 points in 1994 and then rises or falls against that benchmark. Here are the trends in the Democratic Engagement domain.7

Note the sudden dip in the index in 2011. This is driven by indicator E – “Percentage of Members of Parliament’s office budget devoted to sending communications to constituents” which went from 6.79% to 3.35%.8 My attempt to replicate the CIW’s calculation,9 moves the indicator from an index score of 142.6 in 2010 (compared with 100 in 1994) to 70.4 in 2011 – a loss of 72.3 points in 2011. As one might expect, most other indicators move by around 1% or less and and around four others change by less than five points. So while one might doubt that MPs’ communications spend is a good indicator of democratic engagement, being unusually volatile it has far more impact than most indicators throughout the series and in 2011 it completely dominates the domain.

Within the democratic engagement domain, indicator F measures engagement in political activism. This is expressed as the “Percentage of population that volunteers for a law, advocacy, or political group”. This number starts at just 1.5% in 1994. Accordingly as it grows to 2.3% its growth has a 53-point impact on the domain. Together these two of the eight domains explain over half the movement in the index.10 In fact this result itself is the arbitrary artefact of the way the original ratio is expressed initially. If one’s metric of political engagement were expressed in the index as the ratio of unengaged to engaged rather than engaged to unengaged its impact would have gone from 53 points to 0.8% points.11

So much for equal weighting.

Diminishing returns by assumption

In virtually all the reporting I’ve seen, the CIW is taken to demonstrate that total wellbeing is growing less strongly than the economy. 12  As the most recent report on the index concludes:

“The divergence in the CIW and GDP tells us emphatically that we have not been making the right investments in our people and in our communities — and we have not been doing it for a long time.”

For the record, I’m not seeking to underplay the emerging difficulties of our age. My own personal list of grave concerns include these:

  • the potential for catastrophic global warming;
  • growing income and wealth inequality, though it has been far milder in Australia than in the United States;
  • the plunging functionality of and trust in our elites, particularly, but not exclusively, in politics; and
  • multi-generational social dysfunction.

But, however we measure it, where wellbeing deviates from GDP, the first point anyone seeking to inform the public on is this: in just the way that diminishing marginal returns set in for any commodity, this is also true of income itself. It’s both commonsense and basic economics that the utility or wellbeing we each get from each additional dollar of income can be expected to decline with each dollar as it goes from meeting urgent needs to less urgent wants.13

The CIW’s reporting on its own account fails to highlight this. But the problem runs much deeper. Just as the CIW inadvertently gives outlandishly large weighting to citizen involvement in political organisations as a result of ultimately arbitrary methodological decisions, so the deviation between GDP and wider national welfare is a methodological artefact quite independent of any phenomena in the world it aspires to reflect.

To report one’s own wellbeing one does it on a bounded scale – typically from one to ten. Wherever it starts (the median in the Australian Unity Survey is above 7), this number can’t keep growing at a constant rate. GDP, by contrast, is unbounded.14 By my count 52 of the 64 indicators of wellbeing in the CIW are already contained within some bound – usually 1-100%.15

The problem is often perpetuated into the academy proper. Thus for instance, in a consultancy to ANDI in 2013, well respected scholars of social research at the Social Policy Research Centre create the impression of buying into the logic of CIW and ANDI while using words which, if read carefully, need mean no such thing.

“The CIW has attracted considerable attention in Canada and internationally not only because of the innovative ideas, methods and data used in its construction, but also because of the ways in which the findings it has generated have been disseminated. Figure 1 compares movements in CIW and GDP over the period 1994-2010 and highlights the very different picture of progress that is revealed by the two indexes. In overall terms, the 28.9% increase in GDP over the period was around five times greater than the 5.7% growth in CIW. And as the report notes, while GDP declined as a result of the recession induced by the global financial crisis in 2008 by 8.3% but had started to recover by 2010, the decline in CIW was almost three times greater (at 24 per cent), with no sign of recovery evident by 2010. These differences illustrate vividly what difference the measure adopted makes to any conclusions about how a particular society is travelling – over the medium-term but also in response to short-run external shocks.”

Policy implications

“Human knowledge and human power meet in one; for where the cause is not known the effect cannot be produced. Nature to be commanded must be obeyed.” — Francis Bacon, Novum Organum, 1620

An index is a tool for evaluating what is, and so for comparing it to some alternative. However it isn’t built to identify causal links between the phenomena it measures. So while it can highlight emerging issues, it’s of no greater use on its own than anything else in determining how to address it. The reason I argue that something analogous to greenhouse emissions abatement curves should be considered is that they embody causal stories – spend this amount and get this much in return. I think the most promising way to do this is with multiple criteria in mind. So the search would identify a number of actions from most to least promising that we could expect to generate measurable wellbeing benefits whilst promoting existing objectives such as GDP growth. This is partly a political strategy for maximising agreement between people with different priorities. And it’s also a strategy for dealing with our own ignorance on the grounds that one can have greater confidence in policies that satisfy multiple evaluative criteria.

But if we’re serious about promoting wellbeing whether we set our priorities via a ‘wellbeing cost curve’ or some other means, we’ll want to build our knowledge of how to promote wellbeing. That requires us to investigate the causes of improving wellbeing. This suggests an agenda of measuring the wellbeing effects of different policy alternatives. Yet there’s a large gap between what’s been done in this regard compared with the grand pronouncements from on high about new frameworks. I’m unaware of the Australian Treasury taking any interest in such work in the period during which its wellbeing framework was in place. I don’t know of anything being done in New Zealand of this kind, though, as we’ve seen in Part One, the UK What Works Centre for Wellbeing has published guides for managing for wellbeing at the local level.

Investigation of the wellbeing impacts of different policies might be useful in choosing between them, in optimising existing policies’ impact on wellbeing or in designing new policies. In any of these cases, if one were championing a wellbeing index, one would ideally want to measure that impact at the local level in ways that are consistent with the methodology governing the national wellbeing index. Using the HALE as an example, one might capture self-reported wellbeing before and after an intervention using the same standard of self-reported wellbeing around which the HALE is built (the Australian Unity Index survey). If one did so, the HALE methodology could then be used to generate a dollar value of the wellbeing such programs generated.16 This then helps the process of targeting improved wellbeing and informs that process, enabling judgements between relative priorities to be made in that search. It helps both in the choice between programs but also within programs as those delivering them seek to optimise their impact.

Though touted as a policy tool, as something that can fuel “evidence-based and community-focused decision-making” and as something that “places wellbeing at the heart of policy”, the CIW does not make good on these claims at any level of detail. There’s no investigation of the kinds of programs that deliver collateral wellbeing benefits in addition to other outcomes. Instead, we get something with which people will be familiar in our age of TV talking heads and strategy retreats. Big picture thinking. Under the heading “Creating a vision for positive change” we find that:

“the CIW invited 18 people with expertise in one or more of the eight domains to a workshop to reflect on potential policy directions that would enhance the wellbeing of all Canadians. They were asked to consider the findings in this report — … in all domains — to identify connections among domains, and to propose strategies and policy directions that could address multiple challenges simultaneously.

“A central theme that emerged … was inequality, not just in income, but in health, in access to community resources, and in opportunities for leisure and culture. In response, the group recommended policy directions that considered the impact on multiple domains of wellbeing — an innovative and integrated approach to policy that would create multiple benefits for Canadians and reaffirm their core values.”

These are fine sentiments with which I broadly agree, but the work is not in announcing such a vision or “innovative and integrated approach” but in doing the work to show what it is. Alas this isn’t on the cards. The document continues to endorse:

  1. a universal basic income and extension of benefits to low-income Canadians;
  2. build on the strength of the education domain and develop a Pan-Canadian education strategy;
  3. focus on an “upstream” approach to health;
  4. leverage the collaborative power of communities for social change;
  5. provide universal access to leisure and culture; and
  6. improve the collection of social and environmental data.

Some of these approaches – for instance 3 and 4 – are widely supported in the literature, though the latter is often poorly delivered by existing governmental institutions and so it is difficult to bring about by edict, or report. By contrast, a universal basic income is highly contentious both in the community and amongst experts. At least in the short run, it looks like an extremely expensive way to improve wellbeing. Alas there’s no evidence of further exploration of these issues, or of ranking of priorities in the policy roundtable that the CIW reports on.17

Meanwhile, “extending benefits to low income Canadians” is easily agreed to in principle. More generally, the CIW’s policy package looks extremely expensive, but says nothing about how governments would raise the revenue to pay for it, nor about what one imagines are the economic and/or wellbeing costs of doing so, or how to minimise them.

Perhaps most importantly of all, the link between the index and the policy recommendations accompanying it seems both tenuous and highly generalised – lacking in detail. Because of its arbitrary construction, the index offers no considered insights on the relative importance of inequality or even how much it has increased. And even if it did, it sheds little light on how important it is to wellbeing and none on the most effective way of addressing it. The report offers no thoughts on the extent to which wellbeing as measured by the index might be expected to respond to different policies.

Conclusion

As ANDI’s recent report claims, one of ‘key lessons’ from the global progress movement thus far is this:

“Societies need to give urgent consideration to the implications of these new progress measures and how they can be best put into practical application, use and understanding. This may involve some significant changes to current practices, but over time, it is likely to bring many benefits in government planning, policy making and transparency, and provide a better guide to long term development than current measures and decision-making cycles.”

How urgent can you get?


1. As the OECD reports, the ABS Measuring Australia’s Progress project (see Section III below)  was world leading and “an extensive consultation and review was undertaken to decide on the main dimensions (society, economy, governance and environment) and themes that underpin this framework.

2. In fact, as I’ve written once or twice, I’d like us to take the beauty of our built environs more seriously, but judging from the complete absence of this consideration in these kinds of exercises, let’s write it off as a niche eccentricity.

3. Fukuda-Parr, S., 2003. “The Human Development Paradigm: Operationalizing Sen’s Ideas on Capabilities”, Feminist Economics, 9(2):301–317.

4. See this technical paper.

5. Community vitality, democratic engagement, education, environment, healthy populations, leisure and culture, living standards and time use.

6. The Better-Life project itself appears to go to some lengths to avoid publishing a league ladder based on equal weighting between the domains against what one can imagine would be comms directors’ advice (league ladders are invariably lapped up by the media). Sadly, despite it’s best efforts, the media immediately calculate the league ladder in any event, making it the feature of their reporting.

7. In the latest iteration of the CIW, the current eight indicators of democratic engagement are these:
A = Percentage of voter turnout at federal elections
B = Ratio of registered to eligible voters
C = Gap in percentage turnout between older and younger voters
D = Percentage of women in federal Parliament
E = Percentage of Members of Parliament’s office budget devoted to sending communications to constituents
F = Percentage of population that volunteers for a law, advocacy, or political group
G = Percentage of population that is very or fairly satisfied with way democracy works in Canada
H = Percentage of population with a great deal or quite a lot of confidence in federal Parliament.

8. There was an election in May 2011, so perhaps the number was generated in the financial year immediately following it.

9. This indicator wasn’t used in some earlier versions of the index and the information only exists from 2001 (when it is measured at 4.76%). I’ve assumed the methodology would use this figure as the starting 1994 figure.

10. By my calculations if one sums the maximum value they rose to minus the minimum value they fell to since each started out indexed to 100 points in 1994, you get 155.7 points compared with 287.0 points for all eight indicators.

11. It’s a little unclear from the notes, but it looks like the OECD’s Better Life Index may avoid this problem by ‘normalising’ the numbers it uses, though the means by which it normalises the numbers raises further methodological issues by moving scores away from directly observed phenomena towards where those phenomena rank against other countries.

12. The CIW’s 2016 report comments that “[f]rom 1994 to 2014, GDP grew by 38.0%; yet, our wellbeing rose by only 9.9%.” An earlier CIW report was widely reported in its comment that “[d]espite years of prosperity, our economic growth has not translated into similarly significant gains in our overall quality of life”.

13. This was a commonplace of orthodox economics at least within the English speaking tradition  at the turn of the century championed by leading lights like Marshall and Pigou, though the quest for ‘scientific’ foundations for economics has seen it fall into eclipse.

14. Global GDP cannot grow indefinitely without gradually dematerialising so as to be environmentally sustainable, but this subject is not under discussion here.

15. Such important caveats may exist within the bowels of the CIW or ANDI documentation, but I’ve not encountered them.

16. Thus, if a particular program raises the wellbeing of carers by half a point out of 10 one could use the HALE methodology to generate a claim that “at the cost of $x, a program generated wellbeing worth $y”.

17. For instance the paper referred to in the CIW report that endorses universal basic income concludes its executive summary as follows:Lifting all Canadians to at least the poverty line of $21,810 per person (LIM-AT 2016) via a negative income tax is possible but would cost between $49 billion and $177 billion in new spending depending on the clawback rate. Broadly speaking, cancelling existing income transfer programs in favour of a single basic income results either in dramatically higher levels of poverty, or ethically and politically unsupportable compromises where seniors are pushed into poverty to lift up adults and children. The more acceptable and feasible approach would be to set up a new basic income on top of the 33 transfers that already exist, thus creating only winners, though the main beneficiaries would be middle-aged Canadians. To address poverty among other groups requires other strategies. For instance, policies that help increase wages and lower unemployment for youth, as well as better financial support for seniors, will likely be more effective at conquering poverty for those groups than a basic income approach.In the next cited work we read this:If we really want to solve these problems, we should not fixate on a basic income policy. When we compare against the strengths and weaknesses of our current system, we can see that most proposals for a basic income are both prohibitively expensive and leave many people with very low incomes worse off. That’s not a good basis for a massive transformation of social policy.

  • This is very interesting. All the problems set out here confirm my sense that well-being indices are not a good strategic choice to guide the formulation of public policy. A similar dilemma, concerned with the formulation and selection of policy and programmes, was described in 1959 by Charles E. Lindblom in The Science of “Muddling Through” (https://faculty.washington.edu/mccurdy/SciencePolicy/Lindblom%20Muddling%20Through.pdf. He described how various factors that now form part of complex systems thinking, although not unified in a body of work in his day, mean that we can never find all possible options to improve a situation or weigh them against all the criteria we believe matter, two critical assumptions of the methods we use to develop and select policy options.

    I think part of the problem is the assumption that we need to encode matters such as “the relative importance of ‘the environment’ and ‘the economy’” in a unitary index. It clearly does not work. Leaving aside the danger of drifting towards putting a price on a life, in the extreme, and despite the seductive allure of hierarchical weighting schemes and trade-offs expressed in or embodying financial measures, there are fatal flaws in the approach.

    The point is well made that “the relationship between the Big Picture and the details is dialectical, not hierarchical”. The priorities of community stakeholders are not static so only a mechanism that is in close and sustained contact with these stakeholders can gauge the current state of the community let alone maintain its relevance over time. The use in Canada of a reference group of 18 experts is a very narrow channel of communication between the community as a whole and the formulation of policy guidance. In addition, there is no reason to believe that the priorities of those experts at anything but the highest level, uncontroversial descriptions of goodness, will be shared by the community. Then there is the matter of heterogeneity within that community.

    The implication in the conclusion that the key weakness of these ideas is that the implications of well-being indices have not been brought into practical applications misses the point that they are not a good idea in the first place. There are hints of progress in opening the channel between the community and policy development, such as engage.vic.gov.au. The challenge is to scale such communication, sometimes addressed at great cost and with other limitations using sentiment analysis and similar technical solutions, and make space for the diversity of community views so that the way information is gathered does not effectively bias or filter it.

    I do not think the way out of this hole is to dig it deeper and polish the walls as we go deeper but to pull back and reconsider the entire idea of small groups of experts deciding what is in the best interests of large numbers of people. Techniques that go under the general heading of sense making seem likely to provide an alternative although heaven knows how long it will take for anyone in power to have the courage to loosen the reins let alone taking a lead from the community.

    • Nicholas Gruen

      Thanks for your comment Stephen,

      I don’t agree that wellbeing forwards aren’t worth trying. (I say that as one who’s tried.) But they need to be done with an acknowledgement of their profound limitations. And if they can’t then help guide policy not just at the top (abstract nouns) level, but right down to the level of action, then you’re mostly just whistling Dixie.

      • Given all the limitations and misgivings, what do you believe is the alternative?

        • Nicholas Gruen

          Well the HALE index built by Lateral Economics for Fairfax isn’t THE answer, but it is illustrative of the kind of index that could be helpful. If you are serious about saying that we want to target wellbeing rather than GDP, then you start with no-regrets measures where the two reinforce one another. And I think there are oodles of areas which generate such opportunities – an embarrassment of riches it seems to me. I pointed some of them out in the second of these pieces here: https://www.themandarin.com.au/85012-nicholas-gruen-hacking-wellbeing-biggest-returns/

          Once you have an index which makes the (Herculean) assumptions required to provide some commensurability between dollars and utils or wellbeing, the next thing is to build the methodology down to the coalface. When you deliver some program like the NewPin program for young kids for instance, you then seek to measure its impact on the things that your index uses to try to equate dollars and wellbeing. The HALE uses the Australian Unity self-reported wellbeing index and calibrates this against dollars with the income of the person involved in mind. You can do all this and then you can measure the output of the NewPin program or any other program. And you work out what gives you the best wellbeing bang for your buck.

          Of course this is mechanical, as such measurement will mostly be and will have to be if it’s going to ‘commensurate’ wellbeing and dollars. But it gives you a ‘cost curve’ and you can interrogate the methodology at any point to find fault with it and possibly improve it. I hope that makes sense.

          None of this should be regarded as anything other than some provisional rendering of the complexity of life into one hopefully useful abstract domain. In the absence of better ways of ordering relative merit, this gives you something concrete to start with rather than waving your arms around.

          • Sorry if I lost the thread across the three parts of this work. It is certainly interesting.

            I spent a decade or so in industry in the 1980-90s trying to use MAUT to make sense of major technical systems with tangible or measurable attributes, mainly financial and performance characteristics, as well as intangible attributes, mainly relating to client and user perceptions. I came to the conclusion that the decomposition of a figure of merit into a hierarchy of components was quite valuable, what I would now call a type of sense making exercise. However, efforts to map out utilities, characterise trade-offs and choose aggregation rules did not succeed. I could do it myself and reach a consensus with an academic from whom I was taking guidance but once we sought to engage a wider group it would fall apart either because they could not be engaged in the very idea of the process or we started to drown in the diversity of views existing within the groups.

            The only working example we encountered being used in anger was offered by a Russian academic who described hierarchies of factors used to guide public policy decision making. It was all pretty heavy going and methodical. It seemed to rest on a fondness for systems and a systematic approach to policy, a tractor factory for decision making.

            That breakdown at the point where human beings and their opinions become involved seemed to me to be a fatal flaw and I have since come to feel that complexity science might not suffer from the same problem. It took me a few years to turn my mental processes round to adopt the key ideas such as the context influencing the stakeholders and the stakeholders influencing the context, meaning that any assumption of order in anything beyond the short term is unlikely to be borne out. That is why I feel that approaches that accept this as their starting point are likely to be a better strategic option, embedding systematic analyses within an understanding of the disposition of the community.

            I might not live to see this accepted into public sector policy development but there have been a few sparks, mainly of the burning platform variety (nothing else is working so let’s try that). The inertia in the systems analytic approach is so massive that it will take a while to accumulate a critical mass that is self sustaining.