Comparing the Census to alternative data or information: What is the right counterfactual?


Picture: Getty Images

Our inaugural blog post for the Valuing the Census project provided an overview of our strategy for estimating the benefits of the Census to the Australian community.

From here, we’ll be blogging about various issues that we’re grappling with in making this assessment.

To assess the value of the Census, we need to assess the additional value it generates over alternatives in its absence.

These alternatives are counterfactual scenarios to assess the benefits of the Census against.

So, that value is of the extra precision or insight for a given use, compared with the next best alternatives.

(In later blog posts we’ll get into what Census-related data or information means, as well as what the impacts of this extra precision or insight might be.)

But what are the alternatives to the Census?

They might include personal experience, or stylised facts, or the ABS’s survey-based official statistics, or market research data, or many other administrative or non-official data.

Alternatives will obviously vary by issue and use, and not all alternative data will be available to all people (for example, due to commercial or confidentiality restrictions) or be easy for them to understand and/or use.

In a hypothetical world without a Census, would the ABS do something else to fill some of the gaps created?

Would the ABS or other agencies create a new next best alternative source of data for some uses and if so at what cost?

The potential counterfactual scenarios are varied and prompt consideration of a wide range of issues.

For example, consider these hypothetical scenarios:

  1. That data today sourced from the Census because unavailable. Users will need to make best use of whatever alternative data is currently available from ABS or other sources. This saves the Australian government the whole incremental cost of the Census (which were around $500m in 2016).
  2. A Census is conducted less frequently (for example, every ten years as in the US). Census data would become less useful as it became more out-of-date, but it would also reduce costs (presumably  by around half) some of which savings could be invested in expanded ABS surveys to plug the worst gaps.
  3. The traditional Census every five years is replaced with greater use of ongoing large-scale sample surveys (closer to the French rolling census) and integrated administrative data in official statistics.  It is unclear what the costs of this would be, or how quality would be affected.

Our current thinking is that, given the purpose of this exercise, it would be the most simple and straightforward to consider the first scenario that is ‘with’ and ‘without’ the Census.

This reduces the number of permutations and uncertainties in play.

Since our valuation will draw in part from stakeholders’ perspectives on alternatives to Census-related data, the thought experiment needs to be clear and consistent for them, too.

We are hoping that stakeholders will be able to understand this ‘with’ and ‘without’ scenario.

For example, since 2006 the Census has incorporated information of educational attainment (i.e., highest year of school completed or level of highest non-school qualification).

This helps users to investigate the relationship between levels of education and employment outcomes, income and other socioeconomic variables. It is often used as a proxy measure of socioeconomic status.

While various sample surveys also look at this issue, their sample size is too low to give sufficient detail for small population groups and for small regions or localities (for example, with respect to Aboriginal and Torres Strait Islander peoples).

This is the sort of comparison between Census-related and alternative data and information that we’ll be looking to explore further with stakeholders.

In doing this, we acknowledge that simply discontinuing the Census is not a realistic scenario with the kinds of options set out in the second and third scenarios being more likely (and which statistical agencies worldwide are debating).

However, we think it is a reasonable position to take for a first and already challenging Australian assessment of Census value, and one that is not trying to assess the relative desirability of different Census models.

It is also the position taken, with varying degrees of explicitness, in recent UK and NZ analysis on similar topics.

Do you agree with our current thinking?

Are there other approaches we could take that will be both valid and practical?

If so, please let us know by commenting on this post or emailing us at [email protected].

 

About the author
Premium

The essential resource for effective public sector leaders

Check out the Latest