Modernising our national data resources: a never-ending challenge

By David Kalisch

August 28, 2018

This is a lightly edited version of an address given by David W Kalisch, Australian Statistician to the Australian Market and Social Research Society Conference in Melbourne on August 10, 2018.

The Australian Bureau of Statistics, including its predecessor organisation, has now been delivering quality, trusted data to the Australian community for over 110 years.

Over this time, and in many ways, the ABS has modernised what it has done and how it works.

David Kalisch

The ABS statistical program has evolved from the early days where effort was concentrated in population censuses, commerce statistics, production statistics, and vital statistics into other areas including financial statistics, cost of living, labour and wage statistics.

The Commonwealth Bureau of Census and Statistics gradually took on more of the roles previously performed by State Statistical Bureaus, starting with Tasmania in 1924 with other states to follow in 1956. Since this time, the ABS has continued to provide a strong statistical service to all governments, maintaining a strong presence in all states and territories, undertaking national lines of business from all capital cities and now Geelong.

The ABS has been an early adopter of computer technology through the decades, from the first computer installation in 1964, when the ABS actively assisted health and trade Departments to computerise administrative records that could then be used for statistical purposes.

The development of surveys in the late 1960s, alongside developments in statistical methodology, enabled broadening of the social statistics program and provided the backbone for our household-based labour force surveys (first quarterly, and then monthly since 1978).

The “new” ABS Act in 1975 set out the functions of the ABS, the role and responsibilities of the Australian Statistician, and established the Australian Statistics Advisory Council to advise the Minister and the Statistician.

Over the past decade, the ABS has developed its ability to produce new statistical information from linked data sets. Technological developments now enable efficient use of very large data sets. Policy makers and researchers also recognised the opportunities to produce new statistics and information insights from greater use of existing data.

In recent times, the ABS has improved our measurement of the CPI, the labour market, productivity and population estimates, but has also reduced or stopped a number of other statistics in response to persistent Budget cuts over the past decade.

Informing a nation

The ABS purpose is to inform Australia’s important decisions by partnering and innovating to deliver relevant, trusted, objective data, statistics and insights (ABS 2017a). The ABS informs Australia’s important decisions by Governments, by businesses, and by individual Australian citizens in their everyday lives.

ABS data inform fiscal and monetary policy settings, and shape social support programs and infrastructure spending. ABS data inform many pertinent public policy debates, such as jobs and unemployment, housing affordability, income and wealth inequality, cost of living, energy prices, population, the quality of life in our cities and regions, education and health outcomes, needs-based school funding, outcomes for migrants, and so much more.

In addition, ABS data are key to effective functioning of our democracy, with population data helping establish fair electoral boundaries and our official statistics informing choices by voters and political aspirants.

In 2017-18 the ABS released 615 statistical products relating to economic, social, population and environment domains. There were 17.9 million visits to the ABS website and 2.8 million downloads of ABS data. The National Accounts alone publish 12,500 statistical series each quarter. ABS has a significant social media presence compared to other Government agencies, only eclipsed by the BOM weather site.

ABS delivering as much value as we can to the public

The operating environment for the ABS is critical context for what we do and how we work.

National statistical offices are constantly challenged to measure a changing economy, population, society and environment. Globalisation and the digital economy introduce new measurement challenges. We need to further improve our measurement of the growing and increasingly important service sector, which includes health and education. Policy makers and service providers are confronting wicked problems across social policy and the environment that require a more sophisticated evidence base.

Innovation across the data sources we can utilise and our operating methods, alongside ongoing development of our professional expertise will be key to future success. Thankfully, ABS staff enthusiastically support continued innovation of what we do and how we work.

Surveys will nonetheless remain an important part of our operations, because of the insights they produce. Business surveys are generally completed via electronic means, and we are looking to reduce respondent burden where possible while still delivering quality statistics required by business. With household surveys, the ABS is currently achieving world’s best practice, with responses above 90%, but this will become increasingly more difficult to sustain as we confront household availability and attitudes alongside cost constraints.

Most ABS revenues come direct from the annual Commonwealth Government Budget, supplemented by user funding of between 10-20 per cent of our overall revenues. Our Budget Appropriation has declined over the past decade while user funding has increased slightly.

When we consider the public value delivered by the ABS, and where to use our scarce resources, we make choices about our work program so it is most likely to deliver the best outcomes for the nation, and consciously make choices across five main dimensions.

  1. We continue to produce our official national statistics to high standards. Everyone expects us to deliver “perfect” statistics, first time, every time, with extensive public scrutiny of our work by experts and the media.
  2. We also seek to ensure our work program is relevant, and evolves to meet emerging information needs. As noted, our economy, society and environment keep changing, so this is an ongoing challenge.
  3. We put effort into making sure our data are available for important uses, while also making sure we do not compromise the secrecy of individual personal and business information provided to us on trust. Enhanced confidentiality approaches, alongside our Virtual DataLab and adoption of the international Five Safes framework, enable a better balance between increased use and still safe use of sensitive data.
  4. We continuously seek to improve the efficiency of our operations, including data capture. We recognise the burden we place on households and businesses to respond to surveys and we look to capitalise as much as possible from existing and emerging information. We are largely reliant upon voluntary compliance of households and businesses to our surveys that, together with reserve compulsion powers, generally delivers response rates above 90 per cent.
  5. At the same time as we are delivering our current statistical program, we are also building the future capability of the organisation so the ABS is well placed to continue operating as an effective national statistical agency into the future.

While we inevitably do make trade-offs in terms of how we use our available (and generally declining) resources across each of these dimensions, my strong sense is that ABS has delivered more public value across all of these areas over recent years.

We have continued to deliver high quality statistics, reprioritised our statistical program, enhanced data access for users alongside strengthened data secrecy protections, made greater use of available big data options rather than imposing directly on households and businesses, and improved the efficiency and capability of the ABS as an organisation.

We cannot do everything that either data users or we would like, but in the context that we have been operating, this is a remarkable achievement.

Confronting a New Information Age

In 1966, US President Robert Kennedy included a reference in a speech to a Chinese curse along the lines of “may you live in interesting times”. It is a comment I have often made to staff, together with a desire that they should have interesting and challenging jobs.

There are many features of this current information age that have elements of opportunity and curse:

ABS has traditionally made extensive use of government administrative data to produce our national statistics, and this is expanding as more government data collections are assessed as a potential substitute for business and household survey data collections. The increasing cost of traditional survey collection methods is also encouraging this shift, but only where alternative substitute data sources are available.

New data sources are emerging. ABS is using other sources of information such as scanner data and web-scraping for compilation of the CPI. We have an active program considering satellite data, GPS and telco data for producing selected statistics.

Making effective use of new data sources for statistical purposes will be a feature of the future, but does take time and effort as key data users prefer consistent statistical time series. ABS has been building its own expertise and also drawing upon international best practice across respective statistical collections.

We also need to be a discerning consumer of new information, assessing whether data quality is fit for purpose. Some information generated from the internet, or through other means, can be very good, providing accurate information. In other cases, big data will provide a misleading perspective as it is very biased, and this data should not be a key input to our national statistics.

ABS is using machine learning to improve its statistical processes, being further developed through our collaboration with QUT. We are also testing Application Programming Interfaces to enhance our dissemination of statistics to key users. More of our users are accessing ABS data from a range of mobile devices, and via social media.

“ABS has been building its own expertise and also drawing upon international best practice across respective statistical collections.”

Computing capability is now enabling more effective use of very large and linked data sets compared to just a decade ago. New statistical developments have improved the quality of data linkage and introduced new confidentiality approaches to enable safe use of sensitive data without compromising the secrecy of sensitive personal and business information.

New linked data resources (such as the Data Integration Partnership for Australia) increases the opportunity for governments to better design evidence-based policy and service strategies for the community, and more comprehensively evaluate government programs.

While new data sources and methods will enable innovative and constructive use of data for community benefit, privacy and social license to collect and use data will remain a challenging but important issue for us to navigate.

Australians have complex and diverse perspectives towards privacy of information. Australia is quite unusual in having regular debates over privacy with our Census collections since the 1970s, whereas other developed countries that also undertake a five-yearly Census including Canada, New Zealand, the Netherlands and Ireland all retain Census data indefinitely for regulated statistical purposes.

Public sentiment testing through the latest 2016 Census privacy debate in Australia showed that overall the community remained resolute to provide full and accurate Census returns, while a small but vocal proportion of the community were extremely concerned.

Recent ABS transformation

Recognising this challenging and evolving environment in which we operate, the ABS has been pursuing a major transformation since 2015.

This is a wide-ranging and ambitious transformation program, seeking to enhance our organisational effectiveness through improved partnerships, strategy, governance, people, culture and infrastructure.

In 2013, my predecessor, Brian Pink, pointed to the fragile ageing statistical infrastructure that the ABS was using and the risks this introduced to our national statistics (ABS 2013). He initiated internal work to develop a business case, and in 2015 the Government agreed to provide most of the necessary funding for this endeavour. We are in the process of implementing these new systems alongside increased attention to better managing statistical risk, especially for our highest priority statistics.

Risk management has become a more important consideration across our decision making, from our choices about resource allocations, organisational structures, governance forums, advisory arrangements and the allocation of senior staff. Our experience is that effective engagement of our professional skills across ABS, and drawing on external expertise and partnerships as required, will deliver the usual program of around 500-600 quality statistical releases every year.

“Data users need to more actively engage with the community about their data practices over coming years.”

The ABS is now more engaged with a range of data users and data suppliers as we have continued to prioritise our work program. We receive more extensive insights around which statistics are more important than others, how our statistics are used, greater insights around the quality of key data inputs, and anticipated future information requirements.

ABS continues to have a professional workforce, building on our traditionally strong graduate recruitment and professional development. Over recent years, we have broadened our specialist expertise and drawn in staff with a broader range of expertise and organisational experiences, complementing those with deep ABS expertise. Our flexible work arrangements and positive workplace culture assist with staff attraction and retention as data skills become more highly valued across the economy.

While considerable progress has been achieved with the ABS transformation to date, more progress will be necessary as the environment in which we operate is expected to further evolve.

ABS as a learning, innovating organisation

The 2016 Census was a “teachable moment” for the ABS, with some lessons also for many other organisations across Australia and internationally.

The unavailability of the online form for nearly two days slowed down our collection process, temporarily, as ABS actions prioritised the security of Census data provided to us. However, our Census collection was back on its expected trajectory within five days of the August 9 events, and in following weeks we had over a million more Census returns than was originally expected.

Not surprisingly, the media gave little attention to the successful collection of quality Census data by the ABS, yet again in 2016 just like past Censuses. The 2016 Census did not fail and it was not the worst Census ever. 2016 Census data is of comparable quality to past Australian and international Censuses and can be used with confidence (Census Independent Assurance Panel, 2017). There has subsequently been extensive use of the quality 2016 Census data and the new insights it provides about Australia, including by the media.

The ABS owned the 2016 Census process errors, has reflected upon the learnings from this experience and put in place revised operating arrangements across the ABS.

ABS executives have also spoken to many diverse audiences about the 2016 Census experience and the learnings, most of which have applicability beyond the ABS to the public and private sector.

ABS expects that more organisations – across the public and private sector – now have greater understanding of the challenge of cyber security and the risks they may or may not be mitigating effectively.

Broader insights about community expectations, media and communications challenges, risk and issues management, and the importance of leadership and management have applicability to many other organisations.

Drawing upon our 2016 Census experience, and early implementation of these learnings, the ABS delivered an impeccable Australian Marriage Law Postal Survey just one year later. This included improved approaches to customer engagement and communication, procurement, partnerships, cyber security, digital service options, agile work practices and risk and issues management.

People in this audience are probably well placed to understand the challenge of conducting the same-sex marriage survey and the risks that needed to be managed. The Finance Minister, his office, and the partners who helped us with this venture also now have such an understanding, and I think have increased regard for the ABS. On the other hand, I expect the public, who overall had such a smooth and easy personal experience with the survey, have limited understanding of what was required behind the scenes to deliver such an outcome.

Nevertheless, the success of the Australian Marriage Law Postal Survey has helped restore community trust in the ABS. We were able to harness a whole-of-ABS effort and work with many partners to deliver a smooth process and community participation that exceeded community expectations, all in less than 100 days and with savings to taxpayers of $40m.

My prevailing view is that the ABS would not have delivered such a seamless Australian Marriage Law Postal Survey if it had not experienced and learnt from the 2016 Census process dimensions.

Likely future developments for National Statistical Offices

The ABS is one of the world’s most respected National Statistical Offices, delivering essential information for important decisions, as well as contributing to global statistical developments and regional statistical capability.

While it is always somewhat fraught to predict the future, I expect a number of key dimensions will shape how the ABS operates over the next decade and beyond:

Key data users are likely to further increase their expectations and demands. The ABS can further develop its engagement with key users about their information requirements, informing our prioritisation decisions, exploring opportunities for greater collaboration and additional funding. Official statistics should become more valued in the future as a source of truth, compared to assertions and ‘fake news’.

There are a number of areas where our national statistics could be improved, if resources were available. We have not undertaken a time use survey or a survey of mental health for over a decade. Our measurement of the service sector is still relatively modest, given its significance within the Australian economy. Improved national statistics can contribute to improved community understanding of globalisation and structural adjustment. Greater attention could be given to improving the timeliness or frequency of some of our essential national statistics, such as a monthly CPI.

If the ABS continues to be subject to efficiency dividends over the next decade, at the same trajectory as it has for the past decade, some of the core information currently taken for granted by governments, business and the community may no longer be available. Our capacity to continue producing all of the detailed statistics around our labour market, industry activity and population would be increasingly at risk.

New statistical and analytical techniques may be able to draw more value from new and existing data sources, with considerable effort focussing on data integration, confidentiality and big data. ABS should continue to be a discerning user of new information, making judgments around what is fit for purpose. ABS and other National Statistical offices across the world will need to make better use of reliable administrative and commercial data sources, partially substituting for increasingly more difficult and expensive survey approaches. This will require ongoing development of our statistical methods.

In the face of likely proliferation of new data options and increasingly sophisticated ways of using data, there also needs to be a commensurate improvement in the transparency and engagement with the community about how their data is being used for community benefit and protections that underpin its safe use. The ABS should continue to be transparent about its data practices and will need to continue upgrading its multi-faceted security protections, in the face of likely increasing cyber security and other risks. Data users need to more actively engage with the community about their data practices over coming years.

Data skills are expected to be in greater demand across the economy, especially at the more sophisticated end of the data analysis spectrum. More routine data analysis may be increasingly undertaken by software packages. Data expertise required and developed by the ABS will be more highly sought after by more organisations in Australia and overseas, introducing greater challenges for us to attract and retain staff with these essential data skills.

Concluding remarks

Thank you for the opportunity to share with you the journey of the Australia’s national statistical office and the critical role we have to provide essential data for our nation.

I also want to leave you with a sense of optimism and hope as we navigate through this Information Age.

We should not fear good data or its proper use. The professionalism, expertise and integrity of ABS staff, and the organisation as a whole, is an amazing asset to this nation. ABS should continue to develop its professional expertise and protections about our data practices, and be transparent, in order to maintain community trust in and support for what we do.

There will be expanding opportunities to use information for the benefit of the community. I see the prospect of improved information, and improved use of current information. Important public policy decisions, business decisions and community awareness should also benefit from improved information.

I recognise that ABS is not the only organisation across government or the community that provides quality data. My expectation is that we will see a proliferation of more information and information providers into the future, some reliable and some not.

Many of the challenges and opportunities that are facing the ABS, such as demands for more information, the costs of producing quality information, opportunities to leverage new information sources and statistical techniques, the requirement that we operate transparently and maintain community trust, are not unique to the ABS or government.


ABS (2005) Informing a Nation: The evolution of the Australian Bureau of Statistics, 1905-2005

ABS (2013) Annual Report: 2012-13

ABS (2017) Annual Report: 2016-17

ABS (2017) Corporate Plan: 2017-18 to 2020-21

ABS (2017) Forward Work Program: 2017-18

ABS (2018) Submission to the Independent Review of the Australian Public Service, August

Census Independent Assurance Panel (2017) Report on the Quality of 2016 Census Data, published on

Kalisch, D. W. (2016), 2016 Census Learnings, Institute of Public Administration Australia ACT Branch, December

Kalisch, D.W. (2017), 21st Century Leadership in the Public Sector and Learnings of Census 2016, Institute of Public Administration Australia, Tasmania Branch, September

Kalisch, D. W. (2017), Data transformation, Committee for the Economic Development of Australia, Victoria, October

Kalisch, D. W. (2018), Tales from the Corporate Battlefield, Census 2016 and AMLPS 2017, Australian Institute of Company Directors Governance Forum, March

Kalisch, D. W. (2018), Building trust in statistics through communications, Conference of European Statisticians, Geneva, June

Kalisch, D. W. (2018), The importance of reliable statistics to good economic policy making, Australian Conference of Economists, Canberra, July

About the author
Inline Feedbacks
View all comments

The essential resource for effective
public sector professionals