Archives boss: billions going begging if we let data slip away

By Stephen Easton

Wednesday October 29, 2014

Once little more than “a big building with boxes of papers in it”, the National Archives of Australia is now front and centre in government information policy, according to its director-general David Fricker.

Formerly ASIO’s chief information officer and, later, deputy director-general, Fricker (pictured) took up the role at the start of 2012, six months after the federal government’s transition to digital record-keeping began. When it’s over at the end of 2015, all digital government documents will stay in digitised format and the archives will no longer keep hardcopies. This transition has also brought information policy to the fore, and the archives, as the lead agency, along with it.

“We’ve moved right upstream, so rather than being the perceived end of the line for records, we’re now right at the very beginning,” Fricker told The Mandarin. “We’re now having a much more active role in the way records are made, in digital format, the metadata that’s attached to those records, the access and management of those records, and of course their long-term storage and stewardship.”

While storing documents on hard drives is cheaper than in boxes — and efficiency is the driver behind the digital transition policy — “the real payoff” of digital record-keeping is in accessibility, according to Fricker. Access, after all, is what archiving is all about. The value of the information remains the same but the easier the access, the more powerful it becomes, and the possibilities for accessing digital archives are immense.

Fricker imagines a world where well-curated information is widely available through augmented reality, for example, but first a definitive best practice for digital preservation must be developed. As such, he believes the NAA’s digital preservation program should be a higher priority:

“If we do not drastically increase our investment in the preservation of digital government records, this country will have a big black hole of national amnesia …”

“We have to increase our investment by orders of magnitude into digital preservation, especially of government records. Commonwealth government records are just essential. If we do not drastically increase our investment in the preservation of digital government records, this country will have a big black hole of national amnesia and in the decades and centuries to come people will look back at this period of history and think: ‘What the hell were they doing? How did they let all of that slip away?'”

The explosion in digital technology has meant a staggering quantity of information is being captured and created all the time. But it is also being stored, altered, combined, deleted, corrupted and generally treated in ways that make archiving difficult. The purpose of new technology, after all, is generally to increase productivity, not to create strong and reliable data with a long life.

“Paper is patient,” Fricker said. “Paper will sit there and if it remains undisturbed and intact, the integrity is pretty good. Now, we’re creating such a large volume of data and, because technological obsolescence is a necessary part of technological advancement, the data is not surviving from one generation of technology to the next. So while paper is patient, and will wait until the archivist gets it and makes it available to the public, digital data is not so patient and it’ll deteriorate over time. It’s extremely vulnerable to cyber attack, to criminal attack — or just plain neglect.”

Due to the uncertainty over the best ways to store digital records and guarantee their integrity for the long term, some archives around the world still convert digital records to microfilm, on the basis that we will always have light sources, magnifying lenses and eyeballs. The challenge for archivists is to develop similarly robust digital archives, with all the benefits of widespread access via the internet.

Towards a pro-disclosure culture

Open government advocates may be surprised to find that, despite being the former second-in-charge at ASIO, Fricker is on their side. He believes the public service needs to develop a pro-disclosure culture, and says the digital transition policy aims to support that.

“We’ve got to find ways to make it easy for public servants to do the right thing, in terms of creating information, such that if it can be in the public domain, it will be. If it should be protected and kept out of the public domain, it will be. And over time, there will be an orderly release of information into the public and from one day to the next, government will get all the information it needs to make better decisions.”

But what about the massive growth over recent decades in the amount of classified government information, and the associated growth in the percentage of public servants required to hold clearances? “This is a problem,” according to Fricker, and he sees it becoming a more pressing issue as time goes on.

He says information is often held in higher classification levels “just to facilitate the business of government” but that obviously makes it harder to reuse that information or make it public down the track. The archives is already dealing with a major backlog in examining records for release under the 30-year rule — and the rise of routine secrecy will only make this harder.

“In the future, when digital records come up to the open period, this volume of work to be done examining records for release is going to just be huge. So what we need to be doing right now is building much more sophisticated systems for proper marking up of data so that we’re creating data ready for its eventual release at the right time,” said Fricker.

“Everybody wants security. We want government data to be secure because, at the end of the day, it affects the extent to which government can keep me safe and secure, but at the same time, if stuff belongs in the public domain, it shouldn’t be held up, it should be in the public domain.

“I reckon the only way to do that is good record-keeping at the time the information is created with the application of the right metadata, and then the implementation of smart systems that will control that all the way through its lifecycle.”

Thinking bigger, globally

The enthusiastic Fricker became president of the International Council on Archives on October 16, after nominating himself and being elected unopposed in April, and he’s just come back from its annual conference in Spain. He sees international collaboration as vital; the challenging aspects of the digital revolution — like finding reliable, standardised methods of digital preservation — are faced by the global community as a whole.

“When I came into the role in 2012, one of my first duties was to host the four-yearly International Congress of the ICA, in Brisbane,” he recalled. “It was the first time we’d had the congress in the southern hemisphere, so straight away I was deeply immersed in the organisation.” He found the ICA “vigorous, enthusiastic and important” and put his hand up to become president of the organisation’s forum of national archivists straight away.

With about 1000 delegates from 200 countries, the ICA International Congress is “a major, major thing”, according to Fricker, who also sees a role for himself in pushing ahead conversations about what he calls “info-political” issues. The 2012 congress featured former MI5 boss Stella Rimington and Julian Assange’s lead counsel Balthasar Garzon.

“There was fantastic chemistry between them,” he said. “It was perfect. It was exactly the right sort of debate — around big information issues and access issues and why we keep records; what should be in the public domain and what shouldn’t.”

The benefits of good data collection extend beyond the public sector and the “right to know”. Good quality digital archives with data stored in useful, standardised formats can drive innovation and economic growth.

“The opportunity for data mining across Commonwealth data is just enormous,” Fricker said. “It is worth billions of dollars a year to the Australian economy, if we can do a few things. If we can get the data in a format which is usable and reusable, and if we can have some kind of standardisation around the metadata on those Commonwealth records, so that datasets can be combined and meshed together — that requires metadata to make all the data interoperable.

“We need digital continuity standards that mean we’re not creating data which is compatible with the version of the software which I’m using today, we’re creating data which is future-proof … so it’s being built to be discoverable, usable and reusable not for one or two years, but for the next 50 or 500 years.”

About the author
Inline Feedbacks
View all comments