There’s more information flowing around than ever before. But finding ways of recording this data, while critical in its own right, is not enough.

To truly understand its citizens, governments need to think about both internal and external sources of data and consider how this information can be leveraged to best serve citizens.

The changing paradigm

For governments trying to provide better services for citizens, there are several challenges in their path to effective big data analysis, according to Murray Bruce, director of IBM Analytics Industry Solutions, Public Sector.

Historically, Bruce says, governments generally moved to centralise information and create a single source of data that they would base their decisions on. But this is starting to change and departments are looking at other sources for information.

“What we’re seeing is a move from internal data to external data sourced from other government departments and the private sector,” he explained.

The shifting paradigm is also prevalent in the types of information which is being gathered. Instead of drawing solely upon “structured” data (think of databases built on completed forms), agencies are now moving to blend this with “unstructured” data sources.

“When you’re looking at unstructured data it includes everything from video feeds and CCTV cameras, to Facebook and Twitter,” said Bruce. “It could also be the records of when I purchased and sold a property, or the invoices for what I’m claiming, certificates from an employer or service orders.”

Bruce warns that you can’t solve a problem using structured or unstructured data alone.

“What you really want to do is bring the information to the knowledge worker and allow them to make the right decisions,” he said.

Cognitive computing, which involves systems with the ability to learn and interact with people, such as IBM Watson technology, is a powerful complementary tool, says Bruce.

“It brings the structured data and the unstructured data together and enables you to solve more problems. It’s like adding another 50% to your tool kit and brings you much more flexibility in how you solve problems for clients,” he said.

How big data can serve citizens

Establishing a coherent view of a citizen is one of the big advantages of data analysis and insight, says Bruce.

Previously, if you were moving interstate, you would need to change your home address, vehicle registration and Medicare details separately. Now the capability exists for the government to deliver these changes in a seamless way, says Bruce.

Many tax agencies in the region are developing systems making it easier for citizens to fill out their tax returns. “They’re getting their money in quicker and they’re becoming a simpler and better way to deal with the taxpayer,” said Bruce.

“What if the tax agencies knew the relevant information about you and they created a tax return for you and they pre-populated it and pushed it out to you pre-completed? It would take a fraction of the amount of time to complete your tax return.

“It’s about master data management and that is a deterministic view of who each person is. If you don’t have your data managed and you’re not sure of who each person is, you can never bring it together to do that.”

The case for better data management

The core business case for instituting big data changes is primarily about serving citizens better, says Bruce. But tracking finances and assisting with national security are also prime benefits.

“One part of the business case is about collecting and spending money,” said Bruce. “How you make sure you’re only paying what you should be, or collecting what you should.”

Agencies are becoming much more adept at detecting fraud and ensuring compliance thanks to big data, says Bruce.

“They’re seeing strong increases in the number of cases they’re prosecuting,” he said. “There’s also far fewer instances of citizens saying they’ve already provided the information they’ve been asked for.”

Big data analysis also has a big part to play in national security, not just in terms of identifying potential threats, but in reducing impediments for everyday citizens.

“In terms of national security, how do I make citizens who are doing the right thing, not even see us,” Bruce said. “But for someone who has indicators of concern, we can track them down and make sure the right legal processes are adhered to.

“By instituting these changes you are serving the clients and it’s cheaper since you require fewer resources.”

Utility v agility

When installing new capabilities, organisations are presented with a dilemma: do you go with utility (build it once and use it many times) or agility (build it once and build it quickly)?

“Some clients deliver the utility, but they find out they don’t have many uses they can benefit from it,” he said. “On the other hand we have some clients who have built up some good quick wins, but they don’t have the fundamentals under it, so the data isn’t good quality and people don’t trust it.”

Building data infrastructure is all about striking this balance, says Bruce, and a number of organisations are adapting approaches which enable them to deliver sustainable business benefit.

“You can’t subscribe to one camp or the other, you need to balance between utility and agility and that requires some solid decision making to work out how to balance priorities,” said Bruce. “But if you don’t have the right pieces in place, you keep building new capabilities on questionable foundations.”

Culture and computing

Just building new capabilities won’t solve the problem your organisation is facing singlehandedly — the transformation goes hand-in-hand with a shift in the ways agencies work with data.

“We see lots of organisations who have embraced that change and are leading,” Bruce said. “You go to some of the leading agencies and they’re being very proactive and matching best practice in the private sector.

“It’s not just technology, it’s the way they run the processes and the culture to use the information properly, and changing that takes time.”

The important thing, Bruce says, is to remember that managing big data is not about managing technology — technology is an enabler that allows you to solve problems faster, cheaper and more effectively.

Written by Jacob Robinson