THE HORIZON is a curated guide to published research, books, reports and podcasts with utility for practitioners in the public sector. With Australia’s public sector celebrating Innovation Month in July, throughout this month Mandarin Premium is looking at research and thought leadership on public sector innovation.
Measuring value of your innovation projects
Let’s get practical. Leaders need to know public value is being created from any taxpayer-funded project. Every jurisdiction has its own internal red-tape to ensure value for taxpayers, and they may not fully capture the value created by innovation projects, but there are many tools that can.
Douglas Hubbard‘s book series How to Measure Anything is an ideal text for non-statisticians who need to know everything from where to start, what data collection traps to watch out for, and most importantly how to quantify the ‘immeasurables’ usually excluded from return on investment calculations. He focuses on his own Applied Information Economics methodology, which incorporates a combination of quantitative analysis including Bayesian analysis and Monte Carlo simulations. I like it because you can download the Value of Information Analysis tools straight into a spreadsheet.
Take an example from the US Environmental Protection Agency (EPA), which contemplating a combined spend of US$3.5 million on three proposed system modifications to the Safe Drinking Waters Information System (SDWIS), the central system for tracking drinking water safety in the United States and ensuring quick response to health hazards. It needed to prove which of these improvements was really justified and, of those, the best priority. Here’s how they broke down the improved public health value of the three modifications using a spreadsheet model:
The Office of Management and Budget already required the EPA to produce economic arguments for any proposed environmental policy, additionally the EPA would also compute costs of compliance and benefits to the public for each policy it wanted to enforce. From this they already had the economic impact of different types of the the common drinking water contamination, but the EPA often resorted to a willingness-to-pay argument, using only workdays lost in calculating the cost of contamination. They ran workshops with EPA managers, where it became apparent they were used to considering impacts in isolated systems, not as part of a whole. Some had to be dragged kicking and screaming into the process, and didn’t want to pull people away from what they were doing to do a study like this. The end result was translating software to environmental and health impacts, with one deputy chief information officer noting: “The fact that software modules could be traced through a chain of events to some benefit to the public was assumed but never quantified. I think people were frankly stunned anyone could make that connection.”
Jane Keathley, a member of the American Society for Quality’s board of directors and former chair of its innovation division, says well-thought-out value measurements can manage the high levels of uncertainty inherent in innovation. There are many value analysis tools out there, including the value matrix, value factor analysis, the innovation ambition matrix, the innovation maturity model, the willingness to pay model and the value analysis field of engineering. Keathley has written a summary of each in the March 2019 edition of Quality Progress journal, and adds that measurements of value can help us understand where value is lacking or missing so we can identify pain points and opportunities for innovation:
“For example, in a challenging situation such as getting school children from the drop-off point to their classrooms each morning in an orderly fashion, a value assessment of each activity from drop-off to “in their seats” will yield insights into the lowest value activities: Those activities that—if modified—could add the most value to the process. Perhaps it’s the way the students are initially greeted, the route taken to their classrooms, or the way their backpacks and coats are handled. Value analysis, perhaps using a value stream or business process map, can help identify the key points at which things break down. These are the best opportunities for innovating and adding value.
“When devising innovative solutions, just as with any problem-solving challenge, it’s good to have multiple choices. As you gather information and data about these multiple solutions, value analysis can help you compare them and narrow the list to those with the best value proposition, which can be further developed.
“As you refine and hone your selected solution, assess the value of each change to ensure you continue to create and enhance value, not detract from it by errantly veering away from or overdeveloping the planned value. Remember: Users like simplicity and dislike complexity.
“For higher-level decisions—such as strategic directions, pro-gram portfolios and supplier relationships—value analysis helps set direction and align organisational initiatives around those that will provide the greatest innovative benefit.”
Benefits of measuring value
|Element of innovation||Benefit of value measures|
|Understand the problem or need||+ Identify trends and patterns that indicate opportunities.|
|+ Develop clear picture of where the problem lies.|
|Identify problem-solving solutions||+ Uncover the problem areas with biggest impacts.|
|+ Evaluate relative benefits of potential solutions.|
|Compare and prioritise alternative solutions||+ Identify solution most likely to succeed.|
|+ Understand benefits versus costs of solutions.|
|Monitor solution development||+ Avoid drift away from the optimal solution.|
|+ Continue to refine solution for best user experience.|
|Deploy the solution||+ Make additional value-adding refinements.|
|+ Identify new opportunities.|
|Develop organisational strategies||+ Evaluate value of strategic initiatives to meet organisational excellence goals.|
|+ Identify innovative solutions for key strategic challenges.|
How not to analyse value
Keathley also warns what not to do when making value assessments:
- Vanity metrics and gut feelings. It’s easy to come to conclusions that you or a project sponsor want to hear, but these metrics can lead away from the hard truths of value assessment.
- Impersonal methods of surveying the customer. A survey can be helpful, but can miss the most pressing issues unless carefully designed. Instead, work and walk with customers to understand first-hand the problems they are experiencing.
- Thinking short-term. Longer-term innovations are higher risk, but they also provide higher gains. Consider a mix of both short and long-term initiatives.
- Big data. The potential is well known, but is costly in many ways (time, skills, technology, security management) and is only useful if the data is of high quality. Have a clear business objective, source the appropriate and relevant data, and pay careful attention to the information that will get the answers you need — don’t go beyond that.
For leaders: ask better questions
Geoff Tuff and Steven Goldbach, both from Deloitte, have authored a book called Detonate: Why – and How – Corporations need to Blow up Best Practices (and Bring a Beginner’s Mind) to Survive (Wiley, 2018). Originally I picked it up because of how dependent the public sector is on someone else telling it what best practice is, then slavishly following that guidance regardless of local circumstances and rarely testing better ideas from the shop floor. Then I noticed they addressed the power of leaders’ questions, specifically that executive questions have the ability to immediately change behaviour, intentionally or not: Miscommunications happen from thinking out loud in meetings, or overly hasty responses to emails.
Bad question: What’s the ROI on this investment?
“When people hear this question, their immediate stance becomes one of defending the investment rather than considering it, and they feel in the hot seat of making it their problem, rather than the organisation’s problem. So instead of feeling as if they are getting to a better answer, team members feel their backs are against a wall. Hear this question enough, and you’ll learn to come in armed with all the facts and data you can imagine to prove your position. Now, bludgeoned with facts and data, the executives are overwhelmed with PowerPoint slides and can’t make heads or tails of the rationale for the idea in the first place. We’ve created a vicious cycle of questions creating defensiveness and colossal waste while rarely getting to the actual conditions that drive results in the real world…”
Bad question: Has anyone else in our industry done this before?
“This question kills innovation, but we see it all the time. In many situations, it’s about assessing risk – not about risk that exists the marketplace, but the personal risk that decision makers face. It’s about preserving “Me Inc.” This question is really asking, “Am I going to look crazy for doing this?” or “How hard is this going to be to get through the organisation?” Those are legitimate concerns that the organisation can address through company dynamics and culture, not through asking this question, which creates unnecessary and wasteful work. It usually leads to something to the effect of “let’s have an offline conversation,” the kiss of death in any meeting…”
Bad question: How can we prove this will work?
“Don’t ask for proof. In fact, if you’re a senior executive, eliminate this word from your lexicon. There’s no such thing as proof in business, especially in a highly uncertain world … Given their desire not to be embarrassed in front of their boss, they hear “proof,” and they come back to you trying to be “bulletproof.” If you’re looking for more information, simply ask, “How could we learn more about this?” This clarifies that you, too, are in exploratory mode, and it puts you as part of that journey…”
Good question: What might be another possible way to tackle this problem?
“…by forcing team members or the organisation to consider how they might solve a problem another way, it also forces them to come back to clarify what problem they are trying to solve.”
Good question: Which customers will love this? which customers will hate this?
“Many organisations fall prey to being internally focused, and questions that cause them to ask questions about why something will resolve a customer need is good.”
Good question: What behaviour are we trying to change?
“Whether it’s the behaviour of a prospective customer or a line employee, it’s always critical to know what’s the purpose of your actions, and most of the time, it is someone’s behaviour.”
Good question: How might we move faster? if we had to try something today, what would we try?
“It separates the need-to-have from the nice-to-have, and it brings into focus what a minimally viable move might look like.”
Public value is co-created
Our research director Maria Katsonis also looked at putting value creation back into public value, based on a working paper from Mariana Mazzucato and Josh Ryan-Collins.
“The paper proposes that public value is collectively generated by a range of stakeholders including the market, the state and civil society. The emphasis is on value creation at the core – not just public value but value itself. In this view value is co-created rather than created inside business and only facilitated by the public sector.”
Save the date: project success by design
Project Governance and Control Symposium; August 20-21, Canberra: the PGCS is one of the rare multi-day conferences for government types where the discussion is immediately beneficial for both project managers and executives. This year’s theme is project success by design. Attended by defence and infrastructure sector, central agency and coordination leaders, contractors and consultants, project leaders across government.
Join Australia’s public sector leaders at Mandarin Premium. Only $5 a week for a limited time*.
A year ago, after listening to the views of hundreds of Mandarin readers, we launched Mandarin Premium.
Our readers told us they had an appetite for another layer of Mandarin journalism that went well beyond the news – to analyse the nuts and bolts of managing and implementing government policy.
Put simply, our readers wanted something that understood their jobs and would help with it.
And that’s exactly what Mandarin Premium is and does.
We need your support.
For a limited time, sign up to Mandarin Premium for a year at only $5 a week. That’s just $260 a year, and a saving of $180.
Join today by using promo code 5AWEEK at checkout.