It’s a paradox of the information security world: Everyone knows how important it is to protect systems and data, but most also argue that the value of security initiatives can’t be measured. As a result, security expenses are often prioritized based on gut feeling or the latest headlines.
Some experts say this is no way to run a business, and contributes to security's lack of credibility in communicating to CEOs and boards of directors, many of whom are numbers-driven.
Jack Jones, co-founder of RiskLens and former Nationwide CISO
“It is remarkable how often I run into someone who regurgitates arguments against quantification, but who has never examined the basis for those arguments,” said Jack Jones, former CISO at Nationwide Insurance and now executive vice president of research & development and co-founder of security technology company RiskLens Inc.
“It’s like arguing with a bunch of lemmings.”
Jones believes hard value measurement is possible for these solutions. One such method is risk reduced per unit cost (RRUC).
The method works by comparing the measured value of risk reduction to the amount spent on some particular control, according to Pete Lindstrom, vice president of security strategies at International Data Corp. (IDC), and an advocate of the RRUC approach.
A report by IDC, “Choosing a Security Solution Using Risk Reduced Per Unit Cost,” noted that RRUC “provides enlightening information about security decisions to more appropriately identify the potential return on a security investment.” It offers a means to describe alternative options in a similar way so that enterprises can choose between multifactor authentication and security event management products, for example.
(See "Do the Math" below for sample calculations.)
Even with the inherent ambiguity associated with predicting an uncertain future, the report said, RRUC can formalize the decision-making process, create an opportunity for discussion among stakeholders as they conduct an evaluation, and provide a basis from which to get better at future security decisions.
When virtually every aspect of the business is quantitative...having the CISO give red/yellow/green heat maps is debilitating [to] decision-making.Perhaps most important, IDC said, RRUC “provides a means to test individual components and ultimately the full approach of a security program in the age of digital transformation that is driving value in the face of increased risk, perceived or otherwise.”
To the security professional steeped in conventional wisdom and qualitative decisions, calculating risk may seem like a huge mountain to climb, the report said. “Ultimately, however, it is the correct approach, and the reason why comes when we are trying to compare and contrast resource allocation options for the security program,” it said.
Proponents say RRUC is the way forward for prioritizing security controls. “It operates on business terms and provides the most practical way to determine a security program’s success,” Lindstrom said.
RRUC “can be a very effective approach because it gets to the heart of any investment in security controls,” said Jones.
“The only purpose for any security control is to affect the frequency and/or magnitude of loss,” Jones said. “That being the case, it stands to reason that if we can’t understand, measure, and articulate the fundamental value proposition of an investment, then it seems ludicrous to make the investment.”
Pros and consThe method is not without challenges and weaknesses, partly because of a need for cultural change when it comes to perceptions about risk measurement.
“The hardest part has almost nothing to do with the method itself, but rather with overcoming bad habits in risk measurement and cultural/political barriers thrown up by people who buy into the fallacy that cyber and technology risk is a special snowflake that can’t be measured quantitatively,” Jones said.
It is indeed difficult to develop appropriate assumptions and collect data around risk, Lindstrom said. “But it’s hard because it is unusual, not because it is tedious,” he said. One of the biggest hurdles of RRUC is getting people to understand and believe the information, Lindstrom said.
IDC's Pete Lindstrom
The weaknesses of the method are the imprecise nature of some of the measurements, and the fact that the measurements often don’t make decisions clear, Jones said. There are two key things to keep in mind about these though, he said.
One is that the 'high/medium/low' or 'red/yellow/green' qualitative risk measurements that the risk management profession has used for years “are inherently less precise than anything you’d find in good quantitative measurements, even when data are sparse,” Jones said.
“So quantitative analysis can only be an improvement in that regard.”
Furthermore, in quantitative analysis organizations can faithfully represent the uncertainty in both inputs and outputs using distributions and ranges. “This uncertainty isn’t even part of the conversation when people use qualitative measurements, which means decision makers are blind to whether that “medium” risk — whatever medium means — is based on solid data or weak data.”
With quantitative measurements, if one or more inputs has higher degrees of uncertainty around them, that uncertainty can be expressed and explained so decision-makers can factor that into their decision-making. “This is a little-recognized, but crucial aspect of risk measurement and reporting,” Jones said.
A second factor to consider is that decisions are always based on a balance between opportunity, cost, and risk. “This is true whether we’re talking about cyber risk within an enterprise, or personal decisions like buying a new car,” Jones said. “In these decisions, you have data — sometimes great data, and sometimes weaker data. Regardless, you have data and you apply those data to a model in order to come up with an analytic result.”
This is the case whether it’s through formal quantitative analysis or using a “gut feel” method. “In either case, the decision you make is a function of the results of that ‘analysis’ combined with your personal goals, fears, etc.,” Jones said. “The bottom line is that analysis results, whether quantitative or qualitative, are simply one factor in a decision. They don’t dictate the decision.”
The main point is that the weaknesses in a quantitative approach apply to qualitative as well, and quantitative methods are an improvement regardless, Jones said. “Therefore they can only be viewed [as] weaknesses by people who don’t understand the fundamental nature of measurement and decision-making,” he said.
It might take some time before RRUC is widely used within organizations.
“Many CISOs tend to be conservative and averse to risk — naturally — so anything new and which they don’t understand will be viewed skeptically, particularly if there’s dogma that said it isn’t possible,” Jones said.
“Many of them also tend to be herd animals that gravitate to whatever is viewed as ‘commonly accepted practice,’ regardless of whether the common approach is effective or stands up to scrutiny.”
Fortunately, CISOs also tend to recognize the danger that comes with being out of step within the eyes of the executives they serve, Jones notes. One thing executives can do to open the minds of their CISOs is explain that running any organization is a matter of constantly making economic trade-offs, because there’s always too much to do and there are always limited resources, he said.
“When virtually every other aspect of the business is quantitative in nature—revenue, growth, operational costs, other forms of risk, etc.— having the CISO give them red/yellow/green heat maps is debilitating from a decision-making perspective, and hurts the organization’s ability to succeed,” Jones said.
“As a result, it should simply be viewed as no longer acceptable.”
Executives can also point their CISOs to the growing body of evidence that quantitative methods for cyber risk “are not only feasible, but are clearly the direction of the industry,” Jones said.
“The bottom line is that senior executives should make it clear that if there’s an approach that better supports decision-making, then a CISO who wants to remain relevant and employed had better get on the bus.”
Do the Math: Risk Reduced Per Unit CostCalculating risk reduced per unit cost (RRUC) starts with a more familiar equation, assigning a dollar value to a given risk:
Risk = (potential loss in dollars) * (likelihood of the event)
For example, consider a company that currently faces a $100,000 loss in the event of a virus infection, which is 10% likely to happen — yielding a residual risk of $10,000.
By adding a new security control, the company might cut the likelihood of infection to 5% — yielding a reduction worth $5,000.
If that new security control costs $5,000, then the RRUC equals one — $1 of risk reduced per dollar spent. It's a wash. If the control costs half as much, or if the cost remains the same but the risk is reduced significantly more, then the investment becomes much more appealing.
The following chart from IDC's report compares the RRUC scores for three different solutions that could be applied to a single security risk. Here, alternative 2 is most expensive, but also yields a better RRUC score. It is more efficient than the other two in terms of bang for the buck.
As noted in the article above, both Lindstrom and Jones agree that there is potential imprecision in both the likelihood score and the dollar impact score assigned to security events. However, they argue that this method is still superior to qualitative methods most commonly used today for evaluating and reporting security investments.