How Do You Measure Security?

How Do You Measure Security?

Friday 29th September 2017

Security, or at least successful security, can be measured in many different ways, with the intended final result being "Confidence" (also called Assurance).

The extent of confidence in any security system is based on so many factors:

  • How well is it built?
  • What is its track record in providing adequate defences?
  • Who built it?
  • What testing has been done to prove it?
  • What experts have been used to assess it?
  • If things have gone wrong with it, do they get corrected really quickly?
  • Theoretically, how good does it look?
  • Has it been built from known and trusted components?
  • How do people feel about it?
  • Do it only do what we need it to do, or are there lots of extra redundant features we don't know about?

All of the above produce good answers, but not one of these answers in itself provides the full extent of confidence in the security of a "thing."

Attack Path Analysis

Take for instance Attack Path Analysis; this is a mechanism that can be used to look over the structure and implementation of a system in a piecemeal fashion in order to determine whether there are any weak paths that would allow compromise. Attack Path Analysis is a technique that we personally use quite extensively.

Attack Path Analysis will plot out various combinations of possible attack on a system and determine whether an attacker can get a foothold on the system, compromise it and then achieve their end result (money gain, data theft, denial of service etc.).

By using an Attack Path Analysis an attacker can determine their route(s) to success, but this technique can also be used to help is assessing the extent of Confidence someone can have in their own systems.

There is a significant problem in only using Attack Path Analysis though; for an attacker they only need to establish one route to achieve an attack, but for a system owner they need to establish all possible routes against which to defend;this results in a very costly exercise if this is the only security measurement technique to be used.

Extrinsic Assurance

Other techniques give different types of confidence. Extrinsic Assurance is another term sometimes used to describe the amount of testing and proving that is done on a system, and this approach allows a system owner to know how their system respond to all of the tests being thrown at it. Extrinsic Assurance includes techniques such as vulnerability hunting, penetration testing, boundary value testing, soak testing and many more.

Extrinsic Assurance in itself only describes the confidence that one can have in a system based upon the actual tests that were performed. Despite any desire to do so, test results cannot be extrapolated beyond the tests that were performed. So these tests are not the silver bullet for security measurement despite the fact they produce really important facts about a system's behaviour.

Intrinsic Assurance

Another security measurement looks at the Intrinsic Assurance nature of how a system is built. This can include factors such as how qualified are the engineers building the system, how much expert review is built into the development lifecycles, how much control on the stability and change management cycles is exercised and what quality regime is the system built under.

Although Intrinsic Assurances do not provide solid facts about how the system behaves or can be attacked, it provides more of a qualitative "warm feeling" that the system is being built well and being built by people who know what they're doing. Intrinsic Assurances are used quite extensively to give "Brand Value" to a product - the instinctive view of buying a product because the company is good is a great marketing tool here.

Another, less expensive means of providing security success and measurement results is to actually measure how good the system has performed over time. This would typically result in metrics that show that "system A had 180 attacks in a 2 week period, and 179 were successfully defended" or that "system fixes were developed and fixed within a 2 hour window for 98% of discovered faults." Metrics such as these give further confidence that the system is behaving well or can be corrected within certain performance windows.

Looking at many of the approaches to measuring security, it's quite clear that there is no one approach that sorts out all problems. However, covering all approaches to the full extent can be very expensive and it's not always hard facts that win over qualitative assessments.

Given that dichotomy, security assurance should be based on a balanced view of each of the elements of Intrinsic, Extrinsic, Operational and Implementation Assurances that are pertinent, the budget available, the value of information being protected and nature of the threat being faced.