Part 3: How To Create A Risk-Based Statistical Compliance Auditing System

In this last article in my 3-part series on statistical compliance auditing, I demonstrate how the statistical methods and sampling tools outlined in Part 1 and Part 2 can be used to create a world-best-practice, risk-based statistical sampling system.

Risk-based compliance auditing

Risk-based compliance auditing starts with an understanding that some types of non-compliance are more serious than others.   For example, prescribing the wrong medication to a patient is likely to carry more risk than getting their address wrong.

Based on the above understanding, it stands to reason that high-risk items should be audited more rigorously (sample-size and frequency) than low-risk items. This is the essence of risk-based compliance auditing.

The key benefits of risked-based compliance auditing are

  • limited auditing resources are focused on areas of highest risk areas
  • better auditng confidence and accuracy for higher risk items
  • quicker mitigation and control of high risk non-compliances

Risk-based auditing is now considered world-best-practice by the world’s major standard bodies and accounting firms and is incorporated into most of the popular management standards i.e. ISO 9001, ISO 14001, ISO 2700 etc.

Even smaller enterprises that haven’t traditionally considered themselves highly regulated are starting to appreciate the benefits of risk-based compliance auditing.    

Compliance risk assessment

Every risk-based compliance audit starts with a risk assessment of the items to be assessed using a risk matrix similar to that outlined in Figure 1.

The important thing to remember when selecting a risk assessment matrix is that its capable of separating items into different risk-categories based on their potential consequence and likelihood of non-compliance.

Figure 1: Example, Risk Assessment Matrix

Statistical risk-based compliance auditing

Statistical risk-based compliance auditing (SRBA) is a quantitative form of risk-based auditing that uses statistical methods to determine the sample-size and auditing frequency requirements for each risk-category; refer to the following example.

As demonstrated in the following example, statistical risk-based compliance auditing enables enterprises to;

  • specify a maximum non-compliance levels for items in different risk-categories i.e. risk appetite
  • objectively determine the sample-size and auditing frequency requirements for each risk-category
  • reliably assess whether non-compliance levels have exceeded specified limits for each risk-category
  • quickly determine the appropriate level of action needed to mitigate identified non-compliance risks

Example: ABC Insurance Ltd

Jacob is responsible for assessing the extent to which his company’s 20,000 computers and personnel devices comply with prescribed cybersecurity policies and requirements.

Because inspecting every device is impractical, Jacob creates an SRBA system that will enable him to mainly focus his limited resources on those devices with the highest cybersecurity risk.  He’ll still audit lower-risk devices; just at a lower level.

To begin, Jacob uses the 4 x 4 risk assessment matrix outlined in Figure 1 to assign each type of device a risk-category i.e. High, Medium, Low. Past compliance records and subject-matter-expert advice are used to complete the risk assessment.

Next, each risk-category is assigned a confidence level, confidence interval, audit frequency and maximum non-compliance limit ( i.e. risk appetite); which is developed in consultation with senior management.

Finally, a statistical sampling tool (Refer to Part 2 ) is used to calculate the sample-size requirement for each risk-category; refer to Table 1 below.

Risk  CategoryEstimated PopulationEstimated NCRConfidence LevelConfidence IntervalAudit FrequencySample-Size*
High1,0005%99%2%6 mth441
Medium6,0008%95%3%12 mth299
Low13,00012%90%5%24 mth114

Table 1: SRBA Sampling Inputs and Outputs

To complete the data-collection process a random sample of devices from each risk-category is assessed for compliance with the company’s cybersecurity policies and requirements.

Audit analysis

Once all the audit data has been collected the total number of non-compliant devices in each risk-category sample is summated and used to estimate the percentage of non-compliant devices in the larger population;  refer to Table 2 below.

Risk
  Category
Sample-
Size*
Total
Non-Compliant Items
Percent
Non-Compliance^
High441297%
Medium2994013%
Low1152118%

Table 2: Audit Results Analysis

Jacob is able to include the following statements in his cyber-security report;

  1. There is a 99% probability that 5% to 9%. of high risk-category devices are non-compliant (i.e. 7% +/- 2%)
  2. There is a 95% probability that 10% to 16% of medium risk-category devices are non-compliant (i.e. 13% +/- 3%)
  3. There is a 90% probability that 13% to 23% of low risk-category devices are non-compliant (i.e. 18% +/- 5%)

Risk-based decision making

Using the maximum non-compliance limits set by senior-management, Jacob is able to make a series of recommendations in relation to what level of corrective actions, if any, is needed to address the non-compliances identified in each risk-category; refer to Table 3.

Risk Category Maximum
Non-Compliance Limit#
Percent
Non-Compliance^
Action
Required?
High2%7%Yes
Medium10%13%Yes
Low20%18%No

Table 3: ABC Insurance’s Non-Compliance Risk Limits

ExaminingTable 3, it becomes immediately clear the maximum non-compliance limit for high-risk and medium-risk devices has been exceeded; but not for low-risk devices. Based on this insight Jacob is able to include the following objective recommendations in his cybersecurity report.

  1. All high-risk devices are immediately inspected to identify and correct cybersecurity non-compliances
  2. Before any corective action is taken further sample of medium-risk devices is inspected to achieve a more accurate audit result i.e. confidence level = 99%, confidence interval = 2%.
  3. Owners of low-risk devices are instructed on how to esnure their devices comply with company cybersecuurity requirements

Conclusion

In this 3-part series on statistical compliance auditing, I have demonstrated how easy it is for enterprises to use these methods to improve the efficiency and effectiveness of their compliance audits.

Ignoring the statistics behind compliance auditing doesn’t make them go away.   Luckily, making the transition to statistical compliance auditing is not as difficult as a lot of people think. Especially, with the help of an online sampling tool; as outlined in Part 2.

If you’d like to know more about how your enterprise can make the transition to statistical compliance auditing contact us at www.compliance-master.com