In this last article in my 3-part series on statistical compliance auditing, I demonstrate how the statistical methods and sampling tools outlined in Part 1 and Part 2 can be used to create a world-best-practice, risk-based statistical sampling system.
Risk-based compliance auditing
Risk-based compliance auditing starts with an understanding that some types of non-compliance are more serious than others. For example, prescribing the wrong medication to a patient is likely to carry more risk than getting their address wrong.
Based on the above understanding, it stands to reason that high-risk items should be audited more rigorously (sample-size and frequency) than low-risk items. This is the essence of risk-based compliance auditing.
The key benefits of risked-based compliance auditing are
- limited auditing resources are focused on areas of highest risk areas
- better auditng confidence and accuracy for higher risk items
- quicker mitigation and control of high risk non-compliances
Risk-based auditing is now considered world-best-practice by the world’s major standard bodies and accounting firms and is incorporated into most of the popular management standards i.e. ISO 9001, ISO 14001, ISO 2700 etc.
Even smaller enterprises that haven’t traditionally considered themselves highly regulated are starting to appreciate the benefits of risk-based compliance auditing.
Compliance risk assessment
Every risk-based compliance audit starts with a risk assessment of the items to be assessed using a risk matrix similar to that outlined in Figure 1.
The important thing to remember when selecting a risk assessment matrix is that its capable of separating items into different risk-categories based on their potential consequence and likelihood of non-compliance.
Statistical risk-based compliance auditing
Statistical risk-based compliance auditing (SRBA) is a quantitative form of risk-based auditing that uses statistical methods to determine the sample-size and auditing frequency requirements for each risk-category; refer to the following example.
As demonstrated in the following example, statistical risk-based compliance auditing enables enterprises to;
- specify a maximum non-compliance levels for items in different risk-categories i.e. risk appetite
- objectively determine the sample-size and auditing frequency requirements for each risk-category
- reliably assess whether non-compliance levels have exceeded specified limits for each risk-category
- quickly determine the appropriate level of action needed to mitigate identified non-compliance risks
Example: ABC Insurance Ltd
Jacob is responsible for assessing the extent to which his company’s 20,000 computers and personnel devices comply with prescribed cybersecurity policies and requirements.
Because inspecting every device is impractical, Jacob creates an SRBA system that will enable him to mainly focus his limited resources on those devices with the highest cybersecurity risk. He’ll still audit lower-risk devices; just at a lower level.
To begin, Jacob uses the 4 x 4 risk assessment matrix outlined in Figure 1 to assign each type of device a risk-category i.e. High, Medium, Low. Past compliance records and subject-matter-expert advice are used to complete the risk assessment.
Next, each risk-category is assigned a confidence level, confidence interval, audit frequency and maximum non-compliance limit ( i.e. risk appetite); which is developed in consultation with senior management.
Finally, a statistical sampling tool (Refer to Part 2 ) is used to calculate the sample-size requirement for each risk-category; refer to Table 1 below.
| Risk Category | Estimated Population | Estimated NCR | Confidence Level | Confidence Interval | Audit Frequency | Sample-Size* |
| High | 1,000 | 5% | 99% | 2% | 6 mth | 441 |
| Medium | 6,000 | 8% | 95% | 3% | 12 mth | 299 |
| Low | 13,000 | 12% | 90% | 5% | 24 mth | 114 |
Table 1: SRBA Sampling Inputs and Outputs
To complete the data-collection process a random sample of devices from each risk-category is assessed for compliance with the company’s cybersecurity policies and requirements.
Audit analysis
Once all the audit data has been collected the total number of non-compliant devices in each risk-category sample is summated and used to estimate the percentage of non-compliant devices in the larger population; refer to Table 2 below.
| Risk Category | Sample- Size* | Total Non-Compliant Items | Percent Non-Compliance^ |
| High | 441 | 29 | 7% |
| Medium | 299 | 40 | 13% |
| Low | 115 | 21 | 18% |
Table 2: Audit Results Analysis
Jacob is able to include the following statements in his cyber-security report;
- There is a 99% probability that 5% to 9%. of high risk-category devices are non-compliant (i.e. 7% +/- 2%)
- There is a 95% probability that 10% to 16% of medium risk-category devices are non-compliant (i.e. 13% +/- 3%)
- There is a 90% probability that 13% to 23% of low risk-category devices are non-compliant (i.e. 18% +/- 5%)
Risk-based decision making
Using the maximum non-compliance limits set by senior-management, Jacob is able to make a series of recommendations in relation to what level of corrective actions, if any, is needed to address the non-compliances identified in each risk-category; refer to Table 3.
| Risk Category | Maximum Non-Compliance Limit# | Percent Non-Compliance^ | Action Required? |
| High | 2% | 7% | Yes |
| Medium | 10% | 13% | Yes |
| Low | 20% | 18% | No |
Table 3: ABC Insurance’s Non-Compliance Risk Limits
ExaminingTable 3, it becomes immediately clear the maximum non-compliance limit for high-risk and medium-risk devices has been exceeded; but not for low-risk devices. Based on this insight Jacob is able to include the following objective recommendations in his cybersecurity report.
- All high-risk devices are immediately inspected to identify and correct cybersecurity non-compliances
- Before any corective action is taken further sample of medium-risk devices is inspected to achieve a more accurate audit result i.e. confidence level = 99%, confidence interval = 2%.
- Owners of low-risk devices are instructed on how to esnure their devices comply with company cybersecuurity requirements
Conclusion
In this 3-part series on statistical compliance auditing, I have demonstrated how easy it is for enterprises to use these methods to improve the efficiency and effectiveness of their compliance audits.
Ignoring the statistics behind compliance auditing doesn’t make them go away. Luckily, making the transition to statistical compliance auditing is not as difficult as a lot of people think. Especially, with the help of an online sampling tool; as outlined in Part 2.
If you’d like to know more about how your enterprise can make the transition to statistical compliance auditing contact us at www.compliance-master.com.
