In Part 1 in this 3-part series on statistical compliance auditing I outline why enterprises should avoid using ad-hoc and rule-of-thumb methods to determine the sample-size requirements for their compliance audits and instead use tried-and-tested statistical sampling methods.
I also outlined how anyone with a basic understanding of some simple statistical terms and concepts, and help from an online sampling tool, can use these methods to design a highly efficient and effective statistical compliance auditing plan.
Statistical sampling terms and concepts
Before you begin using any online statistical sampling tool you will need to familiarise yourself with the following statistical terms and concepts;
- Population – the total number of produced items (products, services, activities, data, records, reports, etc.) from which the sample is to be selected. An estimate is satisfactory for most compliance audits.
- Population Parameter – the characteristic to be assessed. Depending on your perspective this will be either the “percentage of compliant items” or “percentage non-compliant items” in the population.
- Estimated Population Variation – an estimate of the population’s compliance or non-compliance level. To the uninitiated, this might sound strange but before you can use a sampling tool to calculate the sample-size for a compliance audit you will need to estimate the level of compliance or non-compliance in the population. You can do this using past assessment results, subject-matter-expert advice, or the outcome from a small initial test audit.
- Sample-Size – total number of items to be randomly selected from the population which are then assessed for compliance with specified requirements.
- Confidence Level – the probability that the sample statistic closely approximates the parameter of the entire population. Sometimes referred to as audit reliability. This number (90%, 95% or 99% ) is entered into a sampling tool together with the confidence limit (refer below) to calculate the required sample-size; refer to the example below.
- Confidence Limit – the range of values about the sample statistic which contains the true value of the population. Sometimes referred to as confidence interval or precision e.g. +/- 5%. This number is entered into a sampling tool together with the Confidence Level (refer above) to calculate the required sample-size; refer to the example below.
- Sample (Audit) Result – the measure of the sample that is used to make assumptions about the population parameter. This result is calculated by dividing the total number of compliant/non-compliant items in the sample by the total sample-size and multiplying the result by 100; refer to the example below.
For example, an audit result of 91% compliance obtained with a confidence level of 90% and a compliance limit of 5% means there is a 90% chance that the true population parameter resides somewhere between 86% and 96% i.e. 91% +/- 5%.
Selecting a suitable compliance sampling tool?
There are a lot of statistical sampling tools that can be found on the web capable of calculating the optimum sample-size for a compliance audit but for the purposes of this demonstration, I’ll be using the free sampling tool provided on the Compliance Master website.
The Compliance Master sampling tool is designed specifically for attribute type audits only. With an attribute type audit, the result is always binary i.e. “yes” or “no”; which lends itself nicely to compliance auditing.
With an attribute type compliance audit, the total number of compliant items found in a sample can be used to estimate the percentage of compliant items in the population = compliant items/sample size x 100.
The other is a variables audit which deals with variations in a measurement common to the items in a population i.e. height, length, colour, volume, time, etc. These types of audits are mostly used in quality control. Accordingly, the following examples deal only with attribute compliance sampling.
Example: Statistical sampling tool
Patricia works for ACME Health, a medium-sized health services company. Part of Patricia’s role is to produce an annual report including an estimate of the percentage of services supplied to customers that complied with government and company requirements.
Because Patricia doesn’t have the resources required to check every service supplied to customers (ACME Heath services over 20,000 customers per annum) Patricia decides to estimate the true compliance rate by examining a sample of randomly selected service records.
Patricia uses the Compliance Master statistical sampling tool to calculate and compare the sample the sample-size requirements for six confidence level and confidence limit combinations; refer below.
Sampling Tool Inputs:
Population size = 20,000
Estimated non-compliance = 6% (based on past audit results)
Sample-Size Options:
| Audit Option | Confidence Level (Reliability) | Confidence Limit (Accuracy) | Calculated Sample-Size |
| 1 | 90% | 5% | 61 |
| 2 | 95% | 5% | 86 |
| 3 | 99% | 5% | 149 |
| 4 | 90% | 2% | 377 |
| 5 | 95% | 2% | 527 |
| 6 | 99% | 2% | 896 |
Patricia quickly dismisses Options 5 and 6 because they fall outside her budget and time constraints. Options 1, 2 and 3 are also quickly dismissed because do not deliver enough confidence and accuracy in the final audit result; which leaves Option 4.
Audit Analysis:
Having randomly selected and examined 377 customer records Patricia determines 363 fully comply with government and company requirements, which leaves 14 that are non-compliant. This translates to a sample result of 96.3% compliance (i.e. 14/377 x 100) or 3.7% non-compliance.
Based on the above audit result Patricia is able to report with a 90% level of confidence that ACME Health’s compliance performance for the past 12 months falls somewhere between 94.2% and 98.3% (i.e. 96.3% +/- 2%).
This type of one-size-fits-all statistical sampling works well if every item (product, activities, service, etc.) carries the same non-compliance risk or importance. But what if some non-compliances have more severe consequences than others? This is where risk-based auditing can be highly advantageous.
Risk-based compliance auditing
Unlike normal compliance auditing, which treats every non-compliant item the same, risk-based auditing separates the items in a population into different risk categories. Each risk category is then audited separately with different levels of confidence and accuracy.
The biggest advantage of risk-based compliance auditing when compared to normal compliance auditing is that it focuses an enterprise’s limited auditing resources on its areas of highest compliance risk. Thereby, providing higher audit confidence and accuracy where it is most needed.
If you’d like to learn more about how to use a statistical sampling tool to create a risk-based auditing system make sure you check out the last article in Part-3 in my series on statistical compliance auditing here.
