Sampling Theory and Audit Confidence
December  2010

One function of the underwriting staff is to perform audits of our clients’ recently issued business. Not only is this a good quality control practice but it also ensures that the policies we reinsure are being underwritten in the manner agreed upon when we priced the deal.

When performing these audits, a common question is how many cases should the audit staff review so that results represent the client’s overall book of business. To help provide an answer, we will review the statistics of sampling theory.

The Central Limit Theorem

Given a large population from which to draw a random sample, we can use the Central Limit Theorem as the basis for determining the appropriate number of cases to review. Prior to describing this principle, we will first define some variables. Let: n be the sample size; p equal the percentage of the population having some characteristic that we want to measure; and have q = 1 – p represent the percentage of the population not having the characteristic.

For an underwriting audit, p would be the percent of cases where an exception was made in underwriting (the characteristic to be measured). This could be anything from a business decision to a misinterpretation of lab results. We will call this the exception rate. The value of p measured from Transamerica Reinsurance’s past underwriting audits is usually between five and 15 percent. The sample size n is simply the number of randomly chosen cases that we reviewed.

Given these definitions, the Central Limit Theorem says (in terms of our audit): If we draw multiple independent random samples of the same size n from a large book of recently underwritten cases and calculate the exception rate from each sample, then the pool of sample exception rates will be normally distributed with a mean equal to p and a standard deviation equal to the square root of [(p * q) / n].

Figure 1 shows that the distribution of sample exception rates around the true population rate p is dependent upon the sample size. For a sample size of 5,000 there is only a small chance that a sample exception rate is very far off from the true rate shown by the value at the peak of each curve.

Confidence Interval and Confidence Level

Before proceeding, we need to explain what is meant by two key parameters that need to be chosen by the underwriter before deciding upon an audit’s sample size. The confidence interval is a plus-or-minus percentage range surrounding the exception rate determined from the audit results. The plus-or-minus value is often called the sampling error.

The confidence level is the likelihood that the true rate inherent in the client’s entire book of business is somewhere in the confidence interval. For example, assume the results of an audit show a 15-percent exception rate. If the confidence interval is plus-or-minus two percent and the confidence level is 90 percent, then there is a 9 out of 10 chance that the book’s exception rate is between 13 and 17 percent. In other words, there is only a 10 percent chance that the client’s true exception rate is actually less than 13 or greater than 17 percent.

Determining the Sample Size
Now, we are ready to calculate the sample size. The “Z statistic” from a normal curve is used to determine the sampling error for a given confidence level. As a percent of sample size, the sampling error is equal to [Z * Sqrt (p * q) / Sqrt (n)]. Using a little algebra, we see that sample size equals [Z / (sampling error)]2 * p * (1 – p).

Unfortunately, this value depends on the exception rate p in the book, which we can’t estimate until we perform the audit. One popular and conservative approach is to use a value of p that produces the largest standard deviation and, therefore, the largest sample size. Figure 2 demonstrates that the largest standard deviation is produced when p is 50 percent.

If you don't (or can't) know "p", assuming it is 50 percent is the most conservative
approach.

Our sample size formula is appropriate for populations of 20,000 and over. For smaller populations, an adjustment factor (beyond the scope of this article) can be incorporated into the formula. Figure 3 shows sample sizes for various combinations of confidence interval and confidence level for a large population using this approach. 

Smaller sample sizes may mean widening your confidence level or interval   

If previous knowledge or experience is available, it may be used to “guesstimate” the value of p in our sample size formula. Later, after audit results are in, we can adjust our final metrics with the true rate. This approach may result in a smaller or more practical sample size estimate and is the method we will use in the next section.

An Example of Sampling Theory in Action

We want to audit a large client’s book of recently underwritten business. The auditors want to review an appropriate number of cases so that they are confident the results are representative of the entire book. After discussions with client underwriters, the auditors want the exception rate determined by the audit to be within two percent of the client’s true rate with a 90 percent confidence level. Based upon prior audits, they estimate that the exception rate is likely to be around 10 percent.

Figure 4 indicates that the auditors should review 591 cases. Having an estimate of the exception rate greatly reduces the recommended sample size from the conservative value shown in Figure 3. 

Having a pre-audit estimate greatly reduces
the recommended sample size.    

With pre-audit metrics in hand, the auditors proceed with their review but, due to time constraints, they only look at  a random sample of 300 cases. The exception rate from these cases turns out to be 13 percent. We can now adjust the input parameters and determine the final audit metrics, shown in Figure 5.

Given that audit results show an actual exception rate of 13 percent, and knowing that we looked at 300 cases rather than the recommended 591, the confidence interval increases to a little over 3 percentage p​​oints. Thus, the auditors’ report indicates that they are 90 percent confident that the exception rate for entire book of recently underwritten business is between 10 and 16 percent.

Conclusion

While this article has discussed sampling theory in regards to an underwriting review, the results are equally valid for many other applications. The same concepts used by insurance auditors are also used by pollsters, television program rating services, and quality control engineers. Understanding the limitations to measurements taken from a random sample gives us a better appreciation for why the results are always footnoted with a comment such as “sampling error is ±1.5 percentage points.”