Outliers in Quality Control and Proficiency Testing
I have been interested in ways to compare different assays and have a product for this (1). The recent guest essay on the Westgard web site on this topic stimulated the following comments.
The title of the essay is: “The Quality Goal Index – Its Use in Benchmarking and Improving Sigma Quality Performance of Automated Analytic Tests” The essay starts with:
“The long-term goal of six sigma quality management is to achieve an error rate of 3.4 or less per million opportunities for all laboratory processes. In percent terms, that’s an error rate of less than 0.001%.”
A few sentences later, the author is describing how to calculate “sigma performance” as a measure of quality performance and states that for the CV estimate:
“Data integrity can be assured only when procedures are in place and rigorously practiced to exclude erroneous quality control results due to procedural blunders and statistical outliers.”
This sentence causes me to question whatever follows with suspicion. The author excludes two types of errors: blunders and statistical outliers. From a clinician point of view, interest is in obtaining the correct answer (meaning “correct enough”). If an incorrect answer is produced by a blunder, it is nevertheless wrong. So right off the bat, one knows that whatever is being measured by the author is a subset of real quality performance and one has no doubt heard that the majority of clinical laboratory errors come from pre or post analytical problems.
But perhaps the author intends to measure a subset of quality performance – that due to the analytical process. Then I have a problem with excluding statistical outliers. There is no justification for this exclusion. From a simple numbers view, assume that excluded data are greater than 3 standard deviations from the average. This means excluding about 0.3% of the data, if the data are normally distributed. Well, that’s one way to get to an error rate of less than 0.001%! One might argue that exclusion of a specific outlier is ok because the outlier must have been due to a blunder, but that is speculation and it is possible that the outlier occurred as part of the analytical process - one simply does not know.
From another point of view, if one constructs a Clarke or Parkes type of error grid (e.g., similar to that used for glucose), then dangerous errors will be (by definition of this grid) large errors in certain regions and are likely to be outliers (in a statistical sense) and could easily be excluded. But these are the very errors that should be measured (see essay on FDA's new waiver guidance).
Along these lines, note the risk matrix in the figure below (from my risk analysis FMEA software). A severity of 5 means that the potential error can cause serious harm or death. Patients are not directly “connected” to a clinical assay so such harm is indirect. But there is little question that serious assay errors are associated with patient harm. Also, since most assays are highly reliable, the bucket of most concern is the square at the upper left, corresponding to a severity of 5 and probability of 1 (e.g., probability of 1 has the lowest frequency of occurrence). This is precisely the type of values that
- are likely to do the most harm
- are likely to be called a statistical outlier
The same arguments apply to proficiency testing which often have automated outlier rules that “clean” the data.
This same trend to exclude outliers has been prevalent in discussions to publish a CLSI standard for GUM (Guide to the Expression of Uncertainty in Measurement).
But not excluding outliers messes up our analysis
Welcome to the real world. That’s true. This is why I recommend a method to assess quality performance based on quality control data which:
- does not exclude data
- measures distance from target.
- the number of values outside of medically acceptable limits (either out low or out high)
- the estimated total analytical error
- the lower and upper 95% uncertainty intervals (non parametric estimation)
- the contribution of bias as a percent of total analytical error
- the contribution of imprecision as a percent of total analytical error
Note: My software product manual, which is in part a tutorial may be downloaded at no charge.
Even if outliers are included
Remember that even if outliers are included, the quality performance measured is still a subset of the analytical performance because random biases, especially those due to patient interferences will be missed because:
- quality control material instead of patient samples are being tested
- the usual frequency of quality control testing will miss some intermittent errors
This is described in more detail in another essay.
In the list of essays, I used Outlier.................s. I first saw this in: Beckman, R. J., and R. D. Cook, (1983). Outlier...s. Technometrics, vol. 25, pp. 119-149.