|
Krouwer Consulting
|
|
|
It's time to stop advocating the use of the wrong model in clinical chemistry
Introduction
A recent article in Clin Chem News described instrument evaluation methods (1). Surprisingly absent from this article was any reference to evaluation methods that measure (vs. model) total error, especially since many of the methods cited in this article are NCCLS standards and there is an NCCLS standard specifically devoted to measuring total error (2). This editorial will discuss some of the implications of this deficiency and the use of incorrect models.
The Importance of Diagnostic Assays
When I started in the field over 20 years ago, I asked several clinicians how they used diagnostic assay results. A common answer was “to confirm a clinical diagnosis.” These days that answer has changed for many clinicians to, “to establish a clinical diagnosis.” Putting things another way, a test result can be the prime reason that a clinician recommends a further diagnostic procedure such as a biopsy, a certain treatment, or to withhold treatment. This puts a great burden on manufacturers, laboratorians, and regulators to ensure that diagnostic assay results are as free from error as possible. As an example, take a case that appeared in the media (3) about a woman who had repeated elevated hCG results and was treated for cancer. The hCG results were falsely elevated due to HAMA interference and the woman had no cancer. Moreover, this was not an isolated case – it had occurred repeatedly (4).
Measuring quality vs. modeling quality
There are two possible approaches to determining the quality of diagnostic results: measuring or modeling.
Measuring has the advantages of being simple, requires no assumptions (other than a representative sample), and requires no model, and models can be wrong. Measuring has the disadvantages of requiring more samples to give the same confidence as modeling (assuming of course that the modeling is correct). Measuring also does not provide causes for observed errors.
Modeling has the advantage of providing a cause for errors. This is crucial for quality improvement. Modeling has the disadvantage of requiring the model and its assumptions to be correct. Whereas this may be an obvious requirement, it is difficult to prove that a model and its assumptions are correct. Whereas modeling provides more information than measuring, it also requires considerably more effort.
Measuring methods
Measuring methods are described by the NCCLS standard EP21 (2) or a recent review (5). Basically, one assays a series of patient specimens by a candidate and reference assay. Calculation of the total error is by a parametric or nonparametric method.
Modeling methods
Now here is where there is a major problem (6). There are several well known clinical chemists who propose that average bias plus imprecision equals total error. Some of these authors suggest that:
It just isn’t true. These authors ignore a more complete model of an assay (7) which takes into account the random interferences that often occur in patient samples. Moreover it is just these random interference errors that
· cause clinical problems because the errors are so large and · they cannot be detected by quality control
The reason that these problems can’t be detected by traditional quality control is that the model associated with quality control does not (and cannot) deal with these interferences. Continually advocating a wrong model perpetuates its use by others - as an example see (8) - and has the potential to retard quality improvement.
The companies for which I have worked used more complete models such as that described in reference 7 but manufacturers typically don’t provide to the public their evaluations methods. Not all laboratorians use the wrong model. A recent article about cholesterol compared the measurement method with the wrong model (9).
Outliers
Finally, a word about outliers. The word outlier is somewhat unfortunate because outliers often seem to get special treatment as in “oh don’t worry about that result, it’s an outlier.” Outliers might better be thought of simply as errors, albeit large errors. In a measuring method, outliers are not discarded whereas in modeling methods they are often discarded. Yet in real life they appear.
Conclusions
One can always simply measure the errors in an assay (e.g., without modeling). If one chooses a modeling method, then it’s time to use a more complete model and abandon the model that’s been shown to be wrong.
References
|