Krouwer Consulting
 

 

More on GUM

I have critiqued the use of GUM (Guide to the expression of uncertainty in measurement) for commercial diagnostic assays (1) and also commented on a Letter about GUM (2).

To review why I don’t favor the use of GUM for commercial diagnostic assays:

  • GUM is an extremely complicated modeling method with respect to the capabilities of most clinical laboratories
    • This leads to “simplified” versions of GUM for clinical laboratories which are completely inadequate (3). Whereas one can argue that these simplified methods aren’t GUM, they may nevertheless be claimed as such.
  • GUM requirements won’t be met in many cases. For example:
    • many assays don’t meet the definition of a well defined physical quantity
    • one must correct known errors, which is impractical if not impossible for users of commercial diagnostic assays, who must know what the errors are and how to fix them, and many assays do have problems (although most results are within medically acceptable limits).
  • GUM typically estimates the 95% limits of the error distribution. Whereas this is useful information, GUM provides no information about the remaining 5% of errors – note that the assumption that all data is or has been transformed to Normality is a big stretch.
    • This focus on 95% of the error distribution goes against the patient safety movement of focusing on the largest errors (e.g., the remaining 5%).
  • GUM is unnecessary as one can simply count errors in various severity categories to get rates without the use of complicated modeling with assumptions that may be wrong.

Having said all this, I am still onboard for use of GUM for reference materials.

This essay is about another GUM article for which I published a Letter (4), which prompted a reply from the authors (5). Their article was about use of GUM for serological assays (6). What follows was sent as an eLetter to Clinical Chemistry.

I appreciate the response by Dr. Dimech and understand that analyzing real data is never easy. Of course, I was unaware of Dr. Dimech’s response - I can only react to the words on the paper, not material that is omitted for whatever reason - thus my Letter.

Here is my response to Dr. Dimech’s reply to my Letter combined with his original paper.

Right after the statement to exclude outliers comes the advice:  "It is suggested that results reported by each laboratory are checked for normality by use of a bar graph (See Fig. 1 in the online Data Supplement) or a statistical method such as Grubbs test."

Normality is usually tested graphically with histograms and / or normal probability plots, not bar graphs. Grubb's test is not a test for normality - it is a test for outliers and requires normal data! Statistical tests for normality include the Shapiro-Wilk, Kolmogorov-Smirnov, and Anderson-Darling tests.

Perhaps more importantly, consider the authors’ first sentence in the paper:  ”Most regulatory authorities that use International Organization for Standardization (ISO) Standards to assess laboratory competence require an estimate of the uncertainty of measurement (MU) of assay test results.”

At best this sentence is ambiguous. Perhaps the authors mean that one of the components of laboratory competence is an uncertainty interval but one could also interpret this sentence to equate an uncertainty interval with laboratory competence, even though to a clinician, laboratory competence would suggest an acceptable rate of errors from all sources.

In the case of laboratory data, the distribution of errors can be of any shape and can contain large errors, which may or may not be detached from the rest of the error distribution. To a clinician, wrong answers are dangerous, regardless of their source. So, blunders such as the typographical error are part of the population of interest to a clinician. Now for certain purposes, one can define a subset of the population of errors that contain only analytical error sources and exclude pre- and post- analytical error sources. However, this subset can be quickly confused with the total population and the first sentence in this paper will add to this confusion.

References

  1. Krouwer JS Critique of the Guide to the Expression of Uncertainty in Measurement Method of Estimating and Reporting Uncertainty in Diagnostic Assays Clin. Chem. 2003;49:1818 – 1821.
  2. Stöckl D, Van Uytfanghe K, Rodríguez Cabaleiro D, Thienpont LM, Patriarca M, Castelli M, Corsetti F, and Menditto A Calculation of Measurement Uncertainty in Clinical Chemistry Clin Chem 2005 51: 276-277
  3. White GH and Farrance I Uncertainty of Measurement in Quantitative Medical Testing: A Laboratory Implementation Guide Clin Biochem Rev 2004;25:Suplement ii,S1-S24 available at http://www.aacb.asn.au/pubs/Uncertainty%20of%20measurement.pdf
  4. Krouwer JS Uncertainty Intervals Based on Deleting Data Are Not Useful Clin. Chem. 2006;52:1204 - 1205.
  5. Dimech W Uncertainty Intervals Based on Deleting Data Are Not Useful: Reply Clin. Chem. 2006; 52:1205.
  6. Dimech W, Francis B, Kox J, Roberts G. Calculating uncertainty of measurement for serology assays by use of precision and bias. Clin Chem 2006;52:526-529