1	Improving Method Evaluations Jan S. Krouwer, Ph.D. Krouwer Consulting *: info@KrouwerConsulting.com
2	Outline Introduction Problems with current evaluations Suggested improvements Better way to estimate total error Outliers Process capability Understanding published evaluations Remedies to improve assays
3	Introduction
4	Introduction Diagnostic assay quality has improved At the same time, clinicians rely more on test results to make clinical decisions Elevated cholesterol → statin therapy Elevated PSA → prostate biopsy Elevated troponin → ACS treatment
5	Introduction Manufacturers evaluate their assays Huge amounts of data are collected internally and externally Design improvements Leads to error prevention algorithms Package insert lists performance claims FDA reviews part of manuf. data (510(k), PMA, CBER) Hospital labs are required to evaluate assays Most evaluations are short “verification” assays Some hospital labs publish larger evaluation results But, …
6	Introduction There are still problems Lifescan SureStep glucose meter 2700 complaints → 61 hospitalizations → 105 million dollars in criminal and civil fines and settlements University of Washington hCG assay Unnecessary treatment due to test error, (not the only hCG case) → 16 million dollar lawsuit St. Agnes Prothrombin assay Lab error leads to wrong coumadin dose for 920 patients → 5 deaths Weekly product recalls are listed on the FDA web site http://www.fda.gov/opacom/Enforce.html
7	Introduction How can manufacturers, hospital labs, and regulators improve evaluations to better detect problems? We need to: ask better questions provide better data analysis
8	The finances behind evaluations
9	A common and problematic evaluation Total error (TE) = average bias + 1.96 x imprecision Total error is calculated from a model which is wrong Implies that if you meet bias and precision goals, then you will meet total error, not true In general, this analysis won’t detect the aforementioned problems NCEP (National Cholesterol Education Program) Bachorik PS and Ross JW. Clin Chem 1995 41: 1414-1420 Peterson PH, Stöckl D, Westgard JO Clin Chem Lab Med 2001;39:589-595
10	The problem with this method
11	Modeling Total Error
12	Modeling Total Error Many approaches to simulations (and software such as Simscript, SAS, Excel add-ins) GUM approach is recent, international guideline Guide to the Expression of Uncertainty in Measurement, (ISBN 92-67-10188-9) ISO, Geneva (1995).
13	Modeling Total Error Glucose meter model is too simple! (2 error sources) “Worst case”: 3 sd errors occur for both error sources Tempted to improve assay for both error sources But, two worst case errors at the same time only 0.09% Better models have more than 2 error sources Must use probabilities to account for frequency of error Means that occasional 3 sd error is ok for an error source If error reduction is needed, use cost allocation Reduce error in source that is least costly – not equal reduction for all sources Boyd JC and Bruns DE Clin Chem 2001 47: 209-214. Krouwer JS Clin Chem 2001 47: 1329-1330
14	Modeling Total Error Effect of last slide Also says simulation is not easy: requires modeling characterizing distributions programming verifying results
15	Better estimate of total error Krouwer JS Clin Chem 2002;48:919-927
16	Better evaluation goals Correctly asses total error Estimate outlier rate And we (manufacturers, hospital labs, and regulators) can do this with the same amount of data
17	Outliers Total error accounts for 95% (or 99%) of differences This leaves 5% (or 1%) of differences largely unspecified For a busy lab that reports 1 million results, this could mean unknown quality for 50,000 (or 10,000) results per year That’s why we need outlier limits and rates
18	Outliers Outliers often get a special designation (as in, “Oh that error is due to an outlier”) Outliers often get special treatment They are removed from many parameter estimates (that’s ok) but never show up anywhere else (that’s not ok) Outliers are errors just like all other errors, they are simply larger errors
19	Total error and outlier specifications
20	Differences from reference But there is no reference for our assay There is always a reference, although it may not be an accepted reference True reference Existing commercial (field) assay used by clinicians But, we don’t know which assay is correct. True, but clinicians will observe the difference Clinician’s expectation of patient result Replicating the reference result is needed
21	Total error direct estimation - parametric Bland – Altman method Get differences from reference method Transform to normal distribution, if needed Get standard deviation of differences TE = average difference ± t x sd of differences (t is approximately 2 for 95% total error) Bland JM and Altman DG Lancet 1986; 307-10
22	Total error direct estimation - nonparametric Mountain plot method Get differences from reference method TE = desired percentiles from ordered distribution of differences (2.5 and 97.5 for 95% total error) Krouwer JS and Monti KL Eur J. Clin. Chem. Biochem. 1995;33:525-27 Proposed Guideline NCCLS EP21P. NCCLS, 771 E. Lancaster Ave. Villanova, PA, 2002
23	Better total error estimate examples
24	Better total error estimate examples
25	Better total error estimate examples
26	Better total error estimate examples
27	Better total error estimate Note what isn’t done Scatter plot r (correlation coefficient) Slope and intercept Nothing wrong in getting these but, they don’t answer the total error question
28	Comparison of total error methods
29	Parallel Universes in Evaluations
30	Getting remote data
31	Estimating outlier rates Two methods to estimate outlier rates Modeling GUM = Guide to Uncertainty in Measurement and others Guide to the Expression of Uncertainty in Measurement, (ISBN 92-67-10188-9) ISO, Geneva (1995). Counting Krouwer JS Clin Chem 2002;48:919-927 EP20 (NCCLS guideline on outliers – in preparation)
32	Estimating outlier rates - Counting
33	Process Capability Cpm – A unitless measure of distance from target Allows for comparison among assays Can be estimated during evaluations with patient samples or during routine assay use with controls Equivalent to total error scaled by specification limits Visualized with mountain plot Specification limits are medically acceptable limits
34	Process capability example
35	Process capability example
36	Published Evaluations Let’s define some types of financial support Completely independent – Site buys all materials Partially independent – Site receives free materials from manufacturer(s). Study design, data collection, analysis, and report done by site. Not independent – Manufacturer gives site materials. Study design, data collection, analysis, and report done by some combination of site and manufacturer.
37	Published Evaluations Conclusion: “Accu TnI is a sensitive and precise assay for the measurement of cTnI” But, inside the article… Assay fails ESC / ACC recommendations for precision at medical decision limit In 3 two way comparisons with other assays, 6%, 9%, and 9% of the time, the 2 assays give results on different sides of the medical decision limit Uettwiller-Geiger, Wu, Apple, et. al., Clin Chem 2002;48:869-876
38	There are always two remedies Manufacturer: Improve assay Hospital lab: Imprecision – run more replicates Bias – re-standardize assay Interferences – Check for interfering substance in patient sample HAMA – Add blocking agent to sample and rerun High dose hook effect – dilute and rerun Krouwer JS. Arch Pathol Lab Med 1992;116:726-731
39	Remedies must be evaluated Lab: Run more replicates (duplicates) for troponin Costs Assay cost is (at least) double – if all results are replicated Workflow is disrupted Benefits Some positives may be classified as negative, saving treatment costs Some negatives may be classified as positive, possibly saving lives and malpractice lawsuits Evaluation: Find out how many “some” is and which costs more Leape LL Clin. Chem. 2002 48: 1871-1872
40	Summary
41	Thank you