Jan S. Krouwer

Towards more complete specifications for acceptable analytical performance - a plea for error grid analysis.

Abstract We examine limitations of common analytical performance specifications for quantitative assays. Specifications can be either clinical or regulatory. Problems with current specifications include specifying limits for only 95% of the results, having only one set of limits that demarcate no harm from minor harm, using incomplete models for total error, not accounting for the potential of user error, and not supplying sufficient protocol requirements. Error grids are recommended to address these problems as error grids account for 100% of the data and stratify errors into different severity categories. Total error estimation from a method comparison can be used to estimate the inner region of an error grid, but the outer region needs to be addressed using risk management techniques. The risk management steps, foreign to many in laboratory medicine, are outlined.

The problem with total error models in establishing performance specifications and a simple remedy.

Abstract A recent issue in this journal revisited performance specifications since the Stockholm conference. Of the three recommended methods, two use total error models to establish performance specifications. It is shown that the most commonly used total error model – the Westgard model – is deficient, yet even more complete models fail to capture all errors that comprise total error. Moreover, total error models are often set at 95% of results, which leave 5% of results as unspecified. Glucose meter performance standards are used to illustrate these problems. The Westgard model is useful to asses assay performance but not to set performance specifications. Total error can be used to set performance specifications if the specifications include 100% of the results.

Why specifications for allowable glucose meter errors should include 100% of the data

The recently published ISO 15197 guideline [1], and the CLSI POCT12-A3 guideline [2] are two new glucose meter performance guidelines. Unfortunately, each of these guidelines fails to include error limits for 100% of the data. For the CLSI guideline, specifications are given for 98% of the results and for the ISO guideline, 99% of the results. The purpose of this article is to discuss why a specification for 100% of the results is appropriate. I have previously commented [3] that in clinical chemistry, errors are often thought of as a distribution of continuous variables and not as discrete error events. Thus, these glucose meter guidelines provide one set of error limits for 95% of the data and a wider set of limits for either 98% or 99% of the data. There is nothing wrong with this construct. However, one can also consider glucose meter errors as discrete failure events. For example, if a glucose sample reference result was 1.67 mmol/L (30 mg/dL), a serious glucose meter failure event would be if the meter result for that sample was 16.65 mmol/L (300 mg/dL) (e.g., within the E zone of a Parkes error grid [4]). And for each sample, this failure event can either occur or not occur. It makes sense to specify a goal of zero percent of data in the E zone of the Parkes error grid. The ISO and CLSI goals can be thought of as specifying the percentage of results that fall within the A and perhaps B region of an error grid. Thus, the ISO and CLSI goals are based on a distribution of a continuous variable and goals for the higher zones in an error grid, which are missing in the ISO and CLSI guidelines , are based on discrete failure events. The balance of the data (2% for CLSI and 1% for ISO) is allowed to fall in intermediate but not the highest error zones. Note that in other fields, where people think in terms of discrete failure events, one would never set goals in the way that they are done for glucose meters. Thus, one would not specify that nuclear power plants should run without meltdowns 99% of the time nor that correct site surgery should occur 99% of the time. Perhaps a difficulty arises because one can never (statistically) prove the occurrence of zero failures but the goal itself is valid. Manufacturers worry about the approval process and in performing method comparisons, the possibility of an outlier is perhaps a rationale for the ISO and CLSI glucose meter specifications. However, it is the wrong way to think about the problem. If there were one E zone result in a 125 sample method comparison, this error rate of 0.8% would be acceptable according to either glucose meter standard. On the one hand, if this glucose meter had 1% of the glucose meter market (total market = 7.9 billion results per year in the US), this implies over 632,000 dangerous glucose results per year in the US [5]. On the other hand, if this glucose meter company ran 10 million samples with no results in the E zone, this only proves (with 95% confidence) that no more than 29 dangerous results would occur [6]. Thus, proving that rare events do not occur by running method comparisons is not practical. What is done both in other fields and in clinical chemistry is to perform Failure Mode Effects Analysis (FMEA) and fault tree analysis to enumerate the sources of possible failure events and to implement mitigations where appropriate. These techniques also do not prove zero failures but complement method comparisons which need to be performed anyway. If an E zone event does occur in a method comparison, this by itself is not a reason to consider the meter to be unacceptable. The most important question is the cause of the outlier and it is possible that no cause will be found. There is a financial constraint to collecting data to decide on meter approval and the use of post-market data, the size of which dwarfs method comparison studies should also be used as a way to monitor meter performance.

The chronic injury glucose error grid: a tool to reduce diabetes complications.

Traditional glucose error grids provide error limits for glucose meters. These criteria help to assess the meter’s suitability to prevent acute injury. We present a rationale for an error grid that provides a different set of error limits to help prevent chronic injury in diabetes. For example, glucose values in the no treatment zone of a traditional error grid could be harmful in diabetic retinopathy. The same method comparison data informs both the acute and chronic injury error grids. All of the data are used in an acute injury error grid, whereas only long-term biases populate a chronic injury error grid. These biases can be due to reagent lots and patient specific interferences. An example of a chronic injury glucose error grid is provided using simulated data.

Recommendation to treat continuous variable errors like attribute errors

Abstract Clinical laboratory errors can be considered as either belonging to attribute or continuous variables. Attribute errors are usually considered to be pre- or post-analytical errors, whereas continuous variable errors are analytical. Goals for each error type are different. Error goals for continuous variables are often specified as limits that contain 95% of the results, whereas attribute error goals are specified as allowed error rates for serious events. This leads to a discrepancy, because for a million results, there can be up to 50,000 medically unacceptable analytical errors, but allowable pre- and post-analytical error rates are much lower than 5%. Steps to remedy this are to classify analytical error rates into severity categories, exemplified by existing glucose error grids. The results in each error grid zone are then counted, as has been recommended by the Food and Drug Administration (FDA). This in effect transforms the continuous variable errors into attribute errors. This is an improvement over current practices for analytical errors, whereby the use of uncertainty intervals is recommended that include only 95% of the results (i.e., leaves out the worst 5%), and it is precisely this 5% of results that are likely to be in the most severe zones of an error grid. Clin Chem Lab Med 2006;44:797–8.

Why the Details of Glucose Meter Evaluations Matters

In an article in the Journal of Diabetes Science and Technology, Macleod and coworkers describe an evaluation of LifeScan glucose meters that focus on the effects of sample types and comparison methods. They make a valid point that these factors influence the accuracy observed in evaluations and recommend the comparison method be the one recommended by the manufacturer for the sample type in the intended use statement. Yet, the recommended comparison method is not a reference method. The accuracy hierarchy of definitive, reference, and field methods originally described by Tietz should remind one that virtually all glucose meter evaluations use commercially available field methods as the comparison method. Finally, one should not neglect the FDA adverse event database as a way to assess glucose meter performance.

Analysis of “Seven Year Surveillance of the Clinical Performance of a Blood Glucose Test-Strip Product”

The article titled “Seven Year Surveillance of the Clinical Performance of a Blood Glucose Test-Strip Product” by Setford and coworkers in this issue of Journal of Diabetes Science and Technology is an impressive study showing that over 7 years in three clinics, using multiple reagent lots, a total of 73u2009600 samples met the ISO 15197 2015 standard with no results in the D or E zones of a Parkes glucose meter error grid. Three requirements are suggested for a clinically acceptable glucose meter. The authors provide strong evidence for meeting two requirements but fail to provide summarized data about the number of nonnumeric results. Finally, the authors overstate some results, called “spin” by some which is not necessary. The superb results should stand on their own.

Why the Diabetes Technology Society Surveillance Protocol for Glucose Meters Needs to Be Revised

The Diabetes Technology Society surveillance protocol provides a seal of approval for a glucose meter if a sufficient number of a candidate glucose meter’s results meet ISO 15197:2013 limits. The protocol provides clear details about how to conduct this study and analyze the data but has two flaws. There is no specification about the size of glucose meter errors that are outside of ISO limits. A meter that has a result in the E zone of a glucose meter error grid could receive the DTS seal of approval. In addition, the protocol uses the ISO standard, which could be considered a “state of the art” standard instead of an error grid, which is a clinical standard. Remedies for these problems are to replace the ISO standard with an error grid and to include requirements for errors found in C or higher zones of an error grid.

Improving the Glucose Meter Error Grid With the Taguchi Loss Function.

Glucose meters often have similar performance when compared by error grid analysis. This is one reason that other statistics such as mean absolute relative deviation (MARD) are used to further differentiate performance. The problem with MARD is that too much information is lost. But additional information is available within the A zone of an error grid by using the Taguchi loss function. Applying the Taguchi loss function gives each glucose meter difference from reference a value ranging from 0 (no error) to 1 (error reaches the A zone limit). Values are averaged over all data which provides an indication of risk of an incorrect medical decision. This allows one to differentiate glucose meter performance for the common case where meters have a high percentage of values in the A zone and no values beyond the B zone. Examples are provided using simulated data.

Acute Versus Chronic Injury in Error Grids

Klonoff et al recently described a new glucose meter error grid.1 This error grid updates the limits for the various zones that contain clinically significant glucose meter errors. To determine these limits, clinicians were provided scenarios and were asked to describe glucose error levels that would prompt them to treat patients in several ways ranging from no treatment to emergency treatment. One could view this exercise as clinicians responding to a patient’s symptoms, or the threat of acute injury. Yet diabetes is a disease that includes the possibilities of acute injury and chronic injury due to persistent increases in glucose. n nDiabetic retinopathy is an example of serious chronic injury to patients with diabetes. Hemoglobin A1c levels that exceed 5.5% are associated with diabetic retinopathy.2 Yet an A1c level of 5.5% is equivalent to a mean glucose of 111 mg/dL.3 In error grid terms, a meter measuring glucose with a true level of 111 mg/dL, but reading 100 mg/dL, demonstrates an error in the A zone, which is the no-treatment-needed zone (this is the case for all of the popular glucose error grids including Clarke,4 Parkes,5 as well as the new surveillance error grid1). To be fair, the 5.5% level of A1c is the starting point for diabetic retinopathy but for an A1c level of 6.5% (equivalent to a mean glucose of 140 mg/dL), the prevalence of diabetic retinopathy doubles to 20%, yet this level of glucose bias is still in the A zone. n nOne might argue that the possibility of a 10% to 40% consistent bias in modern-day glucose meters is unlikely. Yet, meters can exhibit biases due to interferences. Moreover, a large source of error for any assay, including glucose meters, is lot-to-lot reagent variability. Both of these errors can contribute to produce a consistent or fixed bias. n nClearly one needs to inform about allowable glucose meter deviations associated with the treatment of acute injury and the proposed error grid does that superbly. We propose that another error grid is needed to inform about diabetes complications by providing allowable error limits for long-term bias. This situation has analogies in other areas such as preventive cardiology. An assay such as low density lipoprotein (LDL) cholesterol, which is not used to diagnose acute injury, might have limits set for allowable long-term bias to inform about the risk of coronary events associated with an incorrect LDL measurement.