APS calculator: a data-driven tool for setting outcome-based analytical performance specifications for measurement uncertainty using specific clinical requirements and population data

Introduction

Analytical performance specifications (APS) are used to monitor the performance of examination procedures [1], set goals for in vitro diagnostic medical devices [2], provide limits for external quality assessment programs [3], ensure laboratory work in conformance with acceptable quality [4], and for the management of result release [5].

Models for derivation of APS were first described according to a hierarchy in 1999 at the IFCC-IUPAC Stockholm conference [6]. Conversely, the 2014 EFLM Strategic Conference held in Milan formulated criteria for allocating measurands to different models, considering the measurand’s role in the clinical decision-making process of a particular disease or clinical situation, as well as biological characteristics [7].

ISO 15189:2022 standard section 7.3.1 sub-section (b) states that “The performance specifications for each examination method shall relate to the intended use of that examination and its impact on patient care.” [8]. However, APS that are based on biological variation and state-of-art may not fully address the intended clinical use. APSs based on the impact on clinical outcomes may be more appropriate, mainly for measurands that play a central role in diagnosis and follow-up [9]. Outcome-based studies can be classified into two categories, corresponding to direct and indirect studies [10]. Direct outcome-based studies require randomized controlled trials, which cannot be readily performed [11]. Therefore, indirect outcome-based studies have become the preferred type of study employed to define APS, most of which have been performed using different misclassification rates around clinical decision limits to assess the effect of MU on the diagnostic assignment of results [12].

Measurands are presently assigned to the Milan models by considering their impact on clinical decision-making, biological variability, and achievable analytical performance [9]. Due to the limited number of outcome-based APS [12], laboratory professionals are however inclined to use APS based on biological variation or state-of-the-art [13]. Therefore, there is a need to conduct more studies on APS based on clinical outcomes. However, it is essential to acknowledge that the APS derived from indirect studies may not be universally applicable across all laboratories, as the selected disagreement or agreement rates, clinical decision limits, and population data of concern may directly impact the calculated APS.

To address these challenges and provide laboratory professionals with a reliable and flexible tool for determining APS for measurement uncertainty (MU), our study aimed to develop a web application that enables to calculate APS using selected agreement thresholds, clinical decision limits, and data of a population of concern, thereby facilitating estimate of APS when direct or indirect outcome studies to define APS are not available. To demonstrate the data-driven web application, we tested it using National Health and Nutrition Examination Survey (NHANES) data.

Materials and methods

We developed a data-driven web application called “APS Calculator” that utilizes simulation techniques, as depicted in Figure 1, to determine APS for MU. The APS Calculator was designed to provide users a clear representation of the distribution of the original uploaded data, the APS for MU, and the visualization of the APS using contour plots.

Figure 1:

Summary of the process for determination of analytical performance specifications. NHANES: National Health and Nutrition Examination Survey, APS: analytical performance specifications, CDL: clinical decision limit, u_rel: relative standard measurement uncertainty, Result_O: original concentration of the measurand, Result_M: measured (simulated) concentration of the measurand, n(0,1): a random number generated with a normal distribution having a mean of 0 and a standard deviation of 1.

The application allows the user to upload an Excel or csv file, select analyte data by choosing column name, specify clinical decision limits with their values, and enter agreement thresholds. After activating the “Simulate & Calculate” routine, the application draws a histogram plot to illustrate the distribution of the data (an example of histogram plot was given in Supplementary Materials). Subsequently, the tool conducts computer simulations to determine the APS for MU and visualize them through contour plots.

Contour plots

The web application produces a series of interactive contour plots that visually represent agreement rates as percentages on the x-axis and their corresponding u_rel values on the y-axis (see Figure 2 as an example). Vertical dashed lines are added to indicate the minimum, desirable, and optimal APS for u_rel (%), corresponding to different agreement levels. These interactive contour plots provide a clear illustration of the APS for u_rel (%) for both overall agreement and sub-level agreement. Additionally, the sublevel contour plots present sensitivity and specificity values, effectively showcasing the inherent trade-offs.

Figure 2:

Contour plots defining analytical performance specifications (APS) for fasting plasma glucose relative standard measurement uncertainty (u_rel) based on national health and nutrition examination survey data as an example. (A) Contour plot of APS for u_rel (%) based on overall agreement. (B) Contour plot of APS for u_rel (%) based on the agreement of subgroup “<5.6 mmol/L (<100 mg/dL)” (desirable concentrations). (C) Contour plot of APS for u_rel (%) based on the agreement of subgroup “5.6–6.9 mmol/L (100–125 mg/dL)” (impaired fasting glucose). (D) Contour plot of APS for u_rel (%) based on the agreement of subgroup “≥7 mmol/L (≥126 mg/dL)” (diabetes mellitus). Dashed vertical lines represent the selected agreement thresholds. 90 % agreement represents the minimum APS, the 95 % agreement the desirable APS, and the 99 % agreement the optimum APS, respectively. Black, red, green, and blue colored agreement dots correspond to unacceptable, minimum, desirable, and optimal u_rel value ranges.

Results

Table 1 depicts relevant demographic characteristics of the NHANES population and the main laboratory test data. Table 2 shows the minimum, desirable, and optimal APS values for u_rel (%) of the six measurands as obtained by using NHANES data. In some cases, APS values belonging to different agreement subgroups of the same measurand displayed significant differences and it could be difficult to apply different APS to the same measurand. As a starting practical point, APS values based on overall agreement may provide more balanced goals encompassing all subgroups. As an example, Figure 2 displays the APS contour plots of FPG. To fulfill APS derived using our model, u_rel of FPG should not exceed 3.0 and 1.4 % at minimum and desirable quality level, respectively.

The APS Calculator’s Python scripts have been made publicly available through our GitHub repository at https://github.com/hikmetc/APSCalculator. Furthermore, the APS Calculator can be easily accessed via any web browser using the following URL: https://hikmetc-apscalculator.streamlit.app.

Discussion

In this study, we have developed a data-driven web application for helping laboratory professionals in calculating APS for MU using their own selected agreement thresholds, clinical decision limits, and data of a population of concern. To demonstrate the use of our application, we utilized NHANES data as an example. Our proposal should be considered as an indirect outcome-based model as described in the 2014 EFLM Milan Conference consensus [10].

Various approaches have been proposed to derive indirect outcome-based APS, which can be summarized under obtaining “true” values, generating measured values, and assessing the effect of measurement differences on outcomes [12]. One of the most used approaches for generating measured results is the formula proposed by Boyd and Bruns [12, 16]. Originally, this formula considered imprecision and bias as two distinct error sources. However, according to the implementation of metrological traceability theory, one should act to eliminate a “medically significant” measurement bias through appropriate correction of calibration and then estimate MU at the commercial calibrator and clinical sample levels [30, 31]. Furthermore, when the imprecision component is calculated over a long period, including different sources of systematic variation, such as calibrations, reagent lot changes, and instrument maintenance, it will include all bias associated with such variations [14]. Therefore, we modified the formula offered by Boyd and Bruns by keeping only u_rel as a source of measurement variability. Furthermore, when there is a bias relative to the method used for determining clinical decision limits, bias correction can be performed (if permitted). In such circumstances, the laboratory may calculate the uncertainty of bias correction (ubias) and combine it with other measurement uncertainty components to achieve relative standard uncertainty of a measured value. The user can input this relative standard uncertainty into the APS calculator, which subsequently conducts further simulations per the provided parameters.

It is important to note that the formula employed in this study assumes the absence of pre-analytical sources of error. However, to the best of our knowledge, previous research has also not accounted for pre-analytical variation. On the other hand, APS calculator simulates “re-analysis of the same samples,” which focuses only on the analytical phase; therefore, intraindividual biological variation was excluded from the simulation process due to the absence of re-sampling.

The effects of measurement differences on clinical outcome can be assessed within the different scopes of clinical accuracy, treatment decision, and costs [12]. Most of the indirect outcome studies in the current literature evaluated the clinical accuracy using clinically acceptable misclassification rates as an identifier of APS [9, 12, 13]. However, different misclassification rates were employed in these studies. For instance, Boyd and Bruns used an error rate of 5 % in insulin dosing for glucometers [16]. Similarly, Von Eyben et al. utilized 5 % false result rates for lactate dehydrogenase isoenzyme 1 [32]. Stöckl et al. considered a misclassification of 20 % as an acceptable rate to determine APS of vitamin D [33]. Finally, Ferraro et al. used misclassified results of 0.5–2.5 % to identify APS of serum folate [34]. Overall, there has not been any consensus on the amount of misclassification rate for determining APS in indirect outcome studies, even because the acceptable misclassification is of course dependent on the type of disease to be diagnosed and the harm inflicted by misclassification. Thus, to provide a flexible tool, we left agreement thresholds (i.e., the reverse form of the misclassification rate) as an input to be determined and entered by users.

In the literature, the assessment of the effect of MU on outcomes has been conducted using contour plots that incorporate both sensitivity and specificity values. However, it should be noted that these plots, as demonstrated in a hypothetical example by Smith et al. [12], primarily focus on discrimination between healthy and patient populations, which corresponds to a 2 × 2 confusion matrix where the calculation of sensitivity and specificity values is straightforward. In our study, we opted to include only overall agreement, sublevel agreement, sublevel sensitivity, and sublevel specificity in the assessment of classification performance due to the intrinsic characteristics of confusion matrix statistics. When a single clinical decision limit is employed, the confusion matrix represents the categories of the uploaded data in a 2 × 2 formation. In this scenario, the overall sensitivity, overall specificity, and overall agreement are equivalent to each other, while the sensitivity of one sublevel is equivalent to the specificity of another sublevel. However, when there are two or more clinical decision limits, the confusion matrix typically represents a 3 × 3 or larger formation. In such cases, stable true negatives are absent, resulting in an overall specificity value of zero. Nonetheless, the overall agreement and overall sensitivity remain equal. As a result, to address this confusion, we have excluded the overall sensitivity and overall specificity from the contour plots depicting the overall APS for standard MU and from the overall APS determination process. Instead, the sublevel contour plots illustrate the sublevel sensitivity and sublevel specificity to demonstrate the trade-offs.

The significance of the variations in APS reported in indirect outcome studies can be attributed to the utilization of different data, various decision limits, several misclassification rates, and diverse calculation methods. Loh et al.’s literature review concerning APS in the context of HbA_1c revealed substantial discrepancies not only between results obtained using different models but also within studies using only indirect outcome approaches, with APS ranging from 2 to 25 % [35]. The reported variability in APS underscores the need for a standardized and flexible approach to set APS for standard MU. In this regard, our data-driven web application can be a valuable tool for laboratory professionals. The APS Calculator provides the flexibility to use various data sources relevant to the specific clinical setting of interest. This application also enables users to customize decision limits and agreement rates based on their particular requirements, resulting in APS that are tailored and fit for purpose. In addition to its customization features, the APS Calculator can also foster transparency and reproducibility in APS determination. The application allows for the documentation and sharing of output data and parameters among researchers and stakeholders, thereby enhancing the ability to replicate results and facilitate cross-validation. Overall, the utilization of this application can improve the understanding of reported APS values, ultimately enhancing the quality of laboratory testing and patient care.

NHANES data, a comprehensive and representative sample of the United States population, consists of both healthy and unhealthy individuals across various age groups and demographic characteristics. By utilizing this publicly available dataset and internationally recommended decision limits, we were able to demonstrate the potential of our application to generate APS for various measurands along with visualisation features, as shown in Figure 2 for FPG. However, the APS identified in the current study cannot be extrapolated to all clinical scenarios. For example, if the goal is to adjust insulin dosing, different clinical decision limits may be employed to determine the APS for glucose [16].

Authors previously stressed that the Milan models use different principles and do not constitute a hierarchy [7, 9, 13, 36]. Therefore, some models are better suited for certain measurands than for others, and the attention should primarily direct toward the measurand and its biological and clinical characteristics. Particularly, the outcome-based model applies for measurands that have a central and well-defined role in the decision-making of a specific disease or clinical situation, and test results should be interpreted through established decision limits. This applies to all six selected measurands in our study. For some of them (FPG, total Hb, total cholesterol, and total folate), some information about APS using the indirect outcome model was already available [4, 34, 37], [38], [39], [40]. For the remaining two (triglycerides and HDL), a recent paper was unable to retrieve peer-reviewed literature for outcome studies dealing with their clinical use and evaluating the impact of analytical variability on clinical outcomes [13]. Therefore, the authors temporarily allocated those measurands to the biological variation model to derive their APS. Table 3 compares previously available APS with those defined in the present study. It is noteworthy that FPG, folate, triglycerides, and HDL show very similar APS, even if the two mentioned lipid parameters have APS derived using different Milan models, i.e., indirect outcome-based in this study and biological variation-based in a previous one [13]. The similarity of APS derived from both models for the two lipids seems to confirm the equivalence of the two models advocated by the expert group of the EFLM when measurands have a steady state in the blood of (normolipemic) individuals. For other two measurands, derived APS appear to be different, and some explanations may be put forward. For total Hb, only one previous paper provided information about APS by accepting a false positive rate of 5 % in classifying patients. However, this was done with some methodological limitations, so APS results cannot be definitively compared [39]. For total cholesterol, Petersen and Klee evaluated the influence of analytical variability of measurements on the number of low-risk individuals misclassified as false positive, i.e., individuals at high risk [40]. The reported APS were obtained by accepting a misclassification rate, in terms of false positive results, of approximately 5.0 % (desirable) and 7.5 % (minimum) at the 6.21 mmol/L (240 mg/dL) cut-off. Checking their simulated data in Supplementary Table 2A, the rates of subject misclassification did, however, not change markedly if our APS from the overall agreement group were considered: an APS of 2.3 % (desirable) corresponded to about 5 % false positives, and an APS of 4.8 % (minimum) to approximately 6 % of wrongly classified individuals. Applying APS specifically derived in our study for the similar agreement subgroup (decision limit ≥6.21 mmol/L [240 mg/dL]) appears, however, to amplify the noticed difference, asking for less demanding analytical goals (Table 2).

The APS calculator is designed to estimate APS by considering the entered minimum, desirable, and optimal agreement thresholds and the data provided by the user. However, it is important to consider that when the data related to a specific subgroup contains a low number of results, the estimated APS for standard MU can be inaccuratety high, exceeding the goal of 33 %, which in Table 2 were labeled as “not available”. Even if the APS calculator ensured the reproducibility of the same experiment settings with the same data, a larger laboratory data set of the same experiment settings might result in more reliable estimates of APS for standard MU and aid in overcoming inaccurately high APS for MU due the low sample size problem. Nevertheless, taking into account that overall APS for standard MU encompass all sublevels based on clinical decision limits, their related values may provide a more comprehensive set of goals for each measurand.

Conclusions

The development of a data-driven web application presented in this study may offer a valuable tool to laboratory professionals to calculate APS for standard MU using their own agreement thresholds, clinical decision limits, and, most importantly, data of the population of interest to address actual clinical questions in specific scenarios. In addition, the application can foster transparency and reproducibility in APS estimates and increase the use of APS obtained using EFLM-recommended models. Overall, the application provides a standardized and flexible approach to set APS for MU, enabling users to customize decision limits and agreement rates based on their particular requirements, resulting in APS that are tailored and fit for purpose, as specified by ISO 15189:2022 [8]. By using this tool, laboratories can define their MU as analytically acceptable on the basis of the clinical decision uncertainty personalized by using local data which do right to their own clinical setting.