veRification: an R Shiny application for laboratory method verification and validation

Assessing assay imprecision

The assessment of an assay’s imprecision is perhaps one of the first steps of a method’s verification. In the UK, the Association for Clinical Biochemistry and Laboratory Medicine (ACB) recommends the measurement of at least two levels of internal quality control (IQC) material 5 times a day (spread throughout the day), across 5 different days [1]. Such an experiment allows the calculation of within- and between-day variability (referred to as repeatability and intermediate imprecision, respectively), alongside the total variation across the time course and the expected value. Analysis of these data is commonly performed with a nested ANOVA (analysis of variance), but can equivalently be modeled with a Bayesian linear varying-effects model [16] (Eqs. 4.1– 4.5) in order to make direct probability statements regarding the model’s parameters and propagate our uncertainty regarding their values into our inferences. Bayesian (and frequentist) varying-effects models can be fitted within the “Imprecision” tab. The Bayesian analyses uses the default, weakly informative prior distributions recommended by the rstanarm team [15], scaled to the input data (Eqs. 4.4 and 4.5, where m y and s y represent the mean and standard deviation of the input data, respectively) and the frequentist models – estimated by means of restricted maximum likelihood (REML) – are fitted through the use of the VCA package [20]. This module allows the simultaneous analysis of multiple levels of IQC material, automatically fitting a separate model to each level.

First, the user is prompted to input their data in .csv or .xls(x) format. The inputted data are then displayed to the user and the user prompted to interactively select the columns within the data that represent the days of measurement, the IQC levels, and the measurements themselves. The user can also select whether to test the estimated total laboratory CV against a given claim (e.g. from a manufacturer’s kit insert or another data source) (Figure 1A–C). Once these settings are chosen, the data are plotted with an interactive visualization and the modeling results are presented to the user when complete (Figure 2A). A number of basic checks of the Bayesian model’s validity are performed and the results of these (pass or fail) are also shown. These checks determine if the Markov chains (MCMC) have converged (parameter R ˆ values ≤1.1 and that the effective sample sizes are adequate (≥10% of the total number of posterior samples) [29], [30], [31]. The MCMC traces for each of the model’s parameters are shown within the “Bayesian model diagnostics” tab, alongside their full posterior distributions (Figure 2B). The user is strongly encouraged to check these for consistency [29, 30] and this remains relevant for each of the Bayesian models fitted within the application. If the user-defined credible intervals – which default to 89% – of the posterior distribution of the total laboratory CV for a given level of QC is higher than the user-provided claimed CV value (i.e. a one-tailed test), then this is highlighted to the user. An equivalent inference is performed if the frequentist method is chosen. It cannot be overstated that the choice of interval is entirely arbitrary and the user is strongly encouraged to view and inspect the entire posterior distribution in the “Bayesian model diagnostics” tab. Note that the credible intervals for the remaining parameters are not displayed for clarity, as it is assumed that the coefficients of variation are the parameters of most interest. If required, the user can view the full posterior distributions in order to assess the uncertainty in their values (Figure 2C).

Figure 1:

The application homepage and imprecision module input. (A) The welcome screen to the application. The different modules can be accessed through the side bar. Guidance for each of the modules is provided on the welcome screen under dropdown boxes shown. (B) The data input screen for the “Imprecision” module. (C) Once data are uploaded, they can be viewed interactively. The user is prompted to select which columns within the data represent days, measured values, and QC levels. The user can then determine whether the data should be tested against imprecision values provided by a manufacturer (or other source) and input these as relevant.

Figure 2:

Imprecision module output. (A) An interactive visualization of one QC level of the input data under the “Plots” tab. Boxplots are displayed for each day of data, showing the median, 25th and 75th percentiles, and tails representing 1.5 times the interquartile range. Red points represent the mean of each day. Individual data points are shown in black – a small amount of random jitter is added for increased clarity. (B) The summary of the posterior distributions of each of the imprecision model’s parameters under the “Summary statistics” tab. (C) Visualizations of the Markov chains and full posterior distributions of the model’s parameters. The dark, shaded areas of the posterior distributions represent the user-defined credible interval chosen under the “Data input” tab.

Assessing trueness against external quality assurance materials

The trueness of an assay, defined as the closeness of agreement between the arithmetic mean of a large number of test results and the true or accepted reference value [32], is fundamental to its clinical utility. This is pragmatically tested through comparison of the values obtained using the assay in question to those reported through an external quality assurance (EQA) scheme. Ideally, this comparison encompasses a large range of clinically-relevant concentrations of the analyte and is preferably performed in duplicate to account for within-assay variability [1]. The application allows the user to analyze these data in a number of different ways within the “Trueness (EQA)” tab using both frequentist (ordinary least-squares, Deming, or Passing–Bablok regression analyses) or Bayesian methods, depending on the user’s preference (Figure 3A and B). The Bayesian linear regression models take the form shown in Eqs. 4.6– 4.10 (where s x represents the standard deviation of all the EQA results) if no duplicate measurements are present and a full measurement error model (Eqs 4.11– 4.17) if they are available. In this measurement error model, the standard deviation of each measured data point is estimated from the duplicate measurements. Once fitted, the data are visualized and the modeling results presented to the user (Figure 3C). The user is again strongly encouraged to view the posterior distribution and MCMC traces of each parameter in their entirety under the “Bayesian model diagnostics” tab.

Figure 3:

Trueness assessed against EQA material. (A) The data input options for the EQA module. (B) Interactive tabulation of the uploaded data. The user is prompted to dynamically select the columns that represent the EQA means, uncertainty values, and the measured values. (C) Visualization of the input data (black points and associated error bars) and a randomized sample from the posterior distributions of the fitted Bayesian model (blue lines). The posterior distributions of the model’s parameters are summarized and presented to the user, alongside the results of the model checks.

Assessing trueness against reference materials

In addition to the assessments vs. EQA material, the trueness of an assay can also be assessed by comparison to a reference material whose analyte concentration has been assigned through the use of a reference method (or appropriate substitute). Such analyses are typically performed through measurement of the reference material on 3–5 occasions in duplicate [1]. These data are then often analyzed through the use to assess the significance of the difference between the measured and assigned values in a Neyman–Pearson, frequentist framework. For reasons discussed elsewhere [5, 16], Bayesian methods provide a formal and more intuitive way through which the probability of a difference between the reference and test methods can be calculated. This is especially true in this case, where a relatively strong prior distribution for the results is already known (the assigned value ± an assigned uncertainty). Within the “Trueness (reference)” tab, the application prompts the user to upload their data, enter the assigned value and associated uncertainty, and choose the relevant columns of their uploaded data (Figure 4A and B). A model is fitted to the data using the assigned value and uncertainty as the prior distribution (Eqs. 4.18– 4.21, where m p and s p represent the reference material’s assigned mean and SD, respectively, and ν represents the degrees of freedom of the Student’s t distribution). The Student’s t distribution is used here in order to allow for more extreme observations [30]. Thus, the model directly assesses the question: what is the probability that the investigated assay is providing different results to those expected for the reference material, given the data we have collected? If duplicate measurements are included in the data set, then a varying-effects model is fitted to the data in order to take into account the within-assay variation (Eqs. 4.22–4.24, where α r e p , i represents the mean for the i th pair of measurements and σ r e p represents the between-replicate variation), using the same likelihood function and residual prior (Eqs. 4.18 and 4.21, respectively). The results of the modeling are visualized for the user and a Bayes factor is presented as a measure of evidence against the assigned mean of the reference material (Figure 4C) [27]. Once again, the full posterior distributions and MCMC traces are shown in the “Bayesian modeling diagnostics” tab.

Figure 4:

Trueness assessed against reference material. (A) The data input options for the trueness against reference materials module. (B) The user is prompted to dynamically select the columns in the data that represent the measurements and their duplicates, if present. (C) Visualization of the prior, data, and posterior distributions. The posterior distribution for the mean parameter is summarized and the Bayes factor vs the prior distribution is shown alongside the result of basic model checks.

Assessing trueness against a reference assay

Comparisons of two methods for the measurement of a given measurand are commonly performed within clinical laboratories. The current UK ACB guidelines [1] suggest that at least 20 samples consisting of patient material should be assayed with each method in a timely manner (preferably in duplicate). The application allows the user to analyze these data in a number of different ways using both frequentist (ordinary least-squares, Deming, or Passing–Bablok regression analyses) and Bayesian methods, depending on the user’s preference, within the “Method comparison” tab (Figure 5A and B). As with the EQA data analysis, the Bayesian linear regression models here take the form shown in Eqs. 4.6– 4.10 (where s x represents the standard deviation of the reference method’s results) if no duplicate measurements are present, and a full measurement error model (Eqs. 4.11– 4.17) if they are available. The standard deviation of each measured data point is again estimated from the duplicate measurements present in the data. Once fitted, the data are visualized and the modeling results presented to the user (Figure 5C). This section of the application also performs a traditional Bland–Altman analysis [33], allowing the user to determine whether they wish to explore absolute (additive) or relative (proportional) differences between the two methods’ results (Figure 5D). In this application, relative differences are presented as log₂ fold-ratios of the results from the two methods – analogous to MA plots in high-throughput transcriptomics/genomics analyses [34] – as these more readily represent the direction and magnitude of the differences on a multiplicative scale and are conveniently symmetrical around zero.

Figure 5:

Method comparison module. (A) The data input options for the “Method comparison” module. Several regression methods are available and the user can select which type of Bland–Altman analysis they wish to perform. (B) Visualization of the input data and dynamic selection of the columns that represent the measurements and their duplicates, if present. (C) Visualization of the input data and a randomized sample from the posterior distributions of the fitted Bayesian model (blue lines). The posterior distributions of the model’s parameters are summarized and presented to the user, alongside the results of the model checks. (D) Bland–Altman analysis. The mean, SD, and SEM of the differences are shown.

Assessing the diagnostic performance of a test

The final tool within the application is designed to assess the performance of a given test to correctly predict a binary categorical outcome using a continuous variable (e.g. predicting “healthy” vs “disease” through the measurement of a biomarker). This is often achieved through the use of receiver operating characteristic (ROC) analysis, which typically involves the derivation of a hard analyte threshold based on minimizing a loss function that balances a test’s sensitivity and specificity. This type of analysis is problematic in medicine, however, for several reasons. Firstly, as clinical decision makers, we are most often analyzing scenarios in which there is significant stochasticity in a given clinical outcome due to measurement error, biological variation, and sampling variability. As such, there is often a significant overlap between the two groups being compared and thus probability estimates are most appropriate to best quantify the tendency towards one group or the other – i.e. directly inferring and interpreting the forward probability, P ( d i s e a s e | d a t a ) [35, 36]. Secondly, ROC – and indeed the alternative precision-recall curve – analyses do not directly inform the analyst as to the clinical utility of a new test, but instead focus on the prediction of a dichotomous classification through minimization of a loss function that is assumed to be equal across individuals [35], [36], [37], [38]. Lastly, ROC analyses are insensitive to the outcome’s distribution and thus are inappropriate for scenarios in which one category is much less frequent than the other. The “Diagnostic performance” module in this application instead first fits a Bayesian logistic regression model (Eqs. 4.25–4.30) to convert the predictor variable’s measurements to probabilities of the given outcome. The module then uses the output from this model (and the associated uncertainty in its parameters) to perform a decision curve analysis [37, 38] to best assess the clinical utility of the predictor variable.

As with the previous modules, the user is prompted to enter their data set in.csv or.xls(x) format and select the columns in their data that represent the analyte measurements, binary outcome measure, and which category of the outcome measure is considered a “positive” (Figure 6A and B). The results of the analysis are then generated within the “Plots and analysis” tab and include: (i) posterior draws of the expected value of the posterior predictive distribution vs the input data; (ii) a summary of the posterior distributions of the model’s parameters; and (iii) the results of a decision curve analysis (including 100 posterior draws of expected value of the posterior predictive distribution in gray in order to display uncertainty) (Figure 6C and D).

Figure 6:

Diagnostic performance module. (A) Data input for the “Diagnostic performance” module. (B) Visualization of the input data and dynamic selection of the measurements, outcome measure, and what is considered a “positive” result. (C, left panel) Posterior draws of the expected value of the posterior predictive distribution from the model (blue line) are plotted against the input data (black points). The chosen width of credible interval is shown as a gray ribbon. (C, right panel) A summary of the posterior distributions of the model’s parameters with the chosen credible interval. (D) The results of the decision curve analysis. The results of intervening in all and no patients are shown in purple and green, respectively. The decision curve analysis performed using expected value of the posterior predictive distribution of the model is shown in blue, with 100 posterior draws shown in gray in order to visualize the uncertainty in the analysis.