Lot-to-lot variation and verification

Introduction

A quantitative laboratory result is generally derived from a reaction between the measurand in the patient sample and the components of a reagent. The intensity of the signal generated from the reaction is then detected by the analytical instrument and subsequently compared against a calibration model, previously constructed using standards containing known concentrations of the measurand and corresponding signal intensity.

Reagent and calibrators lots manufactured in batches under similar conditions, are assigned the same manufacturers’ reference number (or ‘lot number’) [1]. However, in practice, manufacturing conditions may not always be identical between different batches. This may produce reagents and/or calibrators that interact differently with patient samples, subsequently altering analytical performance [2]. Other sources of variation may include the calibration model fitting procedure [3], manufacturing errors [4], logistical and warehousing issues, and processing errors, such as aliquoting and processing errors. These changes in analytical performance contribute to the lot-to-lot variation observed in quality control processes. Changes in analytical performance may be present in the form of bias and imprecision, contributing to overall measurement uncertainty. When such changes are significantly large, this may affect clinical interpretation resulting in suboptimal patient care [5, 6].

Lot-to-lot verification is undertaken by the laboratory to evaluate the magnitude of change in analytical characteristics between an existing (in-use) lot and a new (candidate) lot of reagents to ensure they meet predefined acceptance limits [1]. This exercise is important to ensure the long-term clinical performance of the laboratory tests. The Clinical Chemistry and Laboratory Medicine journal has served as an active platform for advancing the practice of lot-to-lot verification [7]. In this narrative review, key points related to lot-to-lot verification are discussed, with an emphasis on the literature published in this journal for the 60th Anniversary Special Issue. Additionally, these discussions may more broadly be applied to inter-method and inter-instrument comparisons.

Impact of undetected clinically significant lot-to-lot variations

To appreciate the vital importance of performing lot-to-lot verification procedures, one can review the literature of when there have been instances of significant undetected reagent lot differences, which highlight potential patient impacts.

In the first example, a gradual but significant cumulative positive lot-to-lot bias remained undetected over multiple years for insulin-like growth factor-1 [8]. This resulted in an increase in the number of subjects with results greater than the upper reference limit, which did not correlate with the clinical presentation. Clinician’s feedback regarding these clinically discordant results to the laboratory team, brought to light the long-term positive drift in the measurement procedure.

Following on from this a small but significant positive bias was present in the new lots of a prostate-specific antigen reagents [9]. The positive bias was not detected by routine lot-to-lot verification procedures and the reagent lots were put into clinical use. The positive bias caused a number of patients who had previously undetectable prostate-specific antigen measurements, to now have low detectable measurements. For some of these patients who had undergone radical prostatectomy for eradication of prostate cancer, a low detectable prostate-specific antigen measurement may indicate presence of residual prostate tissue or a recurrence of the prostate cancer according to recent local clinical practice [9]. Falsely detectable prostate-specific antigen measurements could have inappropriately prompted invasive procedures such as biopsies and further compounded patient distress.

These two illustrative examples are instructive in appreciation of the importance of the different aspects to lot-to-lot verification. In the insulin-like growth factor-1 case, the cumulative lot-to-lot biases were missed due to the use of an experimental design and statistical analysis, there were underpowered to detect small analytical changes [8]. Additionally, the cumulative effect of lot-to-lot change was not effectively identified and monitored by existing procedures. In the case of the prostate-specific antigen example there were some compounding factors, an under appreciation of the clinical significance of a small analytical bias, lack of appropriate follow up action for technical control rules indicating the presence of bias, and the need for clinical review and retesting of previously reported results that may have been potentially affected [9].

Another important observation from these two examples, is the significant change in analytical performance was observed by clinicians, who subsequently notified the laboratory. This led to the proposal of a quality system that directly monitors the longitudinal trend of laboratory results over time, such as patient-based quality control, becoming an important tool for lot-to-lot performance monitoring.

Pre-implementation considerations

Before establishing a lot-to-lot verification practice for a measurement procedure, the laboratory should systematically consider several important aspects: clinical, analytical, operational, and financial.

Clinical aspects the laboratory should seek to understand are the biology, pathophysiology, and clinical interpretation of the measurand [10]. The latter is particularly important as the clinical use of a measurand may evolve over time – as in the prostate-specific antigen example highlighted [9]. As such, it is important to keep abreast of the latest clinical practice guidelines concerning the measurand of interest. An appreciation of the clinical utility of the measurand will help determine the number of samples and concentration range to include in the verification experiment [1]. It will also assist the laboratory in development of the appropriate analytical performance specifications, interpretation of a failed verification exercises and appropriate troubleshooting and follow up actions including patient review [11].

The laboratory should regularly review method evaluation and routine internal quality control data to gain an understand of analytical performance of the measurement system. This will provide an appreciation of the baseline performance of the measurement procedure and guide the determination of appropriate analytical performance specifications, as well as any subsequent troubleshooting actions. If the baseline analytical performance of the measurement procedure had shown a recent change, then appropriate adjustments (i.e. reducing) to the analytical performance specification for lot-to-lot verification may be considered, with the aim to keep to the overall analytical performance within the quality requirements of the laboratory.

The operational and financial aspects of lot-to-lot verification also need to be considered when establishing lot-to-lot verification procedures. The laboratory should consider if it has the resources to identify, prepare and appropriately store remainder patient samples, that meet the number and concentrations selected for lot-to-lot verification, or whether commercial quality control materials (ideally with commutability demonstrated) may serve the purpose [12]. With consideration, patient samples may be pooled to meet the volume requirement for high replicate analysis procedures. Other considerations include the financial costs of performing non-revenue generating verification testing. Once the above pre-requisites have been duly considered, the laboratory can begin to plan the lot-to-lot verification process.

There are four key factors that influence the capability of lot-to-lot verification exercises to detect significant performance changes: analytical performance specification models and derivations; the testing material and matrix; the number of samples and replicates along with concentration range examined; the statistical approach and interpretation. In the following sections, these factors are discussed in greater detail.

It must be emphasized that the troubleshooting and follow up actions are what determine the outcome of a defective reagent lot [11]. The laboratory should have in place a standard protocol to systematically investigate any unacceptance change in analytical performance and decide whether to accept or reject the reagent lot. The laboratory should have an open disclosure policy and remedy any laboratory results that may have been affected by any missed errors arising from erroneously accepted defective reagent lots [9].

Analytical performance specification models and derivation

Analytical performance specifications for a lot-to-lot verification procedure, describes the limits within which any difference between the existing and new reagent lot is considered fit for purpose [1]. Analytical performance specifications are always a balance between rejecting/accepting a new lot number and one of the main factors that determines if a non-conforming reagent lot is identified by the verification exercise. Strong consideration should be given to the consequences of false acceptance whereby a non-conforming lot number is put into routine clinical use, which may result from an inappropriately wide analytical performance specification. When this happens, erroneous laboratory results that impacting clinical care may be produced. On the other hand, an inappropriately narrow analytical performance specification risks false rejection, whereby a suitable reagent lot is erroneously rejected, which may disrupt the provision of clinical services provided by the laboratory.

Appropriate analytical performance specifications should be selected a priori as the first step when establishing lot-to-lot verification procedures [1, 11], however, this can be a challenging task. The Milan Consensus provides guidance on sources of analytical performance specifications and include those based on outcomes study, biological variation, and state-of-the-art [13, 14], with biological variation and state-of-the-art analytical performance specifications being most commonly in use [15, 16]. Some available common resources and references for deriving analytical performance specifications includes the biological variation database [17], external quality assurance programs [18], regulatory requirements [19] and data produced during method evaluation or long-term quality control data [20]. Expert opinion, either locally [11], regionally, or internationally, can also be sought when selecting an appropriate performance specification. Nonetheless, laboratories also use arbitrary criteria that may not have any justifiable clinical or statistical basis [2].

Traditionally, the analytical performance specification for lot-to-lot verification is based on the total error model, which includes bias and imprecision components [21]. In general, analytical performance specifications for lot-to-lot verifications are not partitioned into these sub-components. However, some lot-to-lot verification experiments focuses solely on the bias component, such as comparing the mean of replicate/overall measurements made between the existing and the new lots. Subsequently a generous allowance is made for a single component of the performance specifications, such as bias [22].

More recently, the measurement uncertainty concepts have been applied to lot-to-lot verification exercises [22, 23]. Under this model [23], the allowable within-laboratory precision is allocated to between-reagent lot verification, based on the need for long-term stability of the measurement variability for a measurand. Mathematically, this is represented by [between-reagent lot variation ≤ analytical variation/(n + 1)] [23].

The selection of analytical performance specification should consider the clinical utility of the test and the potential clinical impacts of any significant deterioration in performance [24]. Drawing from the prostate specific antigen example above [9], the reporting of very low concentrations for detection of laboratory recurrence/residual cancer following radical prostatectomy, requires appropriately stringent lot-to-lot analytical performance specifications at this concentration to ensure small but clinically significant biases do not evade detection. These considerations also apply for other high sensitivity, qualitative or/semi-quantitative laboratory assays, such as high-sensitivity troponin and serological markers.

As indicated earlier, there is room to consider a more dynamic approach in the setting of analytical performance specifications for lot-to-lot verification to compensate for changes in analytical performance to ensure the overall quality specifications are continuously fulfilled.

Material and matrix selection

The material used for lot-to-lot verification should ideally have demonstratable commutability with patient samples. This ensures that results of the lot-to-lot verification reflect the analytical performance when testing patient samples [25]. Due to the scarcity of affordable, easily accessible, commutability-certified materials, laboratories often have to prepare comparison samples in-house [26]. This often entails the laboratory identifying and collecting suitable leftover patient samples at sufficient volumes while covering clinically relevant concentrations.

The preparation, storage, and retrieval, including appropriate thawing process, requires meticulous attention to ensure measurand integrity and avoid introducing pre-analytical artefacts that may affect the results of the verification, and this process overall may itself require validation. This is a resource-intensive process for any laboratory, that is further compounded by the number and complexity of the assays performed by the laboratory.

On the other hand, commercially available materials such as internal quality control products are pragmatic alternatives but may not be commutable with patient samples [27], [28], [29]. In such scenarios, there is a possibility that a significant change in analytical performance may go undetected by the lot-to-lot verification procedure. Of note, the commutability of a manipulated matrix for a commercial material, e.g. stripped serum, cannot be assumed since the processing of such samples (removal/spiked substances) may significantly alter the matrix, for example, removal of certain binding proteins or lipoproteins. Similarly, material from a different matrix (e.g. whole blood vs. plasma) may behave differently even though they are collected from the same source of peripheral blood [29]. The laboratory needs to carefully weigh the ability for error detection, costs, and operational practicality for a selected material.

The use of commutable material is particularly important for commercial quality control materials that have been shown to produce false flags or missed errors when used in internal quality control strategies [27, 28]. The laboratory should also avoid using commercial quality materials that are produced by the same vendor as the instrument, calibrators or reagents, more so if they share similar manufacturing processes. A pragmatic approach may include using a mixture of commercial and remainder patient samples to ensure coverage at clinically relevant concentrations.

If only commercial materials are used, then the laboratory should consider augmenting the monitoring of lot-to-lot performance changes using patient-based quality control [27, 28], bearing in mind that lot-lot changes will be identified retrospectively, and only after a number of patient samples have been tested to generate an alarm. Additional vigilance should also be paid to performance in external quality assurance programs in these settings.

Number of samples, replicates and concentration range

One reason for failure of a lot-to-lot verification procedure to detect a clinically significant change in assay performance is conducting an underpowered experiment. The statistical power of a verification procedure in detection of clinical meaningful change is heavily influenced by the number of samples and replicates tested.

A greater number of samples and replicate tests increases the statistical capability (power) to detect a difference through reducing the uncertainty surrounding the estimates [8, 30–32]. On the other hand, significant differences can evade detection when an inadequate number of samples and replicates are performed during the verification exercise. Hence, in detection of a small differences, a large number of samples and/or replicate tests needs to be performed.

However, there are finite resources that a laboratory can expend on lot-to-lot verification, with diminishing returns relative to the gain in statistical power with increasing number of samples or replicate testing. Consultation of power function graphs and published tables that provide guidance on these parameters can aid selection of the optimal sample size and replicates for the given resource constraints of the laboratory [16, 32].

Beyond the number of samples and replicates, the concentration range of the samples play an important role in determining whether a difference is detected by the verification exercise [1]. The selection of the concentration of the measurand should be informed by its clinical utility and should include the reference intervals and/or medical decision limits. The concentration should also span the interval commonly encountered in clinical practice. Additionally, if there had been a change in the application of the test, for example the lowering of reporting limit such as the case of the prostate specific antigen mentioned above, specific attention should be dedicated to the critical level using dedicated samples that are ideally commutable.

Statistical approaches and analysis

Lot-to-lot verification exercises can be undertaken in with in different forms of experimental designs. Operationally, these involve calibrating the measurement procedure using the existing and new reagent lot, ensuring a valid (in control) quality control run before testing the verification samples in a single run (same conditions). An appropriate statistical analysis should be applied to ensure correct interpretation of the observed data [2]. Similar to the a priori selection of analytical performance specifications, statistical analysis of the lot-to-lot verification data should be determined before the start of the experiments [1, 11]. If formal statistical tests are applied, the level of statistical significance (alpha) and power (1 – beta) should be determined based on the clinical risk of the test, the laboratory risk tolerance and resource availability.

The experimental design may involve measurement of several samples with or without replicate testing [2, 11, 16]. The difference in the laboratory results obtained from the existing and new reagent lot are then compared against a predefined analytical performance specification, either at the level of the individual samples or as a mean of all the samples from each reagent lot respectively. Alternately, a t-test can be performed to detect a statistically significant difference at a predefined level of alpha.

In another approach, the 90% confidence intervals of the average percentage difference in measurements between the existing and new reagent lots are compared against a predefined analytical performance specification [11].

Regression-based approaches have also been employed by some laboratories [2, 30, 31]. This often involves measuring a selection of samples across the measurement range and fitting a linear regression model to the data obtained from the existing and new reagent lots. The regression parameters derived include the slope, intercept, coefficient of correlation (r) and coefficient of determination (R²). These parameters are then compared against predefined acceptance limits that are often arbitrarily defined. Other approaches include modified Bland-Altman analysis and t-tests on the regression parameters [30, 31]. These approaches have poor statistical power as the sample size required to meet a reasonable statistical power (e.g. 90%) can be hard to achieve by routine laboratories [30–33].

It is important that the long-term trends in the analytical performance of a measurement procedure are monitored, and not just the discrete lot changeover differences. Small differences in performance, can compound when accumulated over time, resulting in large differences [8]. In the past, the cumulative effect of lot-to-lot changes has been sub optimally monitored by existing statistical approaches and procedures [8]. More recently, a statistical approach that considers the cumulative effect of lot-to-lot changes [33] and a graphical approach [11] have been described. Patient-based quality control (more details below) is also uniquely placed to detect long-term changes in performance and acts as a continuous quality monitoring process.

Collaborative efforts for lot-to-lot verification

Lot-to-lot verification can be a resource-intensive exercise when performed by an individual laboratory, which limits the number of samples or replicate that can be tested [1]. A collaborative approach, whereby several laboratories performed the verification exercise using a common set of protocol and procedures, can allow pooling of data for more effective statistical analysis and detection of significant changes [1, 11]. Additionally, forming such coalition within a local laboratory community can also improve logistical resilience, since a higher quantity of reagents of the same lot can be reserved and stored. This approach can also be expanded regionally or nationally, under the auspices of external quality assurance programs [11].

Patient-based quality control

Patient-based quality control is a laboratory quality practice that uses patient results to monitor the analytical performance of the measurement procedure [34], [35], [36], [37], [38]. It is more commonly described as a continuous (‘real-time’) quality control tool. This approach can also be adapted for long-term performance monitoring across reagent lots within the laboratory. An advantage of using patient-based quality control is the lack of target value adjustment that may be necessary for internal quality control. The target value adjustment can disrupt the long-term performance trend of the measurement procedure and mask important changes in assay performance [39]. Another important advantage is that because patient results are used, this approach does not suffer from lack of commutability.

At the same time, the patient-based quality control concept can also be applied more broadly to multi-laboratory settings similar to that of external quality control programs. A European-driven ‘Percentiler’ international program was an attempt to use patient medians (a form of patient-based quality control) of thyroid function tests and successfully monitored the long-term stability of calibrators and reagents [40].

Follow up actions

Just like any other aspect of laboratory quality practices, the outcome of any errors detected or missed, depends on the follow up actions, with this point well illustrated in the prostate specific antigen example. When a lot-to-lot verification fails, it is important to examine the process for any possible causes that may explain the observation. If a false rejection is suspected, the laboratory may consider repeating the experiment, in part, or in whole, with more patient samples and/or replicates to improve the statistical power, where appropriate. It is important that the laboratory does not alter the analytical performance specification or data analysis strategy to fit the observed performance.

Once the laboratory is satisfied with the presence of true lot-to-lot failure, there are several options the available to the laboratory. The first is an outright rejection of the new reagent lot and request an alternative lot number from the manufacturer. However, this option may not always available as the manufacturer may not have alternate reagent lot readily available locally. This may lead to service disruptions where the laboratory needs to collaborate with another laboratories meeting its own quality requirements for sample referral to ensure continuity of clinical care. This may be challenging for highly specialized assays. In the meantime, the laboratory also needs to work out the internal logistics of preparing, storing, and transporting the samples to the alternate laboratory service, ensuring that sample integrity is maintained.

Alternately, the laboratory may consider introducing post-analytical correction/adjustment factors in collaboration with the manufacturer [41–44]. It is important that these factors are derived using a sufficient number of commutable (patient) samples to ensure the adjustment factors are applicable for patient samples. This approach should be considered judiciously and managed with careful oversight, since it has large systematic impacts on results interpretation. Helpful guidance on this important topic has been published elsewhere [42].

These follow up actions should be protocolized and documented to ensure consistent decision making and appropriate measures are taken to remedy any errors that affected patient results. The verification parameters, process and follow up actions should be regularly reviewed and revised where necessary.

Role of the manufacturer

There are significant challenges facing manufacturers regarding lot-to-lot variation and include determining the balance between degree of manufacturing precision and meeting the regulatory and clinical requirements. Underpinning this is the analytical performance specification and validation procedures the manufacturer uses to determine the acceptability of the lot-to-lot variation prior to releasing to customers for routine use [1]. This information is generally not available to the laboratory customers [42], which impedes their ability to interpret findings relative to the specifications of the manufacturer. On the other hand, at times manufacturers are limited by a lack of access to sufficient commutable materials. Closer collaboration between the manufacturers and the laboratory profession is required to better define the analytical performance specifications as well as increase access to commutable testing materials, in an effort to mitigate lot-to-lot differences.

Role of external quality assurance programs

External quality assurance providers that serve a large number of laboratories, are uniquely positioned to overcome the challenge of insufficient/small sample sizes that limit the statistical power faced by many individual laboratories [1]. These providers can pool returned results for combined analysis to increase power. However, for this approach to provide an advantage over routine laboratory practices, the external quality assurance provider needs to meet several criteria: have commutable materials available, clinically relevant concentrations covers, have a sufficiently high frequency of sample distribution, a record of the reagent lot number, a sufficiently number of participating laboratories, and capabilities for stratified data analysis by reagent lot, and analytical method [45, 46]. A successful pilot of such an attempt has been reported recently [11].

Role of regulatory bodies

Through post-market surveillance programs, regulatory bodies can also play an important role in clinical risk mitigation for lot-to-lot changes [1]. Regulatory bodies may also set mandatory manufacturing and quality standards that manufacturers must ensure compliance with. A tripartite collaboration between regulatory bodies, manufacturers and laboratory medicine professional bodies is necessary to ensure the requirements of all stakeholders are appropriately considered and met within the resourcing allocations of the health care systems.

Conclusions

Lot-to-lot verification is an integral component for monitoring long-term stability of a measurement procedure. The practice is challenged by resource requirements as well as uncertainty surrounding experimental design and subsequent statistical analysis that is optimal for individual laboratories, although guidance is becoming increasingly available. Collaborative verification efforts as well as application of patient-based monitoring are likely to further improve identification of any differences in performance in a timely manner. Appropriate follow up action of failed lot-to-lot verification is a mandatory requirement. Manufacturers need to increase transparency surrounding release criteria and work with laboratory professionals to ensure acceptable reagent lots are released to end users. A collaboration between all stakeholders including regulatory bodies, manufacturers, and laboratory medicine professional bodies is key to developing a balanced system whereby regulatory, manufacturing and clinical requirements are met, to minimize differences between reagent lots and ensure patient safety. Clinical Chemistry and Laboratory Medicine has served as a fertile platform for advancing the discussion and practice of lot-to-lot verification in the past 60 years and will continue to be an advocate of this important topic for many more years to come.