The internal quality control in the traceability era

Implementing and maintaining metrological traceability

Laboratory Medicine effectively contributes to the medical decision-making process when results provided on clinical samples are accurate, i.e. associated to a clinically tolerable measurement error, and equivalent, i.e. no matter where and how they are obtained [19]. Basic is the implementation of the traceability of IVD measuring systems to recognized high-order references in order to express the results obtained by the commercial procedure in terms of the values obtained at the highest available level of the metrological hierarchy [20]. The European legislation (the European Union [EU] Directive and, in the near future, the new EU Regulation 2017/746 on IVD medical devices) requires IVD manufacturers to accomplish this [21], [22].

In implementing traceability, IVD manufacturers have a key role and unique responsibilities (Table 2). Information regarding reference materials and reference measurement procedures, useful for assigning traceable unbiased values to commercial calibrators, can be obtained from the Joint Committee for Traceability in Laboratory Medicine database [23]. The value assignment to calibrator materials should be managed by correcting any bias against references and, once the bias has been corrected, the uncertainty of this correction should be included in the MU of calibrator value (u_cal) [13]. Manufacturers have the duty of estimating such MU that must include all uncertainties pertaining to the selected calibration hierarchy for the specific measurand, beginning with the highest available reference down to the assigned value of the calibrator for the commercial IVD medical device. The u_cal should represent a portion of the total MU budget allowed for medical laboratories in order to leave enough uncertainty budget usable by individual laboratories to produce clinical results of acceptable quality [6], [24].

IVD manufacturers are also asked to provide end-users with technical documentation, instructions for use and a QC material suitable for post-market surveillance of the measuring system performance, when working according to the manufacturer’s indications. End-users must strictly observe these indications, as only operating in conformity with them the intended purpose of the marketed measurement system can be warranted, including the performance declared in terms of traceability [1]. IVD manufacturers are therefore requested to provide QC materials as a qualified part of the measuring system; these materials, representing the IQC component I, should be designed for daily monitoring of the measuring system alignment, with appropriate target values and acceptability range. This relies on the concept that, if the traceability of the measuring system to higher-order references is granted by the manufacturer, QC materials, which are part of the whole system, have to be a suitable surrogate of the employed (and declared) reference to permit routine checking of the correctness of system alignment to such reference.

Estimating measurement uncertainty

In lay terms, MU represents the interval of possible values for a measurand for which an x result was obtained. Recently published documents have clarified how MU should be calculated and why this information matters in medical laboratories [13], [25], [26]. MU associated with laboratory results should combine two uncertainty components: the u_cal and the MU accounting for random sources (u_Rw) [13], [26]. As mentioned earlier, the former combines the MU of the higher-order reference selected by the IVD manufacturer for implementing traceability with the MU deriving from the process for assignment of calibrator values, including the bias correction, if any.

u_Rw gives information about the stability of the measuring system over time and its variability when employed by an individual laboratory. Major variables that may affect the imprecision of the measuring system and individual laboratory performance in using it are summarized in Table 3. As the measuring system imprecision is an intrinsic characteristic of the system itself, it is under the manufacturer’s responsibility and it is not controllable by end-users. IVD manufacturers should be aware, however, that if the system imprecision is intrinsically too high, it does not leave margin for MU to the end-users causing u_Rw to exceed the desirable performance specifications [27]. When the individual laboratory estimates u_Rw, through the IQC component II (see below), the two components (i.e. intrinsic system imprecision and individual laboratory performance) are evaluated together.

Deriving analytical performance specifications according to Milan 2014 consensus

As mentioned before, objective criteria for defining analytical performance specifications (APS) for different measurands were defined during the EFLM Strategic Conference held in Milan in 2014 [28]. Three models based on (a) the effect of analytical performance on clinical outcome, (b) the components of biological variation of the measurand and (c) the state of the art of the measurement (defined as the highest level of analytical performance technically achievable), respectively, were proposed [16]. One of the innovative aspects of this recommendation was to emphasize that certain models are better suited for some measurands than for others, with attention primarily directed toward the biological and clinical characteristics of each measurand [29]. Grading different levels of quality (i.e. minimum, desirable and optimum) for APS was also highlighted as very important because it may stimulate IVD manufacturers to work for improving the quality of assays in case of unacceptable or minimum performance [17].

Characteristics of IQC component I material

Table 4 lists the characteristics of QC materials that should be provided by the IVD manufacturer as an integral component of the CE (“Communautés Européennes”)-marked measuring system. Firstly, material concentrations should be in line with clinically relevant decision limits of the measurand. Recently, using the highly sensitive assay for measuring cardiac troponin T (hs-TnT) as an example, we showed the relevance of monitoring the performance of the measuring system at all clinically relevant thresholds [30]. In patients admitted to the emergency department with symptoms suspicious for acute coronary syndrome (ACS), the diagnosis of non-ST-segment elevation acute myocardial infarction can be ruled out, with a negative predictive value of >99.5%, on a single hs-TnT measurement if the marker concentration is undetectable [31]. Even relatively small variations in assay performance at concentration near the limit of detection (LOD) may hence affect the classification of patients with suspected ACS. Accurate calibration of the hs-TnT assay in the low range of concentrations is therefore of the utmost importance and, among the tools that laboratories need to check the performance of hs-TnT, is a QC material with concentration near the assay LOD to monitor baseline drifts following assay calibration [32]. Unfortunately, QC materials offered by the manufacturers together with highly sensitive troponin assays do not usually cover very low marker concentrations, leaving assays vulnerable to potential drifts, associated with reagent and calibrator lot changes, platform-to-platform variability and different schemes of instrument maintenance, as well as instrument decline and malfunctioning, that may pass unnoticed.

Secondly, for effective surveillance of traceability, QC target values must be assigned in order to confirm that the measuring system performance is properly unbiased. Note that for this type of materials commutability is not required because they are designed to monitor the specific measuring system for which they are provided. In practice, however, manufacturers of measuring systems commonly derive mean values of their QC materials just from replicates performed by independent laboratories using the same measuring system, with no explicitly certified quality, and frequently provide them as approximate targets, only for convenience, with no trueness claims regarding assigned values. Our recent experience with the use of the Alinity c measuring system for alanine aminotransferase (ALT) illustrates the importance of the issue and possible practical mishaps. The method (Activated ALT, code no. 08P1820) uses a three-level IQC material for system alignment verification provided by Abbott Diagnostics as part of the CE-marked measuring system (Multichem S Plus Technopath, ref. 08P88). In our daily practice, we noticed a systematic relevant underestimation for our ALT measuring system, declared traceable to the IFCC reference procedure, at level-1 QC, corresponding to average physiological ALT concentrations. Figure 1 displays sequential actions we took to correct the problem, including repetition on new aliquots of the QC material, recalibration of the system and then QC repetition, replacement of reagent tank, change of the reagent lot and, finally, manufacturer technical intervention. As all these actions were ineffective, trueness investigation became mandatory to show if the repeated out-of-control results were due to a true system misalignment from the declared IFCC reference. The investigation showed a very good correlation between the ALT patient results by our procedure and those by the IFCC reference procedure, confirming the perfect standardization of the Alinity ALT measuring system, despite the marked underestimation of level-1 QC value, which was clearly wrongly assigned [33]. In verifying what the QC manufacturer declared in its insert, we found no accuracy claims regarding the assigned values and a sentence stating that ALT mean values were derived from independent laboratories without any trueness check, confirming the high vulnerability of this approach adopted to assign QC values in terms of metrological traceability. Overall, taking care of the metrological quality of the assigned values to offered QC materials is not an issue for manufacturers, with very few exceptions [34].

Figure 1:

Level-1 quality control (QC) results by alanine aminotransferase (ALT) measuring system in the authors’ laboratory, and unsuccessful actions implemented to correct the problem.

See text for more information.

Thirdly, the acceptability range, which defines the tolerance of value deviation from the target, should permit the suitable application of test results in clinical conditions. Information reported on QC data sheets shows that the acceptability range provided by manufacturers is usually based on the statistical dispersion of data obtained by n laboratories (e.g. ±2 SD or ±20% of the mean value), with no relationship with clinically suitable APS. Sometimes data from interlaboratory programs are used in the determination of validation range, with the risk to include results from laboratories working under biased analytical conditions, and quite often both mean QC values and ranges are provided only “as guide”, with the recommendation that each laboratory should establish its own acceptability limits. Finally, the same IQC statistical limits are commonly used for all tests carried out on the same measuring system, even if their clinical use and biological behavior are different. This explains why on many analytical platforms it is only possible to set statistical dispersion parameters for IQC program and not predefined measurand-dependent APS.

Lastly, enough stability of the QC material is required to monitor the performance of the measuring system under all variables influencing or deteriorating it. Sometimes it could be difficult to obtain the optimal stability in all different QC manufactured batches. This may further create difficulties in the interpretation of possible decreasing trends in QC results, by confounding a real drift of the measuring system with the analyte instability in the QC material.

Criteria for interpreting IQC component I and establishing validation limits

In general, IQC programs should detect situations when IVD measuring systems are out of control and results of poor quality are at risk of being produced [18]. This is fully dependent on the criteria applied to the interpretation of IQC results and on the definition of desirable quality. In the last 40 years, a huge amount of literature has been published about recommended interpretative approaches for IQC to be implemented by end-users of IVD medical devices [35], [36], [37], [38], [39], [40]. The most popular approach, originally proposed by Westgard et al. [35], is based on a combination of multiple interpretative statistical rules based on different multiples of SD from the QC mean. Although not very recent and limited in the number of surveyed laboratories, some audit has shown that ~60% of medical laboratories used multirules for data analysis, whereas the remainder reported only a single 2-SD rule to define an out-of-control situation [41]. Doubtless, the latter, i.e. checking if the single QC value is (or is not) in the acceptable range (a sort of “macroevaluation”), represents the first interpretative criterion adopted by medical laboratories to accept/reject analytical runs. QC results outside the acceptability range promote immediate corrective actions to bring the situation under control again and before patient results related to the samples analyzed in the affected run are released. What is clear, however, is that this approach, if used in isolation, is not enough sensitive and precocious enough to prevent poor quality results on clinical samples. The interpretation of QC results by checking their temporal trend without waiting for out-of-control signals is surely more effective. As the effectiveness, in terms of early warning, of this “longitudinal evaluation” may be influenced by the frequency of QC testing, the establishment of this frequency is central. Parvin recommended to adapt the frequency to events, both expected and unexpected, potentially influencing the quality of results [42]. Although the determination of the optimal QC frequency may require the application of sophisticated statistical models, the organization of laboratories for measuring clinical samples can drive decisions [18]. For clinical samples assayed in batch, QC should be at minimum tested at the beginning and end of each analytical run. In continuous mode, which is now typical for laboratories using total automation, QC materials should be measured periodically along with clinical samples. CLSI C24 called this type of QC schedule “bracketed QC” and suggests considering several potential influencing factors in defining QC frequency [18]. In our core-lab facility, working by a continuous first-come/first-serve processing of samples [43], we perform QC component I three times per day, i.e. every 8 h.

The longitudinal QC evaluation can be done using two main judging criteria: (a) checking for significant trend of QC values and (b) evaluating how much the measuring system misalignment influences u_Rw, used for the calculation of MU of patient results. In applying the first criterion, decreasing or increasing trends of IQC results toward the upper or lower limit of the acceptability range are investigated, even if the macroevaluation of individual QC results shows that they are still within the acceptability range. This approach allows laboratories to implement corrective actions earlier, before the measuring system definitively gets worse. The second criterion relates to the assessment of the influence of variability of measuring system alignment on u_Rw. Shifts in QC results due to poor calibration or to change in reagent or calibrator lots, which occur during daily activity, may once again go undetected if QC values are judged against the acceptability range. However, the sudden change in the alignment of the measuring system may be responsible of unacceptable u_Rw. One may easily note that what discussed earlier for longitudinal QC evaluation is a type of visual interpretation that was objectified by the Westgard’s rules [35]. What is lacking is the link with the new scientific background described in the previous chapter “The new background”. To obtain this, the acceptability range for QC component I should correspond to APS for MU derived according to the appropriate Milan model and it should be set based on unbiased target value of the material obtained by the manufacturer as the mean of replicate measurements on the same measuring system optimally calibrated to the selected reference. Figure 2 displays a case study where, by using the acceptability range provided by the manufacturer, the QC shift, causing an unacceptable u_Rw, was not alarming. On the contrary, when the acceptability range was defined according to the appropriate APS for MU, the shift of the measuring system alignment, significantly affecting MU of patient results, was noticed and corrective actions immediately put in place.

Figure 2:

Example of the impact of the different definition of acceptability limits for quality control (QC) component I.

Measuring system: Architect c4000 enzymatic assay for hemoglobin A_1c (Abbott Diagnostics), as described in ref. [44]. QC material: BioRad Lyphochek Diabetes control material (ref. 740), mean value: 76.3 mmol/mol. The upper panel displays the situation using the defined acceptability range provided by the manufacturer. The QC shift (arrow), causing an excessively elevated measurement uncertainty (MU) on clinical samples, was not identified. However, if the acceptability range was defined according to the appropriate analytical performance specification (APS) for MU (±3.8%, expanded MU for a coverage factor=2), the shift of the measuring system would have been identified and corrective actions immediately put in place. Dashed lines in the graphs show the upper and lower QC component I acceptability limits.

Estimate u_Rw using IQC component II

From a statistical point of view, the u_Rw represents the dispersion of replicated measurements performed on a QC material, expressed as SD or CV. The ISO 20914:2019 technical specification defines the optimal condition for obtaining u_Rw as that estimating “intermediate within-laboratory precision” [26]. Accordingly, QC measurements should be performed over a sufficiently extended period able to include all changes in measuring conditions influencing u_Rw reported in Table 3. The recommended time span is 6 months, as it allows to cover those significant sources of u_Rw [13], [26]. Table 5 summarizes the steps to be followed to obtain a robust estimate of u_Rw. Once u_Rw has been estimated, it must be combined with u_cal, provided by the manufacturer of measuring system, to obtain MU on clinical samples as follows: √(u_cal²+u_Rw²) [13], [26].

Characteristics of IQC component II material

The characteristics of the QC material to be used for u_Rw estimate have been previously described in detail [5], [6], [13]. The material must be different from that used for the verification of system alignment, i.e. IQC component I material. The latter is used to validate analytical runs and should have different characteristics, described in the previous section of this paper. Importantly, if the component I fails, no other measurements should be performed until the situation is brought back under control. Differently, measurements of component II material for u_Rw estimate must be part of an already validated analytical run, as similarly as possible to those of clinical samples. Importantly, the QC component II material should be commutable as results obtained on non-commutable materials may not reflect performances achieved by the same measuring system on clinical samples in terms of random MU [45]. Hage-Sleiman et al. [46] showed the misleading results obtained in the estimate of precision profile when using a non-commutable IQC material, taking as example a highly sensitive troponin assay. As the use of a single biological sample (commutable by definition) for the evaluation of MU over an extended period is not feasible, the use of adequate commercial QC materials or sample pools is unavoidable [47], [48]. Lastly, QC materials for component II should have analyte concentration levels close to clinical decision limits or, at least, to employed reference limits. This is important because, for most, if not all laboratory tests, MU varies with the analyte concentration, usually decreasing with increasing concentrations.

Criteria for interpreting IQC component II and establishing clinical suitability of estimated MU

Once obtained, IQC component II data should be first critically reviewed and proper decisions related to their management taken before moving on to the u_Rw estimate. Figure 3 shows three examples of component II data evaluation with respective outcomes.

Figure 3:

Examples of evaluation of internal quality control (QC) component II data and their related outcomes.

u_Rw, standard measurement uncertainty due to random error sources.

The ISO 20914 is categorical in stating that “estimating the uncertainty of the results produced is of very limited value unless it can be compared with the allowable MU based on the quality of results required for medical use” [26]. Therefore, in order to ascertain if estimated MU for a given laboratory result may affect its interpretation, clinically allowable MU should be defined. As mentioned before, APS for MU of clinical results should be derived by using one of the three EFLM recommended models [16]. Ceriotti et al. clarified the criteria for assigning different measurands to one of these models [29]. For instance, for cardiac troponin, a marker having a pivotal role in the diagnosis of myocardial infarction, the outcome-based model should be used. Using this model, a standard MU ≤10% at the concentration corresponding to the 99th percentile upper reference limit represents the desirable goal [32]. According to the classical Fraser’s paradigm for deriving APS for random variability [49], the quality level can be further modulated to minimum (10%+½ 10%=15%) and optimal (½ 10%=5%) APS. The biological variation model is used when the measurand has a high homeostatic control. By using published data about the average intra-individual biological variation of measurands with these characteristics and the quality level grading, APS for standard MU can be established [23], [33]. Finally, if the measurand has neither central diagnostic role nor enough homeostatic control, the state-of-the-art model should be used. As an example, we can refer to the human chorionic gonadotropin (hCG) measurement, intended as a test to detect pregnancy. hCG is not used for clinical diagnosis and it is not normally produced in a healthy status. As the definition of “state of the art” requires defining the highest level of achievable analytical performance [16], we compared standard MU of three widely used measuring systems (Roche Modular Evo, Abbott Architect i2000SR and Alinity i), estimated by the approach described in this paper by using a QC component II material, obtaining values of 2.3%, 5.4% and 3.8%, respectively. As a result, the obtained Modular Evo performance (2.3%) can be considered as representing the state of the art of the measurement to be employed for the definition of APS for hCG MU. If we consider this APS as desirable, we can also modulate the quality level to, e.g. minimum goal (2.3%+½ 2.3%=3.5%).