Overcoming challenges regarding reference materials and regulations that influence global standardization of medical laboratory testing results

Introduction

Laboratory tests are essential for many medical decisions [1, 2]. There is a long development pathway between newly discovered potential biomarkers, clinical validation and medically meaningful laboratory tests [3].

Metrological traceability requires defining the measurand and using fit-for-purpose reference materials in the calibration hierarchy. The medical laboratory community has developed a roadmap for an infrastructure intended to enable equivalent laboratory test results for a measurand from all in vitro diagnostic (IVD) medical devices (IVD-MDs), also called end-user medical laboratory measuring systems, that is described in the International Organization for Standardization (ISO) standard 17511:2020 In vitro diagnostic medical devices – Requirements for establishing metrological traceability of values assigned to calibrators, trueness control materials and human samples [4]. The Joint Committee for Traceability in Laboratory Medicine (JCTLM) lists in its online database [5] commercially available certified reference materials (CRMs) for approximately 160 analytes and reference measurement procedures (RMPs) for approximately 180 measurands that conform to the applicable ISO standards [6], [7], [8], [9], [10]. Other providers of reference materials, including the World Health Organization, offer an additional few hundred reference materials intended for metrological traceability but do not submit these to independent review by the JCTLM.

Higher-order CRMs and RMPs are unfortunately only available for a minority of the thousands of measurands used in medicine. Even when the available metrological traceability resources are used, results for clinical samples measured using IVD-MDs from different manufacturers may differ enough to pose a risk for incorrect medical decisions [11], [12], [13], [14], [15]. A contributing factor is that some matrix-based CRMs are not commutable with clinical samples and thus introduce a bias when used in the calibration hierarchy of an IVD-MD [16, 17]. Noncommutable matrix-based CRMs may be in current use because the importance of commutability was not fully appreciated at the time the materials were produced [18, 19] or new IVD-MDs came to market after commutability of a CRM was assessed.

Manufacturers of IVD-MDs have a central role in ensuring that results are standardized. Two situations are encountered: an established biomarker needs improvement, and a new biomarker becomes available. A CRM is unlikely to be available when an IVD-MD for a new biomarker is initially developed and introduced. Consequently, an IVD-MD manufacturer utilizes the best available reference material in its calibration hierarchy for an IVD-MD for a new measurand (the quantity intended to be measured). When other manufacturers introduce IVD-MDs for the same measurand, the ideal practice is to try to make results equivalent to those already on the market. However, because higher-order CRMs or RMPs may not be available, differences in calibration hierarchies lead to differences in patient sample results among different IVD-MDs used in clinical laboratories. A need for standardization typically becomes appreciated when a measurand is used with decision values defined by medical practice guidelines.

A workshop was held in December 2021 to address the challenges associated with the availability and use of CRMs and with meeting regulatory requirements in many countries when recalibration to achieve standardized results is desirable. The workshop was organized by the Scientific Division of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC), the International Consortium for Harmonization of Clinical Laboratory Results (ICHCLR), and the JCTLM. The workshop had approximately 400 participants from 65 countries. The meeting agenda, speakers, and organizing committee are included in the supplementary material that accompanies this report of the consensus discussion and recommendations from the workshop.

The medical need for standardized results

In clinical pathways, laboratory tests support decisions related to screening/assessing health risks, diagnosis/differential diagnosis, selection of treatment, monitoring of health status or response to treatment, and prognosis. Laboratory tests may be used as part of a triage approach to select patients for further investigations such as imaging or to refine a diagnostic process. Laboratory tests, if not used in the right patient population for answering the right medical question or if misinterpreted, could potentially cause harm to patients and lead to wrong medical decisions and actions, unnecessary further investigations, and unnecessary costs to both the patient and the health care system.

Test results need to be equivalent between different IVD-MDs used by different laboratories to ensure consistent interpretation of laboratory results, for evidence-based medicine, to develop and implement decision values, and to avoid harm to patients from diagnostic errors. Additional benefits of equivalent results include facilitating patients moving across health care institutions without re-testing, combining data in medical databases, and medical audit of laboratory data. Standardized test results improve clinical effectiveness and cost-effectiveness of care because the analytical performance of tests impacts their medical performance.

Challenges for producers and users of matrix-based CRMs

Matrix-based CRMs are frequently used in the calibration hierarchies for end-user medical laboratory IVD-MDs. Ensuring that such materials are globally available for more measurands is a key goal for fit-for-purpose calibration hierarchies. The workshop identified five key challenges for CRM producers, including National Metrology Institutes (NMIs), and developed recommendations to address these: 1) measurand prioritization; 2) collaboration between NMIs to develop standardization resources; 3) collaboration between IVD industry and CRM producers; 4) the best way to address commutability requirements of CRMs; and 5) the need for a shared vision on how different bodies could work most effectively together to achieve their common goals for standardized results.

Currently, there is no single global prioritization and planning system for harmonization in laboratory medicine. Examples of current processes for prioritization include: IFCC Scientific Division review of ad hoc standardization proposals, lists of national priority analytes provided by clinicians, information from EQA providers to identify problematic IVD-MDs [20], national health care systems identifying high-frequency tests and those leading to highest costs in terms of reimbursement, international public health organizations identifying biomarkers for risk identification, input from consortia and other expert groups, and surveys of stakeholders. The ICHCLR was created in 2013 in response to a 2010 workshop recommendation for a global process to prioritize measurands in need of harmonization [21, 22]. The ICHCLR maintains a database of measurands that indicates their harmonization status and priority for harmonization based on medical impact and frequency of use. Further clarity on the process used by ICHCLR was requested.

The workshop participants agreed that prioritization of measurands for standardization should be based on their impact on medical decisions. High priority measurands are those that are part of the definition of a disease, such as creatinine for estimating glomerular filtration rate; and HbA_1c and glucose for diagnosis of diabetes. High priority measurands also include those that have well-defined guideline-driven decision limits for medical action, such as prothrombin time/international normalized ratio for anticoagulant treatment monitoring; low-density lipoprotein cholesterol for monitoring lipid-lowering therapy; and drugs with a narrow therapeutic window or target concentration for treatment. Measurands that are used more frequently should be prioritized higher for standardization than those used less frequently.

No single institute or country has the resources to produce all required CRMs. Nearly all IVD-MD manufacturers operate globally, which raises a challenge for national institutes to collaborate most effectively. The IVD industry prefers one continuing supplier of CRMs for individual measurands, with an alternate supplier desirable, to provide lot consistency to assist with manufacturing controls and regulatory conformity. An essential need for the IVD industry is a commitment to the ongoing provision of a CRM over time. One CRM producer reported that IVD-MD manufacturers are involved in working groups of the IFCC Scientific Division and requests from the working groups constituted an important mechanism for shaping their activities.

IVD-MD manufacturers raised concerns about maintaining metrological traceability when a CRM or a replacement batch was no longer available. IVD-MD manufacturers supported development of a standard format technical dossier for new (or modified) CRMs that they could use as supporting documentation for submissions to regulatory authorities and/or notified bodies. There is currently little interaction between CRM producers and the IVD industry at the early stages of new IVD-MD development, partially influenced by the reality that new biomarkers do not always lead to a medically usable product, and that commercial considerations may require confidentiality agreements with potential ethical implications.

Once a new biomarker is identified as medically useful, participants suggested the product development cycle in the IVD industry is usually 2–3 years, although for rapidly evolving diseases such as COVID-19 the test development cycle was 2–3 months. IVD-MD manufacturers are generally required to develop internal standardization as part of the initial product development process, sometimes in a very short timeframe. Consequently, a common occurrence in laboratory medicine is standardization which involves the challenge of “retrofitting” existing calibration hierarchies to subsequently developed higher-order reference systems. Challenges can occur in this approach, for example, two calibration hierarchies were developed independently by two laboratory medicine organizations for C-peptide in response to a medical need for standardized results [23, 24]. However, manufacturers have been reluctant to recalibrate their IVD-MDs because a consensus was needed on which calibration hierarchy to apply and the need to fully justify the efforts and costs associated with regulatory compliance.

The ICHCLR harmonization status review process has also found that there were some measurands with higher-order calibration hierarchies for which implementation was disappointing. CRM producers reported that targeting the most frequently performed tests had challenges, including medical decision limits that were established many years ago and the possibility that change could lead to confusion amongst physicians. In addition, acceptance of new CRMs was, in some cases, limited by the financial challenges of recalibration for IVD-MD manufacturers. The participants expressed interest in the possibility of incorporating new CRMs early in the product development cycle to reduce the challenges associated with the “retrofit” mode, and this approach remains an option for further consideration.

Demonstrating the commutability of a matrix-based CRM with specific IVD-MDs is a prerequisite for the material’s uptake and use [18, 19, 25–28] but is also highly resource-intensive for the CRM producer. Participants reported that commutability studies constitute a major component (as much as 70%) of resources required for CRM production. Finding efficient and cost-effective ways of conducting commutability studies is key to the sustainability of the process. CRM producers would benefit if more information on successful production processes for commutable materials was available and if simplified processes for commutability assessment for replacement batches of CRMs were developed.

International schemes (e.g., Bureau International des Poids et Mesures [BIPM] – Mutual Recognition Arrangement comparisons and IFCC External Quality Control for Reference Laboratories) exist for demonstrating the accuracy of reference material value assignment using RMPs. However, there is currently no analogous scheme for demonstrating commutability. CRM producers reiterated that the resources required for demonstrating the commutability of their materials constituted a challenge that could be eased with collaborative activities with IVD industry and/or producers of commutable EQA materials. For example, the National Institute for Standards and Technology, in collaboration with IVD-MD manufacturers, conducted an interlaboratory study through the Vitamin D Standardization Program to assess commutability of CRMs and EQA samples for measurement of serum total 25-hydroxyvitamin D [29]. Similarly, a collaboration among Laboratoire National de Métrologie et d’Essais, Instand reference laboratory service, hospital laboratories and IVD-MD manufacturers evaluated commutability for CRM and EQA samples for HbA_1c [30]. Participants questioned if IVD-MD manufacturers could continue to use non-commutable materials as part of their calibration hierarchies when alternative commutable CRMs were available. Such considerations become important when a well-established measurand with IVD-MD traceability to an existing comparator device is in use and a more fit-for-purpose commutable CRM becomes available.

Challenges to meeting regulatory requirements in different countries or regions

Many countries and regions have regulations requiring IVD-MDs to pass a formal pre-market review for safety and effectiveness prior to being sold. Obviously, regulation of IVD-MDs is an essential function in the interest of safety and effectiveness for the populations served. However, regulations are national or regional, and while intended to accomplish the same safe and effective assurance, they vary among countries and regions. For example, different requirements are typically specified for demonstrating IVD-MD performance related to numbers of reagent lots, calibrator lots, devices, clinical samples, replication, and criteria used for assessment. Furthermore, the regulatory requirements are continually, but independently, updated by countries and regions. This situation is a major challenge for IVD-MD manufacturers because of the number of different experimental designs and data portfolios needed for different regulatory agencies that add substantial costs to a standardization program. The significant logistical and cost burden for regulatory submissions in multiple countries or regions has made IVD-MD manufacturers reluctant to implement some newly developed standardization initiatives, including free-thyroxine or C-peptide [24]. Differences in regulatory requirements and standards between different countries or regions may lead to delayed availability of useful new IVD-MDs, affecting patient care and safety; e.g., high sensitivity troponin [31].

In most cases, recalibration to achieve standardized results is done to enable the proper use of medical decision limits. An important discussion item was that recalibration of an IVD-MD to achieve equivalent results among various IVD-MDs for the same measurand is often accomplished by adjusting a mathematical factor in the calibration hierarchy for each respective IVD-MD. Such mathematical adjustments change the clinical sample results for each IVD-MD to align all IVD-MDs with the chosen standardization target, e.g., a RMP, a CRM, or a consensus harmonization protocol. This change due to a calibration adjustment factor will often not change many aspects of performance such as selectivity, interferences, or precision. Other aspects, such as reference intervals and linear response interval, change in a mathematically predictable way based on the calibration adjustment factor. Consequently, transitioning to a recalibrated IVD-MD requires education regarding revised reference intervals or medical decision limits that were previously established. In principle, only the performance parameters affected by recalibration should need to be evaluated and validated. However, all calibration changes are currently expected to follow established regulatory frameworks which frequently turn out to be as time- and money-consuming as those required for initial applications. IVD-MD manufacturers are required to satisfy the different regulations in each country or region which adds costs and influences their willingness to recalibrate to achieve standardized results once a product is on the market.

Since standardization, including harmonization, has the goal of optimizing patient safety, national regulations should favor international standardization since medical practice guidelines are increasingly applied globally and the IVD industry is increasingly international. The risk assessment for data to support the implementation of a standardization activity should consider why a recalibration is made and how it affects patient care. Simplifying or waiving submission requirements for the components of IVD-MD performance that are not affected by recalibration should be considered to make implementation of standardization more easily achieved.

IVD-MD regulators recognized that standardized laboratory measurements support trade. Simplification of the level of evidence needed for regulatory purposes is a desirable goal to lower costs without jeopardizing the effectiveness of the review of IVD-MDs. Regulators at the workshop supported a risk-based assessment on a case-by-case basis. Agreeing on requirements for documentation when a calibration hierarchy is modified, and no other changes are made to an IVD-MD, would make regulatory submission simpler and more cost-effective.

Collaboration in an international forum would be useful to enable regulators to agree on a common approach to specifying what data under what conditions would be suitable to review in the situation of recalibration to achieve uniformly standardized results. For example, clarification and shared agreement when an update to an IVD-MD manufacturer’s technical file is sufficient evidence would be helpful. A desirable approach would be for a standardization working group to interact with regulators in an international consortium to determine in advance specific modifications needed for and the risks associated with recalibration of IVD-MDs for a particular measurand, and agree on what data will be required to fulfill regulatory requirements for a group of countries/regions. In addition, agreement on the education of laboratories and providers on the impact of standardization is desirable. Ideally, working groups conducting international standardization projects, e.g., in the IFCC Scientific Division, should include regulators in the team to help plan the studies to collect the data from all participating IVD-MD manufacturers needed to support regulatory requirements.

Recommendations

A standardization initiative requires collaboration and planning among all interested stakeholders. The workshop participants recommend:

1. Manufacturers (or researchers) with a “first in class” IVD-MD maintain calibrators, comparator measurement procedures if used and representative clinical samples to facilitate comparison with RMPs or other IVD-MDs as they are developed.

2. Developing higher order reference system components, i.e. CRMs and RMPs, as early in the life cycle of an IVD-MD as possible.

3. Performing cost-benefit analysis which includes the impact of current variability in results on patient care, prioritization in relation to other standardization projects, assurance of resources and funding, and consideration of regulatory requirements for implementing the calibration adjustments needed. The effects of differences in health care systems in different countries must be investigated.

A global prioritization process for standardization and CRM production should be further developed. The workshop participants recommend:

1. Developing a common understanding and global strategic vision on best practices for prioritization including a risk analysis for achieving successful standardization outcomes.

2. Interested parties should make use of ICHCLR listings and provide feedback to ICHCLR with information supporting adjustment of a listing.

3. ICHCLR should further develop the processes required to bring interested parties together to work on shared interest measurands, and ensure entries are regularly reviewed with feedback sought from all stakeholders including the IVD industry.

International collaboration should develop an effective mechanism for supply of CRMs worldwide. The workshop participants recommend:

1. A coordinated global system be further developed amongst all interested parties to develop and maintain reference measurement system components, i.e. CRMs and RMPs for value assignment.

2. Collaboration among CRM providers to ensure a primary and backup supply of CRMs for given measurands without unreasonable duplication of effort.

3. The material composition of CRMs should be as consistent as possible between batches (lots) to ensure consistent calibration hierarchies used by the IVD-MD manufacturers.

4. CRM producers should consider developing, with IVD-MD manufacturers and regulators, a standard format technical dossier for new (or modified) CRMs that could be used as supporting documentation for regulatory review.

5. JCTLM should consider if and how materials that would be considered useful for international standardization but may not meet all ISO requirements could be referenced or listed by the JCTLM.

6. Matrix-based CRMs known as not commutable with clinical samples when used with particular IVD-MDs should be cataloged in a central database to avoid their inappropriate use.

Cost-effective ways of completing commutability assessments for matrix-based CRMs should be explored. The workshop participants recommend:

1. NMIs and other providers of CRMs, EQA providers, IFCC, and the IVD industry should consider how they can cooperate with coordinated experimental designs to share the cost for commutability assessment.

2. The BIPM Consultative Committee for Amount of Substance: Metrology in Chemistry and Biology should establish the extent to which demonstrations of commutability and the stability of commutability of CRMs can be coordinated with processes for comparisons of measurement results among NMIs.

More uniform regulatory submission requirements are desirable to facilitate and accelerate recalibration to achieve internationally standardized results from IVD-MDs. The workshop participants recommend:

1. National and regional regulatory agencies should favor internationally coordinated standardized traceability and the use of consistent data submission requirements that appropriately ensure that recalibrated IVD-MDs are safe and effective.

2. Improved coordination between international standardization and international regulatory harmonization efforts for IVD-MDs, recognizing that all governments spend substantial resources to support internationally standardized measurements in most areas of commerce including for IVD-MDs, and that the full benefits of these activities will be realized with internationally harmonized regulatory submission requirements.

3. Regulatory agencies in all parts of the world should collaborate to create and maintain international consensus requirements for data to support standardization to achieve equivalent results that can be adopted for use in national and regional regulations, thus simplifying the process IVD-MD manufacturers have to follow to implement a standardization activity.

4. International working groups for the standardization of measurands should include members from regulatory agencies to advise regarding validation data suitable to support regulatory submissions.

5. International working groups for the standardization of a measurands should inform regulatory agencies of standardization activities and seek consensus on appropriate data submission expectations. A suitable international scientific organization, such as the IFCC Scientific Division or ICHCLR, should maintain a registry of active projects.