Recommendations for IVDR compliant in-house software development in clinical practice: a how-to paper with three use cases

Introduction

The right treatment starts with the right diagnosis. Patients and healthcare professionals therefore need to be able to trust medical devices that help them make a reliable diagnosis, such as in vitro diagnostic medical devices. With the protection of health for patients and users of in vitro diagnostic medical devices in mind, the European Union (EU) recently updated its standards for quality and safety for in vitro diagnostic medical devices [1]. The In Vitro Diagnostics Regulation (IVDR) will be effective in May 2022 by which in-house developed tests need to apply to the general safety and performance requirements defined in Annex I of the IVDR ruling [1, 2].

Compared with the In Vitro Diagnostic Directive (IVDD), two significant changes impact current laboratory practice. First, IVDR now specifically refers to laboratory software, including in-house developed software [1]. Furthermore, the IVDR includes specific conditions for in-house developed devices. These conditions state that healthcare institutions can still develop their own tests without the need for CE marking [1].

Currently, laboratories are investigating their practices regarding software to comply with IVDR. Article 16.2 of the IVDR states that “software shall be developed and manufactured in accordance with the state-of-the-art taking into account the principles of development life cycle, risk management, … ” [1]. This article can be hard to interpret and implement, particularly as laboratories are unfamiliar with quality standards for software development. This requires a multidisciplinary collaborative effort, as bioinformaticians and developers of digital health products enter the 21st century laboratory environment that used to be dominated by (wet-)laboratory specialists. Especially in highly specialized clinical facilities that already harbor such innovative teams, questions regarding the opportunities and limitations of IVDR may arise.

At the UMC Utrecht division of Laboratories, Pharmacy and Biomedical Genetics, we have recently aligned IEC-62304 for medical software development with the ISO-15189 medical laboratory quality management system to be able to leverage the potential of article 5.5 and create in-house developed software in an IVDR compliant manner [1]. The IEC-62304 is a harmonized standard that covers safe design, development and maintenance of medical software. In this paper we provide recommendations for IVDR compliant in-house developed software and a practical insight into our novel standard operating procedures using three examples.

What is IVDR software?

In article 2 of IVDR, software is specifically mentioned in the formal definition of an in vitro diagnostic medical device. Briefly, software is considered IVDR software if it is intended to be used (alone or in combination) for the examination of specimens derived from the human body to provide information on physiological or pathological processes, predisposition of diseases and/or to predict treatment responses [1]. IVDR software can often be seen as software that will drive or inform clinical management, i.e. the results from the software are used to diagnose a patient or to provide information about which treatment to start.

Several guidelines have been created by the Medical Device Coordination Group (MDCG) to help interpret the IVDR. The infographic “Is your software a Medical Device?” uses decision steps to assist in qualification of IVDR software [3]. Of particular interest is the guidance on qualification and classification of software (MDCG2019-11 [4]). This guidance states that a laboratory information system is not considered IVDR software as it is merely a storage system. Any addition to the system that performs calculations that exceed basic operations (e.g. transpose US to EU metrics) may be considered an IVDR software module. Websites containing risk calculators and Excel sheets with fixed or flexible formulas are thus subject to the IVDR. Noteworthy, Excel is widely used and provides an intuitive and easy to use interface for (laboratory) specialists. Within our division >50 Excel files contain formulas that generate information on which a diagnosis can be made or which creates/alters a treatment plan, and are thus considered IVDR software. Moreover, with the arrival of big data within the clinic, more complex examples have arrived, where specialized software and algorithms, developed by bioinformaticians and data scientists have become an essential part of diagnostic laboratories.

If no equivalent CE-labeled software product is available for the specific requirements of the laboratory, health institutions that are established in the EU are allowed to develop their own IVDR software if they meet specific conditions and comply with the general safety and performance requirements in the IVDR detailed in article 5.5 and Annex I [1]. For example, in-house developed software cannot be transferred to other legal entities and needs to be manufactured under an ‘appropriate quality management system’.

Below, we share recommendations on how to meet IVDR requirements for software in a laboratory environment, synergizing good laboratory practices and quality management systems that already are part of our day-to-day work.

Recommendations for organizational structure

Successful innovation teams are able to bridge the gap between research and clinical practice efficiently. Rather than appointing specific innovation officers, health institutions can start from existing quality-aware personnel, by freeing up time from different stakeholders to participate in multidisciplinary development teams. In our hospital, software engineers, laboratory specialists and clinical specialists work together to translate diagnostic questions to specific requirements for software and clinically relevant intended purposes. Depending on the software that is being built, ‘software engineers’ in this case includes laboratory specialists (for formulas in Excel), laboratory information system administrators (for modules or simple rules) and/or bioinformaticians (for complex pipelines). Either way, developing software works best in an agile way, where improvement cycles build towards a minimal viable product so end users can co-steer the development based on early results. Embedding multidisciplinary teams within clinical care further facilitates a short quality feedback loop after the developed IVDR software is put into service. Formal higher management support for this approach, including compliance to appropriate standards, is of paramount importance, as responsibility for in-house development and its quality is at a health institution level.

Recommendations for standards to use

Recommendations for documentation

Use case 1: pharmacokinetics of tacrolimus

There is a large intra- and interindividual variability in the pharmacokinetics for tacrolimus so monitoring of tacrolimus blood concentrations, with multiple sampling moments after intake, is useful for therapeutic drug monitoring. With multiple samples the systemic exposure can be predicted by calculating the area under the curve (AUC) using the trapezoidal rule [12, 13]. The hospital pharmacy uses an Excel file to calculate the pharmacokinetics of tacrolimus using the trapezoidal rule for the AUC and to calculate the new dose. As monitoring of levels of medicinal products is considered class C in the IVDR, this Excel file is considered IVDR software. When implementing the IVDR in the UMC Utrecht, this simple Excel file was one of the first we encountered and its revised version served as a template for the whole division. The revised version consists of two additional sheets for QMS-related documentation (Supplementary Material 2). The documentation sheet includes (i) Intended purpose of the module; (ii) risk classification; (iii) requirements; (iv) risk analysis; and (v) Reference to the evidence of clinical performance, which includes a national guideline and two references. The validation sheet includes (i) several tests, which must be repeated with each new version of the Excel file; (ii) evaluation; and (iii) conclusion. The test includes time stamps and concentrations with a verified outcome. One could debate whether testing formulas in Excel is verification or validation. We recommend validation, as the requirements of the end users are tested. Working in an ISO-15189 accredited laboratory, the Quality Management System (QMS) of the lab handles the versioning of this Excel file itself, including the date. Even though we do not expect this Excel file to change in the near future, if it does, all its content will be reconsidered and updated for a version 2. The QMS also handles feedback.

Use case 2: automated diagnosis of acute kidney injury

Acute kidney injury (AKI) is associated with high morbidity and mortality [14] and although its definition is based on simple differences in serum creatinine levels and urine output [15], AKI is often missed. In collaboration with a multidisciplinary team with nephrologists, internists, laboratory specialists and data scientists, we recently implemented a rule-based algorithm for AKI in our Laboratory Information System (LIS), with the outcome visible in the electronic patient record (EHR). One document, containing the complete QMS related documentation, has been created for this module, which includes the following sections (outline in Supplementary Material 3): (i) Intended purpose of the module; (ii) risk classification; (iii) requirements, including a decision tree leading to a yes or no for showing the alert; (iv) risk analysis, including false positive and false negatives outcome and risks for end users; (v) verification of the module; interpreted as an integration test; i.e. does the code provide results; (vi) validation; do the results comply with the end user requests. Specific patients have been selected in order to follow all routes through the decision tree. In the test, patient identifiers for these patients are mentioned and followed in the different steps through LIS and EHR. The acceptance criteria are referring to the decision tree; which should work as requested. The results of the tests came back perfectly and no anomalies were detected (evaluation of the test results). For validation, the people creating and performing the tests have been documented. (vii) Reference to the evidence of clinical performance, which includes 11 references. In summary, it was stated in the document that the AKI alert could be used for diagnostic purposes. The document was verified and authorized by qualified personnel. Similar to use case 1, the QMS of the lab handles the versioning, verification and authorization of the document itself.

Use case 3: bioinformatics pipeline for variant calling in sequencing data

Bioinformaticians have developed a pipeline for variant calling in genetic sequencing data. A pipeline consists of a series of different tools and functions arranged such that the output of each consecutive element is the input of the next. IVDR classifies the pipeline as class C as it is part of genetic testing, and we classified the pipeline as class A following the flowchart in the IEC-62304. Class B of the IEC-62304 seems to be the correct classification, as there is no risk of death when the variant calling is incorrect. Incorrect variant calling could lead to incorrect interpretation which could subsequently lead to incorrect treatment. IEC-62304, however, allows the use of risk mitigation e.g. control measures to be taken into account, which we implemented as follows (1): quality checks throughout the workflow outside of the bioinformatics pipeline that indicate whether or not the data can be trusted and (2) the results of the bioinformatics pipeline are interpreted by a laboratory specialist before release to a geneticist. Thus unexpected results, i.e. a significant change in the number of variants after filtering, will be noticed and the final classification is A.

In our software development procedure, the documentation for each release is combined in an Excel file, where we use a separate sheet for an overview, risk analysis table, requirements, configuration management, SOUPs, related procedures etc. The overview provides the intended purpose, the classification, where the software is installed, links to verification information (on Github) and validation document, information of the last release, such as version, date, commit number, link(s) to Github. For each update of the software all the sheets are updated, to keep the documentation of the software up to date. For software development we use Github, a Git repository hosting service which allows us to track code changes but also to request code reviews, and track comments about these reviews; which is part of our verification procedure. We track issues using Azure DevOps, which we also use for planning. We have a general email address where we get questions and requests for new features, which end up as an issue, and there are multidisciplinary meetings where bioinformaticians meet the lab personnel, where feedback can be gathered. Genome in a Bottle is used for clinical evidence and performance evaluation [16]. Finally, to validate the results of our pipeline, a comparison is made with results from previous versions of the pipeline, or results from other techniques.

Discussion

Being part of several national working groups of the IVDR and software, we have noticed that there is a gap of knowledge in medical laboratories about how to deal with in-house developed software, using ISO-15189 and IEC-62304 in synergy. Indeed, software development is not incorporated in the ISO-15189, and even in a preview of the ISO-15189:2022 there is no information about documentation for in-house developed software. This paper provides recommendations about what to document, and where to find more information regarding documentation of software as a medical device.

The use cases described in this paper definitely do not represent all IVDR software, however they span across the software spectrum from a calculation in Excel and a simple module to a bioinformatics pipeline. The MDCG 2019-11 [4] is a very good guidance document on what software is considered a medical device. One example that is explicitly mentioned in this guidance is software which is intended to determine the Human HbA_1c concentration in serum from the results obtained with a Human HbA_1c ELISA. Even though it is considered a trivial task to create a calibration line in, for example Excel, and receive results for diagnostic purposes, it is considered software as a medical device and should be treated and documented as such. Furthermore this guidance states: “Software intended to modify the representation of available in vitro diagnostic medical device results is not considered an in vitro diagnostic medical device, e.g. basic operations of arithmetic (e.g. mean, conversion of units) and/or plotting of results in function of time, and/or a comparison of the result to the limits of acceptance set by the user.” Whether or not the formula, in for example Excel, is considered a basic operation is up to the developer and laboratory specialist. We highly recommend documenting the reasoning when the formula is considered a basic operation.

With IVDR article 5.5a, it is no longer allowed to transfer the devices to another legal entity. This provides a dilemma. In-house developed software is often used for both healthcare and scientific research. In the last few years it has become increasingly important to publish papers with open source code, to be able to contribute to the open science movement and to comply with FAIR principles for software [17].

To comply with IVDR, there should be distinguished steps, such as a stable IT-infrastructure, inclusion in a QMS, and proper validation, between software used for research and the one that is used in a diagnostic setting. In our opinion, this is not a sustainable nor an efficient way of working. In the upcoming years, competent authorities and health institutions should work together in finding creative solutions to this issue that facilitate innovation while honoring the principles of the IVDR.

Conclusions

IVDR is going to significantly change in-house development of IVDR software. Complying with IVDR through creating synergy between ISO-15189 and IEC-62304 is complicated, but it will create value for diagnostics, through multidisciplinary teamwork and careful documentation.