How is test laboratory data used and characterised by machine learning models? A systematic review of diagnostic and prognostic models developed for COVID-19 patients using only laboratory data

Literature search results

A PubMed search for the years 2020–2021 yielded 123 publications, and a Scopus search yielded 156. Only 115 publications were reported in both databases, and 98 were eliminated because they did not match the selection criteria, leaving 68 studies to be included in this study (Figure 2).

Figure 2:

Initial number of PubMed and Scopus publications on COVID-19 patient with diagnostic and prognostic purposes using artificial intelligence approaches (years 2020–2021), number of publications excluded due to selection criteria and final number of papers included in the review.

Of the 68 eligible publications, 34 developed ML models for diagnostic purposes (the studies labeled D1, D2, D3, …, D34), and 34 developed ML models for prognostic tasks (the studies labeled P1–P34).

The complete reference list for the studies chosen is displayed as Supplementary data (Supplementary Table 1).

The first ML diagnostic study applied to laboratory medicine for the management of COVID-19 patients was published in June 2020 (D6), and the first prognostic study was published barely one month later, in July 2020 (P3), according to the literature search (Supplementary Table 1). There are 38 papers published in biomedical journals, 23 in IT-specific publications, and seven IT conference papers (reported solely in Scopus) among the 68 papers chosen for this review (Table 1, Supplementary Table 1). The Materials and Methods sections of the majority of the studies (92%) lack a detailed explanation of the analytical method and/or instruments used in the laboratory analyses data to construct ML models (Table 1). Only six papers mention the instruments used, whereas 39 papers report the unit of measurement. Twelve papers in medical journals do not include any information about the laboratory test that was employed (Table 1). Figures 3 –5 describe the characteristics of in the articles reviewed (dataset, purpose of study, features selected, ML models) for diagnostic and prognostic studies, respectively.

Figure 3:

The top and bottom panels, respectively, offer an overview of the variables contained in the diagnostic (D) and prognostic (P) datasets (number and type of individuals and their origin, data collecting time).

The task’s purpose is displayed for P studies.

Figure 4:

Overview of the variables included in the diagnostic (D) and prognostic (P) studies (demographic characteristics, laboratory tests, pharmacological treatment, vital signs, clinical symptoms, comorbidities).

Figure 5:

Overview of the variables included in the diagnostic (D) and prognostic (P) studies of the machine learning applications (type of algorithm, mode of data validation, performance metric).

Population characteristics

Description of prognostic tasks

Prognostic models, as shown in Figure 3, have been developed for a variety of tasks, including the ability to predict the length of time spent in the hospital, the requirement for admission to an intensive care unit, the risk of severity and/or sepsis and the prediction of mortality or a favorable prognosis.

It is worth noting that not all studies use the same definition of “risk of severity”: in some, it is defined as intensive care unit admission, mechanical ventilation and/or death, while in others, it is defined as ventilation or death, as well as critical care unit admission or death.

The P8 and P9 research projects produced ML models to predict ICU admission and mortality for distinct purposes, P19 and P23 predicted mechanical ventilation demand and mortality separately, and P28 predicted the duration of stay and mortality. The P5 research project created models for four separate purposes: hospitalisation, admission to an intensive care unit, the need for mechanical ventilation and mortality.

Parameter (features) descriptions

ML models description

Performance metric description

Patient selection heterogeneity

The papers reviewed show a great deal of variation in patient selection. The majority of the diagnostic models were developed using emergency department (ED) patient groups, with inpatients and outpatients being included in the remaining investigations. It is worth noting that the patients’ origins were not specified in some studies (D8, D11–12, D15, D20–25, D27, D30–D34).

In several papers, the ML model has been evaluated in cohorts of unselected patients, with the only inclusion criteria being the availability of the molecular swab result; some biochemical tests and, sometimes, age, sex or ethnicity (D1, D3, D7–10). This inaccuracy in patient characterization is common in studies that used the same Brazilian dataset that is freely available online [28]. Inclusion criteria shape the training distribution, and hence affect its representativeness with respect to the target population from which instances will be drawn upon which the machine will produce new prediction/classifications. Therefore, inclusion criteria can affect prospective (actual) accuracy (or better yet robustness) in that they limit the range of types of clinical cases (i.e., instances or phenotypes) about which the machine will be capable to exhibit a certain performance. In other words, inclusion (and exclusion criteria) in the collection of the training dataset represent sources of bias to take into account to expect a specific actual accuracy on the new cases after training.

In a few studies, the authors may assert that they incorporated some clinical parameters into their study, but they do not provide any of them (D5, D12).

In certain cases, only a partial description (the existence of COVID-19-like symptoms) (D3, D8, D14–15, D17, D21–22, D25–26) is provided, and/or some exclusion criteria, such as the presence of specific comorbidities, are also mentioned (D2). Finally, in certain research, subjects’ clinical presentation was taken into account and accurately documented (D4), and/or the pattern was analysed in distinct subgroups (the entire sample or the subgroup of asymptomatic vs. symptomatic patients) (D4).

Even the clinical characteristics of a COVID-19 patient, such as clinical presentation, prevalence by age (which has changed over time), the introduction of new therapies (e.g., hyper-immune plasma) and biochemical and instrumental tests, show extreme heterogeneity. From this perspective, a model constructed using a dataset from a specific subgroup that achieved high diagnostic accuracy in one clinical environment may perform poorly in another. "The Importance of Being External" is the provocative title of a recent study. According to the authors, there is currently a gap in the literature regarding how to evaluate external validation results and, hence, assess the robustness of ML models [30]. Via the use of eight external validation CBC datasets collected across three continents during different pandemic waves, authors reported that the correlation between accuracy and similarity should serve as a warning sign that reproducing good performance across very heterogeneous settings can be overambitious and unrealistic [30].

These considerations are essential in the case of predictive models.

The predictive models examined here include a great deal of variation in the population’s inclusion criteria and the study’s prognostic end-point.

Mortality (at various time points) (P2–3, P5, P7), a severe form of COVID-19 characterized by respiratory failure requiring mechanical ventilation and/or admission to the intensive care unit (P1, P4–5), the risk of sepsis (P30) or delirium (P13), the likelihood of being hospitalized and length of stay (P28) or a favorable prognosis (P6) were all considered as end-points (Figure 3). Comparisons among models are challenging due to the variability of end-point selection.

How the ML models use medical laboratory data

The selection of laboratory tests is a second crucial part of applying the ML model to a patient with a suspected (diagnostic models) or certain (predictive models) diagnosis of COVID-19.

It is important to remember that, while rRT-PCR is considered the gold-standard diagnostic approach, it has limits in terms of sensitivity and specificity, and some additional criteria (e.g., CT and/or clinical presentation) are frequently used to correctly classify patients [23]. As a result, the analytical procedure for COVID-19 diagnosis utilized in the various studies with which the ML model is evaluated is undoubtedly another source of variability.

The laboratory professional should play a vital role in the selection of laboratory tests to be employed in ML models. Although the ML model can theoretically highlight a previously unknown link between a marker and pathology, the features selection should be interested in, or at least include, variables that have already been reported in the scientific literature as associated or altered in the various stages of COVID-19 disease.

A vast number of laboratory tests have been reported to be changed, with putative relevance for monitoring, stratification and prognosis. Specifically, there have been hematological (leukocytosis, lymphopenia, neutrophilia, anemia, thrombocytopenia), biochemistry (hypoalbuminemia, elevated lactate dehydrogenase, aspartate aminotransferase, alanine aminotransferase, total bilirubin, creatinine, troponin, C-reactive protein), infection indicator (interleukins and procalcitonin) and coagulation (increased D-Dimer and prothrombin time) alterations [18], [19], [20], [21, 31, 32]. However, during COVID-19, rapid changes in these parameters are possible, which could be especially problematic for the ML model and its application. The importance of specific laboratory tests may thus change not only during different stages of the disease but also during different eras of the pandemic [14, 30].

The selection of lab tests, similar to patient selection, has a significant degree of variety. There are ML models that use only CBC parameters (D1–2, 6–7, D13–15, D18, D20, D28, P18, P20, P25, P33), others use CBC and clinical chemistry and others also use coagulation tests and/or blood gas tests (Figures 3 and 4).

In addition to laboratory tests, prediction models took into account additional factors, such as clinical history, comorbidities and pharmacological therapy. There are parsimonious models with a small number of variables (less than 10 in D1, D29, P2 and P18) and, more typically, models with a large number of predictors (more than 50 in D4, D10, D21–25 and D27 and P3, P12, P14–15 and P30–31).

The selection of parameters is typically contentious, and it does not appear to be done in consultation with a laboratory medicine expert. For example, a diagnostic model that includes a marker that cannot be requested urgently and has little or no diagnostic significance (such as urea, which was included in various studies, or cholinesterase, included in D12), as well as one that has particularly long analytical times or is excessively expensive, is not very applicable. Curiously, a prognostic study (P27) based on 28 laboratory parameters (including CBC, coagulation and clinical chemistry tests) detected Urea as the most important feature to predict the mortality for patients with COVID positivity. Similarly, a diagnostic or predictive model containing a marker that is rarely found in a hospital laboratory would be useless (i.e., glucose-6-phosphate dehydrogenase included in P28, ILs included in P30 (IL1, IL6, IL8, IL10) and BNP, proBNP, or even atrial natriuretic peptide (ANP) included in P28). Some papers included generic “troponin” (P8, P16, P23) or a generic “bilirubin” (D10) without stating which one (total, direct or indirect bilirubin) it is.

An overview of the laboratory tests, summarised in macro-categories as hematological clinical chemistry and coagulation tests, included in the diagnostic (D) and prognostic (P) studies shown in Figure 6. To the heterogeneity of the numbers of tests used by the different studies, the variety of the expression of the data is also added. For example, WBC differential count data are expressed as percentages, absolute counts or both or in an undeclared unit. Similarly, prothrombin time (PT) is expressed in s, in INR, in terms of percent or in both in s and INR, and in more than 40% of the papers, the unit is not specified (Figure 6, Table 1).

Figure 6:

Overview of the laboratory tests included in the diagnostic (D) and prognostic (P) studies.

Many factors can influence the representativeness of datasets obtained from hemato-chemical parameters for assessing the robustness of a model: differences in testing equipment (concepts of harmonization), reference ranges/ethnic variability, disease manifestations/phenotypic variability and how humans react to contextual factors (biological variation) make the reference population incredibly vast and diverse; thus, very different datasets can be obtained [30, 33]. This could explain why ML based on lab data has not taken off as it has in other medical fields [13].

The analytical aspect is, in reality, another significant factor to consider (unit of measurement, reference interval, analytical method, instrumentation, traceability). Consider, for example, the non-specificity of Jaffe methods as compared to enzymatic methods in measuring serum creatinine, a parameter used in the majority of the studies examined, as well as the impact that the standardisation process has had on the quality of its measurement [34, 35]. It is clear that, at least for some laboratory parameters, information about analytical methods is essential. In this regard, a mini-review dedicated to D Dimer outcomes in COVID-19 patients is clearly interesting. In this mini-review, there are misunderstandings in terms of how the data are reported, resulting in significant misrepresentation [36]. In our review, more than 90% of the papers selected do not provide sufficient information to adequately characterise laboratory data, and more than 40% do not declare the units of measurement (Table 1).

The use of only CBC data in algorithms developed via ML for diagnostic and prognostic purposes seems to be more appropriate, especially given the robustness of the model based on its reproducibility, rather than the practicability and economic issues [23, 37], [38], [39], [40]. In fact, compared to other hemato-chemical parameters, CBC data are characterized by limited within-subjects and between-subjects biological variation [41], [42], [43], which supports the reproducibility of results for the same patient at different times, as well as negligible analytical variation [44], which ensures the reproducibility of the same data across laboratories, equipment and heterogeneous populations.

Most diagnostic and predictive models do not offer information on analytic methods and/or instruments, with the exception of a handful (D4, D13, D20, D28, P20) (Table 1). Reviewing the cited papers gives the impression that the authors did not rationally select laboratory parameters but, rather, developed the various models simply by using all of the data available in the management systems of individual healthcare facilities.

Due to the large expansion, in recent years, of research using AI techniques, including ML, and the resulting poor reproducibility of the models created [45], it has become imperative to draft standards that could improve the quality of AI studies in the medical area. The MINIMAR (MINimum Information for Medical AI Reporting) guidelines, which were recently published, were established with this goal in mind [46]. MINIMAR includes a minimum list of information that should be included in publications, but it makes no mention of how to record laboratory test findings, which is crucial to a study’s reproducibility, with this being, ironically, the guidelines’ goal. Haymond S et al., in an opinion just published [47], summarize practices that should be applied in the development, reporting and review of ML applications without any indication of how clinical lab data should be characterised.

It is no surprise that recommendations for studies involving laboratory medicine results [48], [49], [50], [51] always include a request for a thorough description and characterisation of the methods employed to generate them (instrument used, analytical principle, unit of measurement, method optimization, generation, specificity). The analytical processes should measure the same quantity in order to achieve comparable and reliable results, and a description of the analytical approach used to gather laboratory data is required to identify the measurand precisely [52]. Even if the various challenges linked to standardisation and harmonisation and, consequently, the reliability and reproducibility of laboratory results, are not addressed here, as laboratory experts, we are also aware that, if these factors are not considered, the potential for laboratory data to be misused is extremely significant.

ML model types and their validation

The majority of the algorithms utilised are supervised algorithms, with only three diagnostic studies and one prognostic study (D16, D30, D33, P21) developing ML models based on unsupervised methods.

In a nutshell, supervised algorithms attempt to create a model from "labelled" training data, which can then be used to generate predictions about future data. In the unsupervised instance, on the other hand, only input data are provided, and the goal is to model the underlying structure or distribution of the data and discover unknown patterns [53].

However, it is worth noting the number of distinct algorithms (of varying complexity levels) that were used (Figure 5). Specifically, RF (23 out of 34 diagnostic models, 15 out of 34 predictive models) and LR (17/34 and 12/34, respectively) are the most commonly employed supervised algorithms (Figure 5).

The validation process employed in the various research projects is a second point to consider. Split-sample validation, that is, hold-out (by splitting the data into training and test), K-fold cross-validation (which is more resilient than the former), K-fold nested/repeated CV, and bootstrapping approaches for obtaining confidence intervals, is the most commonly used technique. These methods have a variety of benefits and drawbacks [54, 55], and they are examples of internal validation, which is particularly valuable in ensuring model stability. External validation, on the other hand, permits measuring a model’s reproducibility and generalisability to the timing and space of data collection by using a dataset that is not the same as that used to train the model. Internal-external validation, a mixture of the two techniques, is also conceivable, and especially effective in situations in which the amount of accessible data is restricted. The authors reported external validation with one or more validation datasets for one-third of the diagnostic models studied (Figure 5). Another important factor to consider is the size of the dataset employed, which influences the validation algorithm and procedure used, as well as the model’s correctness. The sample sizes for the diagnostic models range from 170 (D2) to 115,394 (D10), while the predictive models have sample sizes ranging from 87 (P4) to 64,733 (P24).

Model implementation

Two other factors are critical to the model’s application: the dataset’s representativeness and availability, as well as the algorithm’s explainability, robustness and fairness. It is also crucial to make the models publicly available by developing online tools that allow users to evaluate the algorithms that have been developed.

The availability of all of the data used to create the model enables a conclusive evaluation of the model’s validity and the verification of its applicability. This is especially important in the context of Big Data, machine learning techniques and other scenarios characterised by great computational complexity: it is a necessary component of real reproducible science [56]. To do this, the original data, source code and detailed descriptions of all analytical methods utilised are required. Many authors are currently unwilling to disclose their data for a variety of reasons; nonetheless, such sharing, as well as data openness in general, is a necessary condition for science to progress on a solid foundation. Although only one research project (D10) made its source code available (at https://github.com/andrewsoltan/CURIAL-manuscript), the datasets for some others are publicly available (D4, D8, D14–15, D21, D23–24, D27, D30, D33–34; P10–12, P15, P20, P26, P29, P32).

The availability of IT tools that allow the model to be implemented by the end-user (the physician) is a second essential factor. These tools can be realised as apps that are directly integrated into the management systems of specific operating units or the laboratory’s middleware. Once the laboratory findings are obtained, this will have the advantage of automating the compilation of input fields, lowering both reaction time and the rate of input errors. In this regard, it is worth noting that some of the diagnostic and predictive models under consideration have been applied in real-world applications that can be found on the Internet (D4, D14, D17, P10, P26).