Abstract
Objectives
To compare the artificial intelligence algorithms as powerful machine learning methods for evaluating patients with suspected sepsis using data from routinely available blood tests performed on arrival at the hospital. Results were compared with those obtained from the classical logistic regression method.
Methods
The study group consisted of consecutive patients with fever and suspected infection admitted to the Emergency Department. The complete blood counts (CBC) were acquired using the Mindray BC-6800 Plus analyser (Mindray Diagnostics, Shenzhen, China). Cell Population Data (CPD) were also recorded. The ML and artificial intelligence (AI) models were developed; their performance was evaluated using several indicators, such as the area under the receiver operating curve (AUC), calibration plots and decision curve analysis (DCA).
Results
Overall, all the tested approaches obtained an AUC>0.90. The logistic regression (LR) performed well compared to the ML/AI models. The naïve Bayes and the K-nearest neighbour (KNN) methods did not show good calibration properties. The multi-layer perceptron (MLP) model was the best in terms of discrimination, calibration and clinical usefulness.
Conclusions
The best performance in the early detection of sepsis was achieved using the ML and AI models. However, external validation studies are needed to strengthen model derivation and procedure updating.
-
Research funding: None declared.
-
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
-
Competing interests: Authors state no conflict of interest.
-
Informed consent: Informed consent was obtained from all individuals included in this study.
-
Ethical approval: Approval from the Ethics Committee Board of our hospital was obtained.
References
1. Angus, DC, van der Poll, T. Severe sepsis and septic shock. N Engl J Med 2013;369:840–51. https://doi.org/10.1056/nejmra1208623.Search in Google Scholar
2. Singer, M, Deutschman, CS, Seymour, CW, Shankar-Hari, M, Annane, D, Bauer, M, et al.. The Third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016;315:801–10. https://doi.org/10.1001/jama.2016.0287.Search in Google Scholar PubMed PubMed Central
3. Gyawali, B, Ramakrishna, K, Dhamoon, AS. Sepsis: the evolution in definition, pathophysiology, and management. SAGE Open Med 2019;7:2050312119835043. https://doi.org/10.1177/2050312119835043.Search in Google Scholar PubMed PubMed Central
4. Reinhart, K, Daniels, R, Kissoon, N, Machado, FR, Schachter, RD, Finfer, S. Recognizing sepsis as a global health priority — a WHO resolution. N Engl J Med 2017;377:414–7. https://doi.org/10.1056/nejmp1707170.Search in Google Scholar
5. Brun-Buisson, C. The epidemiology of the systemic inflammatory response. Intensive Care Med 2000;26:S064–74. https://doi.org/10.1007/s001340051121.Search in Google Scholar PubMed PubMed Central
6. Graber, ML, Patel, M, Claypool, S. Sepsis as a model for improving diagnosis. Diagnosis 2018;5:3–10. https://doi.org/10.1515/dx-2017-0036.Search in Google Scholar PubMed
7. Faix, JD. Biomarkers of sepsis. Crit Rev Clin Lab Sci 2013;50:23–36. https://doi.org/10.3109/10408363.2013.764490.Search in Google Scholar PubMed PubMed Central
8. Lippi, G. Sepsis biomarkers: past, present and future. Clin Chem Lab Med 2019;57:1281–3. https://doi.org/10.1515/cclm-2018-1347.Search in Google Scholar PubMed
9. Schuetz, P, Plebani, M. Can biomarkers help us to better diagnose and manage sepsis? Diagnosis 2015;2:81–7. https://doi.org/10.1515/dx-2014-0073.Search in Google Scholar PubMed
10. Wang, X, Zhu, T, Xia, M, Liu, Y, Wang, Y, Wang, X, et al.. Predicting the prognosis of patients in the coronary care unit: a novel multi-category machine learning model using XGBoost. Front Cardiovasc Med 2022;9:764629. https://doi.org/10.3389/fcvm.2022.764629.Search in Google Scholar PubMed PubMed Central
11. Lu, J, Bu, P, Xia, X, Lu, N, Yao, L, Jiang, H. Feasibility of machine learning methods for predicting hospital emergency room visits for respiratory diseases. Environ Sci Pollut Control Ser 2021;28:29701–9. https://doi.org/10.1007/s11356-021-12658-7.Search in Google Scholar PubMed
12. Layeghian Javan, S, Sepehri, MM, Layeghian Javan, M, Khatibi, T. An intelligent warning model for early prediction of cardiac arrest in sepsis patients. Comput Methods Progr Biomed 2019;178:47–58. https://doi.org/10.1016/j.cmpb.2019.06.010.Search in Google Scholar PubMed
13. Singh, YV, Singh, P, Khan, S, Singh, RS, Gupta, SK. A machine learning model for early prediction and detection of sepsis in intensive care unit patients. J Healthc Eng 2022;2022:1–11. https://doi.org/10.1155/2022/9263391.Search in Google Scholar PubMed PubMed Central
14. Wang, D, Li, J, Sun, Y, Ding, X, Zhang, X, Liu, S, et al.. A machine learning model for accurate prediction of sepsis in ICU patients. Front Public Health 2021;9:754348. https://doi.org/10.3389/fpubh.2021.754348.Search in Google Scholar PubMed PubMed Central
15. Urrechaga, E, Bóveda, O, Aguirre, U. Role of leucocytes cell population data in the early detection of sepsis. J Clin Pathol 2018;71:259–66. https://doi.org/10.1136/jclinpath-2017-204524.Search in Google Scholar PubMed
16. Urrechaga, E, Bóveda, O, Aguirre, U. Improvement in detecting sepsis using leukocyte cell population data (CPD). Clin Chem Lab Med 2019;57:918–26. https://doi.org/10.1515/cclm-2018-0979.Search in Google Scholar PubMed
17. Martens, RJH, van Adrichem, AJ, Mattheij, NJA, Brouwer, CG, van Twist, DJL, Broerse, JJCR, et al.. Hemocytometric characteristics of COVID-19 patients with and without cytokine storm syndrome on the sysmex XN-10 hematology analyzer. Clin Chem Lab Med 2021;59:783–93. https://doi.org/10.1515/cclm-2020-1529.Search in Google Scholar PubMed
18. Urrechaga, E, Aguirre, U, España, PP, García de Guadiana, L. Complete blood counts and cell population data from Sysmex XN analyser in the detection of SARS-CoV-2 infection. Clin Chem Lab Med 2021;59:e57–60. https://doi.org/10.1515/cclm-2020-1416.Search in Google Scholar PubMed
19. Harte, JV, Mykytiv, V. A panhaemocytometric approach to COVID-19: a retrospective study on the importance of monocyte and neutrophil population data on Sysmex XN-series analysers. Clin Chem Lab Med 2021;59:e169–72. https://doi.org/10.1515/cclm-2021-0096.Search in Google Scholar PubMed
20. Urrechaga, E. Morphometric parameters of leukocytes in the management of sepsis running title: cell population data in sepsis. Highl Med Med Res 2021;9:44–58.10.9734/bpi/hmmr/v9/8560DSearch in Google Scholar
21. Webb, GI. Naïve Bayes. In Sammut C, Webb GI, editors. Encyclopedia of machine learning. Boston, MA: Springer; 2011: 713–4 pp.10.1007/978-0-387-30164-8_576Search in Google Scholar
22. Hastie, TT, Robert, Friedman, J. The elements of statistical learning. New York, NY: Springer; 2009.10.1007/978-0-387-84858-7Search in Google Scholar
23. Chen, T, Guestrin, C. XGBoost. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. New York, NY: Association for Computing Machinery; 2016:785–94 pp.10.1145/2939672.2939785Search in Google Scholar
24. LaValle, SM, Branicky, MS, Lindemann, SR. On the relationship between classical grid search and probabilistic roadmaps. Int J Robot Res 2016;23:673–92. https://doi.org/10.1177/0278364904045481.Search in Google Scholar
25. Kohavi, R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: International joint conference on artificial intelligence IJCA; 1995, vol. 2:9 p.Search in Google Scholar
26. Ruopp, MD, Perkins, NJ, Whitcomb, BW, Schisterman, EF. Youden index and optimal cut-point estimated from observations affected by a lower limit of detection. Biom J 2008;50:419–30. https://doi.org/10.1002/bimj.200710415.Search in Google Scholar PubMed PubMed Central
27. Rufibach, K. Use of brier score to assess binary predictions. J Clin Epidemiol 2010;63:938–9. https://doi.org/10.1016/j.jclinepi.2009.11.009.Search in Google Scholar PubMed
28. Huang, Y, Li, W, Macheret, F, Gabriel, RA, Ohno-Machado, L. A tutorial on calibration measurements and calibration models for clinical prediction models. J Am Med Inf Assoc 2020;27:621–33. https://doi.org/10.1093/jamia/ocz228.Search in Google Scholar PubMed PubMed Central
29. Appenzeller, C, Liniger, MA, Weigel, AP. The discrete brier and ranked probability skill scores. Mon Weather Rev 2007;135:118–24. https://doi.org/10.1175/mwr3280.1.Search in Google Scholar
30. Lundberg, SM, Erion, G, Chen, H, DeGrave, A, Prutkin, JM, Nair, B, et al.. From local explanations to global understanding with explainable AI for trees. Nat Mach Intell 2020;2:56–67. https://doi.org/10.1038/s42256-019-0138-9.Search in Google Scholar PubMed PubMed Central
31. Pedregosa, F, Varoquaux, G, Gramfort, A, Michel, V, Thirion, B, Grisel, O, et al.. Scikit-learn: machine learning in Python. J Mach Learn Res 2011;12:2825–30.Search in Google Scholar
32. Portell, RIM, Alarcón, RR, Benayas, BMP, Avivar, C. Analysis of leukocyte cell population data (CPD) as biomarkers in the diagnosis of sepsis. Hematol Transfus Int J 2022;2:33–7.10.15406/htij.2022.10.00278Search in Google Scholar
33. Buoro, S, Manenti, B, Seghezzi, M, Dominoni, P, Barbui, T, Ghirardi, A, et al.. Innovative haematological parameters for early diagnosis of sepsis in adult patients admitted in intensive care unit. J Clin Pathol 2018;71:330–5. https://doi.org/10.1136/jclinpath-2017-204643.Search in Google Scholar PubMed
34. Zhang, W, Zhang, Z, Pan, S, Li, J, Yang, Y, Qi, H, et al.. The clinical value of hematological neutrophil and monocyte parameters in the diagnosis and identification of sepsis. Ann Transl Med 2021;9:1680. https://doi.org/10.21037/atm-21-5639.Search in Google Scholar PubMed PubMed Central
35. Buoro, S, Carobene, A, Seghezzi, M, Manenti, B, Pacioni, A, Ceriotti, F, et al.. Short- and medium-term biological variation estimates of leukocytes extended to differential count and morphology-structural parameters (cell population data) in blood samples obtained from healthy people. Clin Chim Acta 2017;473:147–56. https://doi.org/10.1016/j.cca.2017.07.009.Search in Google Scholar PubMed
36. Tang, H, Jing, J, Bo, D, Xu, D. Biological variations of leukocyte numerical and morphologic parameters determined by UniCel DxH 800 hematology analyzer. Arch Pathol Lab Med 2012;136:1392–6. https://doi.org/10.5858/arpa.2011-0679-oa.Search in Google Scholar PubMed
37. Hoffmann, J. New hematology analyzer parameters and their clinical relevance. EFLM Newslett 2018;1:3.Search in Google Scholar
38. Seghezzi, M, Buoro, S, Previtali, G, Moioli, V, Manenti, B, Simon-Lopez, R, et al.. A preliminary proposal for quality control assessment and harmonization of leukocytes morphology-structural parameters (cell population data parameters). J Med Biochem 2018;37:486–98. https://doi.org/10.2478/jomb-2018-0005.Search in Google Scholar PubMed PubMed Central
39. Rajula, HSR, Verlato, G, Manchia, M, Antonucci, N, Fanos, V. Comparison of conventional statistical methods with machine learning in medicine: diagnosis, drug development, and treatment. Medicina 2020;56:455. https://doi.org/10.3390/medicina56090455.Search in Google Scholar PubMed PubMed Central
40. Zhang, Z, Hong, Y. Development of a novel score for the prediction of hospital mortality in patients with severe sepsis: the use of electronic healthcare records with LASSO regression. Oncotarget 2017;8:49637–45. https://doi.org/10.18632/oncotarget.17870.Search in Google Scholar PubMed PubMed Central
41. Rawson, TM, Hernandez, B, Moore, LSP, Blandy, O, Herrero, P, Gilchrist, M, et al.. Supervised machine learning for the prediction of infection on admission to hospital: a prospective observational cohort study. J Antimicrob Chemother 2019;74:1108–15. https://doi.org/10.1093/jac/dky514.Search in Google Scholar PubMed
42. Hu, C, Li, L, Huang, W, Wu, T, Xu, Q, Liu, J, et al.. Interpretable machine learning for early prediction of prognosis in sepsis: a discovery and validation study. Infect Dis Ther 2022;11:1117–32. https://doi.org/10.1007/s40121-022-00628-6.Search in Google Scholar PubMed PubMed Central
43. Kijpaisalratana, N, Sanglertsinlapachai, D, Techaratsami, S, Musikatavorn, K, Saoraya, J. Machine learning algorithms for early sepsis detection in the emergency department: a retrospective study. Int J Med Inf 2022;160:104689. https://doi.org/10.1016/j.ijmedinf.2022.104689.Search in Google Scholar PubMed
44. Ghias, N, Haq, SU, Arshad, H, Sultan, H, Bashir, F, Ghaznavi, SA, et al.. Using machine learning algorithms to predict sepsis and its stages in ICU patients. medRxiv 2022:2022.03.15.22271655. https://doi.org/10.1101/2022.03.15.22271655.Search in Google Scholar
45. Zeng, Z, Yao, S, Zheng, J, Gong, X. Development and validation of a novel blending machine learning model for hospital mortality prediction in ICU patients with Sepsis. BioData Min 2021;14. https://doi.org/10.1186/s13040-021-00276-5.Search in Google Scholar PubMed PubMed Central
46. Kuhn, M, Johnson, K. Applied predictive modeling. New York, NY: Springer; 2013.10.1007/978-1-4614-6849-3Search in Google Scholar
47. Dankowski, T, Ziegler, A. Calibrating Random Forests for probability estimation. Stat Med 2016;35:3949–60. https://doi.org/10.1002/sim.6959.Search in Google Scholar PubMed PubMed Central
48. Cabitza, F, Rasoini, R, Gensini, GF. Unintended consequences of machine learning in medicine. JAMA 2017;318:517–8. https://doi.org/10.1001/jama.2017.7797.Search in Google Scholar PubMed
49. Martins, EC, Silveira, LdF, Viegas, K, Beck, AD, Fioravantti Júnior, G, Cremonese, RV, et al.. Neutrophil-lymphocyte ratio in the early diagnosis of sepsis in an intensive care unit: a case-control study. Revista Brasileira de Terapia Intensiva 2019;31:64–70. https://doi.org/10.5935/0103-507x.20190010.Search in Google Scholar PubMed PubMed Central
50. Drăgoescu, AN, Pădureanu, V, Stănculescu, AD, Chiuțu, LC, Tomescu, P, Geormăneanu, C, et al.. Neutrophil to lymphocyte ratio (NLR)—a useful tool for the prognosis of sepsis in the ICU. Biomedicines 2021;10:75. https://doi.org/10.3390/biomedicines10010075.Search in Google Scholar PubMed PubMed Central
Supplementary Material
The online version of this article offers supplementary material (https://doi.org/10.1515/cclm-2022-0713).
© 2022 Walter de Gruyter GmbH, Berlin/Boston
