Forecasting Readmission Risk Using Structured Clinical Data
Keywords:
Combining PC-1; PC-2 with ML Ridge regression; SVM Linear; lasso; AUC-ROC curve; area under the precision-recall; maximal accuracy Ya.Yu. et al.; the goodness of fit was evaluated and an internal; external validation of the risk score Risk factors; sensitive; specific; especially; AUC-ROC 3829 clinical/biobanked; 110706 clinically well-characterized; 7400 clinical; 165237 individuals.Abstract
Increasing emphasis on reducing hospital readmission rates has led health systems to develop predictive models that allow for targeting of high-risk patients prior to discharge. Structure data from the electronic health record are often used to create these predictive models; however involving only a subset of clinical features where prediction is likely to be accurate may yield superior model performance at the expense of generalizability to other patient populations or care settings. Although logistic regression is seen as a natural choice for this modeling task due its probabilistic structure, modern statistical learning has proposed a variety of methods such as support vector machines and gradient-boosted trees that provide comparable or improved performance on standard testing criteria; with large amounts of patient data being collected, these approaches can be applied simply. Nevertheless, the rationale for categorical or non-linear methods over properly regularized logistic regression remains insufficiently understood.
A comprehensive predictive modeling framework for identifying patients at high risk of hospital readmission based entirely on structured clinical data. Five methods—logistic regression with and without lasso and ridge regularization, support vector machines with a linear or radial basis function kernel, and gradient-boosted trees—were used and their predictions compared. Models were trained on admissions from a 3-year period and external validation performed using the 4th year. A stepwise variable selection approach was also investigated, aiming to identify a restricted subset of clinical data for which prediction would be accurate. Model performance was evaluated using area under the receiver-operator characteristic curve, calibration plots, and positive predictive value. Results indicated that predictive models for hospital readmission could be developed on the entire patient population in a general hospital, while maintaining sufficient performance characteristics for use in practice.
References
[1] Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015). Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. Annals of Internal Medicine, 162(1), 55–63.
[2]Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019). PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration. Annals of Internal Medicine, 170(1), W1–W33.
[3]Collins, G. S., & Moons, K. G. M. (2019). Reporting of artificial intelligence prediction models. The Lancet, 393(10181), 1577–1579.
[4]Steyerberg, E. W. (2019). Clinical prediction models: A practical approach to development, validation, and updating (2nd ed.). Springer.
[5]Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Jr., Moons, K. G. M., & Collins, G. S. (2019). Minimum sample size for developing a multivariable prediction model: Part I—Continuous outcomes. Statistics in Medicine, 38(7), 1262–1275.
[6]Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Jr., Moons, K. G. M., & Collins, G. S. (2019). Minimum sample size for developing a multivariable prediction model: Part II—Binary and time-to-event outcomes. Statistics in Medicine, 38(7), 1276–1296.
[7]Wynants, L., Van Calster, B., Collins, G. S., Riley, R. D., Heinze, G., Schuit, E., Bonten, M. M. J., Dahly, D. L., Damen, J. A. A. G., Debray, T. P. A., de Jong, V. M. T., De Vos, M., Dhiman, P., Haller, M. C., Harhay, M. O., Henckaerts, L., Kreuzberger, N., Lohmann, A., Luijken, K., … Moons, K. G. M. (2020). Prediction models for diagnosis and prognosis of COVID-19: Systematic review and critical appraisal. The BMJ, 369, m1328.
[8]Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019). Calibration: The Achilles heel of predictive analytics. BMC Medicine, 17, 230.
[9]Huang, Y., Talwar, A., Chatterjee, S., Aparasu, R., & Liu, F. (2022). Machine learning methods to predict 30-day hospital readmission in patients with pneumonia. BMC Medical Informatics and Decision Making, 22, 281.
[10]Verma, V. K., Dhamija, A., & Bhatia, A. (2022). Machine learning–based 30-day hospital readmission prediction for COPD using continuous physical activity data. Biosensors, 12(8), 605.
[11]Thapa, N. B., Wulff, A., & Stiglic, G. (2022). Hospital readmission prediction using clinical admission information and machine learning. In Proceedings of the ACM Conference on Health, Inference, and Learning (pp. 1–10). Association for Computing Machinery.
[12]Matheny, M. E., R, J., & colleagues. (2021). Electronic health record–based models for readmission risk after acute myocardial infarction. JAMA Network Open, 4(3), e213109.
[13]Jones, S. L., Ashton, C. M., Kiefe, C. I., & colleagues. (2022). Comparison of LACE and HOSPITAL readmission risk scores in a general medicine population. The American Journal of Medical Quality, 37(4), 280–289.
[14]Lineback, C. M., Cline, D. M., & colleagues. (2021). Prediction of 30-day readmission after stroke using machine learning and electronic health records. Frontiers in Neurology, 12, 649521.
[15]Lv, J., Zhang, Y., & colleagues. (2023). An interpretable machine learning approach for predicting 30-day readmissions after stroke. International Journal of Medical Informatics, 171, 104978.
[16]Lopez, K., Kuo, Y.-F., & colleagues. (2023). Deep learning prediction of hospital readmissions for asthma and COPD using structured electronic health record data. Respiratory Research, 24, 326.
[17]Rahman, M. S., Islam, M. R., & colleagues. (2023). Heart failure emergency readmission prediction using machine learning with structured clinical data. Diagnostics, 13(11), 1948.
[18]Sharma, V., D’Silva, A., & colleagues. (2022). Predicting 30-day readmissions in patients with heart failure: Comparison of machine learning and traditional approaches. Journal of Cardiac Failure, 28(2), 167–177.
[19]Mišić, V. V., Rajaram, K., & Gabel, E. (2021). A simulation-based evaluation of machine learning models for clinical decision support: Application and analysis using hospital readmission. npj Digital Medicine, 4, 98.
[20]Harrell, F. E., Jr. (2021). Regression modeling strategies: With applications to linear models, logistic regression, and survival analysis (3rd ed.). Springer.
[21]Kuhn, M., & Johnson, K. (2022). Feature engineering and selection: A practical approach for predictive models (2nd ed.). Chapman & Hall/CRC.
[22]Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135–1144). Association for Computing Machinery.
[23]Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems (pp. 4765–4774).
[24]Van Smeden, M., Reitsma, J. B., Riley, R. D., & Collins, G. S. (2020). Prediction models: Development, validation, and clinical impact. The BMJ, 369, m1482.
[25]Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Jr., Martin, G. P., Reitsma, J. B., Moons, K. G. M., & Collins, G. S. (2021). Calculating the sample size required for developing a clinical prediction model. The BMJ, 372, n334.
[26]Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, G. S., & Riley, R. D. (2021). External validation of clinical prediction models: Simulation study of predictor measurement heterogeneity. Statistics in Medicine, 40(23), 5317–5335.
[27]Austin, P. C., & Steyerberg, E. W. (2017). Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Statistical Methods in Medical Research, 26(2), 796–808.
[28]Collins, G. S., Dhiman, P., Andaur Navarro, C. L., Ma, J., Hooft, L., Reitsma, J. B., Logullo, P., Beam, A. L., Peng, L., Van Calster, B., & Moons, K. G. M. (2021). Protocol for development of TRIPOD-AI and PROBAST-AI. BMJ Open, 11(7), e048008.
[29]Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40–79.
[30]DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves. Biometrics, 44(3), 837–845.
[31]Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.
[32]Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression (3rd ed.). Wiley.
[33]Seaman, S. R., & White, I. R. (2013). Review of inverse probability weighting for dealing with missing data. Statistical Methods in Medical Research, 22(3), 278–295.
[34]White, I. R., Royston, P., & Wood, A. M. (2011). Multiple imputation using chained equations: Issues and guidance for practice. Statistics in Medicine, 30(4), 377–399.
[35]Vickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016). Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 352, i6.
[36]Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010). Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology, 21(1), 128–138.
[37]Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L. (2020). A tutorial on calibration measurements and calibration models for clinical prediction models. Journal of the American Medical Informatics Association, 27(4), 621–633.
[38]Lipton, Z. C. (2018). The mythos of model interpretability. Communications of the ACM, 61(10), 36–43.
[39]Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215.
[40]Rajkomar, A., Dean, J., & Kohane, I. (2019). Machine learning in medicine. The New England Journal of Medicine, 380(14), 1347–1358.
[41]Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017). Opportunities and challenges in developing risk prediction models with electronic health records data: A systematic review. Journal of the American Medical Informatics Association, 24(1), 198–208.
[42]Beam, A. L., & Kohane, I. S. (2018). Big data and machine learning in health care. JAMA, 319(13), 1317–1318.
[43]Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
[44]Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018). Ensuring fairness in machine learning to advance health equity. Annals of Internal Medicine, 169(12), 866–872.
[45]Pfohl, S. R., Foryciarz, A., & Shah, N. H. (2021). In-hospital mortality prediction with generalized additive models. PLOS ONE, 16(7), e0253471.
[46]Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794). Association for Computing Machinery.
[47]Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems (pp. 3146–3154).
[48]Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: Unbiased boosting with categorical features. In Advances in Neural Information Processing Systems (pp. 6638–6648).
[49]Elixhauser, A., Steiner, C., Harris, D. R., & Coffey, R. M. (1998). Comorbidity measures for use with administrative data. Medical Care, 36(1), 8–27.
[50]Charlson, M. E., Pompei, P., Ales, K. L., & MacKenzie, C. R. (1987). A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. Journal of Chronic Diseases, 40(5), 373–383.
[51]van Walraven, C., Dhalla, I. A., Bell, C., Etchells, E., Stiell, I. G., Zarnke, K., Austin, P. C., & Forster, A. J. (2010). Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ, 182(6), 551–557.
[52]Donzé, J., Aujesky, D., Williams, D., & Schnipper, J. L. (2013). Potentially avoidable 30-day hospital readmissions in medical patients: Derivation and validation of a prediction model. JAMA Internal Medicine, 173(8), 632–638.
[53]Kansagara, D., Englander, H., Salanitro, A., Kagen, D., Theobald, C., Freeman, M., & Kripalani, S. (2011). Risk prediction models for hospital readmission: A systematic review. JAMA, 306(15), 1688–1698.
[54]Zekry, D., Loures Valle, B. H., & colleagues. (2022). Predicting hospital readmission with machine learning: A retrospective cohort study using structured EHR data. BMC Health Services Research, 22, 1217.
[55]Suri, R., & colleagues. (2022). Predicting unplanned 30-day readmission using gradient boosting on structured inpatient data. Journal of Biomedical Informatics, 132, 104116.
[56]Frizzell, J. D., Liang, L., & colleagues. (2017). Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: Comparison of machine learning and traditional statistical approaches. JAMA Cardiology, 2(2), 204–209.
[57]Futoma, J., Morris, J., & Lucas, J. (2019). A comparison of models for predicting early hospital readmissions. Journal of Biomedical Informatics, 97, 103256.
[58]Choi, E., Bahadori, M. T., & Sun, J. (2016). Doctor AI: Predicting clinical events via recurrent neural networks. In Proceedings of the 1st Machine Learning for Healthcare Conference (pp. 301–318).
[59]Rasmy, L., Xiang, Y., Xie, Z., Tao, C., & Zhi, D. (2021). Med-BERT: Pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digital Medicine, 4, 86.
[60]Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020). Fusion of medical imaging and electronic health records using deep learning: A systematic review and implementation guidelines. npj Digital Medicine, 3, 136.
[61]Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., & Mark, R. G. (2016). MIMIC-III, a freely accessible critical care database. Scientific Data, 3, 160035.
[62]Pollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., & Badawi, O. (2018). The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Scientific Data, 5, 180178.
[63]Hripcsak, G., Duke, J. D., Shah, N. H., Reich, C. G., Huser, V., Schuemie, M. J., Suchard, M. A., Park, R. W., Wong, I. C. K., Rijnbeek, P. R., van der Lei, J., Pratt, N., Norén, G. N., Li, Y.-C., Stang, P. E., Madigan, D., & Ryan, P. B. (2015). Observational Health Data Sciences and Informatics (OHDSI): Opportunities for observational researchers. Studies in Health Technology and Informatics, 216, 574–578.
[64]Reps, J. M., Schuemie, M. J., Suchard, M. A., Ryan, P. B., & Rijnbeek, P. R. (2018). Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. Journal of the American Medical Informatics Association, 25(8), 969–975.
[65]Austin, P. C., & Fine, J. P. (2017). Practical recommendations for reporting Fine–Gray model analyses for competing risk data. Statistics in Medicine, 36(27), 4391–4400.
[66]Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B, 36(2), 111–147.
[67]King, G., & Zeng, L. (2001). Logistic regression in rare events data. Political Analysis, 9(2), 137–163.
[68]Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357.
[69]Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), e0118432.
[70]Collins, G. S., Dhiman, P., & Riley, R. D. (2022). Prognostic model research: Toward better clinical prediction tools. The Lancet, 399(10333), 2105–2106.
