Smart Feature Engineering for Large Health Datasets
Keywords:
Clinical Feature Engineering, Automated Feature Engineering Systems, Healthcare Machine Learning, Predictive Model Pipelines, Electronic Health Records (EHR), Multimodal Clinical Data, Data Governance in Healthcare AI, Privacy and Regulatory Compliance, Prospective Model Validation, Bias-Aware Feature Design, Data Quality and Preprocessing Pipelines, Candidate Feature Generation and Selection, Feature Scoring Strategies, Reproducible Clinical ML, Biosensor and Time-Series Data, Imaging-Derived Biomarkers, Genomic and Sequencing Features, Point-of-Care Laboratory Data, Clinical Model Generalization, Production-Grade Healthcare AI.Abstract
A rapidly proliferating corpus of clinical research harnessing the power of machine learning has substantial implications for healthcare feature engineering. As a broad umbrella encompassing data preprocessing, quality control, transformation, and generation, feature engineering addresses a major bottleneck in the production of predictive models. Automated feature engineering systems are increasingly deployed at scale to meet the challenges of generating the vast quantity of predictive features necessary for successful, generalizable, and clinically useful machine learning systems. Such clinical feature engineering systems produce features that are applied in a predictive setting after the fact and not explicitly linked to clinical care, but nevertheless involve substantial risk. A principled examination of a clinical feature engineering system can be framed in terms of six core components: data governance; compliance with privacy and regulatory constraints; appropriate validation and prospective evaluation; consideration of data biases; the use of effective data-quality and preprocessing pipelines; and sound candidate feature generation, scoring, and selection strategies. Health systems typically possess an assemblage of rich and diverse, yet underutilized, information with the potential to contribute meaningfully to clinical prediction problems. Electronic health record (EHR) data, comprising clinical notes, laboratory values, medication orders, and procedure codes; over a decade’s worth of length and width dataset and point-of-care laboratory test results; continuous biosensor measurements; DNA sequencing data; biomarkers derived from imaging; and drug compounds targeting genotypes provide raw material for hundreds of prediction problems in diverse specialties. However, machine learning in healthcare exhibits a stunning lack of reproducibility: many predictive models fail to retain their accuracy in different cohorts, and those that do are seldom incorporated into routine clinical care. A significant bottleneck underlying this failure lies with the feature engineering step.
References
1. Chen, I. Y., Joshi, S., Ghassemi, M., & Ranganath, R. (2021). Probabilistic machine learning for healthcare. Annual Review of Biomedical Data Science, 4, 393–415.
2. Kreimeyer, K., Dang, O., Spiker, J., Muñoz, M. A., Rosner, G., Ball, R., & Botsis, T. (2021). Feature engineering and machine learning for causality assessment in pharmacovigilance: Lessons learned from application to the FDA Adverse Event Reporting System. Computers in Biology and Medicine, 135, 104517.
3. Kolla, S. H. (2023). Large Language Model-Driven Enterprise Service Intelligence for Digital Workflow Transformation. International Journal of Research and Applied Innovations, 6(1), 8380-8391.
4. Garnica, O., Gómez, D., Ramos, V., Hidalgo, J. I., & Ruiz-Giardín, J. M. (2021). Diagnosing hospital bacteraemia in the framework of predictive, preventive and personalised medicine using electronic health records and machine learning classifiers. EPMA Journal, 12, 365–381.
5. Yoshida, Y., et al. (2021). Supervised machine learning-based prediction for in-hospital pressure injury development using electronic health records: A retrospective observational cohort study in a university hospital in Japan. International Journal of Nursing Studies, 119, 103932.
6. Wang, H., Wang, L., Lee, E. H., Zheng, J., Zhang, W., Halabi, S., Liu, C., Deng, K., Song, J., & Yeom, K. W. (2021). Decoding COVID-19 pneumonia: Comparison of deep learning and radiomics CT image signatures. European Journal of Nuclear Medicine and Molecular Imaging, 48, 1873–1882.
7. Bandi, V. D. V. K. (2023, November). Production-Grade Machine Learning Pipelines For Healthcare Predictive Analytics. South Eastern European Journal of Public Health, 189–205.
8. Yuan, J., Ran, X., Liu, K., Yao, C., Yao, Y., Wu, H., & Liu, Q. (2022). Machine learning applications on neuroimaging for diagnosis and prognosis of epilepsy: A review. Journal of Neuroscience Methods, 368, 109441.
9. Janjua, Z. H., Kerins, D., O’Flynn, B., & Tedesco, S. (2022). Knowledge-driven feature engineering to detect multiple symptoms using ambulatory blood pressure monitoring data. Computer Methods and Programs in Biomedicine, 217, 106638.
10. Kumaraswamy, N., Markey, M. K., Barner, J. C., & Rascati, K. (2022). Feature engineering to detect fraud using healthcare claims data. Expert Systems with Applications, 201, 118433.
11. Rajendran, R., & Karthi, A. (2022). Heart disease prediction using entropy based feature engineering and ensembling of machine learning classifiers. Expert Systems with Applications, 207, 117882.
12. Mangalampalli, B. M. Generative AI Applications In Healthcare Data Mart Design And Optimization.
13. Garriga, R., Mas, J., Abraha, S., Nolan, J., Harrison, O., Tadros, G., & Matic, A. (2022). Machine learning model to predict mental health crises from electronic health records. Nature Medicine, 28, 1240–1248.
14. Xie, F., Zhou, J., Lee, J. W., Tan, M., Li, S., Rajnthern, L. S., Chee, M. L., Chakraborty, B., Wong, A. I., Dagan, A., Ong, M. E. H., Gao, F., & Liu, N. (2022). Benchmarking emergency department prediction models with machine learning and public electronic health records. Scientific Data, 9, 658.
15. Sadasivuni, S., Saha, M., Bhatia, N., Banerjee, I., & Sanyal, A. (2022). Fusion of fully integrated analog machine learning classifier with electronic medical records for real-time prediction of sepsis onset. Scientific Reports, 12, 5711.
16. Davuluri, P. S. L. N. (2023). Integrating artificial intelligence into event-driven financial crime compliance platforms. International Journal of Finance, 36(6), 707-736.
17. Wu, Y., et al. (2022). Improved prediction of body mass index in real-world administrative healthcare claims databases. Pharmacoepidemiology and Drug Safety, 31, 1019–1028.
18. Yang, S., Varghese, P., Stephenson, E., Tu, K., & Gronsbell, J. (2023). Machine learning approaches for electronic health records phenotyping: A methodical review. Journal of the American Medical Informatics Association, 30(2), 367–381.
19. Sun, H., & Wang, X. (2023). High-dimensional feature selection in competing risks modeling: A stable approach using a split-and-merge ensemble algorithm. Biometrical Journal, 65(2), e2100164.
20. Zeng, D., et al. (2023). A general framework of nonparametric feature selection in high-dimensional data. Biometrics, 79(2), 951–963.
21. Yandamuri, U. S. (2023). An Intelligent Analytics Framework Combining Big Data and Machine Learning for Business Forecasting. Zenodo.
22. La Cava, W. G., et al. (2023). A flexible symbolic regression method for constructing interpretable clinical prediction models. npj Digital Medicine, 6, Article 116.
23. Ben Miled, Z., Dexter, P. R., Grout, R. W., & Boustani, M. (2023). Feature engineering from medical notes: A case study of dementia detection. Heliyon, 9(3), e14636.
24. Ben-Assuli, O., Heart, T., Klempfner, R., & Padman, R. (2023). Human-machine collaboration for feature selection and integration to improve congestive heart failure risk prediction. Decision Support Systems, 172, 113982.
25. Margeloiu, A., Simidjievski, N., Liò, P., & Jamnik, M. (2023). Weight predictor network with feature selection for small sample tabular biomedical data. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8), 9195–9203.
26. Bhattacharjee, A., Basak, S., & Kumari, P. (2023). A two-step feature selection procedure for relevant markers of squamous cell lung carcinoma using different survival models. Healthcare Analytics, 3, 100168.
27. Kolla, S. K., & Reddy, V. A. R. (2023). Deep Learning Architectures For Multimodal Medical Data Integration. South Eastern European Journal of Public Health, 248–260.
28. Wang, X., et al. (2023). Deep feature screening: Feature selection for ultra high-dimensional data via deep neural networks. Neurocomputing, 538, 126186.
29. Mukherjee, P., Humbert-Droz, M., Chen, J. H., & Gevaert, O. (2023). SCOPE: Predicting future diagnoses in office visits using electronic health records. Scientific Reports, 13, 11005.
30. Inness, C., et al. (2023). Predicting disease onset from electronic health records for population health management: A scalable and explainable deep learning approach. Frontiers in Artificial Intelligence, 6, 1287541.
31. Kumar, V. D. V. B. (2023). Automated feature engineering systems in large-scale healthcare data environments. Journal of Neonatal Surgery.
Additional Files
Published
Issue
Section
License
Copyright (c) 2024 European Advanced Journal for Science & Engineering (EAJSE)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Articles published in the European Advanced Journal for Science & Engineering (EAJSE) are made freely available online immediately upon publication under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited. Authors retain copyright of their work. By submitting to EAJSE, authors grant the journal the right of first publication. For details, visit: https://creativecommons.org/licenses/by/4.0/
