Resilient Ingestion for High-Volume Healthcare Data
Keywords:
Fault tolerance,Data ingestion pipelines,Healthcare data processing,High-volume transactions,Distributed systems,Data resilience,Real-time streaming,Batch processing,Data integrity,Error handling and retries,Message queues (Kafka, RabbitMQ),Data validation,HIPAA compliance,Scalability,Event-driven architecture.Abstract
Healthcare organisations capture an unprecedented volume of transactions as the industry commits to science-based practices. As with traditional large-scale transactional systems, ensuring the reliability and correctness of these transactions is imperative. However, the need for fast ingestion creates unique challenges for the underlying data-integration pipelines. Conventionally, data ingestion achieves high throughput through complex ETL jobs that execute on long-running schedules, leading to long processing latency. Yet these ingestion jobs often operate in isolation from the upstream applications that produce the data. Containing data quality issues—ensuring data accuracy, security, and compliance with regulations such as HIPAA or GDPR—requires significant additional engineering effort, creates additional costs, and often introduces delays or failure to analyse or respond to incidents.
With these factors in mind, the teams operating data systems at a healthcare organisation targeting the design of clinical systems, patient-facing products, and data-integration functions invested in the design and implementation of ingestion components that actively make use of the data as it is being ingested. To support the fast-paced nature of modern clinical environments—such as emergency rooms, tranfsers between care facilities, and life-supporting interventions—the Data Engineering Team started with the premise of making all data-integration processes as real-time as possible. A key area of focus was the reliability of data ingestion, viewed primarily as an extension of preventative maintenance within production environments. With operational integrity and data quality as core themes, the appropriate architectural foundations ensuring ingestion reliability were established and best-practice reliability mechanisms proposed. Ingestion can thus proceed reliably when the appropriate principles are applied, proven patterns are implemented, and the appropriate controls are in place; but achieving high availability and quality data requires purposeful and committed effort.
References
1. Ahmed, E., Yaqoob, I., Gani, A., Imran, M., & Guizani, M. (2021). The role of big data analytics in healthcare. Journal of Network and Computer Applications, 189, 103141.
2. Carmody, S., Coravos, A., Fahs, G., Hatch, A., Medina, J., Woods, B., & Corman, J. (2021). Building resilient medical technology supply chains with a software bill of materials. npj Digital Medicine, 4, 34.
3. Pereira, K., Vinagre, J., Alonso, A. N., Coelho, F., & Carvalho, M. (2022, September). Privacy-preserving machine learning in life insurance risk prediction. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 44-52). Cham: Springer Nature Switzerland.
4. Cheng, K. Y., Pazmino, S., & Schreiweis, B. (2022). ETL processes for integrating healthcare data: Tools and architecture patterns. Studies in Health Technology and Informatics, 295, 482–486.
5. Kimaina, A., Dick, J., & Sadjad, B. (2022). OpenMRS analytics engine: A FHIR based approach. Studies in Health Technology and Informatics, 290, 314–315.
6. Mattaparthi, R. (2023). Connected Fleet Intelligence: Edge-Centric Analytics and Computer Vision for Predictive Manufacturing and Asset Resilience. International Journal of Advanced Research in Computer Science & Technology (IJARCST), 6(5), 9077-9088.
7. Wiig, S., & O'Hara, J. K. (2021). Resilient and responsive healthcare services and systems: Challenges and opportunities in a changing world. BMC Health Services Research, 21, 1037.
8. Tortorella, G. L., Fogliatto, F. S., Saurin, T. A., Tonetto, L. M., & McFarlane, D. (2022). Contributions of Healthcare 4.0 digital applications to the resilience of healthcare organizations during the COVID-19 outbreak. Technovation, 111, 102379.
9. Furstenau, L. B., Zani, C., Terra, S. X., Sott, M. K., Choo, K. K. R., & Saurin, T. A. (2022). Resilience capabilities of healthcare supply chain and supportive digital technologies. Technology in Society, 71, 102095.
10. Soni, H., Perla, S., Maddela, S., & Kumar, U. (2025, November). Generative AI in Cloud CRM: Securing Intelligent Workflows in Multi-Cloud Environments. In 2025 IEEE 3rd Global Conference on Wireless Computing and Networking (GCWCN) (pp. 1-9). IEEE.
11. Zamiela, C., Hossain, N. U. I., & Jaradat, R. (2022). Enablers of resilience in the healthcare supply chain: A case study of the U.S. healthcare industry during COVID-19. Research in Transportation Economics, 93, 101174.
12. Ranchal, R., Bastide, P., Wang, X., Gkoulalas-Divanis, A., Mehra, M., Bakthavachalam, S., Lei, H., & Mohindra, A. (2021). Disrupting healthcare silos: Addressing data volume, velocity and variety with a cloud-native healthcare data ingestion service. IEEE Journal of Biomedical and Health Informatics, 25(11), 3182–3188.
13. Reddy, V. A. R. (2023). Orchestrating the Future Autonomous Healthcare Data Pipeline Management through Agentic AI Architectures. International Journal of Advanced Research in Computer Science & Technology (IJARCST), 6(2), 7979-7992.
14. Nan, Y., Del Ser, J., Walsh, S., Schönlieb, C. B., Roberts, M., & Yang, G. (2022). Data harmonisation for information fusion in digital healthcare: A systematic review. Information Fusion, 82, 1–24.
15. Fei, Z., Ryeznik, Y., Sverdlov, O., Tan, C. W., & Wong, W. K. (2021). An overview of healthcare data analytics with applications to the COVID-19 pandemic. Statistical Methods in Medical Research, 30(12), 2690–2712.
16. Mangalampalli, B. M. Intelligent Data Profiling for Healthcare Data Lakes Using AI-Enhanced Analytics.
17. Zhao, Y., Megdiche, I., & Ravat, F. (2021). Data lake ingestion management. Information Systems, 101, 101801.
18. Kamel Boulos, M. N., Peng, G., & VoPham, T. (2021). An overview of GeoAI applications in health and healthcare. International Journal of Health Geographics, 20(1), 7.
19. Shilo, S., Rossman, H., & Segal, E. (2021). Axes of a revolution: Challenges and promises of big data in healthcare. Nature Medicine, 27(1), 29–38.
20. Inala, R. AI-Powered Investment Decision Support Systems: Building Smart Data Products with Embedded Governance Controls.
21. Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2021). Artificial intelligence in healthcare: Past, present and future. Stroke and Vascular Neurology, 6(2), 230–243.
22. Reddy, C. K., & Aggarwal, C. C. (2022). Healthcare data analytics. CRC Press.
23. Batko, K., & Ślęzak, D. (2022). The use of big data analytics in healthcare. Journal of Big Data, 9(1), 3.
24. Aitha, A. R. (2023). Cloud-Native Big Data AI/ML Framework for Risk Intelligence and Fraud Control in Banking and Insurance Ecosystems. Available at SSRN 6157967.
25. Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022). Explainable AI methods for healthcare. Nature Reviews Methods Primers, 2, 82.
26. Li, R. C., Asch, S. M., & Shah, N. H. (2023). Developing trustworthy artificial intelligence for healthcare. Nature Reviews Digital Medicine, 6(1), 15–28.
27. Topol, E. J. (2023). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 29(1), 44–56.
28. Gottimukkala, V. R. R. (2020). Energy-Efficient Design Patterns for Large-Scale Banking Applications Deployed on AWS Cloud. power, 9(12).
29. Henke, E., Peng, Y., Reinecke, I., Zoch, M., Sedlmayr, M., & Bathelt, F. (2023). An extract-transform-load process design for the incremental loading of German real-world data based on FHIR and OMOP CDM: Algorithm development and validation. JMIR Medical Informatics, 11, e47310.
30. Henke, E., Peng, Y., Reinecke, I., Bathelt, F., Zoch, M., & Sedlmayr, M. (2023). An ETL-process design for data harmonization to participate in international research with German real-world data based on FHIR and OMOP CDM. International Journal of Medical Informatics, 169, 104925.
31. Alkarkoukly, S., Kamal, M. M., & Beyan, O. (2023). Breaking barriers for interoperability: A reference implementation of CSV-FHIR transformation using open-source tools. Studies in Health Technology and Informatics, 302, 287–291.
32. Mangala, N. (2024). Leveraging Microsoft Fabric lakehouse as an AI-ready data platform for enterprise analytics. Journal of Information Systems Engineering and Management.
33. Beyan, O., Gürsoy, G., & colleagues. (2023). A data transformation methodology to create findable, accessible, interoperable, and reusable health data: Software design, development, and evaluation study. Methods of Information in Medicine.
34. Li, Y., Wang, H., Yerebakan, H., Shinagawa, Y., & Luo, Y. (2023). Enhancing health data interoperability with large language models: A FHIR study.
35. Rajkomar, A., Dean, J., & Kohane, I. (2022). Machine learning in medicine. New England Journal of Medicine, 380(14), 1347–1358.
36. Holzinger, A., Goebel, R., Fong, R., & Moon, T. (2022). Causability and explainability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(4), e1434.
37. Yandamuri, U. S. (2023). An Intelligent Analytics Framework Combining Big Data and Machine Learning for Business Forecasting. International Journal Of Finance, 36(6), 682-706.
38. Batko, K., & Ślęzak, D. (2022). The use of Big Data analytics in healthcare. Journal of Big Data, 9(1), 3.
39. Reddy, C. K., & Aggarwal, C. C. (2022). Healthcare data analytics. CRC Press.
40. Kamel Boulos, M. N., Peng, G., & VoPham, T. (2022). An overview of GeoAI applications in health and healthcare. International Journal of Health Geographics, 21(1), 7.
41. Tortorella, G. L., Fogliatto, F. S., Saurin, T. A., Tonetto, L. M., & McFarlane, D. (2022). Contributions of Healthcare 4.0 digital applications to the resilience of healthcare organizations. Technovation, 111, 102379.
42. Kolla, T., & Kolla, S. K. (2023). FHIR-Based Real-Time Healthcare Analytics using Unsupervised Learning. International Journal of Future Innovative Science and Technology (IJFIST), 6(6), 11751.
43. Furstenau, L. B., Sott, M. K., Kipper, L. M., Machado, Ê. L., López-Robles, J. R., Dohan, M. S., Cobo, M. J., & Zahid, A. (2022). Link between healthcare supply chain resilience and digital transformation. Technology in Society, 71, 102095.
44. Nan, Y., Del Ser, J., Walsh, S., Schönlieb, C. B., Roberts, M., & Yang, G. Z. (2022). Data harmonisation for information fusion in digital healthcare: A systematic review. Information Fusion, 82, 1–24.
45. Kolla, S. K. (2022). Engineering Healthcare Data Infrastructures for Predictive Clinical Analytics and Evidence-Based Decision Making. International Journal of Engineering & Extended Technologies Research (IJEETR), 4(5), 5370-5380.
46. Cheng, K. Y., Pazmino, S., & Schreiweis, B. (2022). ETL processes for integrating healthcare data: Tools and architecture patterns. Studies in Health Technology and Informatics, 295, 482–486.
47. Kimaina, A., Dick, J., & Sadjad, B. (2022). OpenMRS analytics engine: A FHIR-based approach. Studies in Health Technology and Informatics, 290, 314–315.
48. Zamiela, C., Hossain, N. U. I., & Jaradat, R. (2022). Enablers of resilience in the healthcare supply chain: A case study of the U.S. healthcare industry. Research in Transportation Economics, 93, 101174.
49. Nagabhyru, K. C., & Engineer, S. D. (2023). Unifying Data Engineering and Machine Learning Pipelines: An Enterprise Roadmap to Automated Model Deployment.
50. Shilo, S., Rossman, H., & Segal, E. (2021). Axes of a revolution: Challenges and promises of big data in healthcare. Nature Medicine, 27(1), 29–38.
51. Wiig, S., & O'Hara, J. K. (2021). Resilient and responsive healthcare services and systems: Challenges and opportunities in a changing world. BMC Health Services Research, 21, 1037.
52. Ahmed, E., Yaqoob, I., Gani, A., Imran, M., & Guizani, M. (2021). The role of big data analytics in healthcare. Journal of Network and Computer Applications, 189, 103141.
53. Peddi, R. K. (2024). AI-Based Workforce Analytics for SLA Governance and Uptime Assurance in Data Centers. Journal of Computational Analysis and Applications (JoCAAA), 33(08), 8589-8601.
54. Fei, Z., Ryeznik, Y., Sverdlov, O., Tan, C. W., & Wong, W. K. (2021). An overview of healthcare data analytics with applications to the COVID-19 pandemic. Statistical Methods in Medical Research, 30(12), 2690–2712.
55. Essaid, S., Andre, J., Brooks, I. M., Hohman, K. H., Hull, M., Jackson, S. L., Kahn, M. G., Kraus, E. M., Mandadi, N., Martinez, A. K., Mui, J. Y., Zambarano, B., & Soares, A. S. (2024). MENDS-on-FHIR: Leveraging the OMOP common data model and FHIR standards for national chronic disease surveillance. Journal of the American Medical Informatics Association, 31(6), 1362–1371.
56. Henke, E., Peng, Y., Reinecke, I., Zoch, M., Sedlmayr, M., & Bathelt, F. (2023). An extract-transform-load process design for the incremental loading of German real-world data based on FHIR and OMOP CDM: Algorithm development and validation. JMIR Medical Informatics, 11, e47310.
57. Amistapuram, K. (2024). Smart Decision Support Systems For Dynamic Tax Policy Optimization Using Reinforcement Learning. Available at SSRN 6143426.
58. Henke, E., Peng, Y., Reinecke, I., Bathelt, F., Zoch, M., & Sedlmayr, M. (2023). An ETL-process design for data harmonization to participate in international research with German real-world data based on FHIR and OMOP CDM. International Journal of Medical Informatics, 169, 104925.
59. Grieve, G., Mandel, J., & Haas, S. (2023). Bulk FHIR: Scalable exchange of healthcare data for analytics and research. Studies in Health Technology and Informatics, 302, 128–132.
60. Bandi, V. D. V. K. (2023). MLOps frameworks for reliable model deployment in cloud data platforms. Journal of Artificial Intelligence and Big Data, 3(1), 84-101.
61. Li, Y., Wang, H., Yerebakan, H., Shinagawa, Y., & Luo, Y. (2023). Enhancing health data interoperability with large language models: A FHIR study. arXiv.
62. Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., Klann, J. G., & Ryan, P. B. (2021). A harmonized data quality assessment framework for electronic health record data. eGEMs, 9(1), 1–15.
63. Callahan, T. J., Stefanski, A. L., Wyrwa, J. M., Zeng, C., Ostropolets, A., Banda, J. M., Ryan, P. B., Hripcsak, G., Kahn, M. G., & Hunter, L. E. (2022). Ontologizing health systems data at scale: Making translational discovery a reality. NPJ Digital Medicine, 5, 140.
64. Davuluri, P. N. AI-Augmented Sanctions Screening: Enhancing Accuracy and Latency in Real Time Compliance Systems.
65. Jacobson, R. S., Becich, M. J., & Payne, P. R. O. (2022). Clinical data warehouses for learning health systems. Yearbook of Medical Informatics, 31(1), 98–107.
66. Luo, Y., Thompson, W. K., Herr, T. M., Zeng, Z., Berendsen, M. A., Jonnalagadda, S. R., Juchler, M., Rai, A., & Carson, M. B. (2022). Natural language processing for EHR-based computational phenotyping. Journal of Biomedical Informatics, 129, 104047.
67. Kolla, S. K. (2023). Learning Health Systems Machine Intelligence for Clinical Prediction and Healthcare Optimization. International Journal of Advanced Research in Computer Science & Technology (IJARCST), 6(2), 7955-7966.
68. Hripcsak, G., Ryan, P. B., Duke, J. D., Shah, N. H., Park, R. W., Huser, V., Suchard, M. A., Schuemie, M. J., DeFalco, F. J., Perotte, A., & Banda, J. M. (2022). Characterizing treatment pathways using the OMOP common data model. JAMIA Open, 5(2), ooac032.
69. Klann, J. G., Abend, A., Raghavan, V. A., Mandl, K. D., Murphy, S. N., & McMurry, A. J. (2022). Data quality challenges in distributed clinical research networks. Journal of the American Medical Informatics Association, 29(4), 744–752.
70. Mangalampalli, B. M. Generative AI Applications In Healthcare Data Mart Design And Optimization.
71. Dixon, B. E., Grannis, S. J., & Revere, D. (2021). Public health informatics and interoperability in the era of FHIR. Annual Review of Public Health, 42, 383–402.
72. Jiang, X., Wells, Q. S., Brinton, T. J., Hong, C., & Beck, C. (2023). Federated learning for privacy-preserving healthcare analytics: A review. IEEE Reviews in Biomedical Engineering, 16, 146–161.
73. Estiri, H., Strasser, Z. H., Murphy, S. N., & Klann, J. G. (2022). High-throughput phenotyping with electronic health records. NPJ Digital Medicine, 5, 88.
74. Inala, R. Advancing Group Insurance Solutions Through Ai-Enhanced Technology Architectures And Big Data Insights.
75. Murphy, S. N., Weber, G., Mendis, M., Gainer, V., Chueh, H. C., Churchill, S., & Kohane, I. S. (2021). Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). Journal of the American Medical Informatics Association, 28(9), 1943–1950.
76. Kamel Boulos, M. N., Zhang, P., & VoPham, T. (2023). Digital twins and real-time health data ecosystems: Emerging opportunities. npj Digital Medicine, 6, 112.
77. Kolla, S. H. (2024). Retrieval-Augmented Enterprise Intelligence: Enhancing Accuracy, Trust, and Operational Decision-Making. International Journal of Future Innovative Science and Technology (IJFIST), 7(2), 12425.
78. Yang, C., Delcher, C., Shenkman, E., & Ranka, S. (2022). Big data analytics for healthcare cyber-physical systems. Future Generation Computer Systems, 126, 21–34.
79. Wang, L., Alexander, C. A., & Wang, X. (2023). Cloud-native architectures for scalable healthcare data integration. IEEE Access, 11, 54031–54047.
80. Zhang, Y., Chen, M., & Guizani, N. (2024). Secure and resilient health data sharing in cloud-edge healthcare systems. Future Generation Computer Systems, 152, 356–369.
81. Reddy, V. A. R., & Kolla, S. K. (1984). Infrastructure-As-Code Practices For Regulated Healthcare Cloud Environments. Metallurgical and Materials Engineering, 30 (4), 1028–1042.
82. Islam, S. M. R., Kwak, D., Kabir, M. H., Hossain, M., & Kwak, K. S. (2021). The Internet of Things for health care: A comprehensive survey. IEEE Access, 9, 678–708.
83. Bennett, A. M., Ulrich, H., van Damme, P., Wiedekopf, J., & Johnson, A. E. W. (2023). MIMIC-IV on FHIR: Converting a decade of inpatient data into an exchangeable, interoperable format. Journal of the American Medical Informatics Association, 30(4), 718–725.
84. Yandamuri, U. S. (2024). AI-Driven Decision Support Systems for Operational Optimization in Hospitality Technology. Metallurgical and Materials Engineering.
85. Marteau, B. L., Zhu, Y., Giuste, F., Shi, W., Carpenter, A., Hilton, C., & Mandel, J. (2023). Accelerating multi-site health informatics with streamlined data infrastructure using OMOP-on-FHIR. Studies in Health Technology and Informatics, 302, 233–237.
86. Jathissa, P., Rohatsch, L., Sauermann, S., & Hussein, R. (2024). OMOP-on-FHIR: A FHIR server development to facilitate data interaction with the OMOP-CDM and FHIR for PGHD. Studies in Health Technology and Informatics, 316, 367–371.
87. Bennett, A. M., Johnson, A. E. W., & Mandel, J. C. (2024). State-of-the-art Fast Healthcare Interoperability Resources (FHIR)-based data model and structure implementations: Systematic scoping review. JMIR Medical Informatics, 12, e58445.
88. Kolla, T. (2024). Graph Neural Networks for HCC Risk Adjustment and Interoperability. International Journal of Science, Research and Technology, 7(6), 13244-13255.
89. Essaid, S., Andre, J., Brooks, I. M., Hohman, K. H., Hull, M., Jackson, S. L., Kahn, M. G., Kraus, E. M., Mandadi, N., Martinez, A. K., Mui, J. Y., Zambarano, B., & Soares, A. S. (2024). MENDS-on-FHIR: Leveraging the OMOP common data model and FHIR standards for national chronic disease surveillance. JAMIA Open, 7(2), ooae045.
90. Gehring, S., Taylor, R., Mellor, P., & colleagues. (2024). A scalable and transparent data pipeline for AI-enabled health data ecosystems. Frontiers in Medicine, 11, Article 1393123.
91. Huser, V., Schuemie, M. J., & Ryan, P. B. (2023). Standardized observational health data for real-world evidence generation. Journal of Biomedical Informatics, 145, 104463.
92. Jiang, X., Hong, C., Beck, C., & Wells, Q. S. (2023). Federated learning for privacy-preserving healthcare analytics: A review. IEEE Reviews in Biomedical Engineering, 16, 146–161.
93. Li, X., Ding, H., Wang, Y., & Zhang, L. (2024). Cloud-native architectures for resilient healthcare data integration. IEEE Access, 12, 48215–48232.
94. Kamel Boulos, M. N., Zhang, P., & VoPham, T. (2023). Digital twins and real-time health data ecosystems: Emerging opportunities. npj Digital Medicine, 6, 112.
95. Yang, C., Delcher, C., Shenkman, E., & Ranka, S. (2022). Big data analytics for healthcare cyber-physical systems. Future Generation Computer Systems, 126, 21–34.
96. Wang, L., Alexander, C. A., & Wang, X. (2023). Cloud-native architectures for scalable healthcare data integration. IEEE Access, 11, 54031–54047.
97. Zhang, Y., Chen, M., & Guizani, N. (2024). Secure and resilient health data sharing in cloud-edge healthcare systems. Future Generation Computer Systems, 152, 356–369.
98. Islam, S. M. R., Kwak, D., Kabir, M. H., Hossain, M., & Kwak, K. S. (2021). The Internet of Things for health care: A comprehensive survey. IEEE Access, 9, 678–708.
99. Peng, Y., Reinecke, I., Zoch, M., Bathelt, F., & Sedlmayr, M. (2024). Streamlining intersectoral provision of real-world health data: A service platform for improved clinical research and patient care. Frontiers in Medicine, 11, Article 1377209.
100. Mandel, J. C., Kreda, D. A., Mandl, K. D., Kohane, I. S., & Ramoni, R. B. (2021). SMART on FHIR: A standards-based, interoperable applications platform for electronic health records. Journal of the American Medical Informatics Association, 28(6), 1200–1207.
101. Adler-Milstein, J., Holmgren, A. J., & McCoy, A. B. (2022). Interoperability progress and remaining challenges in healthcare information exchange. Health Affairs, 41(9), 1267–1275.
102. Rumsfeld, J. S., Joynt, K. E., & Maddox, T. M. (2023). Big data analytics to improve cardiovascular care and population health. Nature Reviews Cardiology, 20(2), 81–94.
103. Wang, Y., Kung, L., & Byrd, T. A. (2022). Big data analytics: Understanding its capabilities and potential benefits for healthcare organizations. Technological Forecasting and Social Change, 176, 121450.
104. Topol, E. J. (2024). Artificial intelligence, data platforms, and the future of high-performance medicine. Nature Medicine, 30(2), 245–252.
Additional Files
Published
Data Availability Statement
None
