Hospital Network Data Engineering
Keywords:
Health Data Analytics, Healthcare Data Pipeline, Clinical Decision-Making Support, Data Governance, Healthcare Data Silos, HTAP.Abstract
Healthcare is increasingly recognized as a data-intensive industry. Multi-hospital networks, among other organizations, face mounting operational and governance challenges because of rigid data-integration pipelines that support all data sources and destinations in the network. These pipelines have become difficult to modify, causing them to lag behind the changing needs of the clinical operation. Scalable data-pipeline architectures better support clinical decision making, optimize hospital operations, ease data quality and compliance concerns, and contribute to improved patient outcomes. Meeting scalability goals requires breaking up monolithic data-integration pipelines into smaller decoupled components and aligning service-level agreements of pipeline components and source systems.
Parallelization and adoption of distributed data-warehouse technology mitigate the burden of ingesting data into a multi-hospital network. However, latency requirements still warrant the construction of separate pipelines for data ingress from clinical devices, electronic health records, and external laboratory-information systems. Healthcare associations recommend near real-time data availability for a growing list of clinical and operational applications. Mishandling the real-time ingestion of data from clinical devices, in particular, compromises availability and performance. Scalable architectural patterns for real-time streaming Ingestion from heterogeneous data sources, transport processes, and back-end processing structures are detailed.
References
1. Aerts, H., Kalra, D., Sáez, C., Ramírez-Anguita, J. M., Mayer, M.-A., Garcia-Gomez, J. M., Durà-Hernández, M., Thienpont, G., & Coorevits, P. (2021). Quality of hospital electronic health record (EHR) data based on the International Consortium for Health Outcomes Measurement (ICHOM) in heart failure: Pilot data quality assessment study. JMIR Medical Informatics, 9(8), e27842.
2. Tian, Q., Han, Z., Yu, P., An, J., Lu, X., & Duan, H. (2021). Application of openEHR archetypes to automate data quality rules for electronic health records: A case study. BMC Medical Informatics and Decision Making, 21, 113.
3. Razzaghi, H., Greenberg, J., & Bailey, L. C. (2021). Developing a systematic approach to assessing data quality in secondary use of clinical data based on intended use. Learning Health Systems, 6(1), e10264.
4. Mattaparthi, R. (2024). Transformer-Based Fault Diagnosis for Large-Scale Standby Power Generators: Partial Discharge Pattern Recognition at Hyperscale Data Center Installations. Journal of Computational Analysis and Applications (JoCAAA), 33(08), 8781-8799.
5. Li, E., Clarke, J., Neves, A. L., Ashrafian, H., & Darzi, A. (2021). Electronic health records, interoperability and patient safety in health systems of high-income countries: A systematic review protocol. BMJ Open, 11(7), e044941.
6. Ozonze, O., Scott, P. J., & Hopgood, A. A. (2023). Automating electronic health record data quality assessment. Journal of Medical Systems, 47(1), 23.
7. Bernstam, E. V., Warner, J. L., Krauss, J. C., Ambinder, E., Rubinstein, W. S., Komatsoulis, G., Miller, R. S., & Chen, J. L. (2022). Quantitating and assessing interoperability between electronic health records. Journal of the American Medical Informatics Association, 29(5), 753–760.
8. Mangalampalli, B. M. (2024). Transparent Intelligence Explainability Frameworks for AI-Driven Clinical Decision Support in Healthcare Business Intelligence. International Journal of Research Publications in Engineering, Technology and Management (IJRPETM), 7(3), 10566-10579.
9. Cheng, K. Y., Pazmino, S., & Schreiweis, B. (2022). ETL processes for integrating healthcare data—Tools and architecture patterns. Studies in Health Technology and Informatics, 294, 355–359.
10. Romanchikova, M., & Hippolyte, J.-L. (2022). A framework for user-configurable data quality assurance of electronic patient records. Studies in Health Technology and Informatics, 289, 89–92.
11. Schmeelk, S., Kanabar, M., Peterson, K., & Pathak, J. (2022). Electronic health records and blockchain interoperability requirements: A scoping review. JAMIA Open, 5(3), ooac068.
12. Goldstein, N. D., Kahal, D., Testa, K., Gracely, E. J., & Burstyn, I. (2022). Data quality in electronic health record research: An approach for validation and quantitative bias analysis for imperfectly ascertained health outcomes via diagnostic codes. Harvard Data Science Review, 4(2).
13. Loganathan, R., & Kolla, S. H. (2024). Harnessing generative AI for adaptive risk policy generation in multi-regulatory data center environments. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 15(3), 505-515.
14. Diaz-Garelli, F., Long, A., Bancks, M. P., Bertoni, A. G., Narayanan, A., & Wells, B. J. (2022). Developing a data quality standard primer for cardiovascular risk assessment from electronic health record data using the DataGauge process. AMIA Annual Symposium Proceedings, 2021, 388–397.
15. Wiley, K. K., Mendonca, E., Blackburn, J., Menachemi, N., De Groot, M., & Vest, J. R. (2022). Quantifying electronic health record data quality in telehealth and office-based diabetes care. Applied Clinical Informatics, 13(5), 1172–1180.
16. Avendano, J. P., Gallagher, D. O., Hawes, J. D., Boyle, J., Glasser, L., Aryee, J., & Katt, B. M. (2022). Interfacing with the electronic health record (EHR): A comparative review of modes of documentation. Cureus, 14(6), e26330.
17. Reddy, V. A. R. (2024). Generative Intelligence for Healthcare Claims Processing and Personalized Benefits Management. International Journal of Advanced Engineering Science and Information Technology (IJAESIT), 7(3), 14099.
18. Mihăilescu, A. I., et al. (2022). Semantic interoperability in health records standards: A systematic literature review. Health and Technology, 12, 1009–1024.
19. Kaspar, M., Liman, L., Morbach, C., Dietrich, G., Seidlmayer, L. K., Puppe, F., & Störk, S. (2023). Querying a clinical data warehouse for combinations of clinical and imaging data. Journal of Imaging Informatics in Medicine, 36, 715–724.
20. Kohler, S., Boscá, D., Kärcher, F., Haarbrandt, B., Prinz, M., Marschollek, M., & Eils, R. (2023). Eos and OMOCL: Towards a seamless integration of openEHR records into the OMOP Common Data Model. Journal of Biomedical Informatics, 144, 104437.
21. Hechtel, N., Apfel-Starke, J., Köhler, S., Fradziak, M., Schönfeld, N., Steinmeyer, J., & Oeltze-Jafra, S. (2023). Harmonisation of German health care data using the OMOP Common Data Model—A practice report. Studies in Health Technology and Informatics, 305, 287–290.
22. Kolla, S. H., & Peddi, R. K. (2024). Designing Governance-Aligned GenAI Pipelines Using Small Language Models for Enterprise Workflow Intelligence. International Journal of Science, Research and Technology, 7(6), 13256-13268.
23. Oja, M., Tamm, S., Mooses, K., Pajusalu, M., Talvik, H.-A., Ott, A., Laht, M., Malk, M., Lõo, M., Holm, J., Haug, M., Šuvalov, H., Särg, D., Vilo, J., Laur, S., Kolde, R., & Reisberg, S. (2023). Transforming Estonian health data to the Observational Medical Outcomes Partnership Common Data Model: Lessons learned. JAMIA Open, 6(4), ooad100.
24. Williams, N. (2023). Building the Observational Medical Outcomes Partnership's T-MSIS Analytic File common data model. Informatics in Medicine Unlocked, 39, 101251.
25. Uwase, M., Iradukunda, J. D., Rusa, D. U., Ndahimana, R., Mvuyekure, B., Birindabagabo, P., Ruton, H., Mpunga, T., Mugisha, M., Twizere, C., & Tumusiime, D. (2023). Towards improving data quality in electronic medical records: An investigation of data completeness in a tertiary hospital in Rwanda. Studies in Health Technology and Informatics, 305, 390–393.
26. Mangala, N. (2022). Real-Time Data Quality Monitoring and Gating Frameworks in Cloud-Based Data Pipelines. International Journal of Research and Applied Innovations, 5(6), 8197-8219.
27. Oh, S. W., Ko, S. J., Im, Y. S., Jung, S., Choi, B. Y., Kim, J. Y., Choi, W., & Choi, I. Y. (2023). Data quality assessment for Observational Medical Outcomes Partnership Common Data Model of multi-center. Studies in Health Technology and Informatics, 302, 322–326.
28. Madandola, O. O., Bjarnadottir, R. I., Yao, Y., Ansell, M., Dos Santos, F., Cho, H., Dunn Lopez, K., Macieira, T. G. R., & Keenan, G. M. (2023). The relationship between electronic health records user interface features and data quality of patient clinical information: An integrative review. Journal of the American Medical Informatics Association, 31(1), 240–255.
29. Gude, F., et al. (2023). Interoperability of heterogeneous health information systems: A systematic literature review. Journal of Medical Systems, 47, 1–15.
30. Anand, G., & Sadhna, D. (2023). Electronic health record interoperability using FHIR and blockchain: A bibliometric analysis and future perspective. Perspectives in Clinical Research, 14(4), 187–194.
31. Stausberg, J., et al. (2023). An ETL-process design for data harmonization to participate in international research with German real-world data based on FHIR and OMOP CDM. International Journal of Medical Informatics, 169, 104925.
32. Zhang, S., et al. (2024). Cross-standard health data harmonization using semantics of data elements. Scientific Data, 11, 1407.
33. Papapostolou, G., Chytas, A., Rekkas, A., Bigaki, M., Zeimpekis, D., Dermentzoglou, L., Tortopidis, G., & Natsiavas, P. (2024). Real-world data in Greece: Mapping the Papageorgiou General Hospital data to the OMOP Common Data Model. Studies in Health Technology and Informatics, 316, 1324–1325.
34. Fraser, H. S. F., Mugisha, M., Bacher, I., Ngenzi, J. L., Seebregts, C., Umubyeyi, A., & Condo, J. (2024). Factors influencing data quality in electronic health record systems in 50 health facilities in Rwanda and the role of clinical alerts: Cross-sectional observational study. JMIR Public Health and Surveillance, 10, e49127.
35. Förstel, S., Förstel, M., Gallistl, M., Zanca, D., Eskofier, B. M., & Rothgang, E. M. (2024). Data quality in hospital information systems: Lessons learned from analyzing 30 years of patient data in a regional German hospital. International Journal of Medical Informatics, 192, 105636.
36. Mahadevan, S. (2024). Clouding the Future: Innovating Towards Net-Zero Emissions. International Journal of Computing and Engineering, 6(2), 17-23.
37. Tabari, P., Costagliola, G., De Rosa, M., & Boeker, M. (2024). State-of-the-art Fast Healthcare Interoperability Resources (FHIR)-based data model and structure implementations: Systematic scoping review. JMIR Medical Informatics, 12, e55062.
38. Niţulescu, A., & Stoicu-Tivadar, L. (2024). Data standardization in the medical field through FHIR and FAIR implementation: A systematic review. Studies in Health Technology and Informatics, 316, 1128–1132.
39. Hoffmann, N., Nesterow, S., Peng, Y., Henke, C., Barnett, N., Klengel, A., Gruhl, M., Bartos, S., Nüßler, J., Gebler, F., Grummt, S., Seim, J., Bathelt, M., Reinecke, S., Wolfien, M., Weidner, S., & Sedlmayr, M. (2024). Streamlining intersectoral provision of real-world health data: A service platform for improved clinical research and patient care. Journal of Medical Internet Research, 26, e53547.
40. Nesterow, S., et al. (2024). Uncovering harmonization potential in health care data through iterative refinement of Fast Healthcare Interoperability Resources profiles based on retrospective discrepancy analysis: Case study. JMIR Medical Informatics, 12, e57005.
41. Zhang, X., et al. (2024). Development and initial validation of a data quality evaluation tool in obstetrics real-world data through HL7-FHIR interoperable Bayesian networks and expert rules. JAMIA Open, 7(3), ooae062.
42. Kleesiek, J., et al. (2024). Establishing medical intelligence—Leveraging Fast Healthcare Interoperability Resources to improve clinical management: Retrospective cohort and clinical implementation study. Journal of Medical Internet Research, 26, e55148.
