Modern Data Platform Architecture
Keywords:
Maintenance and services, and provisioning of ubiquitous, high-performance, fault-tolerant access to users across multiple organizations.Abstract
IArchitectural frameworks for large-scale Electronic Health Record (EHR) data platforms are described. Existing EHR data platform architectures often leverage multiple cloud-based solutions blended with institutional infrastructures to manage and analyze clinical data at scale. Key design principles governing the scale of existing EHR data architecture include model design, governance structure, data access management, data security/policy/protection, data-information-language-based standardization, and analytics tool alignment, among others.
The rapidly evolving technology landscape and the unprecedented volume of incident and retrospective clinical data being collected and generated within healthcare organizations have led to the emergent need for a dedicated architectural framework to support large-scale computing in the health informatics domain. The application areas of large-scale computing in health informatics include real-time predictive analytics, risk stratification, patient cohort analytics, development of predictive models for specific institutions or population groups, and many more. The use of EHR data for a multitude of decision-making processes in both clinical and non-clinical settings has prompted the establishment of policies prescribing the conditions of access and use of EHR data for non-employed individuals in the organization. Consequently, the demand for accessing, using, and managing EHR data at scale has impacted the overa
References
1. Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. CIDR 2021.
2. Sawadogo, P. N., & Darmont, J. (2021). On data lake architectures and metadata management. Journal of Intelligent Information Systems, 56(1), 97–120.
3. Nandan, B. P., & Chitta, S. S. (2023). Machine Learning Driven Metrology and Defect Detection in Extreme Ultraviolet (EUV) Lithography: A Paradigm Shift in Semiconductor Manufacturing. Educational Administration: Theory and Practice, 29 (4), 4555–4568. International Journal of Scientific Research and Modern Technology, 1(12), 216-226.
4. Diamantini, C., Lo Giudice, P., Potena, D., Storti, E., & Ursino, D. (2021). An approach to extracting topic-guided views from the sources of a data lake. Information Systems Frontiers, 23(1), 243–262.
5. Mashetty, S. (2023). A Comparative Analysis of Patented Technologies Supporting Mortgage and Housing Finance. Available at SSRN 5249181. Mashetty, S. (2023). A Comparative Analysis of Patented Technologies Supporting Mortgage and Housing Finance. Available at SSRN 5249181.
6. Begoli, E., Goethert, I., & Knight, K. (2021). A lakehouse architecture for the management and analysis of heterogeneous data for biomedical research and mega-biobanks. Proceedings of the 2021 IEEE International Conference on Big Data, 4643–4651.
7. Oreščanin, D., & Hlupić, T. (2021). Data lakehouse—A novel step in analytics architecture. Proceedings of the 44th International Convention on Information, Communication and Electronic Technology, 1242–1246.
8. Mandala, G., Reddy, R., Nishanth, A., Yasmeen, Z., & Maguluri, K. K. (2023). Ai and ml in healthcare: redefining diagnostics, treatment, and personalized medicine. International Journal of Applied Engineering & Technology, 5(S6).
9. Tovarnák, D., Raček, M., & Velan, P. (2021). Cloud native data platform for network telemetry and analytics. Proceedings of the 17th International Conference on Network and Service Management, 394–396.
10. Weintraub, G., Gudes, E., & Dolev, S. (2021). Indexing cloud data lakes within the lakes. Proceedings of the 14th ACM International Conference on Systems and Storage.
11. Nadal, S., Abelló, A., Romero, O., Vansummeren, S., & Vassiliadis, P. (2021). Graph-driven federated data management. IEEE Transactions on Knowledge and Data Engineering, 35(1), 509–520.
12. Mashetty, S. (2023). View of A Comparative Analysis of Patented Technologies Supporting Mortgage and Housing Finance. Educational Administration: Theory and Practice.
13. Megdiche, I., Ravat, F., & Zhao, Y. (2021). Metadata management on data processing in data lakes. In SOFSEM 2021: Theory and Practice of Computer Science (pp. 553–562). Springer.
14. Stach, C., Bräcker, J., Eichler, R., Giebler, C., & Mitschang, B. (2021). Demand-driven data provisioning in data lakes: BARENTS—A tailorable data preparation zone. Proceedings of the 23rd International Conference on Information Integration and Web Intelligence, 187–198.
15. Meda, R., & Pamisetty, A. (2023). Intelligent Infrastructure for Real-Time Inventory and Logistics in Retail Supply Chains. Educational Administration: Theory and Practice, 29 (4), 5215–5233.
16. Ehrlinger, L., Schrott, J., Melichar, M., Kirchmayr, N., Wöber, K., & Schafferer, M. (2021). Data catalogs: A systematic literature review and guidelines to implementation. In Database and Expert Systems Applications—DEXA 2021 Workshops (pp. 148–158). Springer.
17. Schoenenwald, A., Kern, S., Viehhauser, J., & Schildgen, J. (2021). Collecting and visualizing data lineage of Spark jobs. Datenbank-Spektrum, 21, 179–189.
18. Taleb, I., Serhani, M. A., Bouhaddioui, C., & Dssouli, R. (2021). Big data quality framework: A holistic approach to continuous quality management. Journal of Big Data, 8, Article 76.
19. Adusupalli, B. (2022). The Impact of Regulatory Technology (RegTech) on Corporate Compliance: A Study on Automation, AI, and Blockchain in Financial Reporting. Mathematical Statistician and Engineering Applications, 71 (4), 16696–16710.
20. Mainali, K., Ehrlinger, L., Himmelbauer, J., & Matskin, M. (2021). Discovering DataOps: A comprehensive review of definitions, use cases, and tools. Proceedings of the International Conference on Data Analytics, 61–69.
21. Giebler, C., Gröger, C., Hoos, E., Schwarz, H., & Mitschang, B. (2021). A zone reference model for enterprise-grade data lake management. Proceedings of the IEEE International Conference on Enterprise Distributed Object Computing Workshops.
22. Kalisetty, S. (2023). Big Data–Driven Cloud Collaboration Models for Enhancing Supplier–Retailer Synchronization in Mod-ern Manufacturing Supply Chains. Journal of Computational Analy-sis and Applications (JoCAAA), 31(4), 2188-2205.
23. Machado, I. A., Costa, C., & Santos, M. Y. (2022). Data mesh: Concepts and principles of a paradigm shift in data architectures. Procedia Computer Science, 196, 263–271.
24. Machado, I. A., Costa, C., & Santos, M. Y. (2022). Advancing data architectures with data mesh implementations. In Intelligent Information Systems: CAiSE Forum 2022 (pp. 10–18). Springer.
25. Pamisetty, V. (2023). Leveraging artificial intelligence for strategic decision-making in tax administration and policy design. Available at SSRN.
26. Hlupić, T., Oreščanin, D., Ružak, D., & Baranović, M. (2022). An overview of current data lake architecture models. Proceedings of the 45th Jubilee International Convention on Information, Communication and Electronic Technology, 1082–1087.
27. Wieder, P., & Nolte, H. (2022). Toward data lakes as central building blocks for data management and analysis. Frontiers in Big Data, 5, Article 945720.
28. Cherradi, M., & El Haddadi, A. (2022). Data lakes: A survey paper. Lecture Notes in Networks and Systems, 823–835.
29. Bauer, D., Giblin, C., Garcés-Erice, L., Pardon, N., Rooney, S., Toniato, E., & Urbanetz, P. (2022). Revisiting data lakes: The metadata lake. Proceedings of the 23rd ACM/IFIP International Middleware Conference.
30. Pandugula, C., & Nampalli, R. C. R. Optimizing Retail Performance: Cloud-Enabled Big Data Strategies for Enhanced Consumer Insights.
31. Zichichi, M., Rodríguez-Doncel, V., D’Angelo, G., & Ferretti, S. (2022). Data governance through a multi-DLT architecture in view of the GDPR. Cluster Computing, 25, 4515–4542.
32. Nadal, S., Jovanovic, P., Bilalli, B., & Romero, O. (2022). Operationalizing and automating data governance. Journal of Big Data, 9, Article 117.
33. Tiwari, S., Sharma, S., Shetty, S., & Packer, F. (2022). The design of a data governance system. BIS Papers, 124.
34. Paleti, S. (2023). Transforming Money Transfers and Financial Inclusion: The Impact of AI-Powered Risk Mitigation and Deep Learning-Based Fraud Prevention in Cross-Border Transactions. Available at SSRN, 5158588.
35. Ataei, P., & Litchfield, A. (2022). The state of big data reference architectures: A systematic literature review. IEEE Access, 10, 113789–113807.
36. Batchu, K. C. (2022). Modern data warehousing in the cloud: Evaluating performance and cost trade-offs in hybrid architectures. International Journal of Advanced Research in Computer Science and Technology, 5(6), 7343–7349.
37. Im, C.-H. (2022). A study on how to build a data catalog to improve data distribution and usability of public and private data platforms. The Journal of Information Technology and Architecture, 19(3), 217–228.
38. Dončević, J., Fertalj, K., Brčić, M., & Kovač, M. (2022). Mask-mediator-wrapper architecture as a Data Mesh driver. arXiv.
39. Zagan, E., & Danubianu, M. (2023). Data lake architecture for storing and transforming web server access log files. IEEE Access, 11, 40916–40929.
40. Hai, R., Koutras, C., Quix, C., & Jarke, M. (2023). Data lakes: A survey of functions and systems. IEEE Transactions on Knowledge and Data Engineering, 35(12), 12571–12590.
41. Gunklach, J., Michalczyk, S., Nadj, M., & Maedche, A. (2023). Metadata extraction from user queries for self-service data lake exploration. Datenbank-Spektrum, 23, 97–105.
42. Recharla, M., & Chitta, S. AI-Enhanced Neuroimaging and Deep Learning-Based Early Diagnosis of Multiple Sclerosis and Alzheimer’s.
43. Stach, C., Eichler, R., & Schmidt, S. (2023). A recommender approach to enable effective and efficient self-service analytics in data lakes. Datenbank-Spektrum, 23, 123–132.
44. Arundel, S. T., McKeehan, K. G., Campbell, B. B., & Bulen, A. N. (2023). A guide to creating an effective big data management framework. Journal of Big Data, 10, Article 146.
45. Gentner, T., Neitzel, T., Schulze, J., Gerschner, F., & Theissler, A. (2023). Data lakes in healthcare: Applications and benefits from the perspective of data sources and players. Procedia Computer Science, 225, 1302–1311.
46. Dittmar, C., & Schulz, P. (2023). Architekturen und Technologien für den Data Lake. In A. Gillhuber, G. Kauermann, & W. Hauner (Eds.), Künstliche Intelligenz und Data Science in Theorie und Praxis (pp. 157–166). Springer.
Additional Files
Published
Issue
Section
License
Copyright (c) 2024 European Advanced Journal for Science & Engineering (EAJSE)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Articles published in the European Advanced Journal for Science & Engineering (EAJSE) are made freely available online immediately upon publication under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited. Authors retain copyright of their work. By submitting to EAJSE, authors grant the journal the right of first publication. For details, visit: https://creativecommons.org/licenses/by/4.0/
