Scalable Medallion Lakehouse with MLOps on Azure Databricks
Keywords:
Medallion Architecture, Databricks, Azure Cloud, MLOps, Data Intelligence, Data Engineering, Data Pipelines, Data Governance, Data Architecture, Machine Learning, Model Deployment, CI CD, Model Lifecycle, Data Processing, Scalable Systems, Analytics Systems, Cloud Platforms, Data Integration, Automation, Enterprise Data.Abstract
This Medallion Architecture design and implementation leverages Databricks and Azure technology while aligning with MLOps principles to engender a comprehensive, scalable solution for enterprise data intelligence, a topic of increasing relevance across commercial sectors. As organizations expand their bases of data assets, the need for an integrated framework that supports both analytics and machine learning with a clear vision for data intelligence emerges as a priority.
Today, machine learning is applied across many industries and is advancing rapidly. However, the majority of ML models remain on pilot status or just a few have been productized. Furthermore, their use, maintenance, and governance sprinkle a wide array of manual and poorly controlled actions. The Achilles heel of these initiatives that are such a big promise of added value is the implementation of a continuous integration/continuous deployment (CI/CD) flow. ML-Ops is the answer to these impediments and allows exploring ways to build ML products properly.
References
1. Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. CIDR 2021 Proceedings.
2. Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., Murching, S., Nykodym, T., Ogilvie, P., Parkhe, M., Xie, F., & Zumar, C. (2021). Delta Lake: High-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment, 14(12), 3411–3424.
3. Kolla, S. K., & Mangalampalli, B. M. (2024). Edge-Based Deep Learning Systems for Point-of-Care Diagnostic Intelligence. Journal of Neonatal Surgery, 13(1), 2387-2399.
4. Chambers, B., & Zaharia, M. (2021). Spark: The definitive guide (2nd ed.). O'Reilly Media.
5. Karau, H., & Warren, R. (2021). High performance Spark (2nd ed.). O'Reilly Media.
6. Reis, J., & Housley, M. (2022). Fundamentals of data engineering. O'Reilly Media.
7. Kreuzberger, D., Kühl, N., & Hirschl, S. (2022). Machine learning operations (MLOps): Overview, definition, and architecture. IEEE Access, 10, 31866–31879.
8. Kolla, S. K. (2024). Clinical Knowledge Intelligence through Deep Learning and Natural Language Understanding in Healthcare Platforms. International Journal of Advanced Engineering Science and Information Technology (IJAESIT), 7(3), 14088.
9. Huyen, C. (2022). Designing machine learning systems. O'Reilly Media.
10. Guller, M. (2022). Big data analytics with Spark. Apress.
11. Damji, J., Wenig, B., Das, T., Lee, D., & Xin, R. (2022). Learning Spark (2nd ed.). O'Reilly Media.
12. Nagubandi, A. R. (2023). Advanced Multi-Agent AI Systems for Autonomous Reconciliation Across Enterprise Multi-Counterparty Derivatives, Collateral, and Accounting Platforms. International Journal of Finance (IJFIN)-ABDC Journal Quality List, 36(6), 653-674.
13. Kleppmann, M. (2022). Designing data-intensive applications (Updated ed.). O'Reilly Media.
14. Burns, B., Beda, J., & Hightower, K. (2022). Kubernetes: Up and running (3rd ed.). O'Reilly Media.
15. Sato, D., & Yamamoto, K. (2022). Cloud-native data engineering using Apache Spark. Journal of Cloud Computing, 11(1), 54–69.
16. Dong, X., Li, J., & Wang, Y. (2022). Enterprise data lakehouse architecture for cloud analytics. Future Generation Computer Systems, 132, 98–111.
17. Kolla, S. H. (2024). Retrieval-Augmented Enterprise Intelligence: Enhancing Accuracy, Trust, and Operational Decision-Making. International Journal of Future Innovative Science and Technology (IJFIST), 7(2), 12425.
18. Chen, L., Xu, P., & Zhao, H. (2022). Scalable cloud-based machine learning pipelines. IEEE Transactions on Cloud Computing, 10(4), 2217–2230.
19. Sharma, P., & Patel, A. (2022). DataOps and MLOps integration for enterprise AI systems. Journal of Systems and Software, 190, 111336.
20. Abadi, D. J. (2022). Consistency trade-offs in cloud data platforms. Communications of the ACM, 65(8), 58–66.
21. Armbrust, M., Franklin, M., Ghodsi, A., et al. (2022). Data lakehouse architecture for modern analytics. Communications of the ACM, 65(9), 40–49.
22. Reddy Segireddy, A. (2024). Federated Cloud Approaches for Multi-Regional Payment Messaging Systems. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 15(2), 442-450.
23. Xin, R., Das, T., & Zaharia, M. (2022). Efficient query optimization for lakehouse systems. Proceedings of the VLDB Endowment, 15(8), 1852–1865.
24. Kreps, J. (2022). Kafka: The definitive guide (2nd ed.). O'Reilly Media.
25. Carbone, P., Katsifodimos, A., & Ewen, S. (2022). Streaming analytics with Apache Spark. IEEE Data Engineering Bulletin, 45(2), 40–55.
26. Kolla, T. (2024). Intelligent Discovery and Governance of Healthcare Data Assets Through AI-Powered Catalog Architectures. International Journal of Emerging Trends in Engineering and Management Research, 9(4), 16083.
27. Hueske, F., & Kalavri, V. (2022). Stream processing with Apache Flink. O'Reilly Media.
28. Sculley, D., Holt, G., Golovin, D., et al. (2021). Hidden technical debt in machine learning systems. Communications of the ACM, 64(7), 36–43.
29. Inala, R. Advancing Group Insurance Solutions Through Ai-Enhanced Technology Architectures And Big Data Insights.
30. Polyzotis, N., Roy, S., Whang, S., & Zinkevich, M. (2021). Data validation for machine learning. Proceedings of Machine Learning Systems, 3, 334–347.
31. Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2021). The ML test score: A rubric for production ML. Proceedings of Machine Learning Systems, 3, 112–125.
32. Kolla, S. H., & Peddi, R. K. (2024). Designing Governance-Aligned GenAI Pipelines Using Small Language Models for Enterprise Workflow Intelligence. International Journal of Science, Research and Technology, 7(6), 13256-13268.
33. Baylor, D., Breck, E., Cheng, H., et al. (2021). TFX: A TensorFlow-based production-scale machine learning platform. Proceedings of the ACM SIGKDD Conference, 1387–1395.
34. Chen, T., & Guestrin, C. (2021). XGBoost: A scalable tree boosting system. Proceedings of the ACM SIGKDD Conference, 785–794.
35. Akidau, T., Bradshaw, R., Chambers, C., et al. (2021). Streaming systems. O'Reilly Media.
36. Mattaparthi, R. (2024). Transformer-Based Fault Diagnosis for Large-Scale Standby Power Generators: Partial Discharge Pattern Recognition at Hyperscale Data Center Installations. Journal of Computational Analysis and Applications (JoCAAA), 33(08), 8781-8799.
37. Nykodym, T., Potamias, M., Mishra, S., et al. (2021). Delta Lake: Building reliable data lakes. Databricks Technical White Paper.
38. Sadalage, P., & Fowler, M. (2021). NoSQL distilled (Updated ed.). Addison-Wesley.
39. White, T. (2021). Hadoop: The definitive guide (5th ed.). O'Reilly Media.
40. Vohra, D. (2023). Apache Spark 3 for data engineering. Apress.
41. Loganathan, R. (2024). GENERATIVE AI-ENABLED COMPLIANCE DOCUMENTATION AND AUDIT TRAIL AUTOMATION FOR GLOBAL DATA CENTER GOVERNANCE. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 15(3), 487-504.
42. Reis, J. (2023). Modern data platform architecture. IEEE Software, 40(3), 42–49.
43. Kreuzberger, D., Hirschl, S., & Kühl, N. (2023). MLOps challenges in enterprise AI. Machine Learning with Applications, 13, 100472.
44. Chen, Y., Wang, J., & Liu, H. (2023). Cloud-native data lakehouse architectures. Journal of Big Data, 10(1), 86.
45. Singh, R., & Gupta, P. (2023). Scalable feature engineering pipelines for cloud machine learning. Future Internet, 15(4), 132.
46. Nagabhyru, K. C. (2022). Bridging Traditional ETL Pipelines with AI Enhanced Data Workflows: Foundations of Intelligent Automation in Data Engineering. Available at SSRN 5505199.
47. Kumar, A., Sharma, S., & Patel, R. (2023). Enterprise MLOps for cloud-native AI systems. IEEE Access, 11, 45233–45251.
48. Verma, S., & Sinha, A. (2023). Data quality management in cloud analytics platforms. Journal of Information Systems Engineering, 38(2), 145–160.
49. Li, W., Zhang, H., & Zhao, L. (2023). Metadata-driven data governance for lakehouse environments. Future Generation Computer Systems, 144, 217–231.
50. Yandamuri, U. S. (2024). AI-Driven Decision Support Systems for Operational Optimization in Hospitality Technology. Metallurgical and Materials Engineering.
51. Gao, J., Liu, M., & Wang, X. (2023). Automated machine learning lifecycle management using MLflow. Software: Practice and Experience, 53(8), 1654–1671.
52. Zhao, H., Chen, L., & Sun, P. (2023). Cloud-native orchestration for scalable AI workloads. Journal of Cloud Computing, 12(1), 104.
53. Wang, J., Li, H., & Xu, Q. (2023). Delta Lake optimization techniques for large-scale analytics. Information Systems, 118, 102278.
54. Reddy, V. A. R. (2024). Generative Intelligence for Healthcare Claims Processing and Personalized Benefits Management. International Journal of Advanced Engineering Science and Information Technology (IJAESIT), 7(3), 14099.
55. Ahmad, M., Khan, S., & Ali, R. (2023). DataOps-driven cloud analytics platforms. Computing, 105(10), 2447–2465.
56. Luo, X., Zhang, Y., & Chen, J. (2023). Intelligent workflow automation for enterprise data engineering. IEEE Access, 11, 103121–103136.
57. Martin, R., & Brown, K. (2023). Scalable Spark optimization for enterprise data lakes. Concurrency and Computation: Practice and Experience, 35(21), e7806.
58. Garapati, R. S. (2022). Web-Centric Cloud Framework for Real-Time Monitoring and Risk Prediction in Clinical Trials Using Machine Learning. Current Research in Public Health, 2, 1346.
59. Fernandez, P., & Kumar, S. (2023). Cloud governance for enterprise analytics platforms. Journal of Cloud Computing, 12(1), 118.
60. Brahmandam, L. M. K. (2024). An empirical evaluation of the medallion architecture on Databricks and Apache Spark with Snowflake: Throughput, latency, and cost for batch and real-time ingestion patterns. International Journal of AI BigData Computational and Management Studies, 5(3), 197–206.
61. Gottimukkala, V. R. R. (2021). Digital Signal Processing Challenges in Financial Messaging Systems: Case Studies in High-Volume SWIFT Flows.
62. Panchal, D., Verma, P., Baran, I., Musgrove, D., & Lu, D. (2024). Reusable MLOps: Reusable deployment, reusable infrastructure and hot-swappable machine learning models and services. arXiv.
63. Yan, B. (2024). MLOps in a multicloud environment: Typical network topology. arXiv.
64. Sharma, N., & Gupta, R. (2024). Cloud-native MLOps pipelines for enterprise AI. IEEE Access, 12, 41235–41252.
65. Bandi, V. D. V. K. (2024). AI-Driven Predictive Risk Modeling Architectures for Financial Systems. International Journal Of Finance, 37(3), 54-78.
66. Wang, T., Li, S., & Chen, Y. (2024). Unified data governance for lakehouse architectures. Journal of Big Data, 11(1), 54.
67. Patel, A., Kumar, V., & Singh, R. (2024). Scalable feature stores for machine learning operations. Future Generation Computer Systems, 152, 18–31.
68. Mangala, N. (2024). Leveraging Microsoft Fabric lakehouse as an AI-ready data platform for enterprise analytics. Journal of Information Systems Engineering and Management.
69. Zhang, X., Zhao, J., & Liu, H. (2024). AI-driven data engineering on cloud platforms. Information Systems Frontiers, 26(2), 487–503.
70. Roy, S., Banerjee, D., & Chatterjee, A. (2024). Enterprise-scale ML lifecycle management using MLflow. Software: Practice and Experience, 54(4), 802–821.
71. Lee, J., Kim, H., & Park, S. (2024). Cloud-native analytics with Apache Spark and Delta Lake. Journal of Cloud Computing, 13(1), 27.
72. Pamisetty, V., & Amistapuram, K. Smart Decision Support Systems For Dynamic Tax Policy Optimization Using Reinforcement Learning.
73. Singh, A., Sharma, V., & Patel, N. (2024). Scalable streaming data engineering using Delta Lake. Future Internet, 16(2), 65.
74. Kumar, R., & Das, P. (2024). Modern enterprise lakehouse design patterns. IEEE Software, 41(2), 76–84.
75. Chen, Z., Wu, L., & Li, Y. (2024). Data observability and governance in AI-enabled cloud platforms. Journal of Systems and Software, 209, 111920.
76. Aitha, A. R. (2021). Dev Ops Driven Digital Transformation: Accelerating Innovation In The Insurance Industry. Available at SSRN 5622190.
77. Brown, T., Wilson, P., & Green, J. (2024). End-to-end MLOps automation for cloud-native AI systems. Machine Learning with Applications, 16, 100611.
78. Garcia, M., Lopez, D., & Perez, R. (2024). Data engineering best practices for enterprise lakehouses. Journal of Big Data, 11(1), 73.
79. Davuluri, P. N. (2020). Event-Driven Architectures for Real-Time Regulatory Monitoring in Global Banking.
80. Ahmed, S., Khan, M., & Hussain, F. (2024). Scalable cloud analytics and AI lifecycle management using Databricks Lakehouse. IEEE Access, 12, 87215–87234.
Additional Files
Published
Data Availability Statement
None
Issue
Section
License
Copyright (c) 2024 Sophia Martinez (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
