Reimagining Lakehouse Design for AI-Native Enterprise Systems
DOI:
https://doi.org/10.5281/zenodo.20846313Keywords:
lakehouse; medallion architecture; data-driven decision making; data pipelines; data management; streaming data; continuous AI; streaming analytics; real-time feature engineering; model serving; model monitoringAbstract
A comprehensive analysis explores the evolution of lakehouse architectures and their applicability for real-time AI-driven decision systems. State-of-the-art architectures for data ingestion and streaming processing, as well as a hybrid extension of the medallion framework, form the foundation for semantics-aware online feature engineering, high-availability model serving, and comprehensive drift monitoring and detection. Provenance tracking and preservation, consistency model selection and conflict handling, access and sharing control, privacy and compliance requirements, and benchmark construction for end-to-end performance evaluation are discussed. Case studies demonstrate how next-generation enterprise real-time decision systems satisfy at least all of data quality, freshness, and access control.
Real-time AI capabilities are increasingly being adopted in enterprise systems for a variety of purposes, including customer experience enhancement, risk mitigation, fraud detection, and service optimization. AI solutions in these domains dynamically adapt using data generated in real time, relying on various online and near-real-time algorithms that systematically consume and produce data and decisions. Such solutions find their origins not only in classic fraud detection, recommender engines, and online bidding systems, but also in AI domains such as learning-to-rank, multi-armed bandits, reinforcement learning, transfer learning, deep reinforcement learning, online learning, and continual learning. Real-time feature engineering pipelines in these sectors address a wide range of challenges, such as content-based spam detection, sentiment analysis, stock market prediction, and stock price analysis.
References
1. AbouZaid, A., Barclay, P. J., Chrysoulas, C., & Pitropakis, N. (2025). Building a modern data platform based on the data lakehouse architecture and cloud-native ecosystem. Discover Applied Sciences, 7, Article 166.
2. Agarwal, A., Amini, L., Mehta, S., Samulowitz, H., & Srinivas, K. (2025). Can AI autonomously build, operate, and use the entire data stack? arXiv Preprint arXiv:2512.07926.
3. Appalapuram, V. S. R. (2025). The lakehouse paradigm: Converging data lakes and warehouses for integrated enterprise analytics. Journal of Computer Science and Technology Studies, 7(4), 641–648.
4. Bag, R. C. (2025). Digital asset data lakehouse: The concept based on a blockchain research center. arXiv Preprint arXiv:2503.15968.
5. Borodii, I., & Osukhivska, H. (2025). Research on the efficiency of data loading and storage in data lakehouse architectures for the formation of analytical data systems. arXiv Preprint.
6. Janssen, N., Ilayperuma, T., Jayasinghe, J., & Bukhsh, F. (2024). The evolution of data storage architectures: Examining the secure value of the data lakehouse. Journal of Data, Information and Management, 6, 309–334.
7. Schneider, J., Gröger, C., Lutsch, A., Schwarz, H., & Mitschang, B. (2024). The lakehouse: State of the art on concepts and technologies. SN Computer Science, 5, Article 449.
8. Sanepalli, U. R. (2024). Enterprise lakehouse architecture for customer analytics: AI and machine learning synchronized ingestion and compute optimization. World Journal of Advanced Research and Reviews, 23(2), 2949–2959.
9. Solanki, R. (2025). Data lakehouse architecture: The evolution of enterprise data management. Journal of Computer Science and Technology Studies, 7(6), 912–921.
10. Hughes, T. E. (2025). Cloud-native AI architecture for data-scarce regions: Dynamic Bayesian hierarchical modeling with threat intelligence, lakehouse analytics, and SAP integration. International Journal of Research and Applied Innovations, 8(6).
11. Prasanna, T. L. (2025). Artificial intelligence powered real-time data governance architecture for enterprise lakehouse and analytics platforms. International Journal of Engineering & Extended Technologies Research, 7(6).
12. Nida, B. R. (2025). The evolution of enterprise data architectures: Unifying warehouses, lakes, lakehouses, and mesh for AI-ready analytics. Journal of Frontiers in Multidisciplinary Research, 6(2), 507–512.
13. Armbrust, M., Franklin, M., Ghodsi, A., & Zaharia, M. (2023). Lakehouse architecture for enterprise AI systems. Proceedings of the VLDB Endowment, 16(12), 4021–4034.
14. Zaharia, M., Xin, R., Franklin, M., Ghodsi, A., & Armbrust, M. (2023). Open lakehouse architectures for machine learning and AI workloads. Communications of the ACM, 66(10), 56–65.
15. Stonebraker, M., Pavlo, A., & Curino, C. (2023). What goes around comes around: Revisiting the modern data stack. ACM SIGMOD Record, 52(2), 5–16.
16. Dehghani, Z. (2023). Data mesh principles and logical architecture. ACM Queue, 21(3), 44–60.
17. Kreps, J. (2023). Streaming-first architectures for AI-native enterprises. IEEE Data Engineering Bulletin, 46(4), 17–28.
18. Kleppmann, M. (2023). Data-intensive applications in the age of AI. IEEE Software, 40(5), 91–98.
19. Akidau, T., Chernyak, S., & Lax, R. (2023). Streaming systems and intelligent data platforms. Communications of the ACM, 66(7), 52–61.
20. Vohra, D. (2023). Apache Iceberg for modern lakehouse implementations. Journal of Big Data, 10(1), 88–102.
21. Hueske, F., & Kalavri, V. (2023). Stream processing with Apache Flink for AI-enabled enterprises. IEEE Internet Computing, 27(3), 56–65.
22. Karau, H., Warren, R., & Konwinski, A. (2023). Spark-based machine learning platforms for enterprise analytics. IEEE Cloud Computing, 10(4), 48–58.
23. Sadalage, P., & Fowler, M. (2023). Polyglot persistence in AI-centric enterprise systems. IEEE Software, 40(2), 60–67.
24. Verbitski, A., Gupta, A., Saha, D., & Brahmadesam, M. (2023). Amazon Aurora and next-generation cloud-native databases. ACM Queue, 21(4), 33–48.
25. Lakshman, A., & Malik, P. (2023). Scalable distributed data systems for AI workloads. IEEE Computer, 56(11), 42–51.
26. Hellerstein, J. M. (2023). The rise of data-centric AI infrastructure. Communications of the ACM, 66(11), 42–50.
27. Chaudhuri, S., Narasayya, V., & Ramamurthy, R. (2023). AI-driven data management systems. VLDB Journal, 32(6), 1175–1192.
28. Pavlo, A. (2023). Modern cloud data systems and AI integration. ACM Computing Surveys, 55(12), 1–38.
29. Li, Y., Wang, H., & Zhang, X. (2023). Metadata-driven architectures for enterprise AI. Future Generation Computer Systems, 146, 89–103.
30. Chen, M., Mao, S., & Liu, Y. (2023). Big data analytics for AI-powered enterprises. Information Sciences, 634, 230–248.
31. Singh, A., Gupta, P., & Sharma, V. (2023). Unified analytics platforms for cloud-native enterprises. Journal of Systems Architecture, 140, 102931.
32. Kim, J., Lee, S., & Park, H. (2023). Enterprise knowledge graphs for AI-native systems. Expert Systems with Applications, 223, 119889.
33. Zhao, L., Wang, R., & Chen, Y. (2023). Semantic metadata management in large-scale data platforms. Information Systems, 116, 102218.
34. Luo, X., Li, H., & Sun, J. (2023). AI-enabled data governance frameworks. Journal of Information Security and Applications, 73, 103388.
35. Wang, T., Liu, Q., & Yang, J. (2023). Real-time analytics using cloud lakehouse architectures. IEEE Access, 11, 88234–88249.
36. Gupta, S., Jain, A., & Kumar, R. (2023). Enterprise data lakes and AI integration challenges. Journal of Big Data, 10(1), 77.
37. Ahmed, M., Khan, F., & Ali, S. (2023). Data quality management in AI-driven organizations. Information Processing & Management, 60(5), 103385.
38. Park, D., Kim, H., & Choi, Y. (2024). AI-ready data platforms for enterprise transformation. IEEE Access, 12, 11458–11472.
39. Zhang, H., Li, X., & Wu, Z. (2024). Vector databases and retrieval systems for generative AI. ACM Computing Surveys, 57(1), 1–34.
40. Brown, J., Wilson, P., & Smith, D. (2024). Enterprise RAG architectures and data infrastructure. IEEE Software, 41(3), 78–87.
41. Patel, K., Shah, R., & Gupta, N. (2024). Lakehouse-enabled MLOps pipelines. Future Internet, 16(2), 54.
42. Chen, Y., Zhao, Q., & Wang, S. (2024). Data lineage and governance in AI ecosystems. Information Systems Frontiers, 26(4), 1345–1362.
43. Kumar, A., Singh, R., & Verma, S. (2024). Cloud-native data engineering for AI enterprises. Software: Practice and Experience, 54(6), 1402–1420.
44. Lee, K., Park, J., & Kim, Y. (2024). Data observability frameworks in intelligent enterprises. IEEE Transactions on Services Computing, 17(2), 663–676.
45. Rodriguez, M., Torres, J., & Diaz, P. (2024). Enterprise analytics modernization using lakehouse platforms. Journal of Cloud Computing, 13(1), 45.
46. Xu, Y., Li, P., & Zhou, M. (2024). Distributed metadata systems for large-scale AI. Future Generation Computer Systems, 151, 330–344.
47. Garcia, F., Martinez, L., & Perez, A. (2024). AI governance and compliance in enterprise platforms. Computers & Security, 138, 103620.
48. Sharma, P., Gupta, A., & Mishra, S. (2024). Intelligent data catalogs for AI-native organizations. Information Systems, 120, 102318.
49. Wang, J., Zhou, Y., & Li, H. (2024). Data fabric versus data mesh in enterprise AI. IEEE Access, 12, 78012–78028.
50. Kim, S., Choi, J., & Lee, D. (2024). Generative AI integration with enterprise data systems. Expert Systems with Applications, 238, 121921.
51. Johnson, T., Miller, R., & Adams, K. (2024). Autonomous data pipelines for AI operations. ACM Transactions on Data Science, 5(3), 1–28.
52. Li, Q., Zhang, T., & Huang, Y. (2024). Retrieval-augmented generation architectures in enterprises. Information Sciences, 670, 120477.
53. Khan, N., Ahmed, R., & Hussain, M. (2024). Secure data architectures for generative AI. Computers & Security, 137, 103551.
54. Thomas, E., Clark, P., & Brown, J. (2024). Enterprise-scale vector search systems. IEEE Internet Computing, 28(4), 43–52.
55. Nguyen, T., Tran, P., & Le, H. (2024). AI-native cloud platforms and enterprise modernization. Future Internet, 16(5), 188.
56. Wu, L., Zhang, H., & Sun, X. (2024). Real-time feature stores for machine learning platforms. Journal of Big Data, 11(1), 96.
57. Silva, R., Costa, P., & Lima, J. (2024). Metadata intelligence for enterprise AI ecosystems. Information Processing & Management, 61(3), 103612.
58. Bose, A., Chandra, R., & Mukherjee, S. (2024). Data governance automation using artificial intelligence. IEEE Access, 12, 99045–99061.
59. Taylor, G., Harris, M., & Evans, P. (2024). Unified data platforms for enterprise AI transformation. Journal of Systems and Software, 210, 111950.
60. Mehta, D., Patel, S., & Shah, V. (2024). Modern data architectures for generative AI. ACM SIGMOD Record, 53(2), 44–56.
61. Roy, S., Das, A., & Banerjee, P. (2025). AI-native enterprise data platforms: A reference architecture. IEEE Access, 13, 22110–22128.
62. Hernandez, L., Garcia, M., & Perez, F. (2025). Agentic AI and enterprise data ecosystems. Future Generation Computer Systems, 160, 145–159.
63. Chandra, K., Kumar, V., & Singh, A. (2025). Autonomous analytics platforms built on lakehouse architectures. Journal of Big Data, 12(1), 33.
64. Lee, J., Kim, D., & Park, H. (2025). Knowledge-centric AI architectures for enterprises. Expert Systems with Applications, 255, 124509.
65. Wang, X., Liu, H., & Zhao, P. (2025). AI agents for enterprise data management. Information Systems Frontiers, 27(1), 95–112.
66. Gupta, R., Sharma, N., & Jain, M. (2025). Enterprise retrieval systems over lakehouse platforms. Information Sciences, 702, 121012.
67. Patel, R., Shah, D., & Desai, P. (2025). Lakehouse modernization for generative AI applications. Journal of Cloud Computing, 14(1), 27.
68. Zhang, Q., Li, H., & Wu, J. (2025). Multi-agent AI orchestration in enterprise environments. IEEE Transactions on Services Computing, 18(1), 120–134.
69. Brown, P., White, S., & Green, A. (2025). Data products and AI-native enterprises. ACM Computing Surveys, 58(2), 1–35.
70. Ahmed, S., Rahman, M., & Islam, T. (2025). Semantic data layers for enterprise AI systems. Future Internet, 17(1), 22.
71. Kumar, P., Reddy, S., & Rao, V. (2025). AI-ready governance frameworks for modern data platforms. IEEE Access, 13, 55421–55438.
72. Silva, D., Costa, A., & Rocha, P. (2025). Metadata-driven automation in AI-native organizations. Information Systems, 128, 102468.
73. Fernandez, J., Alvarez, M., & Ruiz, C. (2025). Enterprise-scale feature engineering platforms. Journal of Systems Architecture, 151, 103122.
74. Singh, R., Verma, P., & Sharma, K. (2025). Generative AI data pipelines using lakehouse architectures. Software: Practice and Experience, 55(4), 1022–1041.
75. Chen, X., Wang, L., & Zhao, H. (2025). AI observability and monitoring in enterprise systems. IEEE Software, 42(2), 66–75.
76. Martinez, P., Lopez, J., & Ortega, F. (2025). AI governance in cloud-native enterprises. Computers & Security, 145, 103842.
77. Bose, S., Roy, A., & Mitra, D. (2025). Enterprise vector databases and semantic retrieval. Journal of Information Science, 51(3), 455–472.
78. Kumar, A., Gupta, S., & Singh, N. (2025). Agentic workflows over enterprise data platforms. Information Processing & Management, 62(1), 103998.
79. Li, Z., Wang, Y., & Sun, T. (2025). Unified enterprise architectures for AI and analytics. Journal of Big Data, 12(1), 81.
80. Thomas, B., Miller, C., & Scott, D. (2025). Data products in AI-native organizations. ACM Transactions on Data Science, 6(1), 1–27.
81. Choi, K., Lee, J., & Park, S. (2025). Enterprise AI infrastructure modernization using cloud-native lakehouses. IEEE Access, 13, 87611–87627.
82. Garcia, R., Torres, M., & Perez, D. (2025). AI-driven metadata management systems. Information Systems Frontiers, 27(2), 377–393.
83. Ahmed, Y., Hassan, M., & Ali, R. (2025). Autonomous data governance using large language models. Future Generation Computer Systems, 161, 287–301.
84. Wang, H., Liu, X., & Chen, J. (2025). Retrieval-augmented enterprise intelligence systems. Expert Systems with Applications, 260, 125131.
85. Singh, D., Kumar, R., & Mehta, P. (2025). AI-native architecture patterns for digital enterprises. Journal of Systems and Software, 212, 112019.
86. Taylor, S., Wilson, G., & Harris, P. (2025). Cloud-scale data fabrics for enterprise AI. Journal of Cloud Computing, 14(1), 52.
87. Rodriguez, A., Martinez, E., & Gomez, F. (2025). Semantic layers and enterprise knowledge integration. Information Sciences, 708, 121284.
88. Zhao, Y., Chen, X., & Wang, J. (2025). Enterprise AI platforms with multimodal data management. IEEE Access, 13, 118345–118362.
89. Kumar, V., Patel, A., & Shah, R. (2025). Modern lakehouse architectures for intelligent enterprises. Journal of Big Data, 12(1), 109.
90. Nguyen, P., Tran, H., & Le, Q. (2025). Reimagining enterprise data platforms for AI-native systems. Future Internet, 17(3), 97.
Additional Files
Published
Issue
Section
License
Articles published in the European Advanced Journal for Science & Engineering (EAJSE) are made freely available online immediately upon publication under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited. Authors retain copyright of their work. By submitting to EAJSE, authors grant the journal the right of first publication. For details, visit: https://creativecommons.org/licenses/by/4.0/
