Lakehouse Streaming for Intelligent Decision Systems
Keywords:
Real-time data pipelines, Azure Databricks, PySpark, streaming, online feature store, micro-batching, anomaly detection, decision systems.Abstract
Real-time data engineering pipelines play a pivotal role in acquiring, processing, and analyzing streaming data to support intelligent decision-making systems. Taking a practical approach, this work examines real-time data pipelines within the Azure Databricks platform using PySpark. It presents a set of foundational concepts alongside a flexible implementation framework to guide the design of real-time data technology workloads. Empirical breadth and repeatability are demonstrated through educational data sources and case studies that showcase working applications of real-time technologies for monitoring and controlling business processes — including anomaly detection in streaming financial indicators and dynamic pricing strategies.
Intelligent decision systems draw on theories and technologies from a range of disciplines, including Artificial Intelligence (AI), Machine Learning (ML), Data Mining, Decision Support Systems (DSS), and Operations Research (OR). Within the Digital Business and Technology domain, real-time data engineering pipelines address the acquisition, processing, and analysis of streaming data to enable data-driven decisions grounded in a company's current business environment. Notably, most intelligent decision systems emphasize the data-driven component of decision-making more than the decision engine itself. Building on this, the following discussion centers on a Digital Business and Technology use case for designing an intelligent decision system — integrating both data engineering and data science aspects of data-driven decision-making — and implementing a series of real-time data technology workloads in Azure Databricks using PySpark.
References
1. Armbrust, M., Das, T., Sun, L., Yavuz, B., Zhu, S., Murthy, M., Torres, J., van Hovell, H., Ionescu, A., Łuszczak, A., Świtakowski, M., Szafrański, M., Li, X., Ueshin, T., Mokhtar, M., Boncz, P., Ghodsi, A., Paranjpye, S., Senster, P., Xin, R., & Zaharia, M. (2020). Delta Lake: High-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment, 13(12), 3411–3424.
2. Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. In Conference on Innovative Data Systems Research (CIDR 2021).
3. Yandamuri, U. S., Loganathan, R., Davuluri, P. S. L. N., Rani, P. R. S., Kolla, S. H., & Nagubandi, A. R. (2026). Adaptive Intelligence Networks for Humancentered Enterprise Automation and Governance. In 2026 Third International Conference on Innovations in Cybersecurity and Data Science (ICICDS) (pp. 678–683). IEEE. 2026 Third International Conference on Innovations in Cybersecurity and Data Science (ICICDS). https://doi.org/10.1109/icicds70526.2026.11604640
4. Akidau, T., Begoli, E., Chernyak, S., Hueske, F., Knight, K. E., Knowles, K., Mills, D., & Sotolongo, D. (2021). Watermarks in stream processing systems: Semantics and comparative analysis of Apache Flink and Google Cloud Dataflow. Proceedings of the VLDB Endowment, 14(12), 3135–3147.
5. Sheng, S., Huang, Q., Wang, S., & Bao, Y. (2021). PR-Sketch: Monitoring per-key aggregation of streaming data with nearly full accuracy. Proceedings of the VLDB Endowment, 14(10), 1783–1796.
6. Pandis, I. (2021). The evolution of Amazon Redshift. Proceedings of the VLDB Endowment, 14(12), 3162–3174.
7. Banoth, S., Santoshi Kumari, M., Lalitha, P., Deepthi, P., Kumar Peddi, R., & Kavitha, P. (2026). An Efficient Hybrid K-Means and Random Forest-Based Approach for Cloud Malware Detection and Privacy Protection. In 2026 6th International Conference on Intelligent Technologies (CONIT) (pp. 1–6). IEEE. 2026 6th International Conference on Intelligent Technologies (CONIT). https://doi.org/10.1109/conit69683.2026.11621822
8. Hai, R., Koutras, C., Quix, C., & Jarke, M. (2023). Data lakes: A survey of functions and systems. IEEE Transactions on Knowledge and Data Engineering, 35(12), 12571–12590.
9. Khatiwada, A., Shraga, R., Gatterbauer, W., & Miller, R. J. (2022). Integrating data lake tables. Proceedings of the VLDB Endowment, 16(4), 932–945.
10. Behm, A., Palkar, S., Agarwal, U., Armstrong, T., Cashman, D., Dave, A., Greenstein, T., Hovsepian, S., Johnson, R., Krishnan, A. S., Leventis, P., Luszczak, A., Menon, P., Mokhtar, M., Pang, G., Paranjpye, S., Rahn, G., Samwel, B., van Bussel, T., van Hovell, H., Xue, M., Xin, R., & Zaharia, M. (2022). Photon: A fast query engine for lakehouse systems. In Proceedings of the 2022 International Conference on Management of Data (pp. 2045–2057). Association for Computing Machinery.
11. Armenatzoglou, N., Basu, S., Bhanoori, N., Cai, M., Chainani, N., Chinta, K., Govindaraju, V., Green, T. J., Gupta, M., Hillig, S., Hotinger, E., Leshinksy, Y., Liang, J., McCreedy, M., Nagel, F., Pandis, I., Parchas, P., Pathak, R., Polychroniou, O., Rahman, F., Saxena, G., Soundararajan, G., Subramanian, S., & Terry, D. (2022). Amazon Redshift re-invented. In Proceedings of the 2022 International Conference on Management of Data (pp. 2205–2217). Association for Computing Machinery.
12. Padma, G., Reddy, V. A. R., KA, S. D., Buvaneswari, B., & Kumar, K. S. (2026, April). Privacy-Preserved Face Recognition Biometric Authentication Using FaceNet and Zero-Knowledge Proofs for Secure, Access Control on Decentralized Blockchain Networks. In 2026 2nd International Conference on Intelligent Systems and Computational Networks (ICISCN) (pp. 1-8). IEEE.
13. Park, K., Saur, K., Banda, D., Sen, R., Interlandi, M., & Karanasos, K. (2022). End-to-end optimization of machine learning prediction queries. In Proceedings of the 2022 International Conference on Management of Data (pp. 587–601). Association for Computing Machinery.
14. Manousis, A., Cheng, Z., Ben Basat, R., Liu, Z., & Sekar, V. (2022). Enabling efficient and general subpopulation analytics in multidimensional data streams. Proceedings of the VLDB Endowment, 15(11), 3249–3262.
15. Sabek, I., Vaidya, K., Horn, D., Kipf, A., Mitzenmacher, M., & Kraska, T. (2023). Can learned models replace hash functions? Proceedings of the VLDB Endowment, 16(3), 532–545.
16. Sabek, I., & Kraska, T. (2023). The case for learned in-memory joins. Proceedings of the VLDB Endowment, 16(7), 1749–1762.
17. Mao, Y., Zhao, J., Zhang, S., Liu, H., & Markl, V. (2023). MorphStream: Adaptive scheduling for scalable transactional stream processing on multicores. Proceedings of the ACM on Management of Data, 1(1), Article 59.
18. Saxena, G., Rahman, M., Chainani, N., Lin, C., Caragea, G., Chowdhury, F., Marcus, R., Kraska, T., Pandis, I., & Narayanaswamy, B. (2023). Auto-WLM: Machine learning enhanced workload management in Amazon Redshift. In Companion of the 2023 International Conference on Management of Data. Association for Computing Machinery.
19. Chen, Z., Gu, Z., Cao, L., Fan, J., Madden, S., & Tang, N. (2023). Symphony: Towards natural language query answering over multi-modal data lakes. In Conference on Innovative Data Systems Research (CIDR 2023).
20. Kulkarni, V., Jyothsna, P., Musku, K., Mattaparthi, R., & Nagaraj, S. S. (2026, July). A Multi-Objective Metaheuristic Framework for Demand-Side Management in Isolated Tropical Photovoltaic Microgrids: Performance Bounds and Battery Trade-off Analysis. In 2026 14th International Conference on Smart Grid (icSmartGrid) (pp. 331-337). IEEE.
21. Sun, Y., Meehan, T., Schlussel, R., Xie, W., Basmanova, M., Erling, O., Rosa, A., Fan, S., Zhong, R., Thirupathi, A., Collooru, N., Wang, K., Agarwal, S., Gupta, A., Logothetis, D., Xirogiannopoulos, K., Dutta, A., Gajjala, V., Jain, R., … Chattopadhyay, B. (2023). Presto: A decade of SQL analytics at Meta. Proceedings of the ACM on Management of Data, 1(2), 1–25.
22. Doshi, L., Zhuang, V., Jain, G., Marcus, R. C., Huang, H., Altinbüken, D., Brevdo, E., & Fraser, C. (2023). Kepler: Robust learning for parametric query optimization. Proceedings of the ACM on Management of Data, 1(1).
23. Verwiebe, J., Grulich, P. M., Traub, J., & Markl, V. (2023). Survey of window types for aggregation in stream processing systems. The VLDB Journal, 32(5), 985–1011.
24. Zhang, S., Soto, J., & Markl, V. (2024). A survey on transactional stream processing. The VLDB Journal, 33(2), 451–479.
25. Fragkoulis, M., Carbone, P., Kalavri, V., & Katsifodimos, A. (2024). A survey on the evolution of stream processing systems. The VLDB Journal, 33(2), 507–541.
26. Chatziliadis, X., Tzirita Zacharatou, E., Eracar, A., Zeuch, S., & Markl, V. (2024). Efficient placement of decomposable aggregation functions for stream processing over large geo-distributed topologies. Proceedings of the VLDB Endowment, 17(6), 1501–1514.
27. Schneider, J., Gröger, C., Lutsch, A., Schwarz, H., & Mitschang, B. (2024). The lakehouse: State of the art on concepts and technologies. SN Computer Science, 5, Article 449.
28. Gorle, R., & Pareek, P. K. (2026). Agentic AI-Based Security Orchestration for Oracle Cloud and Multi-Cloud Platforms. International Journal of Computer Information Systems and Industrial Management Applications, 18(11s), 354-369.
29. Levandoski, J., Casto, G., Deng, M., Desai, R., Edara, P., Hottelier, T., Hormati, A., Johnson, A., Johnson, J., Kurzyniec, D., McVeety, S., Ramanathan, P., Saxena, G., Shanmugam, V., & Volobuev, Y. (2024). BigLake: BigQuery’s evolution toward a multi-cloud lakehouse. In Proceedings of the 2024 International Conference on Management of Data (pp. 334–346). Association for Computing Machinery.
30. Xue, M., Bu, Y., Somani, A., Fan, W., Liu, Z., Chen, S., van Hovell, H., Samwel, B., Mokhtar, M., Korlapati, R. K., Lam, A., Ma, Y., Ercegovac, V., Li, J., Behm, A., Li, Y., Li, X., Krishnamurthy, S., Shukla, A., … Zaharia, M. (2024). Adaptive and robust query execution for lakehouses at scale. Proceedings of the VLDB Endowment, 17(12), 3947–3959.
31. Okolnychyi, A., Sun, C., Tanimura, K., Spitzer, R., et al. (2024). Petabyte-scale row-level operations in data lakehouses. Proceedings of the VLDB Endowment, 17(12), 4159–4172.
32. Schulze, R., Schreiber, T., Yatsishin, I., Dahimene, R., & Milovidov, A. (2024). ClickHouse—Lightning fast analytics for everyone. Proceedings of the VLDB Endowment, 17(12), 3731–3744.
33. Mahadevan, S. (2026). Governed Serverless Automation Ecosystem for AI-Driven Enterprise Integration and Sustainable Cloud Operations. International Journal of Computer Information Systems and Industrial Management Applications, 18(16s), 1561-1570.
34. van Renen, A., Horn, D., Pfeil, P., Vaidya, K., Dong, W., Narayanaswamy, M., Liu, Z., Saxena, G., Kipf, A., & Kraska, T. (2024). Why TPC is not enough: An analysis of the Amazon Redshift fleet. Proceedings of the VLDB Endowment, 17(11), 3694–3706.
35. Harby, A. A., & Zulkernine, F. H. (2025). Data lakehouse: A survey and experimental study. Information Systems, 127, Article 102460.
36. Chandra, R., Chen, H., Matharu, R., Cai, S., Chen, J., Dutta, P., Ghita, B., Greenstein, T., Holla, G., Huang, P., Huo, Y., Ionescu, A., Ispas, A., Januschowski, T., Karajgaonkar, V., Leone, S., Lewis, D., Li, A., Li, N., … Zaharia, M. (2025). Unity Catalog: Open and universal governance for the lakehouse and beyond. In Companion of the 2025 International Conference on Management of Data (pp. 310–322). Association for Computing Machinery.
37. Vakkalagadda, T., & Pareek, P. K. (2026). Self-Learning Artificial Intelligence Platforms for Autonomous Financial Risk Management. International Journal of Special Education, 41(15s), 238-254.
Additional Files
Published
Data Availability Statement
The datasets analyzed in this study are publicly available, including the European Credit Card Fraud Detection dataset and the New York City Taxi Trips dataset. No new data were generated in this study.
