Self-Optimizing Cloud Data Pipelines via Machine Learning
Keywords:
Cloud Data Pipelines, Real-Time ETL Optimization, ML-Driven Resource Scheduling, Cost–Latency Trade-off, Cross-Cloud Data Architecture, Intelligent Autoscaling, Data Ingestion Frameworks, Transformation Pipelines, Enhanced ETL, Cloud Portability, Modular Connection Management, Multi-Cloud Support (AWS, GCP, Azure, IBM), ML-Based Workload Scheduling, Resource Allocation Optimization, Data Engineering Automation, Operational Cost Reduction, Low-Latency Analytics Pipelines, Structured and Unstructured Data Processing, Cloud-Native Data Platforms, Production-Grade Data Pipelines.Abstract
Cloud Data Pipelines enable enterprises to readily ingest, process, clean, and store large amounts of structured and unstructured data in cloud environments to drive analytics, business intelligence, and data-science workloads. However, designing and implementing such pipelines is non-trivial and challenging. Pipelines should be optimized for cost, speed, or any combination of the two but these objectives are at odds with each other. A data pipeline architecture that enables easy prototyping of data ingestion and transformation processes within any cloud platform is presented. Machine Learning (ML) is employed to inform scheduling and resource allocation decisions in order to reduce operational cost while ensuring acceptable latencies.
The objectives of optimizing Ingest, Transformation, and Enhanced ETL cloud data pipelines in real-time for cost and latency are accomplished. The four cloud providers—Google, Amazon, Microsoft, and IBM—are supported, with data volumes ranging from a few megabytes to generating several gigabytes. Latency from minutes to hours can be supported without breaking the bank. ML models inform autoscaling groups, transformation resources, and scheduling. Cross-cloud portability through modular code-based connection-management further optimizes the development phases while improving code quality.
References
1. Bandi, V. D. V. K. (2025). Self-optimizing data pipelines using machine learning for cloud workloads. Journal of Information Systems Engineering and Management, 10(63s).
2. Jakubik, J., Vössing, M., Kühl, N., Walk, J., & Satzger, G. (2024). Data-centric artificial intelligence. Business & Information Systems Engineering, 66, 507–515.
3. Kang, D., Guibas, J., Bailis, P., Hashimoto, T., Sun, Y., & Zaharia, M. (2024). Data management for ML-based analytics and beyond. ACM/IMS Journal of Data Science, 1(1), Article 4.
4. Davuluri, P. S. L. (2023). AI-Augmented Sanctions Screening: Enhancing Accuracy and Latency in Real Time Compliance Systems. AI-Augmented Sanctions Screening: Enhancing Accuracy and Latency in Real Time Compliance Systems (December 15, 2023).
5. Raatikainen, M., Souris, H., Remes, J., & Stirbu, V. (2024). Machine learning lineage for trustworthy machine learning systems: Information framework for MLOps pipelines. IEEE Software, 42(1), 51–58.
6. Zhao, M. Y., Adamiak, E., & Kozyrakis, C. (2024). Cedar: Optimized and unified machine learning input data pipelines. Proceedings of the VLDB Endowment, 18(2), 488–502.
7. Rad, Z. S., & Ghobaei-Arani, M. (2024). Data pipeline approaches in serverless computing: A taxonomy, review, and research trends. Journal of Big Data, 11, Article 82.
8. Sriram, H. K., Challa, K., Gadi, A. L., & Singreddy, S. (2025). AI and Cloud-Driven Transformation in Finance, Insurance, and the Automotive Ecosystem: A Multi-Sectoral Framework for Credit Risk, Mobility Services, and Consumer Protection. Anil Lokesh and singreddy, Sneha, AI and Cloud-Driven Transformation in Finance, Insurance, and the Automotive Ecosystem: A Multi-Sectoral Framework for Credit Risk, Mobility Services, and Consumer Protection (March 15, 2025).
9. Gu, Y., Cheng, F., Yang, L., Xu, J., Chen, X., Cheng, L., et al. (2024). Cost-aware cloud workflow scheduling using DRL and simulated annealing. Digital Communications and Networks, 10(6), 1590–1599.
10. Seenu, A., Sheelam, G. K., Motamary, S., Meda, R., Koppolu, H. K. R., & Inala, R. (2025). AI-Driven Innovations in Infrastructure Management with 6G Technology. In 2025 2nd International Conference on Computing and Data Science (ICCDS) (pp. 1–6). IEEE. 2025 2nd International Conference on Computing and Data Science (ICCDS). https://doi.org/10.1109/iccds64403.2025.11209649
11. Pan, J., & Wei, Y. (2024). A deep reinforcement learning-based scheduling framework for real-time workflows in the cloud environment. Expert Systems with Applications, 249, 124845.
12. Schelter, S., Guha, S., & Grafberger, S. (2024). Automated provenance-based screening of ML data preparation pipelines. Datenbank-Spektrum, 24, 187–196.
13. Dutta, P., Adusupalli, B., Koppolu, H. K. R., Dodda, A., Yağanoğlu, M., Banerjee, J. S., & Chakraborty, A. (2025, March). Textual Social Data Disinformation Analysis Using a Hybrid Context-Enhanced Deep Learning Model. In Doctoral Symposium on Human Centered Computing (pp. 342-352). Singapore: Springer Nature Singapore.
14. Majeed, A., & Hwang, S. O. (2024). Feature stores: A key enabler for feature reusability and availability across machine learning pipelines. Computer, 57(1), 69–74.
15. Li, G., Zhou, X., & Zhao, X. (2024). LLM for data management. Proceedings of the VLDB Endowment, 17(12), 4213–4216.
16. Priestley, M., O’Donnell, F., & Simperl, E. (2023). A survey of data quality requirements that matter in ML development pipelines. Journal of Data and Information Quality, 15(2), Article 11.
17. Narasareddy Annapareddy, V., Pamisetty, A., Malempati, M., Kaulwar, P. K., & Bhardwaj Komaragiri, V. (2025, April). Enhancing Solar Power System Efficiency Through AI-Driven Predictive Maintenance and Cloud-Based Infrastructure Stability Solutions. In International Conference on Smart Computing and Informatics (pp. 328-337). Cham: Springer Nature Switzerland.
18. Liu, R., Park, K., Psallidas, F., Zhu, X., Mo, J., Sen, R., Interlandi, M., Karanasos, K., Tian, Y., & Camacho-Rodríguez, J. (2023). Optimizing data pipelines for machine learning in feature stores. Proceedings of the VLDB Endowment, 16(13), 4230–4239.
19. Kreuzberger, D., Kühl, N., & Hirschl, S. (2023). Machine learning operations (MLOps): Overview, definition, and architecture. IEEE Access, 11, 31866–31879.
20. Vadisetty, R., Nuka, S. T., Kalisetty, S., Pandugula, C., Burugulla, J. K. R., & Annapareddy, V. N. (2025, February). Generative AI for Advanced Recycling Processes in Polyethylene and Polypropylene Manufacturing. In International Ethical Hacking Conference (pp. 269-284). Singapore: Springer Nature Singapore.
21. Faubel, L., Schmid, K., & Eichelberger, H. (2023). MLOps challenges in Industry 4.0. SN Computer Science, 4, Article 828.
22. Xiu, X., Li, J., Long, Y., & Wu, W. (2023). MRLCC: An adaptive cloud task scheduling method based on meta reinforcement learning. Journal of Cloud Computing, 12, Article 75.
23. Challa, S. R., Burugulla, J. K. R., Pamisetty, A., Challa, K., & Paleti, S. (2025, April). AI and ML-Powered Cybersecurity Strategies for Cloud Computing: Ensuring Infrastructure Stability in Financial and Retail Sectors. In International Conference on Smart Computing and Informatics (pp. 315-327). Cham: Springer Nature Switzerland.
24. Zhang, S., Zhao, Z., Liu, C., & Qin, S. (2023). Data-intensive workflow scheduling strategy based on deep reinforcement learning in multi-clouds. Journal of Cloud Computing, 12, Article 125.
25. Zhang, J., Cheng, L., Liu, C., Zhao, Z., & Mao, Y. (2023). Cost-aware scheduling systems for real-time workflows in cloud: An approach based on genetic algorithm and deep reinforcement learning. Expert Systems with Applications, 234, 120972.
26. Challa, S. R., Kaulwar, P. K., Koppolu, H. K. R., Adusupalli, B., & Suura, S. R. (2025, April). Big Data and AI in Wealth Management: Leveraging Cloud Computing for Secure and Scalable Financial Infrastructure. In International Conference on Smart Computing and Informatics (pp. 307-318). Cham: Springer Nature Switzerland.
27. Mampage, A., Karunasekera, S., & Buyya, R. (2023). Deep reinforcement learning for application scheduling in resource-constrained, multi-tenant serverless computing environments. Future Generation Computer Systems, 143, 277–292.
28. Singireddy, S., Pandiri, L., Motamary, S., Kummari, D. N., Somu, B., & Lakkarasu, P. (2025, August). Blockchain-Powered Claims Validation for Enhancing Trust in Health and Auto Insurance Ecosystems. In International Conference on Artificial Intelligence: Theory and Applications (pp. 26-38). Cham: Springer Nature Switzerland.
29. Mangalampalli, S., Karri, G. R., Mohanty, S. N., Ali, S., Khan, M. I., Abduvalieva, D., Awwad, F. A., & Ismail, E. A. A. (2023). Fault tolerant trust based task scheduler using Harris Hawks optimization and deep reinforcement learning in multi cloud environment. Scientific Reports, 13, Article 19179.
30. Cui, T., Yang, R., Fang, C., & Yu, S. (2023). Deep reinforcement learning-based resource allocation for content distribution in IoT-edge-cloud computing environments. Symmetry, 15(1), 217.
31. KOTHAPALLI, S. L., SEENU, A., DILEEP, V., & YASMEEN, Z. (2025). THE FUTURE OF CUSTOMER ENGAGEMENT IN RETAIL BANKING: EXPLORING THE POTENTIAL OF AUGMENTED REALITY AND IMMERSIVE TECHNOLOGIES. INTERNATIONAL JOURNAL, 73(1), 72-79.
32. Barbudo, R., Ventura, S., & Romero, J. R. (2023). Eight years of AutoML: Categorisation, review and trends. Knowledge and Information Systems, 65, 5097–5149.
33. Neutatz, F., Lindauer, M., & Abedjan, Z. (2024). AutoML in heavily constrained applications. The VLDB Journal, 33, 957–979.
34. Jakubik, J., Vössing, M., Kühl, N., Walk, J., & Satzger, G. (2023). Data-centric artificial intelligence. Business & Information Systems Engineering, 65, 507–515.
35. Singireddy, J., Kaulwar, P. K., Somu, B., Meda, R., Dodda, A., & Yellanki, S. K. (2025, August). Reinforcement Learning-Based Asset Allocation in Algorithmic Trading for Banking Institutions. In International Conference on Artificial Intelligence: Theory and Applications (pp. 357-369). Cham: Springer Nature Switzerland.
36. Zha, D., Bhat, Z. P., Lai, K.-H., Yang, F., & Hu, X. (2023). Data-centric AI: Perspectives and challenges. ACM Computing Surveys, 56(1), Article 17.
37. Mohr, F., Wever, M., Tornede, A., & Hüllermeier, E. (2021). Predicting machine learning pipeline runtimes in the context of automated machine learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9), 3055–3066.
38. Ande, R., Mehta, D., Pandugula, C., Krishna AzithTejaGanti, V., & Kalisetty, S. (2025). Modeling the Som Kamla Amba sub-watershed using Artificial Neural Networks (ANN) and the SWAT tool. Available at SSRN 5341755.
39. Marinescu, R., Kishimoto, A., Ram, P., Rawat, A., Wistuba, M., Palmes, P. P., & Botea, A. (2021). Searching for machine learning pipelines using a context-free grammar. Proceedings of the AAAI Conference on Artificial Intelligence, 35(10), 8902–8911.
40. Orr, L., Sanyal, A., Ling, X., Goel, K., & Leszczynski, M. (2021). Managing ML pipelines: Feature stores and the coming wave of embedding ecosystems. Proceedings of the VLDB Endowment, 14(12), 3054–3061.
41. Maguluri, K. K. (2025). Ethical challenges in artificial intelligence. How Artificial Intelligence is Transforming Healthcare IT: Applications in Diagnostics, Treatment Planning, and Patient Monitoring, 132.
42. Renggli, C., Rimanic, L., Gurel, N. M., Karlas, B., Wu, W., & Zhang, C. (2021). A data quality-driven view of MLOps. IEEE Data Engineering Bulletin, 44(1), 11–23.
43. Patel, H., Ishikawa, F., Berti-Equille, L., Gupta, N., Mehta, S., Masuda, S., Mujumdar, S., Afzal, S., Bedathur, S., & Nishi, Y. (2021). Data quality assessment for machine learning. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 3141–3150.
44. Gong, F. (2021). Workflow scheduling based on mobile cloud computing machine learning. Wireless Communications and Mobile Computing, 2021, Article 9923326.
45. Karat, C., & Senthilkumar, R. (2022). Optimal resource allocation with deep reinforcement learning and greedy adaptive firefly algorithm in cloud computing. Concurrency and Computation: Practice and Experience, 34(4), e6657.
46. Gong, F., & colleagues. (2022). Reinforcement learning-assisted autoscaling mechanisms for serverless computing platforms. Simulation Modelling Practice and Theory, 116, 102461.
47. Zhou, R., Pang, J., Zhang, Q., Wu, C., et al. (2022). Online scheduling algorithm for heterogeneous distributed machine learning jobs. IEEE Transactions on Cloud Computing.
48. ADUSUPALLI, B. (2025). Redefining Financial Risk Strategies: The Integration of Smart Automation, Secure Access Systems, and Predictive Intelligence in Insurance, Lending, and Asset Management. JAIBDD.
49. Öztürk, E., Ferreira, F., Jomaa, H. S., Schmidt-Thieme, L., Grabocka, J., & Hutter, F. (2022). Zero-shot AutoML with pretrained models. Proceedings of the 39th International Conference on Machine Learning, 162, 17138–17155.
50. Mohr, F., Wever, M., & Hüllermeier, E. (2023). Naive automated machine learning. Machine Learning, 112, 1131–1170.
51. Pandugula, C., Ganti, V. K. A. T., Kalisetty, S., Ande, R., & Mehta, D. (2025). Modeling the Som Kamla Amba Sub-Watershed Using Artificial Neural Networks (ANN) and the SWAT Tool. Available at SSRN 5341614.
52. Subramanya, R., et al. (2022). A systematic review of machine learning operations and MLOps practices. Proceedings of relevant international software engineering and machine learning research venues.
53. Pölöskei, I. (2021). MLOps approach in the cloud-native data pipeline design. Acta Technica Jaurinensis, 14(4), 385–395.
54. Barrak, A., Petrillo, F., & Jaafar, F. (2022). Serverless on machine learning: A systematic mapping study. IEEE Access, 10, 99337–99352.
55. Faubel, L., Schmid, K., & Eichelberger, H. (2022). A reference architecture for the operationalization of machine learning models in manufacturing. Procedia CIRP, 115, 130–135.
56. Auer, A., et al. (2022). Qualitative assessment of the impact of manufacturing-specific influences on machine learning operations. Procedia CIRP, 115, 136–141.
57. Warg, M., Brazhenko, D., Somenkova, A., Golovkov, A., & Sergunin, I. (2025). MLOps architecture as a future of machine learning. Journal of Computer Information Systems.
58. Nogare, D., Silveira, I. F., Banzai, R., & Alexandre, M. C. (2025). Make or buy strategy for machine learning operations—MLOps. Anais da Academia Brasileira de Ciências, 97(2), e20240924.
59. Wei, H., García Pañeda, X., & Salvachúa Rodriguez, J. (2025). Optimizing machine learning operations in multi-cloud infrastructure: A framework for unified deployment management and topology discovery. Cluster Computing, 28, Article 933.
60. Kramer, K. M., Restat, V., Strasser, S., Störl, U., & Klettke, M. (2025). Towards next generation data engineering pipelines. Proceedings of the VLDB-related data engineering research community.
Additional Files
Published
Issue
Section
License
Copyright (c) 2025 European Advanced Journal for Science & Engineering (EAJSE)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Articles published in the European Advanced Journal for Science & Engineering (EAJSE) are made freely available online immediately upon publication under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited. Authors retain copyright of their work. By submitting to EAJSE, authors grant the journal the right of first publication. For details, visit: https://creativecommons.org/licenses/by/4.0/
