AI-Driven Anomaly Detection for Distributed Enterprise Systems
Keywords:
Enterprise Information Systems, Decentralized Data Architectures, Atypical Event Detection, AI-Driven Anomaly Detection, Heterogeneous Computing Environments, Large-Scale Data Operations, Autonomous Monitoring Systems, Enterprise Data Pipelines, Technical Debt in AI Initiatives, Solution Taxonomy for Event Detection, Data Source Heterogeneity, Operational AI Architectures, Data Governance and Compliance, Privacy-Preserving Analytics, Secure AI Deployment, Regulatory-Aware AI Systems, Intelligent Observability, Scalable Detection Frameworks, Enterprise-Wide AI Integration, Adaptive Monitoring Infrastructure.Abstract
Enterprise information systems typically employ a decentralized architecture based on data stored across numerous heterogeneous computing environments. These environments are also employed by other organizations that provide services to a significant user base. With such user bases come volume and frequency of operations that are unprecedented. Furthermore, systems that allow the execution of user-controlled queries can have unpredictable types and patterns of operations. Consequently, unified enterprise data systems may lag behind such developments. Artificial intelligence can counteract human operators’ limitations on detecting atypical situations in these decentralized systems. However, current applications often result from piecemeal isolated initiatives by data scientists from diverse parts of the organization, which quickly become technical debts. Four decisive aspects of fully supporting the atypical event detection process with artificial intelligence and its implications on enterprise information systems have emerged from a synthesis of the academic literature over the last several decades.
First, a comprehensive taxonomy of current solutions is essential to manage the large number of proposals, as the expressed needs of enterprises imply the possibility of artificial intelligence detecting atypical events in any part of the data systems. Second, defining general characteristics of the enterprise data systems’ architecture is key because many of the proposed solutions are strongly dependent on these characteristics, especially aspects related to the sources of data and their pipelines. Third, the data requirements and operational principles of the adapted technical solutions must be covered. Fourth, underlying requirements for data governance, privacy, and security must be considered, together with the implications of regulatory pressures for the application and development of artificial intelligence.
References
1. Bento, A., Correia, J., Filipe, R., Araujo, F., & Cardoso, J. (2021). Automated analysis of distributed tracing: Challenges and research directions. Journal of Grid Computing, 19, Article 9.
2. Guo, H., Yuan, S., & Wu, X. (2021). LogBERT: Log anomaly detection via BERT. In Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN) (pp. 1–8).
3. Kummari, D. N. (2022). IoT-enabled additive manufacturing: Improving prototyping speed and customization in the automotive sector. Migration Letters, 19(S8), 2084-2104.
4. Le, V.-H., & Zhang, H. (2021). Log-based anomaly detection without log parsing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) (pp. 492–504).
5. Li, Z., Zhao, Y., Han, J., Su, Y., Jiao, R., Wen, X., & Pei, D. (2021). Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 3220–3230).
6. Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021). PLELog: Semi-supervised log-based anomaly detection via probabilistic label estimation. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (pp. 230–231).
7. Krishna AzithTejaGanti, V., Senthilkumar, K. P., Robinson L, T., Karunakaran, S., Pandugula, C., & Khatana, K. (2024, November). Energy-Efficient Real-Time Hybrid Deep Learning Framework for Adaptive Iot Intrusion Detection with Scalable and Dynamic Threat Mitigation. In Proceedings of the 3rd International Conference on Optimization Techniques in the Field of Engineering (ICOFE-2024).
8. Wang, Z., Tian, J., Fang, H., Chen, L., & Qin, J. (2022). LightLog: A lightweight temporal convolutional network for log anomaly detection on the edge. Computer Networks, 203, 108616.
9. Xu, J., Wu, H., Wang, J., & Long, M. (2022). Anomaly Transformer: Time series anomaly detection with association discrepancy. In International Conference on Learning Representations.
10. Annapareddy, V. N. (2024). Leveraging Artificial Intelligence, Machine Learning, and Cloud-Based IT Integrations to Optimize Solar Power Systems and Renewable Energy Management. Machine Learning, and Cloud-Based IT Integrations to Optimize Solar Power Systems and Renewable Energy Management (December 06, 2024).
11. Tuli, S., Casale, G., & Jennings, N. R. (2022). TranAD: Deep transformer networks for anomaly detection in multivariate time series data. Proceedings of the VLDB Endowment, 15(6), 1201–1214.
12. Zhang, C., Peng, X., Sha, C., Zhang, K., Fu, Z., Wu, X., Lin, Q., & Zhang, D. (2022). DeepTraLog: Trace-log combined microservice anomaly detection through graph-based deep learning. In Proceedings of the 44th International Conference on Software Engineering (pp. 255–267).
13. Pandiri, L., Paleti, S., Kaulwar, P. K., Malempati, M., & Singireddy, J. (2023). Transforming financial and insurance ecosystems through intelligent automation, secure digital infrastructure, and advanced risk management strategies. Educational Administration: Theory and Practice, 29(4), 4777-4793.
14. van Ede, T., Aghakhani, H., Spahn, N., Bortolameotti, R., Cova, M., Continella, A., van Steen, M., Peter, A., Kruegel, C., & Vigna, G. (2022). DEEPCASE: Semi-supervised contextual analysis of security events. In 2022 IEEE Symposium on Security and Privacy (SP) (pp. 522–539).
15. Bogatinovski, J., Madjarov, G., Nedelkoski, S., Cardoso, J., & Kao, O. (2022). Leveraging log instructions in log-based anomaly detection. In Proceedings of the 2022 International Workshop on Artificial Intelligence for IT Operations.
16. Sriram, H. K., Challa, S. R., Challa, K., & ADUSUPALLI, B. (2024). Strategic Financial Growth: Strengthening Investment Management. Secure Transactions, and Risk Protection in the Digital Era. Secure Transactions, and Risk Protection in the Digital Era (November 10, 2024).
17. Kohyarnejadfard, I., Aloise, D., Azhari, S. V., & Dagenais, M. R. (2022). Anomaly detection in microservice environments using distributed tracing data analysis and NLP. Journal of Cloud Computing, 11, Article 25.
18. Le, V.-H., & Zhang, H. (2022). Log-based anomaly detection with deep learning: How far are we? In Proceedings of the 44th International Conference on Software Engineering (pp. 2617–2628).
19. Davuluri, P. S. L. (2023). AI-Augmented Sanctions Screening: Enhancing Accuracy and Latency in Real Time Compliance Systems. AI-Augmented Sanctions Screening: Enhancing Accuracy and Latency in Real Time Compliance Systems (December 15, 2023).
20. Mejri, N., Lopez-Fuentes, L., Roy, K., Chernakov, P., Ghorbel, E., & Aouada, D. (2022). Unsupervised anomaly detection in time-series: An extensive evaluation and analysis of state-of-the-art methods. arXiv preprint arXiv:2212.03637.
21. Silva, P. R., Vinagre, J., & Gama, J. (2022). Federated anomaly detection over distributed data streams. arXiv preprint arXiv:2205.07829.
22. Singireddy, S. (2024). The Integration of AI and Machine Learning in Transforming Underwriting and Risk Assessment Across Personal and Commercial Insurance Lines. Journal of Computational Analy-sis and Applications (JoCAAA), 33(08), 3966-3991.
23. Han, P., Li, H., Xue, G., & Zhang, C. (2023). Distributed system anomaly detection using deep learning-based log analysis. Computational Intelligence, 39, 433–455.
24. Hang, F., Guo, W., Chen, H., Xie, L., Zhou, C., & Liu, Y. (2023). Logformer: Cascaded transformer for system log anomaly detection. Computer Modeling in Engineering & Sciences, 136(1), 517–529.
25. Sheelam, G. K. (2024). AI-driven spectrum management: Using machine learning and agentic intelligence for dynamic wireless optimization. European Advanced Journal for Emerging Technologies (EAJET), 2(1), 3050-9742.
26. Jia, P., Cai, S., Ooi, B. C., Wang, P., & Xiong, Y. (2023). Robust and transferable log-based anomaly detection. Proceedings of the ACM on Management of Data, 1(1), Article 64.
27. Han, X., Yuan, S., & Trabelsi, M. (2023). LogGPT: Log anomaly detection via GPT. In Proceedings of the 2023 IEEE International Conference on Big Data (BigData) (pp. 1117–1122).
28. Yang, Y., Zhang, C., Zhou, T., Wen, Q., & Sun, L. (2023). DCdetector: Dual attention contrastive representation learning for time series anomaly detection. In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 3033–3045).
29. Yadav, V. (2024). Predictive Analytics for Preventive Medicine: Analyzing how Predictive Analytics is Utilized for Forecasting Patient Health Trends and Preventive Disease. Progress in Medical Sciences. PMS-1126. Prog Med Sci, 8(4).
30. Xie, Z., Xu, H., Chen, W., Li, W., Jiang, H., Su, L., Wang, H., & Pei, D. (2023). Unsupervised anomaly detection on microservice traces through graph VAE. In Proceedings of the ACM Web Conference 2023 (pp. 2874–2884).
31. Nobre, J., Solteiro Pires, E. J., & Reis, A. (2023). Anomaly detection in microservice-based systems. Applied Sciences, 13(13), 7891.
32. Liu, J., Yang, D., Zhang, K., Gao, H., & Li, J. (2023). Anomaly and change point detection for time series with concept drift. World Wide Web, 26, 3229–3252.
33. Ikudabo, A. O., & Kumar, P. (2024). AI-driven risk assessment and management in banking: balancing innovation and security. International Journal of Research Publication and Reviews, 5(10), 3573-88.
34. Wang, S., Luo, C., & Shao, R. (2023). Unsupervised concept drift detection for time series on Riemannian manifolds. Journal of the Franklin Institute, 360(17), 13186–13204.
35. Jeong, Y., Yang, E., Ryu, J. H., Park, I., & Kang, M. (2023). AnomalyBERT: Self-supervised transformer for time series anomaly detection using data degradation scheme. arXiv preprint arXiv:2305.04468.
36. Zhong, Z., Fan, Q., Zhang, J., Ma, M., Zhang, S., Sun, Y., Lin, Q., Zhang, Y., & Pei, D. (2023). A survey of time series anomaly detection methods in the AIOps domain. arXiv preprint arXiv:2308.00393.
37. Srinivas Kalisetty, D. A. S. (2024). Leveraging Artificial Intelligence and Machine Learning for Predictive Bid Analysis in Supply Chain Management: A Data-Driven Approach to Optimize Procurement Strategies.
38. Huang, J., Yang, Y., Yu, H., Li, J., & Zheng, X. (2023). Twin graph-based anomaly detection via attentive multi-modal learning for microservice system. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (pp. 66–78).
39. Almodovar, C., Sabrina, F., Karimi, S., & Azad, S. A. (2024). LogFiT: Log anomaly detection using fine-tuned language models. IEEE Transactions on Network and Service Management.
40. Singireddy, J. (2024). AI-driven payroll systems: Ensuring compliance and reducing human error. American Data Science Journal for Advanced Computations (ADSJAC) ISSN, 3067-4166.
41. Guo, H., Yang, J., Liu, J., Bai, J., Wang, B., Li, Z., Zheng, T., Zhang, B., Peng, J., & Tian, Q. (2024). LogFormer: A pre-train and tuning pipeline for log anomaly detection. Proceedings of the AAAI Conference on Artificial Intelligence, 38(1), 135–143.
42. Hashemi, S., & Mäntylä, M. (2024). OneLog: Towards end-to-end software log anomaly detection. Automated Software Engineering, 31(2), Article 37.
43. Kolla, S., Meda, R., Balleda, L., & Thimmapuram, C. R. (2024). The utility value of ROX index and modified ROX index in determining the efficiency of HFNC in children admitted with respiratory distress. International Journal of Contemporary Pediatrics, 11(6), 775.
44. Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C. (2024). Impact of log parsing on deep learning-based anomaly detection. Empirical Software Engineering, 29(6), Article 139.
45. Xie, Y., Zhang, H., & Babar, M. A. (2024). LogSD: Detecting anomalies from system logs through self-supervised learning and frequency-based masking. Proceedings of the ACM on Software Engineering, 1(FSE), Article 93.
46. Koppolu, H. K. R. (2024). The impact of data engineering on service quality in 5G-enabled cable and media networks. European Advanced Journal for Science & Engineering (EAJSE), 1(1).
47. Tu, C., Chen, M., Zhang, L., Zhao, L., Wu, D., & Yue, Z. (2024). Towards efficient multi-granular anomaly detection in distributed systems. Array, 21, 100330.
48. Panahandeh, M., Hamou-Lhadj, A., Hamdaqa, M., & Miller, J. (2024). ServiceAnomaly: An anomaly detection approach in microservices using distributed traces and profiling metrics. Journal of Systems and Software, 209, 111917.
49. Mitropoulou, K., Kokkinos, P., Soumplis, P., & Varvarigos, E. (2024). Anomaly detection in cloud computing using knowledge graph embedding and machine learning mechanisms. Journal of Grid Computing, 22, Article 6.
50. Zamanzadeh Darban, Z., Webb, G. I., Pan, S., Aggarwal, C. C., & Salehi, M. (2024). Deep learning for time series anomaly detection: A survey. ACM Computing Surveys, 57(1), Article 15.
51. Ramanakar Reddy Danda, Z. Y., Mandala, G., & Maguluri, K. K. (2024). Smart Medicine: The Role of Artificial Intelligence and Machine Learning in Next-Generation Healthcare Innovation.
52. Correia, L., Goos, J.-C., Klein, P., Bäck, T., & Kononova, A. V. (2024). Online model-based anomaly detection in multivariate time series: Taxonomy, survey, research challenges and future directions. Engineering Applications of Artificial Intelligence, 138, 109323.
53. Mejri, N., Lopez-Fuentes, L., Roy, K., Chernakov, P., Ghorbel, E., & Aouada, D. (2024). Unsupervised anomaly detection in time-series: An extensive evaluation and analysis of state-of-the-art methods. Expert Systems with Applications, 256, 124922.
54. Zhang, Y., Chen, X., Wang, Y., Liu, J., & Pei, D. (2024). A robust Wide & Deep learning framework for log-based anomaly detection. Applied Soft Computing, 153, 111314.
55. Almodovar, C., Sabrina, F., Karimi, S., & Azad, S. A. (2024). Interaction prediction and anomaly detection in a microservices-based telecommunication platform. In Proceedings of the International Conference on Software and Systems Processes.
56. Correia, L., Goos, J.-C., Klein, P., Bäck, T., & Kononova, A. V. (2024). Online model-based anomaly detection in multivariate time series: Taxonomy, survey, research challenges and future directions. Engineering Applications of Artificial Intelligence, 138, 109323.
57. Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C. (2024). A theoretical framework for understanding the relationship between log parsing and anomaly detection. Empirical Software Engineering.
58. Bogatinovski, J., Nedelkoski, S., Acker, A., Cardoso, J., & Kao, O. (2024). A deep graph neural networks approach for service failure analytics. In Proceedings of the 11th International Conference on Future Internet of Things and Cloud.
59. Almodovar, C., Sabrina, F., Karimi, S., & Azad, S. A. (2024). Log anomaly detection using fine-tuned language models. IEEE Transactions on Network and Service Management.
60. Darban, Z. Z., Webb, G. I., Pan, S., Aggarwal, C. C., & Salehi, M. (2024). Deep learning approaches for anomaly detection in multivariate time series: A comprehensive review. ACM Computing Surveys.
Additional Files
Published
Issue
Section
License
Copyright (c) 2024 European Advanced Journal for Science & Engineering (EAJSE)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Articles published in the European Advanced Journal for Science & Engineering (EAJSE) are made freely available online immediately upon publication under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited. Authors retain copyright of their work. By submitting to EAJSE, authors grant the journal the right of first publication. For details, visit: https://creativecommons.org/licenses/by/4.0/
