European Advanced Journal for Science & Engineering (EAJSE)
European Advanced Journal for Science & Engineering (EAJSE) is a peer-reviewed, open access journal publishing original research, reviews, and technical papers across the applied sciences and engineering, including artificial intelligence, IoT and digital transformation, materials science, and sustainable engineering. The journal follows a double-blind peer review process and publishes quarterly. EAJSE is committed to rigorous editorial standards, transparent publication ethics in line with COPE guidelines, and immediate open access to all published content under a CC BY 4.0 license.
Articles published in the European Advanced Journal for Science & Engineering (EAJSE) are made freely available online immediately upon publication under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited. Authors retain copyright of their work. By submitting to EAJSE, authors grant the journal the right of first publication. For details, visit: https://creativecommons.org/licenses/by/4.0/
Intelligent Cloud-Native Infrastructure for Financial Transactions
AI-powered cloud-native financial platforms based on microservices architectures enable secure, real-time, and scalable transaction processing for payment, insurance, brokerage, trading, capital markets, and treasury services. Cloud-native architectures are designed to support the convergence of digital banking, cybersecurity, fraud detection, capital market trading, trading platform services, liquidity management, data-driven pricing, and governmental regulatory compliance. Cloud environments are also the preferred foundation for integrating the machine learning workloads behind AI innovation into financial-function operations. The primary platform enabler is real-time transaction-processing capability, essential for providing privacy risk scoring, instantaneous payment fraud detection, and market-shock liquidity forecasting with embedded pricing models. Cloud-native streaming-data architecture, event-driven processing with near-real-time latency guarantees, and the consistency–availability–partition tolerance framework underpin secure processing of financial transactions. AI methods supporting these platforms incorporate transaction anomaly detection for fraud prevention, risk scoring for the governing approval process, ASIC-generated cash–liquidity forecasting for funding cost management, and data-driven algorithms for pricing bank, insurance, and brokerage products.
Copyright (c) 2024 European Advanced Journal for Science & Engineering (EAJSE) https://creativecommons.org/licenses/by-sa/4.0
Secure Cloud Frameworks for Real-Time Financial Risk Intelligence
Intelligent autonomous financial ecosystems are ecosystems of interacting autonomous systems responsible for managing financial portfolios and executing individual and collective investment strategies. Support for scalability is drawn from the cloud architectural reference models of multi-cloud implementations, hybrid cloud infrastructures, the microservices architectural style, and the serverless computing execution model. Real-time data ingestion, streaming analytics, and risk-aware decisioning endpoints constitute active architectural components. Security, privacy, and compliance considerations follow a zero-trust approach, are co-designed with the system, and are addressed for all data-related operations. The financial AI is designed and monitored for classically relevant properties and its agents are routinely reviewed and retrained. Continuous integration and delivery pipelines accommodate the rapid deployment cycles of financial workloads. Observability, monitoring, and incident-response processes support the requirements of production autonomy, resilience, and incident recovery. Intelligent autonomous financial ecosystems support cloud-native operations that learn, adapt, and evolve under operational conditions. Lesson-learning mechanism support the mechanisms of not only decisioning systems, trained by supervised learning, but also automated optimisation engines, where proper agent interaction is key for the integrated dynamic robustness and performance of the ecosystem. Cyber-physical, hybrid trading agents incorporate the actual market into the investment strategy; their behaviour is continuously monitored, and the expected market response to the agents’ activity is taken into consideration during deployment risk assessment. Cyber-physical agents interact with environments, injecting the decisioning behaviour into the business-cycle evolution itself; risk-aware decisioning supports early detection of departure from the normal operating regime and enables antifragile adaptation.
Reinventing Claims Artificial Intelligence Across the Insurance Value Chain
Over the past decade, claims processing has emerged as a primary area of interest for insurers exploring how artificial intelligence (AI) can reduce costs or enhance service quality. Several factors motivate this focus. First, processing claims represents a major operational expense—one that, if reduced, is likely to translate into greater profitability. Second, claims processing often significantly influences the customer experience and may even determine clients’ future purchase decisions. Finally, AI technologies tailored for claims processing have demonstrated the ability to deliver faster settlement of smaller claims without sacrificing accuracy. Provided that AI is applied within a carefully monitored, ethically governed framework, its integration into claims processing represents a genuine win-win opportunity. But increasing the efficiency of claims processing is only half the story. Unless these savings translate into improved customer experience and satisfaction, the long-term gains may be illusory. Customers today demand – and expect – faster claims settlements. Organisations that can consistently meet these expectations are likely to enjoy a competitive edge, gaining not only customer satisfaction but also long-term loyalty. Conversely, companies that fail to deliver speedy settlements will, over time, see an increased rate of customer attrition, particularly among their high-value clients. Insurers that can harness new AI capabilities to manage these twin objectives – faster settlement with greater consistency – stand to capture both the operating and customer experience benefits.
Intelligent Behavioral Analytics for Cloud Payment Authorization
Cloud-based platforms are increasingly incorporating Payment as a Service (PaaS) that empowers various organizations to swiftly offer payment services to their end users—most in a win-win manner. However, the growing violence and higher money flowing into crime severs the win-win situation and reduces the reception by the organizations. Payment transactions can never be completely anonymized. To keep the cloud-based transaction services flourish in the ecosystem, a technology that tracks criminals’ abnormal behaviors and predicts potential attacks before they happen is essential for sensitive, high-value, and risky transactions. Protecting the merchants and users by incorporating intelligent, machine learning (ML)-powered behavioral analytics for real-time authorization of sensitive transactions is a crucial and hot research topic; and their integrated receiving and sending decision policies should be taken care of in a balanced way. To meet these needs and offer intelligent risk sensing and scoring services, the integration of denser behavioral signals in addition to the transactional and contextual ones is necessary. Various machine-learning-anomaly-detection algorithms are compared to select the most suitable model. Moreover, denser behavioral signals are searched and experimented with to attain ML models of higher accuracy. The normal classes are defined by both supervised learning for benign actions of authorized users and by semi-supervised clustering with a few labeled instances only for malicious actions of non-users. The models are label-free in the testing phase. Enriching high-risk monetary transfer authorization with intelligent decision-making is accomplished by ML-based risk scoring, where pleasing sensitivity and specificity are achieved through careful calibration of the risk scoring threshold.
Copyright (c) 2025 European Advanced Journal for Science & Engineering (EAJSE) https://creativecommons.org/licenses/by-sa/4.0
Rule-Based Decision Engines for Automated Audit Sample Selection
Automation of Audit Sampling Using Rule-Based Decision Systems: an objective, scholarly analysis of automated sampling approaches, evidence-based assessment, and formal structure. Automation and computerization in auditing are ubiquitous and offer unparalleled assistance to auditors in improving efficiency, effectiveness, and overall cost. Audit sampling is an effective means to make audit decisions based on part of the evidence rather than the whole. Rule-based decision systems are popular in many business and accounting areas, but not widely deployed for audit sampling yet. Audit sampling can be automated by creating sampling rules based on audit data, professional guidelines, and/or judgment. Such rules specify sampling conditions and thresholds, the population elements that trigger the rule-set, the sampling specification produced, and the sample sizes required and whether the auditor should consider additional information of other related decisions. The approach also supports a human-in-the-loop function, providing the auditor with automation assistance but allowing judgment to deviate from the rules. The extent of automation can vary to meet auditors’ needs and is not limited only to the mention mode of rule-based systems. Effectiveness and efficiency can be evaluated in terms of accuracy, coverage, false positives, false negatives, processing time, and resource utilization.
Copyright (c) 2026 European Advanced Journal for Science & Engineering (EAJSE) https://creativecommons.org/licenses/by-sa/4.0
Collaborative Federated Perception for Autonomous Fleets
The growing interest in autonomous vehicle technology presents opportunities for manufacturers to pursue plans based on machine learning to develop their own self-driving solutions while maintaining their competitive edge. Collaborative artificial intelligence presents manufacturers with the possibility of cooperation without sharing private user data. Federated learning is a distributed machine-learning framework that enables multiple parties to cooperate on a common model while training locally on their own private data. Despite the similarities in the underlying technology, the application of federated learning for autonomous driving has attracted limited interest. Numerous challenges arise and need to be addressed: Different autonomous driving systems develop independently, and hence their models differ significantly. Vehicle manufacturers may use proprietary data formats that ensure the privacy of user data, and the necessary training data remain in the manufacturer domain. Communication efficiency is critical, as the resources of vehicles are constrained compared with fixed servers. Training data remain local, and agent-specific knowledge cannot be transferred directly. Collaborative autonomous driving powered by federated learning could fill the perceptual gaps during periods of limited visibility and ensure the safe crossing of busy intersections. A federated-learning framework has been developed that integrates several key components and techniques for training across a multi-manufacture federated vehicular network. It transmits only key information that preserves the privacy of the drivers while enabling safe autonomous driving in a limited-visibility environment. Powers and constraints differ across manufacturers, so establishing a top-level data flow is also necessary.
Embedding Governance into Autonomous Decision-Making Agents
Enterprises seek to harness emerging technologies such as generative AI to enable decision automation in business operations while maintaining governance, risk, and oversight. Existing architectures for building Autonomous Agents in Information Systems work well for narrow-band decision problems but fall short when adapting generative AI methods to solve problems requiring high governance and compliance risk mitigation. Existing ServiceNow implementations operate at ChatOps levels of automation but lack the otherwise available major platform functionality. Governance-Embedded Action-Policy Decision Intelligent architecture provides an end-to-end pipeline, embedded policy enforcement, built-in audit/traceability, and governance oversight of decoupled/parameterized ServiceNow Decision Intelligent Workflows. The concept is being explored and tested in a large enterprise with the necessary conditions for success. The initial focus is business-process auto-detection and classification, with connector integration into enterprise systems such as global knowledge repositories, tracking systems, and data quality monitoring. These endpoints provide situational updates for Business-Process Decision Intelligence reflex workflows.
The Rise of Self-Service Portals in Data Centers
Service Level Agreements (SLAs) are key contractual documents describing expected quality of service between a service provider and its customers. In the context of Data Center Provider Services, SLAs define electrical, mechanical, cooling, network, storage, security, and other attributes of the facilities. Both the provider and the customers share the responsibility for realizing the SLA objectives. Customers rely on their Data Center Providers to deliver the agreed-upon service quality and the Providers themselves utilize their own supporting suppliers and partners to ensure demand is met. A shortcoming of the prevailing SLA management paradigm is that it remains largely a suspenseful activity for customers. A complaint-handling, after-the-event exercise can have deep implications on their business. Allowing customers to self-service the entire life cycle and processes around their SLAs via an easy-to-use self-service portal enhances their experience and jointly drives quality with the Data Center Provider. Visualizing and presenting SLAs in an engaging manner for customers is critical. Self-service SLAs enable customer-initiated workflows around creation, amendment, monitoring (via dashboards), ticket requests, and approvals. There are numerous operational support functions that can be formalized between the Data Center Provider and customers such as demand forecasts, spare part reservations, service requests, operational reports, etc. Enabling these requestfulfilling, change management, and knowledge-management processes through a self-service portal further drive quality during the day-to-day life cycle. There are several specific considerations that require attention in a Data Center context. Customers expect real-time position and health of Data Center Services to ensure quality. Any breach of physical or environmental parameters will have a cascading effect on one or more SLAs. Multi-tenancy and customer isolation further add complexity.
Machine Learning-Based Computation of Transient Bed Profiles for cohesive channel bed
This study utilizes machine learning models, including Support Vector Machine (SVM), Gene Expression Programming (GEP), and Artificial Neural Network (ANN), to predict transient bed profiles in cohesive channel beds composed of clay-silt-sand-gravel mixtures with clay content ranging from 10% to 50%. By leveraging 4,200 experimental data points (70% for training and 30% for testing), dimensional analysis and Γ-test for sensitivity analysis were employed to identify key dimensionless parameters: clay percentage (Pc), normalized time (<!-- [if gte msEquation 12]><m:oMath><m:f><m:fPr><m:ctrlPr></m:ctrlPr></m:fPr><m:num><m:r><m:rPr><m:nor/></m:rPr>t</m:r></m:num><m:den><m:sSub><m:sSubPr><m:ctrlPr></m:ctrlPr></m:sSubPr><m:e><m:r><m:rPr><m:nor/></m:rPr>t</m:r></m:e><m:sub><m:r><m:rPr><m:nor/></m:rPr>e</m:r></m:sub></m:sSub></m:den></m:f></m:oMath><![endif]--> ), relative density (<!-- [if gte msEquation 12]><m:oMath><m:f><m:fPr><m:ctrlPr></m:ctrlPr></m:fPr><m:num><m:sSub><m:sSubPr><m:ctrlPr></m:ctrlPr></m:sSubPr><m:e><m:r><m:rPr><m:nor/></m:rPr>γ</m:r></m:e><m:sub><m:r><m:rPr><m:nor/></m:rPr>b</m:r></m:sub></m:sSub></m:num><m:den><m:sSub><m:sSubPr><m:ctrlPr></m:ctrlPr></m:sSubPr><m:e><m:r><m:rPr><m:nor/></m:rPr>γ</m:r></m:e><m:sub><m:r><m:rPr><m:nor/></m:rPr>w</m:r></m:sub></m:sSub></m:den></m:f></m:oMath><![endif]--> ), void ratio (e), moisture content (w), and normalized distance (<!-- [if gte msEquation 12]><m:oMath><m:f><m:fPr><m:ctrlPr></m:ctrlPr></m:fPr><m:num><m:r><m:rPr><m:nor/></m:rPr>x</m:r></m:num><m:den><m:sSub><m:sSubPr><m:ctrlPr></m:ctrlPr></m:sSubPr><m:e><m:r><m:rPr><m:nor/></m:rPr>L</m:r></m:e><m:sub><m:r><m:rPr><m:nor/></m:rPr>max</m:r></m:sub></m:sSub></m:den></m:f></m:oMath><![endif]--> ). GEP demonstrated superior performance, achieving the highest testing accuracy (R2=0.9955, RMSE=1.8456), followed by SVM (R2 = 0.9870, RMSE = 3.2480) and ANN (R2 = 0.9838, RMSE = 3.9143). Taylor diagrams and DDR analysis validated the robustness of Gene Expression Programming (GEP), demonstrating optimal correlation and minimal variance compared to observed data. GEP’s symbolic regression yielded interpretable equations using operators (+, -, *, /, x2, <!-- [if gte msEquation 12]><m:oMath><m:rad><m:radPr><m:ctrlPr></m:ctrlPr></m:radPr><m:deg><m:r>3</m:r></m:deg><m:e><m:r>x</m:r></m:e></m:rad></m:oMath><![endif]--> ), while SVM and ANN exhibited higher deviations for extreme values. This study emphasizes GEP's effectiveness in capturing nonlinear erosion dynamics, making it a reliable tool for predicting bed degradation in hydraulic engineering applications. The results highlight the significance of model interpretability and parameter selection, offering valuable insights for soil erosion management and sediment transport modeling.
Hazi Mohammad Azamathulla Hazi Mohammad Azamathulla
Numerically Modelling 2D Swash Zone Flows SMAC VS VOF
In recent years, numerically modelling the swash-zone has proven a feasible approach in unraveling the zone’s intricacies. Whilst many researchers have used and examined many swash-zone numerical models, no specific research has examined the comparative performance of the Simplified Marker and Cell (SMAC) method relative to the status quo; the Volume of Fluid (VOF) method, in 2D applications of the zone. This study investigates the performance of these numerical methods via comparative simulations of 2D dam-break swash like flows across both models. The results show that an adapted Finite Difference (FD)- Finite Volume (FV) discretization of the SMAC method with reductions in marker particles yields smooth interfacial predictions and clear convergence with mesh refinement, while being less computationally demanding than the VOF method for the test scenario. The study demonstrates the suitability of the SMAC method for simulating swash zone flows and highlights its potential for efficient and accurate 2D simulations.
Collaborative MARL for Real-Time Healthcare Decision Optimization
Adaptive Real-Time Healthcare Interoperability and Decision Optimization Applied multi-agent reinforcement learning to jointly optimize adaptive real-time data integration and uncertainty-aware decision support across multiple healthcare agents. A simulated healthcare environment for a COVID-19 treatment workflow served as a multi-agent reinforcement learning environment. Adaptive real-time interoperability and decision-making were achieved. An implemented clinical testbed demonstrated improved test results and indicated the potential for improved clinical workflows. Adaptive real-time interoperability and adaptive risk-aware decision support will improve clinical workflows and support decision quality, speed, and safety while reducing the effects of gaps in the data provided by other agents, thereby facilitating agents with different data modelling paradigms to interact more efficiently and also enhancing clinical workflows. Multiple innovations improve the usefulness and impact of multi-agent reinforcement learning on healthcare agents. The fundamental goal is to enhance the adaptive real-time interoperability and risk-aware decision-making of multiple healthcare agents in a simulated healthcare ecosystem modelled on a COVID-19 treatment workflow. Performance indications suggest that the framework achieves these goals. Automated and concurrent updates to both adaptive real-time semantic support and uncertainty-aware decision policies enable joint optimisation through multi-agent learning. The integration of message-based data transfer systems with pipeline architecture further enables multi-agent learning to address latency in the triggering of agents’ decision policies.
Intelligent Deployment Pipelines via MLOps
MLOps-enabled intelligent systems automate the deployment of machine learning pipelines that can exploit machine learning capabilities and pre-existing data across data platforms in multiple sectors, enabling automated decision-making with a known (soft) risk or loss function. MLOps concepts enable new forms of machine learning model lifecycle management that may leverage autoML and human-in-the-loop approaches for model development, training, and tuning; they facilitate the continuous operation of pipelines on a 24/7 basis; they address concerns such as privacy, security, model bias, transparency, explainability, interpretability, and inclusivity; and they provide mechanisms for automatic scaling, fault-tolerance, and infrastructure cost optimisation. Automated model deployment represents a major feature of MLOps. Automated deployment refers to the management of machine learning models and pipelines so that the entire lifecycle from development to deployment and operation may be governed from an MLOps perspective. When system performance is measured within a known range and against a soft loss or risk function, virtual data platforms that encompass sectors as diverse as finance, healthcare, retail, and telecommunications possess feature sets supporting such automations. Such automations focus on Machine Learning as a Service (MLaaS) deployments on a 24/7 basis, with the objective of enabling decision making within these soft constraints and scaling the deployment of autoML pipelines in tandem with demand for untrained services.
Governance-Driven MLOps Using Databricks Medallion Architecture
Integrating Data Governance and MLOps support within data management is a recent requirement accelerating in the current decade. Several factors drive the demand for data governance: regulatory compliance, trust in analytics and machine learning insights, the need for data reuse controlling operational risks, and adequate data quality. The multiple dimensions of data management complexity require specialization, with increasingly higher levels of reliability. A maturity model derived from cybersecurity concepts divides data management in data management and data science operations with horizontal and policy layers spanning the intensity and breadth of data management, relying on appropriate degrees of specialization are. MLOps methodologies are still maturing while the requirements from different domains push towards the definition of more specific solution, including most of the aspects demanded from modern-day data management operations. The integration of Data Governance and MLOps strategies for Machine Learning workloads is an evident necessity in the current panorama of data operations. The definition of the Bronze layer for the DataForge helps the coverage of Data Governance needs in Machine Learning operations, presenting a Medallion Architecture supported by Databricks tools able to provide end-to-end coverage of Data Governance Data Management processes and MLOps requirements from Data and Model development until Data and Models production.
AI-Driven Derivatives and Collateral Optimization
Intelligent Financial Integration Using AI for Derivatives and Collateral Optimization Commercial banks are significant intermediaries in the global derivatives market, generating substantial derivative revenues that require effective management to ensure stable and adequate earnings with reduced risk. Within banks, the Treasury function is responsible for collateral management and optimization across the group, including intra-group trades with other market-making businesses. Collateral optimization involves maximizing the utility of collateral through careful management and control of the Securities Financing Transactions (SFT) book with market counterparts—allowing Securities Financing Transactions for any reason during normal market conditions, minimizing funding costs, and being ready for stressed market conditions. AI can enhance decision-making and operational efficiency through automation of derivatives pricing, dynamics, P&L, back-testing, risk modelling, and liquidity management. Three types of microstructure decisions, supported by AI, can significantly improve trading profitability: optimally triggering market orders during volatility bursts; determining the best price for volume-exploratory limit orders; and screening and selecting limit orders positioned close to the market. AI can also optimize trade schedules using market meta-data extracted from liquidity provider flows. Profits from non-transparent or opaque strategies can be improved by early detection and exploitation of statistical arbitrage opportunities. Additionally, AI can detect arbitrage opportunities in collat-eral-required strategies and condition the trades to exploit minimum-risk scenarios, for example, using capital structure information from the Towersy et al. credit-suer model. Intelligent Financial Integration Using AI for Derivatives and Collateral Optimization Commercial banks are significant intermediaries in the global derivatives market, generating substantial derivative revenues that require effective management to ensure stable and adequate earnings with reduced risk. Within banks, the Treasury function is responsible for collateral management and optimization across the group, including intra-group trades with other market-making businesses. Collateral optimization involves maximizing the utility of collateral through careful management and control of the Securities Financing Transactions (SFT) book with market counterparts—allowing Securities Financing Transactions for any reason during normal market conditions, minimizing funding costs, and being ready for stressed market conditions. AI can enhance decision-making and operational efficiency through automation of derivatives pricing, dynamics, P&L, back-testing, risk modelling, and liquidity management. Three types of microstructure decisions, supported by AI, can significantly improve trading profitability: optimally triggering market orders during volatility bursts; determining the best price for volume-exploratory limit orders; and screening and selecting limit orders positioned close to the market. AI can also optimize trade schedules using market meta-data extracted from liquidity provider flows. Profits from non-transparent or opaque strategies can be improved by early detection and exploitation of statistical arbitrage opportunities. Additionally, AI can detect arbitrage opportunities in collat-eral-required strategies and condition the trades to exploit minimum-risk scenarios, for example, using capital structure information from the Towersy et al. credit-suer model.
Copyright (c) 2026 European Advanced Journal for Science & Engineering (EAJSE)
Machine Learning Meets Industrial Big Data
A concept of predictive intelligence, based on industry-scale big data and supervised machine learning models, is put forth, establishing the foundations of a Predictive Intelligence Framework designed to enable trusted predictions in specific application domains. Such predictions encompass various forecasting, anomaly detection, and binary/multi-class classification tasks. An objective, evidence-based analysis of how data, models, and governance serve to enable specific applications provides a framework for the architectural, engineering, and deployment choices required for trusted predictions in the context of real-world industry use cases. Like general artificial intelligence, industry-scale predictive intelligence encompasses multiple paradigms and tasks. These include not only the mini–brain models that perform high-level cognition on limited training data but also the predictive tasks for which data availability has reached industrial scale, enabling the use of Deep Learning on Data-Demanding Tasks. These data-driven systems are of particular interest to industry due to their inherent focus on prediction, given that prediction is often the key component of any task that requires the application of intelligence. Predictive Intelligence focuses on such tasks, whose execution in real-world scenarios has thus far remained limited owing to the need for substantial engineering effort.
Copyright (c) 2025 European Advanced Journal for Science & Engineering (EAJSE)
Agentic Intelligence for Real-Time Diagnosis Coding and Risk Calibration
Generative and multi-agent Artificial Intelligence (AI) solutions empower real-time clinical coding and risk adjustment in interoperable healthcare systems. Coding staff translate clinicians' patients and encounter information into billing codes to support health insurance reimbursement. Progressive Health Information Exchanges, however, make medically coded patient encounter data available to health decision-makers and public health for tracking care delivery and managing care costs. Timely Data-sharing requires coding to occur near the time of the patient encounter. This is a touted advantage of Electronic Health Record (EHR) systems but rarely realised in practice given the demands of real-time coding for clinical support, administration, and billing are beyond what's reasonable to expect coding staff to achieve. Generative AI are now making exciting strides in concurrent real-time coding by Cognizant stakeholders. Investment in coding health outcome and cost risk is now similarly in consideration with much at stake in terms of real outcomes, AI toxicity risk and viability of these health economic business models in support of these investment priorities. Clinical coding systems are uncontrolled user-specific serial processes. Management have limited means to incentivise share price-enhancing coding quality and accuracy, regulators likewise and it's simply too costly to validate intervention work-up procedures or to open new billing coding classes or categories on a case-by-case basis. Risk-adjusted health outcome truths are bootstrapped from patient populations enabled semantically-scanned interoperable data inventory of value 100%—a requirement for the sustainment of publicly funded health systems—implying a bad-enough health outcome event for a paid and closed population at risk ought-online to be equally bad-for-a-similarly-situated population placed one-event post. It is now the turn of Generative and Multi-Agent AI to afford real-time coding and the associated risk investment.
Reimagining Lakehouse Design for AI-Native Enterprise Systems
A comprehensive analysis explores the evolution of lakehouse architectures and their applicability for real-time AI-driven decision systems. State-of-the-art architectures for data ingestion and streaming processing, as well as a hybrid extension of the medallion framework, form the foundation for semantics-aware online feature engineering, high-availability model serving, and comprehensive drift monitoring and detection. Provenance tracking and preservation, consistency model selection and conflict handling, access and sharing control, privacy and compliance requirements, and benchmark construction for end-to-end performance evaluation are discussed. Case studies demonstrate how next-generation enterprise real-time decision systems satisfy at least all of data quality, freshness, and access control. Real-time AI capabilities are increasingly being adopted in enterprise systems for a variety of purposes, including customer experience enhancement, risk mitigation, fraud detection, and service optimization. AI solutions in these domains dynamically adapt using data generated in real time, relying on various online and near-real-time algorithms that systematically consume and produce data and decisions. Such solutions find their origins not only in classic fraud detection, recommender engines, and online bidding systems, but also in AI domains such as learning-to-rank, multi-armed bandits, reinforcement learning, transfer learning, deep reinforcement learning, online learning, and continual learning. Real-time feature engineering pipelines in these sectors address a wide range of challenges, such as content-based spam detection, sentiment analysis, stock market prediction, and stock price analysis.
Azure DevOps CICD for Enterprise Data Pipelines
Data and artificial intelligence (AI) models lack the governance and protective mechanisms that traditional applications enjoy through software engineering (DevOps) practices in industry. The complexities of the enterprise CI/CD pipelines, which span across source control, continuous integration, continuous delivery deployments, testing, monitoring for observability, blocking and telemetry for production, enterprise dashboards with pipelines and project health, for AI models and Data in Enterprise environments is presented. Data at Rest, Metadata and Smart data through automation that provide lineage and tracking automation for ML models are discussed. Cloud DevOps ecosystem built with Azure DevOps repository and Pipelines with extensions, which enables orchestration creation and data pipeline YAML templates that embrace best CI/CD practices guide is discussed with detailed indices for enterprise reference. The enterprise ecosystem provides seamless package management integrated with other Azure DevOps extensions and third-party services availability. The pipelines provide the ability to deploy to any environments read from the pipeline YAML config. The development of pipeline YAML is driven as per the extensions installed and project structure for easy consumption by developers with integrated documentation and additional metadata to help in easy maintainability. These pipelines templates can be used to build, test, and publish data artifacts in enterprise data pipeline, enterprise database automation.
Real-Time Data Quality Monitoring and Gating Frameworks in Cloud-Based Data Pipelines describes real-time data quality monitoring within cloud-based data pipelines using a triage and gating approach, and formulates the main objectives and research questions. Real-time data quality monitoring within cloud-based data pipelines is considered a necessary capability to mitigate, detect, and manage data quality issues. Data pipelines ingest, process, and publish streams of data potentially originating from many geographically dispersed sources and targeting multiple upstream and downstream consumers. An effect of these characteristics is that the best data cleaning options are seldom explored in advance and validated for effectiveness and efficiency. Data noise may, therefore, not be adequately controlled or reduced. Quality gate design principles are introduced, and the concept of streaming gatelets is proposed to support the deployment of micro gates able to monitor data streams and control their onward journey in the data pipeline. The method also defines thresholds for measurements, utilizes severity levels to trigger remedial actions, and supports the fast-track and stop-check gating strategies. Real-time data quality monitoring within cloud-based data pipelines is considered a necessary capability to mitigate, detect, and manage data quality issues. Data pipelines ingest, process, and publish streams of data potentially originating from many geographically dispersed sources and targeting multiple upstream and downstream consumers. An effect of these characteristics is that the best data cleaning options are seldom explored in advance and validated for effectiveness and efficiency. Data noise may, therefore, not be adequately controlled or reduced. Quality gate design principles are introduced, and the concept of streaming gatelets is proposed to support the deployment of micro gates able to monitor data streams and control their onward journey in the data pipeline. The method also defines thresholds for measurements, utilizes severity levels to trigger remedial actions, and supports the fast-track and stop-check gating strategies.
Copyright (c) 2024 European Advanced Journal for Science & Engineering (EAJSE)
Data governance encompasses defining the roles, responsibilities, and accountability needed to safeguard data assets; enabling access control and usage monitoring; and supporting policy-driven data discovery, classification, protection, and retention. The increasing pervasiveness of Artificial Intelligence (AI) in analytics, data science, and machine-learning workloads highlights the importance of managing sensitive information and complying with legal requirements regarding PII. Numerous regulations compel organizations to prevent PII breaches while allowing analytics. Third-party cloud data platforms simplify AI-driven data ecosystems but pose a risk of PII exposure when sensitive information is shared across multiple environments, including untrusted external entities. These issues can be addressed through automated data governance, using policy-driven workflows that define and enforce policies related to PII and data governance. Unity Catalog extends the data platform’s capability to manage metadata across multiple cloud-object-storage accounts, implementing policy-driven automation for PII compliance and data governance through two approaches. The first approach automates key management and PII mapping to Data Loss Prevention tags, independent of an Identity and Access Management cloud service. The second approach enforces policies defined in an external Identity and Access Management service, with the cloud data platform as a service consumer rather than an IAM vendor. Implementation details gleaned from an enterprise production environment illustrate how Unity Catalog can automate PII-data governance and PII-compliance workflows.
Rethinking Health Data as a Living National Fabric
Principle Foundations of Cloud-Based Data Fabrics: Data Virtualization and Fabric Principles Cloud-based services are revolutionizing storage and processing, breaking the strong coupling between applications, data, and infrastructures within a single organization. Different components can now be deployed in different geographic areas, in a flexible and cost-effective manner. A cloud-based architecture for data virtualization, sensitive-data-oriented, based on a fabric model, is presented. The architecture optimizes operational and system-recovery costs by leveraging a cloud resources-market-driven processing environment. A sensitive-data protection mechanism prevents sensitive data from being exposed even during data re-use. A system-cache approach based on a monitoring and statistical model mechanism improves system utilization and performance by offloading data from the system cache to cloud storage. Concepts, principles, phases, and building blocks of the fabric support any type of resource and information service from the cloud, and any resource-market-driven virtualization requirements that improve application-provisioning flexibility and processing cost. The presentation of National Cloud Platform for Data Sharing shows that Public and Private sectors maintain distributed data resources in Taiwan, but most national data cannot be integrated and shared easily, due to different forms, contents, and access mechanisms. Recently, strategy of Cloud Computing was proposed to solve resource-usage optimization and fault-tolerant problems. A Cloud Platform connecting the Information-resource-provide agencies by Cloud links is developed, so that their resources can be shared without unnecessary data-refining and –duplication processes. The platform optimizes last-mile data-sharing processes for data consumers and build a Cloud Sharing Marketplace for Information Resources.
Unified Trust Layers Across Sovereign Cloud Boundaries in Modern Banking
Secure Multi-Cloud Data Orchestration Frameworks for Digital Banking Ecosystems reflect the increasing demand for data services and complex workloads in the digital banking ecosystem. Service and data integration across clouds is a prominent requirement but sharing and cross-cloud geo-replication of sensitive data introduce security and privacy risks that control dispositions across cloud service providers do not adequately address. In this work, Security Data Orchestration in Multi-Cloud Banking Ecosystem proposes an architecture and a Data Orchestration in Multi-Cloud Banking Ecosystem Methodology for achieving a cloud-agnostic Control Plane. The comprehensive control set satisfies independent compliance frameworks and enables secure access to sensitive data while satisfying data policies, residency requirements, and regulatory considerations. Given the interest and demand for Banking-as-a-Service from within the finance industry, a clear implementation roadmap and criteria enable banking institutions to plan, design, and secure a Service “Orchestration” and/or “Data Orchestration” capability. Security Data Orchestration in Multi-Cloud Banking Ecosystem present the Trade-Offs for Centralized, Federated, and Hybrid Architectures of Data Orchestration in Multi-Cloud Banking Ecosystem and Information and Data Aspects of Control Plane.
Orchestrating Smart Movement Through Live Intelligence
The logistics and supply chain domain is a hotspot for Internet-of-Things (IoT) applications and Big Data. Logistics processes are increasingly instrumented, resulting in the generation of massive amounts of data. For humankind, this is an opportunity to achieve sustainability and efficiency increases through the development of Smart Logistics. The Big Data stream processing field offers tools to process data in near real-time and/or at large scales. Yet, the requirement for Smart Logistics is not merely the processing of Big Data streams, but rather the development of a Smart Logistics Operations System that continuously adapts to new information and subsystems. Such an application demands high availability, scalability, reliability, fault tolerance, and the ability to process streams of data in real-time, allowing learned knowledge to be applied instantaneously. Two key decisions must be made for building a stream system that supports these same hard requirements in terms of implementation and solution development. The processing unit, implemented in the Cloud or at the Edge, must guarantee the lowest possible delay, while the use of Microbatch or True Streaming Processing should provide the best response time feasible, without sacrificing, at least at the design stage, reliability and fault tolerance. A third concern, occurring in the Data Ingestion layer of the Smart Logistics stack, is the heterogeneous nature of the data sources and their integration into a common Near Real-Time stream for the application. Finally, the development of a Smart Logistics Operations System relies on true Real-Time Analytics that naturally support decision-making, since optimal strategy changes are executed and orchestrated in the Streams.
Copyright (c) 2023 European Advanced Journal for Science & Engineering (EAJSE)
AI-Enabled IoT Data Fusion for Digital Transformation
Automated digital transformation via AI-enabled streaming pipelines with IoT data fusion is a viable concept that fulfills companies' aspirations to innovate services and business lines. It eliminates substantial hurdles by automating the setup of data pipelines, orchestrating end-to-end solutions, accelerating data governance and quality improvement, and establishing closing feedback loops. Automation relies on creating pipelines for Internet of Things (IoT) data, driven by data sources and offering AI-backed services. Orchestration dynamically deploys AI models in the streaming layer, while data governance is facilitated by an integrating framework for data quality and data privacy and security assurance in streaming contexts. Real-life examples in industries such as manufacturing, energy, transportation and logistics, smart cities, and agriculture illustrate these points. Companies are investing heavily in technology to realize new services and business lines. However, the aspiration for digital transformation still depends on manually developing complex custom data pipelines, integrating AI-driven models, and ensuring data privacy and quality. Automated digital transformation through data fusions is an appealing concept. Automating all the content and process in the topic enables a company to adapt data pipelines required to provide these services dynamically, onboarding the new use cases at high speed and reducing costs. AI modelling, therefore, is required to be produced only once in the data sources, providing services in the pipelines without involving teams in developing reports and dashboards.