Azure DevOps CICD for Enterprise Data Pipelines
DOI:
https://doi.org/10.5281/zenodo.20846274Keywords:
Enterprise data pipelines; CI/CD; Azure DevOps; data versioning; data-artifact promotion; telemetry and monitoring.Abstract
Data and artificial intelligence (AI) models lack the governance and protective mechanisms that traditional applications enjoy through software engineering (DevOps) practices in industry. The complexities of the enterprise CI/CD pipelines, which span across source control, continuous integration, continuous delivery deployments, testing, monitoring for observability, blocking and telemetry for production, enterprise dashboards with pipelines and project health, for AI models and Data in Enterprise environments is presented. Data at Rest, Metadata and Smart data through automation that provide lineage and tracking automation for ML models are discussed. Cloud DevOps ecosystem built with Azure DevOps repository and Pipelines with extensions, which enables orchestration creation and data pipeline YAML templates that embrace best CI/CD practices guide is discussed with detailed indices for enterprise reference.
The enterprise ecosystem provides seamless package management integrated with other Azure DevOps extensions and third-party services availability. The pipelines provide the ability to deploy to any environments read from the pipeline YAML config. The development of pipeline YAML is driven as per the extensions installed and project structure for easy consumption by developers with integrated documentation and additional metadata to help in easy maintainability. These pipelines templates can be used to build, test, and publish data artifacts in enterprise data pipeline, enterprise database automation.
References
[1]J. Humble and D. Farley, Continuous Delivery, Addison-Wesley, 2010.
[2]M. Fowler, “Continuous Integration,” 2006.
[3]P. Duvall et al., Continuous Integration: Improving Software Quality, 2007.
[4]N. Forsgren et al., Accelerate, IT Revolution, 2018.
[5]G. Kim et al., The DevOps Handbook, 2016.
[6]L. Bass et al., DevOps: A Software Architect’s Perspective, 2015.
[7]K. Beck, Extreme Programming Explained, 2000.
[8]M. Shahin et al., “Continuous Integration, Delivery and Deployment: A Systematic Review,” IEEE, 2017.
[9]E. Laukkanen et al., “Problems in CI Adoption,” ICSE, 2015.
[10]T. Chen, “Continuous Delivery: Huge Benefits, but Challenges Too,” IEEE, 2015.
[11]B. Fitzgerald and K. Stol, “Continuous Software Engineering,” JSS, 2017.
[12]M. Hilton et al., “Usage, Costs, and Benefits of CI,” ASE, 2016.
[13]D. Ståhl et al., “CI Practices in Industry,” JSS, 2017.
[14]A. Rahman et al., “CI Failures and Causes,” MSR, 2019.
[15]S. McIntosh et al., “Empirical Study of Build Systems,” ICSE, 2011.
[16]T. M. K. et al., “Automating Deployment Pipelines,” IEEE, 2014.
[17]A. Savor et al., “Continuous Deployment at Facebook,” ICSE, 2016.
[18]M. Shahin et al., “Deployment Pipelines: A Systematic Review,” IEEE, 2016.
[19]A. Gallaba et al., “CI/CD Security Issues,” ICSME, 2018.
[20]S. Neely and S. Stolt, “CI Maturity Models,” Agile Conf., 2013.
[21]Microsoft, Azure DevOps Documentation, 2019.
[22]Microsoft, “Azure Pipelines Overview,” 2020.
[23]J. Buckley et al., “Adopting Azure DevOps in Enterprises,” IEEE, 2020.
[24]R. Brito et al., “CI/CD in Microsoft Environments,” ACM, 2019.
[25]S. Gupta et al., “Enterprise DevOps using Azure,” Springer, 2020.
[26]P. R. et al., “Azure Pipelines for Cloud Deployment,” IEEE, 2020.
[27]M. Villamizar et al., “Cloud DevOps with Azure,” IEEE, 2015.
[28]A. L. et al., “DevOps Adoption in Microsoft Stack,” IEEE, 2018.
[29]K. S. et al., “Automating Builds using Azure DevOps,” IEEE, 2019.
[30]S. K. et al., “Azure DevOps Security Integration,” IEEE, 2020.
[31]M. Villamizar et al., “Infrastructure Cost Comparison in Cloud DevOps,” IEEE, 2015.
[32]P. Jamshidi et al., “Microservices and DevOps,” IEEE Software, 2018.
[33]L. Chen, “Continuous Delivery: State of the Art,” IEEE, 2015.
[34]X. Xu et al., “DevOps for Big Data Systems,” IEEE, 2018.
[35]S. Zhang et al., “CI/CD for Data Pipelines,” IEEE, 2019.
[36]A. S. et al., “Automation in Data Engineering Pipelines,” ACM, 2019.
[37]H. H. et al., “Big Data DevOps Pipelines,” IEEE, 2017.
[38]J. Bogner et al., “DevOps for Enterprise Systems,” IEEE, 2019.
[39]R. Lwakatare et al., “DevOps Adoption in Industry,” IEEE, 2016.
[40]J. Roche, “Adopting DevOps in Enterprise,” IEEE, 2013.
[41]M. Fowler, “Deployment Pipeline,” 2013.
[42]Jenkins Project, Jenkins User Handbook, 2018.
[43]GitLab, CI/CD Documentation, 2020.
[44]Atlassian, Bitbucket Pipelines Guide, 2019.
[45]CircleCI Docs, 2020.
[46]Travis CI Documentation, 2019.
[47]ThoughtWorks, GoCD Guide, 2018.
[48]Red Hat, OpenShift CI/CD Pipelines, 2020.
[49]AWS, CodePipeline Documentation, 2020.
[50]Google Cloud, Cloud Build CI/CD, 2020.
[51]S. Myrbakken and R. Colomo-Palacios, “DevSecOps Review,” IEEE Access, 2017.
[52]F. Angermeir et al., “Security Automation in CI Pipelines,” 2021.
[53]N. R. Mead et al., “Security in DevOps,” IEEE, 2018.
[54]J. R. Goodall et al., “Secure CI/CD Pipelines,” IEEE, 2019.
[55]A. Rahman et al., “Security Smells in CI,” IEEE, 2019.
[56]D. Pletea et al., “Security Issues in CI Systems,” 2014.
[57]OWASP, DevSecOps Guidelines, 2020.
[58]SANS Institute, Secure DevOps Practices, 2019.
[59]M. Behl and S. Behl, “Security Integration in DevOps,” 2020.
[60]G. McGraw, “Software Security in DevOps,” IEEE, 2016.
[61]B. Burns et al., Kubernetes: Up and Running, 2019.
[62]D. Merkel, “Docker: Lightweight Containers,” Linux J., 2014.
[63]Kubernetes Docs, CI/CD Pipelines, 2020.
[64]HashiCorp, Terraform Automation Guide, 2020.
[65]Packer Docs, 2019.
[66]AWS DevOps Whitepaper, 2020.
[67]Google SRE Book, 2016.
[68]Azure Architecture Center, 2020.
[69]M. Villamizar et al., “Containers in DevOps,” IEEE, 2016.
[70]S. Newman, Building Microservices, 2015.
[71]K. Schwaber and J. Sutherland, Scrum Guide, 2020.
[72]M. Cohn, Agile Estimating and Planning, 2005.
[73]D. Leffingwell, SAFe Framework, 2018.
[74]VersionOne, State of Agile Report, 2020.
[75]Puppet Labs, State of DevOps Report, 2019.
[76]Atlassian, Agile + DevOps Integration, 2020.
[77]IBM, DevOps Transformation Guide, 2019.
[78]Capgemini, DevOps Enterprise Adoption, 2020.
[79]Gartner, DevOps Trends Report, 2020.
[80]Forrester, CI/CD Adoption Trends, 2020.
[81]L. Crispin and J. Gregory, Agile Testing, 2009.
[82]K. Beck, Test Driven Development, 2003.
[83]Selenium Docs, 2020.
[84]NUnit Documentation, 2019.
[85]JUnit Documentation, 2018.
[86]Prometheus Monitoring Docs, 2020.
[87]Grafana Labs, 2020.
[88]ELK Stack Documentation, 2019.
[89]Nagios Monitoring Guide, 2018.
[90]Splunk DevOps Monitoring Guide, 2020.
Additional Files
Published
Issue
Section
License
Articles published in the European Advanced Journal for Science & Engineering (EAJSE) are made freely available online immediately upon publication under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited. Authors retain copyright of their work. By submitting to EAJSE, authors grant the journal the right of first publication. For details, visit: https://creativecommons.org/licenses/by/4.0/
