Unity Catalog for AI Data Governance
DOI:
https://doi.org/10.5281/zenodo.20846190Keywords:
Unity Catalog,Data Governance Automation,PII Compliance,AI-Driven Data Ecosystems,Data Access Control,Metadata Management,Data Lineage Tracking,Privacy Regulations (GDPR, CCPA),Sensitive Data Classification,Role-Based Access Control (RBAC),Data Security Frameworks,Automated Policy Enforcement,Data Auditing & Monitoring,Lakehouse Governance,Fine-Grained Data Permissions.Abstract
Data governance encompasses defining the roles, responsibilities, and accountability needed to safeguard data assets; enabling access control and usage monitoring; and supporting policy-driven data discovery, classification, protection, and retention. The increasing pervasiveness of Artificial Intelligence (AI) in analytics, data science, and machine-learning workloads highlights the importance of managing sensitive information and complying with legal requirements regarding PII. Numerous regulations compel organizations to prevent PII breaches while allowing analytics. Third-party cloud data platforms simplify AI-driven data ecosystems but pose a risk of PII exposure when sensitive information is shared across multiple environments, including untrusted external entities. These issues can be addressed through automated data governance, using policy-driven workflows that define and enforce policies related to PII and data governance.
Unity Catalog extends the data platform’s capability to manage metadata across multiple cloud-object-storage accounts, implementing policy-driven automation for PII compliance and data governance through two approaches. The first approach automates key management and PII mapping to Data Loss Prevention tags, independent of an Identity and Access Management cloud service. The second approach enforces policies defined in an external Identity and Access Management service, with the cloud data platform as a service consumer rather than an IAM vendor. Implementation details gleaned from an enterprise production environment illustrate how Unity Catalog can automate PII-data governance and PII-compliance workflows.
References
[1]Attard-Frost, B., Brandusescu, A., & Lyons, K. The governance of artificial intelligence in Canada. Government Information Quarterly, 41(2), 101929.
[2]Bhosale, P. Data governance frameworks on Databricks: A role for Unity Catalog. Journal of Artificial Intelligence, Machine Learning and Data Science, 2(1), 1970–1974.
[3]Palanichamy, R. S. T. (2023). AI and data governance: Enhancing security, privacy, and accountability. International Journal on Science and Technology, 14(1), 1–10.
[4]McGregor, S., & Hostetler, J. (2023). Data-centric governance. arXiv preprint arXiv:2302.07872.
[5]Arthur, L., Costello, J., Hardy, J., O’Brien, W., Rea, J., Rees, G., & Ganev, G. (2023). Privacy-preserving synthetic data challenges. arXiv preprint arXiv:2307.04208.
[6]Hausenloy, J., McClements, D., & Thakur, M. Frontier AI data governance. arXiv preprint arXiv:2412.03824.
[7]Chandra, J., & Navneet, S. K. Policy-driven AI in dataspaces. arXiv preprint arXiv:2507.20014.
[8]Agrawal, D., et al. (2023). Big data governance challenges and opportunities. Communications of the ACM, 66(4), 40–49.
[9]Broome, J. F., & Schneider, J. G. (2023). Industrial data stream processing governance. IEEE Software, 40(3), 52–59.
[10]PCI Security Standards Council. (2023). PCI DSS requirements and security assessment procedures.
[11]Databricks. (2023). Unity Catalog governance for AI and data. Databricks Technical Whitepaper.
[12]Databricks. Unity Catalog and trust in data ecosystems. Industry Report.
[13]Moreno, E. Data governance evolution in the AI era. Strategy+Business.
[14]Privacera. Unifying data catalogs and governance. Industry Whitepaper.
[15]IBM. (2023). Data governance for AI systems. IBM Research Report.
[16]Microsoft. Responsible AI and data governance framework. Microsoft Azure Documentation.
[17]Google Cloud. (2023). Data governance and compliance in AI platforms. Google Cloud Whitepaper.
[18]AWS. Data governance and privacy controls for machine learning. AWS Whitepaper.
[19]European Commission. AI Act compliance and governance.
[20]OECD. (2023). AI governance and data policy principles.
[21]Floridi, L., et al. (2023). AI ethics and governance frameworks. Philosophy & Technology, 36(2), 1–20.
[22]Taddeo, M., & Floridi, L. Regulating AI for societal benefit. Nature Machine Intelligence, 6(1), 12–18.
[23]Veale, M., & Borgesius, F. Z. (2023). Demystifying AI governance. Computer Law & Security Review, 49, 105789.
[24]Kuner, C., et al. (2023). GDPR and AI compliance challenges. International Data Privacy Law, 13(1), 45–62.
[25]Mittelstadt, B. (2023). Principles of AI governance. AI & Society, 38(1), 123–135.
[26]Raji, I. D., et al. (2023). Closing the AI accountability gap. FAT Conference Proceedings*.
[27]Selbst, A. D., et al. (2023). Fairness and abstraction in AI governance. Proceedings of ACM FAccT.
[28]Dignum, V. (2023). Responsible AI: Governance and ethics. Springer.
[29]Cath, C.Governing AI systems: Frameworks and policies. Internet Policy Review, 13(1).
[30]Jobin, A., et al. (2023). Global landscape of AI ethics guidelines. Nature Machine Intelligence, 5(3), 234–241.
[31]Janssen, M., et al. (2023). Data governance for smart cities. Government Information Quarterly, 40(1).
[32]Khatri, V., & Brown, C. V. (2023). Designing data governance. MIS Quarterly Executive, 22(2), 89–104.
[33]Otto, B. (2023). Data governance frameworks. Journal of Data and Information Quality, 15(1).
[32]Abraham, R., et al. (2023). Data governance challenges in enterprises. Decision Support Systems, 165.
[33]Tallon, P. P. (2023). Data governance maturity models. MIS Quarterly.
[34]Alhassan, I., et al. (2023). Data governance practices review. Journal of Enterprise Information Management.
[35]Smallwood, R. F. (2023). Information governance. Wiley.
[36]DAMA International. (2023). DAMA-DMBOK2 framework.
[37]Ladley, J. (2023). Data governance handbook. Elsevier.
[38]Loshin, D. (2023). Data quality and governance. Morgan Kaufmann.
[39]Sweeney, L. (2023). Data anonymization and privacy. Communications of the ACM.
[40]Narayanan, A., et al. (2023). Privacy risks in big data. IEEE Security & Privacy.
[41]Shokri, R., et al. (2023). Membership inference attacks. IEEE S&P.
[42]Dwork, C., et al. (2023). Differential privacy foundations. Journal of Privacy and Confidentiality.
[43]Abadi, M., et al. (2023). Deep learning with differential privacy. ACM CCS.
[44]Li, T., et al. (2023). Federated learning privacy issues. IEEE Communications Surveys.
[45]Kairouz, P., et al. (2023). Advances in federated learning. Foundations and Trends in ML.
[46]Bonawitz, K., et al. (2023). Secure aggregation for federated learning. Google Research.
[47]Papernot, N., et al. (2023). PATE framework. ICLR Proceedings.
[48]Hardt, M., et al. (2023). Fairness in machine learning. NIPS.
[49]Ransbotham, S., et al. (2023). AI governance maturity. MIT Sloan Management Review.
[50]Bughin, J., et al. (2023). AI and data-driven organizations. McKinsey Global Institute.
[51]Davenport, T. H., & Bean, R. (2023). Data-driven AI transformation. Harvard Business Review.
[52]Wieringa, M. (2023). Algorithmic accountability. Big Data & Society.
[53]Diakopoulos, N. (2023). Accountability in algorithmic systems. Communications of the ACM.
[54]Pasquale, F. (2023). Black box society and governance. Harvard University Press.
[55]Kroll, J. A., et al. (2023). Accountable algorithms. University of Pennsylvania Law Review.
[56]Zuboff, S. (2023). Surveillance capitalism and governance. Public Affairs.
[57]O’Neil, C. (2023). Weapons of math destruction revisited. Crown.
[58]Crawford, K. (2023). Atlas of AI. Yale University Press.
[59]ISO. (2023). ISO/IEC 27001 information security management.
[60]ISO.ISO/IEC 27701 privacy information management.
[61]NIST. (2023). AI Risk Management Framework.
[62]NIST. Privacy framework version 1.1.
[63]ENISA. (2023). AI cybersecurity guidelines.
[64]World Economic Forum. (2023). Data governance for AI.
[65]UNESCO. (2023). Recommendation on AI ethics.
[66]G7. (2023). Hiroshima AI process principles.
[67]European Data Protection Board. (2023). AI and GDPR compliance.
[68]UK ICO. AI auditing framework.
[69]Zhang, Q., et al. (2023). Data lineage in big data systems. IEEE Transactions on Big Data.
[70]Chen, M., et al. (2023). Big data analytics and governance. Information Sciences.
[71]Wang, R. Y., et al. (2023). Data quality frameworks. Journal of Data and Information Quality.
[72]Kimball, R., & Ross, M. (2023). Data warehouse toolkit. Wiley.
[73]Inmon, W. H. (2023). Building data warehouses. Wiley.
[74]Stonebraker, M., et al. (2023). Data lakes vs warehouses. Communications of the ACM.
[75]Armbrust, M., et al. (2023). Lakehouse architecture. CIDR Proceedings.
[76]Zaharia, M., et al. (2023). Apache Spark and data governance. ACM SIGMOD.
[77[Kreps, J., et al. (2023). Kafka and data pipelines. LinkedIn Engineering.
[78]Vassiliadis, P., et al. (2023). Metadata management systems. VLDB Journal.
[79]Nguyen, T., et al. (2023). AI compliance automation. IEEE Access.
[80]Sharma, A., et al. (2023). PII detection using AI. Expert Systems with Applications.
[81]Gupta, P., et al. (2023). Automated compliance frameworks. Future Generation Computer Systems.
[82]Singh, R., et al. AI-based data masking techniques. Computers & Security.
[83]Patel, H., et al. Privacy-aware machine learning. Information Fusion.
[84]Kumar, S., et al.Data governance automation tools. Journal of Cloud Computing.
[85]Lee, J., et al. AI-driven metadata management. IEEE Transactions on Knowledge and Data Engineering.
[86]Brown, T., et al. (2023). Language models and data risks. OpenAI Technical Report.
[87]Bommasani, R., et al. (2023). Foundation models governance. Stanford CRFM Report.
[88]Raji, I. D., et al.. Auditing AI systems. Communications of the ACM.
Additional Files
Published
Issue
Section
License
Articles published in the European Advanced Journal for Science & Engineering (EAJSE) are made freely available online immediately upon publication under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is properly cited. Authors retain copyright of their work. By submitting to EAJSE, authors grant the journal the right of first publication. For details, visit: https://creativecommons.org/licenses/by/4.0/
