Provenance Research in Data Management Systems at Microsoft

Explore key developments in enterprise-level data provenance through this 44-minute conference talk from the Dutch Seminar on Data Systems Design. Dive into Microsoft's Gray Systems Lab research on provenance capture, querying, and applications, featuring detailed discussions of OneProvenance and DSProvenance engines. Learn about OneProvenance's implementation in Microsoft Purview for extracting provenance from Azure SQL, and understand how DSProvenance handles both static and dynamic provenance in data science pipelines. Discover PurviewQL, a SQL-based solution for simplified provenance and metadata management in data catalogs. Gain insights into practical applications of provenance across various domains including query optimization, job scheduling, semantic type inference, code synthesis, and data quality. The presentation covers essential aspects of connecting datasets, generation workflows, and metadata for enterprise governance, auditing, and observability purposes.

Syllabus

We hold bi-weekly talks on Fridays from PM to 5 PM CET for and by researchers and practitioners designing and implementing data systems. The objective is to establish a new forum for the Dutch Data Systems community to come together, foster collaborations between its members, and bring in high-quality international speakers. We would like to invite all researchers, especially also Ph.D. students, who are working on related topics to join the events. It is an excellent opportunity to receive feedback early on from researchers in your field.