Unit Testing Data with Deequ - Data Quality Validation Using Apache Spark

Learn about data quality testing in production environments through this 39-minute conference talk from Data Science Conference Europe 2023. Explore how Deequ, a declarative API system, enables efficient data validation using Apache Spark, similar to software engineering unit testing practices. Discover why data volume and cleanliness have become paramount in modern business operations, surpassing the importance of model tuning. Understand the risks of using customers as testing frameworks and how to implement proper data validation strategies. Gain insights into handling growing datasets through incremental validation techniques, ensuring data quality remains consistent across business processes and decision-making systems.