Enriching the Data vs Filtering in Apache Spark

Explore a conference talk that delves into the comparison between enriching data and filtering in Apache Spark. Learn about a loyalty use case at Capital One and discover the advantages and disadvantages of both approaches. Understand the issues with the filtering approach and how enriching data can provide a more efficient solution. Gain insights into practical examples of both methods and their implementation in Spark. Conclude with a summary of the key takeaways and learn about the speaker's background in data engineering.

Syllabus

intro
preamble
capitalone
agenda
loyalty use case in capitalone
filtering the data approach
filtering approach example
issues with filtering approach
enriching the data approach
enriching approach example
advantage of enriching over filtering
conclusion
about gokul
thank you!