Spark Operator - Feature Engineering with Spark on Kubeflow

Learn to transform messy real-world data into machine learning-ready features using Apache Spark with the Kubeflow Spark Operator in this 33-minute conference talk. Discover how to handle diverse data inputs including PDFs, scanned documents, images, ZIP files, and enterprise warehouse data, processing hundreds of terabytes using fully open-source tools. Explore the integration of Apache Spark with Kubeflow Pipelines to bridge the gap in extracting actionable insights from massive volumes of raw data. Master the orchestration of feature engineering workflows through the Kubeflow Spark Operator, addressing real-world machine learning challenges where clean tabular data is rarely available. Gain practical insights into scaling data processing and feature extraction for production ML systems using cloud-native technologies. This session targets data and ML engineers with basic Spark or Kubernetes experience who want to enhance their feature engineering capabilities in cloud-native environments.