Using Scikit-Learn's Interface for Turning Spaghetti Data Science into Maintainable Software

Learn how to transform messy data science code into maintainable software using Scikit-Learn's interface in this EuroPython conference talk. Explore techniques for structuring number-crunching code, especially for data processing, cleanup, and feature construction. Discover how to leverage Scikit-Learn's estimator and composite classes to encapsulate complex machine learning models into a manageable tree of objects. Gain insights into simplifying model development, testing, and validation while combining best practices from software engineering and data science. Follow along with examples demonstrating how to use pipelines for chained transformations, apply the single-responsibility principle to estimators, and create serializable trained models. No prior knowledge of Scikit-Learn is required to benefit from this presentation on improving code structure and maintainability in data science projects.

Syllabus

Intro
Supervised Machine Learning
Training a Machine Learning Model
Training a model
Preparing data for the estimator
Testing out transformer
Composition using Pipelines: Chained transformations
Scikit-Learn offers transformers data preparation
Conclusions Use fit/predict and fit/transform interfaces. • Apply single-responsibility principle to estimators. - Think in small, testable units Have your complete trained model in a serialisable object.