Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Hyperparameter Optimization for Machine Learning

via Train in Data

Overview

The most comprehensive online course on hyperparameter tuning for machine learning. You will learn multiple techniques to select the best hyperparameters and improve the performance of your machine learning models.

If you're disappointed for whatever reason, you'll get a full refund.

Sole is a lead data scientist, instructor and developer of open source software. She created and maintains the Python library for feature engineering Feature-engine, which allows us to impute data, encode categorical variables, transform, create and select features. Sole is also the author of the book "Python Feature engineering Cookbook" by Packt editorial.

Welcome to Hyperparameter Optimization for Machine Learning, the most comprehensive course on hyperparameter tuning available online. In this course, you will learn multiple techniques to select the best hyperparameters and improve the performance of your machine learning models.

Hyperparameters are parameters that are not directly learnt by the machine learning algorithm. They define and control the machine learning model, i.e., how flexible the model is to fit the training data, and they are calibrated to avoid over-fitting and improve generalization.

Some examples of hyperparameters are the regularization constants in linear models and support vector machines (SVMs), the number of estimators and the max-depth in decision trees, and the number of nodes or the learning rate in deep neural networks.

Hyperparameter tuning or hyperparameter optimization is the process of finding the best hyperparameter values for a given machine learning algorithm and a given dataset.

There are various hyperparameter optimization methods, including grid search, random search, and sequential models, usually involving Bayesian optimization.

Hyperparameter optimization algorithms consist of a search space, a search algorithm, a cross-validation scheme to find the optimal values while avoiding over-fitting, and an objective function with the classifier or regression model and the metric to optimize.

Throughout the tutorials, you will learn each and every aspect of the tuning methods.

In this course, you will learn multiple hyperparameter optimization methods to find the best set of hyperparameters for your classifier or regression models.

You will learn the following search algorithms:

We'll take you step-by-step through engaging video tutorials and teach you everything you need to know about to find the best combination of hyperparameters. Throughout this comprehensive course, we cover almost every available approach to optimize hyperparameters, discussing their rationale, their advantages and shortcomings, the considerations to have when using the technique, and their implementation in Python.

Syllabus

  •   Introduction
    • Introduction
    • Course curriculum
    • Course requirements
    • How to approach this course
    • How did you hear about us?
  •   Course materials
    • Course material
    • Jupyter notebooks
    • Presentations
    • Datasets
    • Complementary resources
  •   Hyperparameter Tuning - Overview
    • Parameters and Hyperparameters
    • Hyperparameter Optimization
    • Refer a friend
  •   Performance metrics
    • Performance Metrics - Introduction
    • Classification Metrics (Optional)
    • Regression Metrics (Optional)
    • Scikit-learn metrics
    • Creating your own metrics
    • Using Scikit-learn metrics
  •   Cross-Validation
    • Cross-Validation
    • Bias vs Variance (Optional)
    • Cross-Validation schemes
    • Estimating the model generalization error with CV - Demo
    • Cross-Validation for Hyperparameter Tuning - Demo
    • Special Cross-Validation schemes
    • Group Cross-Validation - Demo
    • Nested Cross-Validation
    • Nested Cross-Validation - Demo
    • Wrap-up
    • How are we doing?
    • Reading resources
    • Extra Treat: Our Reading Suggestion 📕
  •   Basic Search Algorithms
    • Basic Search Algorithms - Introduction
    • Manual Search
    • Grid Search
    • Grid Search - Demo
    • Grid Search with different hyperparameter spaces
    • Random Search
    • Random Search with Scikit-learn
    • Random Search with Scikit-Optimize
    • Random Search with Hyperopt
    • How are we doing?
    • Additional resources
    • Quiz
  •   Bayesian Optimization
    • Sequential Search
    • Bayesian Optimization
    • Bayesian Inference - Introduction
    • Joint and Conditional Probabilities
    • Bayes Rule
    • Sequential Model-Based Optimization
    • Gaussian Distribution
    • Multivariate Gaussian Distribution
    • Gaussian Process
    • Kernels
    • Acquisition Functions
    • Quiz
    • Additional Reading Resources
    • Scikit-Optimize - 1-Dimension
    • Scikit-Optimize - Manual Search
    • Scikit-Optimize - Automatic Search
    • Scikit-Optimize - Alternative Kernel
    • Scikit-Optimize - xgboost (optional)
    • Scikit-Optimize - Neuronal Networks
    • Scikit-Optimize - CNN - Search Analysis
    • How are we doing?
    • Added Treat: A Movie We Recommend
  •   Other SMBO Algorithms
    • SMBO - Using alternative surrogates
    • SMAC
    • SMAC Demo
    • Tree-structured Parzen Estimators - TPE
    • TPE Procedure
    • TPE hyperparameters
    • TPE - why tree-structured?
    • TPE with Hyperopt
    • Discussion: Bayesian Optimization and Basic Search
    • Additional resources
    • Quiz
    • Simulated annealing: improving the random search
    • Simulated annlearning - demo
  •   Multi-fidelity Optimization
    • Multi-fidelity Optimization
    • Successive Halving
    • Successive Halving - demo
    • Hyperband
    • Asynchronous Successive Halving
    • Additional resources
    • Quiz
    • More Wisdom: Our Chosen Podcast Episode 🎧
  •   Python Libraries for Hyperparameter Tuning
    • Python libraries for hyperparameter tuning
  •   Scikit-Optimize
    • Current state of Scikit-Optimize
    • Scikit-Optimize
    • Section content
    • Hyperparameter Distributions
    • Defining the hyperparameter space
    • Defining the objective function
    • Random search
    • Bayesian search with Gaussian processes
    • Bayesian search with Random Forests
    • Bayesian search with GBMs
    • Parallelizing a Bayesian search
    • Bayesian search with Scikit-learn wrapper
    • Changing the kernel of a Gaussian Process
    • Optimizing xgboost
    • Optimizing Hyperparameters of a CNN
    • Analyzing the CNN search
  •   Hyperopt
    • Hyperopt
    • Section content
    • Search space configuration and distributions
    • Sampling from nested spaces
    • Search algorithms
    • Evaluating the search
    • Optimizing multiple ML models simultaneously
    • Optimizing Hyperparameters of a CNN
    • References
  •   Optuna
    • Optuna
    • Optuna main functions
    • Section content
    • Required libraries
    • Search algorithms
    • Optimizing multiple ML models simultaneously
    • Optimizing hyperparameters of a CNN
    • Optimizing a CNN - extended
    • Evaluating the search with Optuna's built in functions
    • Storing multiple values during the search
    • Assignment: monitor different metrics during the search
    • Successive halving
    • Successive halving - demo
    • Hyperband
    • Assignment: SH with different budget options
    • Additional pruners
    • References
    • More examples
  •   Congratulations! You did it!
    • Congratulations
    • Next steps

Taught by

Soledad Galli

Reviews

Start your review of Hyperparameter Optimization for Machine Learning

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.