Online Experimentation and AB Testing

This course introduces online experimentation and A/B testing, with the statistical fundamentals needed to interpret results and recognize common pitfalls. It covers experiment metrics, sample size, segmentation, and applications to web traffic and email.

Syllabus

Intro
Agenda
Background in Clinical Trials
What is A/B Testing?
Obama 2012 Campaign
Testing Call-to-Action Button
Testing Navigation Bar
Jocelyn or Michael?
Away Find - Mobile notifications for priority messages
Online Form
CALIFORNIA CLOSET
Why We Use A/B Testing
Multi-Variate Testing
A/B Testing vs Multivariate Testing
Control and Treatment
Metrics Used For A/B Testing
Null vs Alternate Hypothesis
Type I and Type II Error
The Cook and Smoke Detector
The Boy Who Cried Wolf
Confidence Interval Intuition
Calculating Confidence Interval
Steps in Experimentation
Categories of Metrics
A/B Testing Tools
Questions?