A/B Testing: A Complete Guide

Course Curriculum

Introduction (3 Lessons, 16 min)

• A Warm Welcome
• Guess the Test: Which Variant Won?
• The 8-Step Process for Successful A/B Tests

Avoid the Silent Killers of Good A/B Tests (9 Lessons, 61 min)

• Is A/B Testing What I Need?
• You’re Asking Why Instead of What Happened?
• You Have Low Traffic
• You’re Comparing More Than Two Variables
• Your Audience Definitions are Vague
• You Don’t Have Concrete and Measurable Outcomes
• You Want to Conduct Long-Term Usability Studies or Do Longitudinal Research
• You’re Still Prototyping
• Eight Reasons to Run an A/B Test

Choose Impactful Design Ideas to Test (6 Lessons, 72 min)

• How to Discover Opportunities Where A/B Testing Will Create Value
• Three Problematic Prioritization Methods for A/B Testing (PIE, ICE, RICE)
• The Prioritization Method I Actually Use With Clients
• The Best A/B Testing Hypothesis Formula in the World
• Define Your Success Metrics: Key Performance Indicators (KPIs) in A/B Testing
• Two Examples of Excellent A/B Testing Hypotheses

Determine Sample Size and Testing Duration (4 Lessons, 40 min)

• Visually Design Your Test for Success
• Determine a Statistically Rigorous Sample Size
• How to Run a Properly Timed A/B Test
• Don’t Peek! It’s Far Too Risky

Select Your Testing Platform Carefully (2 Lessons, 28 min)

• The Six Questions to Consider When Selecting an A/B Testing Platform
• How to Set Up and Run Your A/B Test: A Glimpse Inside the Convert Platform

Analyze and Present Compelling A/B Test Results (7 Lessons, 42 min)

• The Three Possible Test Outcomes
• How to Call Your Tests
• Statistical Significance Clearly Explained
• Twyman’s Law and the Winner’s Curse
• Sample Ratio Mismatch (SRM): The Invisible Threat
• Know Your Audience and Speak Their Language
• Wrap-Up and Conclusion