AB-Tests, Lies, Damned Lies, and Statistics

This session explores ways to build A/B tests into a codebase, assess whether changes have a significant effect, and combine experiments with user testing. It also addresses how to test and validate data before relying on it for decisions.

Syllabus

Introduction
Who am I
ABtestdriven development
Doing the right thing
Biases
ABtest
Randomness
Questioning
Making better decisions
KPIs
Page Views
User Testing
Optimizely
Statistics
User Journey
Learning to make better decisions
The naive approach
Dividing concrete
Buckets
Bucket ID
Bucket Cookie
Bucket IDs
Google Analytics
Dashboards
histogram
tracking all tests
red button test
negative testing
nobody uses the data
focus on the wrong metrics
name your data
sampling
significance metrics
broken code
proxy testing
browser mode proxy
selenium
curve for change
double verification
subjectivity
quantify everything
be context aware
feel your curiosity
testdriven development
express development