MySQL Automation at Facebook Scale

This talk examines automation systems for managing Facebook’s large MySQL cluster across multiple data centers. It covers inventory management, automated provisioning and promotions, centralized resource locking, server lifecycle management, local triage, and maintenance at scale.

Syllabus

Intro
What is this talk about?
Terminology
Inventory Management
Directory Service
Automated Provisioning
Automated Promotions
Central Resource Locking
Server's Lifecycle at Facebook
Full Disk
Bad RAID
Old MySQL Version
Metadata Database
Design Overview
Local Triage -MPS' Agent
Main Components
mps, examples
Picking a destination
Chosen Algorithm
Result: Pretty Graphs
Turning Up New Servers
Maintenance at Scale
The Case of the Helpful Janitor
REPLACE ALL THE THINGS!
Prepare for takeoff!
What happened?