Why We Built Our Own Distributed Column Store

This talk explains how Honeycomb built Retriever, a custom distributed data store for real-time queries over large event volumes. It covers Kafka ingestion, column-oriented disk storage, distributed reads, multi-tenancy, quota management, fault tolerance, and recovery.

Syllabus

Intro
Please meet Retriever
Retriever is a special purpose data store
What is Honeycomb?
How Honeycomb works
Honeycomb under the hood
Our requirements
Requirements - summary
Retriever at a glance
Retriever compared to Scuba
Architecture - write path
Architecture - read path
Data model - datasets
Data model - events
Row oriented storage
Column oriented storage
Storage Format - timestamp column
Storage Format - reading
Distributed queries
Distributed reads - calculations
Distributed reads - fanout
Detour - Kafka
Ingestion
Quota management
Fault tolerance
Failure recovery
Bootstrapping new nodes