Building a Serverless Database on S3 - Lessons from Man Group

Learn how to build a high-performance time-series database directly on object storage without traditional server overhead in this 42-minute conference talk. Discover how Man Group, one of the world's largest hedge funds, replaced a massive MongoDB cluster with a "thick client" architecture capable of streaming data at 40GB/s. Explore the challenges of managing hedge fund data at scale and understand why MongoDB's traditional approach created an "arms race" with users. Examine what constitutes a truly serverless database and master techniques for building atomicity and versioning on object storage systems. Analyze performance bottlenecks including CPU, I/O, and Python overhead considerations. Investigate the global state problem when managing millions of keys and dive deep into Set CRDTs, comparing grow-only versus observed-remove approaches. Understand why distributed locks can become problematic and explore future possibilities with atomic operations and conditional writes in S3. Gain insights into retrofitting metadata and compaction strategies through the Q&A session, making this essential viewing for software architects working with distributed systems, database design, and serverless architectures.