Goodbye Distributed Locks - Message Orchestration at Scale with Apache Pulsar
StreamNative via YouTube
AI Adoption - Drive Business Value and Organizational Impact
Learn Backend Development Part-Time, Online
Overview
Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
Learn how to achieve mutual exclusivity in event-driven messaging platforms at massive scale through this conference talk that explores Attentive's architectural evolution from Redis-based distributed locks to Apache Pulsar's subscription modes. Discover how Attentive built a high-throughput messaging platform that successfully handled 3.9 billion SMS and emails during peak shopping events, including 620 million texts on Black Friday alone. Explore the critical challenges of user-level mutual exclusivity in preserving orchestration integrity and delivering consistent user experiences, while examining the limitations of Pulsar's SHARED mode that caused mutual exclusivity violations and "split-brain effects" due to concurrent event consumption. Understand the architectural journey from Redis distributed locks through Pulsar's FAILOVER mode to the final KEY_SHARED subscription implementation, including detailed analysis of trade-offs, operational hurdles, and real-world production incidents that shaped the decision-making process. Gain insights into practical strategies for mitigating head-of-the-line blocking, managing consumer stalling, optimizing throughput during peak loads, and implementing effective observability through distributed log tracing and custom metrics. Master the comparative analysis of different subscription modes (SHARED, FAILOVER, KEY_SHARED) with their respective pros and cons for ensuring mutual exclusivity and scalability in distributed event-driven systems, while learning about future architectural directions toward event-loop-based models and continuous event acknowledgment patterns.
Syllabus
Goodbye Distributed Locks: Message Orchestration at Scale with Apache Pulsar
Taught by
StreamNative