Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
Explore a 31-minute InfoQ talk where Josh Wills, a software engineer at Slack, delves into the company's data architecture and pipeline development strategies. Learn about Slack's technology stack evolution from a PHP monolith to a sophisticated system utilizing Hack-lang, HHVM, and Java/Go binaries, processing hundreds of thousands of events per second through Kafka clusters. Discover how Slack approaches machine learning pragmatically, focusing on practical applications like Learn to Rank (LTR) for search improvement, while maintaining a philosophy of building only what's necessary. Gain insights into their observability practices, emphasizing structured data, tracing, and high cardinality events using tools like ELK, Prometheus, and Grafana. Understand their data infrastructure, which includes S3 storage, Hive metastore, EMR, Presto, Airflow, Snowflake for business analytics, and Quiver for key-value storage, along with valuable lessons on transitioning between IC and management roles in data engineering.
