High Throughput with Low Resource Usage - A Logging Journey

This session examines performance challenges in Fluent Bit logging pipelines at scale. It covers network I/O and TLS, filesystem buffers, routing, and multiplexing, and describes design changes that increased throughput using Linux interfaces.

Syllabus

Introduction
Performance vs Cost
Fluency
Logging
Debug Mode
Logging Workflow
Data Analysis
IO
Data Processing
Output
Pipeline
System Code
Processing
Routing
Network Setup
JSON vs Message Pack
Data serialization
Memory serialization
Memory buffering
Demo