Velocity, Variety & Volume - Gaining Insights from Your Logs at Scale

Explore log management at scale in this GOTO Copenhagen 2017 conference talk. Dive into handling high-velocity, high-variety, and high-volume log data for security, failure detection, and analytics. Learn about Kafka, microservices, event formats, and data models. Discover DevOps methodologies, behavioral detection techniques, and ransomware detection strategies. Understand the Pyramid of Pain concept and how to leverage logs effectively. Gain insights on machine learning applications, GDPR considerations, and handling sensitive data in log management. Follow the journey from Elastic to Greylog and explore open-source solutions for managing logs at scale.

Syllabus

Introduction
What is a Region
Why Log Management
What is a Log
Requirements
What is Kafka
Kafka Consumer Groups
Microservices
Interesting things
Event format
Count
Common denominator
Vector Selector
Time formats
Data model
DevOps methodologies
Homeo
Behavioral Detection
Ransomware Detection
The Pyramid of Pain
Using Logs
Kafka
Machine Learning
GDPR Considerations
Handling Sensitive Data
Open Source
Moving from Elastic to Greylock