Mastering Real-Time Anomaly Detection with Open Source Tools

Learn to build a robust real-time anomaly detection system using open-source tools in this 41-minute conference talk from NDC Copenhagen 2025. Discover how to combine Apache Kafka for streaming data ingestion, Apache Flink for real-time processing, and ARIMA statistical models to detect unusual patterns as they occur. Explore the fundamentals of streaming data collection and management with Kafka, then dive into real-time data processing techniques using Flink integrated with ARIMA-based anomaly detection algorithms. Understand how ARIMA models work by examining their approach to modeling trends, seasonality, and residual noise to identify outliers in time-series data. Gain insights into using Apache Iceberg for efficient historical data storage, enabling retrospective analysis, continuous model evaluation, and performance optimization over time. Master the integration of real-time detection capabilities with long-term storage solutions to create an evolving system that adapts as your data grows. Follow practical examples and step-by-step guidance to implement your own anomaly detection pipeline suitable for fraud detection, system monitoring, IoT device tracking, and other real-time monitoring applications using time-tested statistical methods and reliable open-source technologies.