Where's All My Memory Gone? Mapping Kubernetes Memory Metrics to Physical Resources

This conference talk delves into the critical understanding of Kubernetes memory metrics for effective application management and avoiding Out-Of-Memory (OOM) killer issues. Decode the complexities behind metrics like container_memory_working_set_bytes by following their journey from the kubelet binary to the host's memory control groups. Learn how memory metrics observed through kubectl top or Prometheus are processed through a complex path involving memory control group statistics, calculations by libraries such as opencontainers/libcontainer, cAdvisor or container runtime, and finally the kubelet. Explore the role of cAdvisor and container runtime in memory tracking, understand their interaction with the OOM killer, and discover how control groups (cgroups) versions affect metric calculations. By the end of this 28-minute presentation from Mahé Tardy of Isovalent at Cisco, gain the knowledge to better interpret memory statistics and troubleshoot memory-related issues in Kubernetes clusters.