Go Profiling and Observability from Scratch

This talk builds a mental model of the Go runtime to explain its built-in profilers and runtime tracer. It also covers Linux perf, eBPF, Delve, and the overhead of profiling and tracing across several workloads.

Syllabus

Intro
Target audience
Agenda
Scheduling & Memory Management
Memory Management: Stack
Memory Management: Heap
CPU Profiler: SIGPROF for every 10ms of CPU Time
CPU Profiler: Add Stacktrace to Profile
CPU Profiler: Waiting for Go 1.18
Block Profiler: Channels
Block Profiler: Mutexes
Mutex Profiler
Block vs Mutex
Quick Note on Time: 1 Request - 3 Goroutines
Memory Profiling
Goroutine Profiling
Tracing Manually
Runtime Tracer
Profiling and Tracing: A Faustian Bargain?
Profiling and Tracing Overhead Analysis
Overhead Analysis: SQL Workload
Overhead Analysis: HTTP Workload
Overhead Analysis: JSON Workload
JsoN Workload (Concurrency = 8)
Overhead Analysis: Chan Workload
Overhead Analysis: Error Sources
Metrics
Linux perf
bpftrace
Scheduling / Execution Observability
Memory Management Observability
Recap