There is no faster way to grow as an engineer than spending an evening with a debugger and a project you depend on every day. Here are the libraries that taught us the most and the exact commits worth bookmarking.
Internal tools tend to rot the moment they leave the founding team. We rebuilt our admin from scratch around three principles: every action should be auditable, nothing should require a tribal-knowledge runbook, and there should be exactly one button that resolves the on-call page.
Five years ago this would have been a controversial post. Today we have battle-tested operators, decent storage classes and a community that has finally agreed on backup tooling. Here is the stack we landed on for a fleet of around forty clusters.
Continuous profiling sounds great until you turn it on in a fleet of three hundred pods and your p99 latency doubles. Here is how we instrumented pprof endpoints behind a feature flag, sampled at one percent and shipped flame graphs straight to Grafana.
Every team eventually discovers the joy of shipping behind a flag. Fewer teams discover the misery of cleaning up flags two years later when the original author has long since left the company. A short rant and a longer guide.