Navigating the Complexities of Self-Hosted Observability
As systems grow in complexity, so too does the challenge of maintaining observability – the ability to understand a system’s internal state from its external outputs. At QCon London 2026, Colin Douch, Site Reliability Engineer at DuckDuckGo, addressed the growing pains of observability stacks, advocating for a cautious approach to self-hosting and emphasizing the importance of a coherent telemetry pipeline.
The Paradox of Observability
Douch began by acknowledging a common frustration: observability tooling, intended to simplify debugging, can quickly become as complex as the systems it monitors. While many organizations opt for Software-as-a-Service (SaaS) observability solutions, Douch’s session focused on the realities and challenges of building and operating observability infrastructure internally.
Should You Self-Host?
Douch offered a stark warning: “Should I run my own Observability stack? No, at least not until you have exhausted each and every other option.” He stressed that self-hosted observability requires significant investment – at least two to three full-time engineers and substantial financial resources – before even beginning to build a functional system.
Components of a Self-Hosted Stack
Despite the challenges, Douch outlined the typical components of a self-hosted observability stack. For metrics, he recommended Prometheus, acknowledging its horizontal scaling limitations, or VictoriaMetrics as alternatives. He also highlighted the underutilization of exemplars – a feature in modern metrics systems that links metrics to specific instances of events.
For logs, Douch suggested structuring them and storing them in a columnar database like VictoriaLogs or Loki, noting their differing design philosophies. He cautioned against directly ingesting logs into the database without proper processing, warning that “Sprinkling in logs leads to a soup of unusable data that makes it nigh on impossible to solve problems.”
Tracing and OpenTelemetry
Douch addressed distributed tracing, stating that “traces are just a fancy name for logs with some pre-agreed structure.” He endorsed OpenTelemetry for tracing, despite its complexity, but advised against using it for metrics or logs, favoring Prometheus Text Exposition and JSON instead.
The Importance of Coherent Telemetry
A key takeaway from Douch’s presentation was the need to move beyond treating logs, metrics, and traces as separate entities. He argued that the true value of observability lies in connecting these signals, recognizing that logs are a subset of traces, and metrics are aggregations of the same underlying data. He emphasized that building an observability platform is less about selecting a single tool and more about designing a coherent telemetry pipeline.
Sampling and Collectors
Douch also reviewed sampling techniques, discussing the advantages and disadvantages of head and tail sampling, and the role of collectors in the observability pipeline.
Colin Douch currently works as an SRE at DuckDuckGo, orchestrating solutions for their growing portfolio of services, including search queries and AI chats. He previously led the Observability Team at Cloudflare, bringing nearly a decade of experience in monitoring and observability to his work.