Concepts on Advanced Beginner

Three Pillars of Observability

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers new to Observability concepts Prerequisites: Basic understanding of web application architecture After Reading: You’ll understand the role of each pillar and know when to use which

TL;DR#

Key Summary:

Metrics: “How much?” - Numerically measurable states (CPU 80%, response time 200ms)

Logs: “What happened?” - Detailed records of individual events

Traces: “From where to where?” - Tracking the entire path of a request

The 3 pillars are complementary, most effective when used together

Why Are All Three Pillars Necessary?#

Analogy: A Doctor’s Diagnostic Process

Metrics Fundamentals

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers designing Prometheus metrics for the first time Prerequisites: Three Pillars of Observability After Reading: You’ll be able to select the appropriate metric type and implement it correctly

TL;DR#

Key Summary:

Counter: Only cumulative increase (request count, error count) → Calculate rate of change with rate()

Gauge: Current value (temperature, memory) → Use as-is or avg() for average

Histogram: Distribution measurement (response time) → Calculate percentiles with histogram_quantile()

Summary: Percentile calculation on client (rarely used)

Why Are Metric Types Important?#

Metric types are not just a technical choice. They reflect the nature of the data.

Prometheus Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers who want to operate or deeply understand Prometheus Prerequisites: Metrics Fundamentals After Reading: You’ll understand Prometheus design philosophy and components, and be able to plan operational strategies

TL;DR#

Key Summary:

Pull Model: Prometheus fetches metrics from targets (not Push)

Time Series DB: Label-based multidimensional data model

Service Discovery: Auto-discover targets with Kubernetes, Consul, etc.

Single Server Design: Optimized for single server rather than horizontal scaling (extend with Federation)

Prometheus Overall Structure#

graph TB
 subgraph "Data Collection"
 T1["Target 1<br>/metrics"]
 T2["Target 2<br>/metrics"]
 T3["Target 3<br>/metrics"]
 PG["Pushgateway<br>(for batch jobs)"]
 end

 subgraph "Prometheus Server"
 R["Retrieval<br>(Scraper)"]
 TSDB["TSDB<br>(Time Series DB)"]
 HTTP["HTTP Server<br>(PromQL API)"]
 R --> TSDB
 TSDB --> HTTP
 end

 subgraph "Service Discovery"
 K8S["Kubernetes"]
 CONSUL["Consul"]
 FILE["File SD"]
 end

 subgraph "Alerting"
 AM["Alertmanager"]
 SLACK["Slack"]
 PD["PagerDuty"]
 end

 subgraph "Visualization"
 GF["Grafana"]
 end

 T1 --> |"pull"| R
 T2 --> |"pull"| R
 T3 --> |"pull"| R
 PG --> |"pull"| R

 K8S --> |"target list"| R
 CONSUL --> |"target list"| R
 FILE --> |"target list"| R

 TSDB --> |"alerting rules"| AM
 AM --> SLACK
 AM --> PD

 HTTP --> |"PromQL"| GF

Why Pull Model?#

Two Philosophies of Monitoring#

There are broadly two philosophies for collecting metrics.

Log Aggregation

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs designing log systems Prerequisites: Three Pillars of Observability After Reading: You’ll be able to select a log collection system and design effective logs

TL;DR#

Key Summary:

Loki: Label-based, lightweight, excellent Grafana integration

ELK: Powerful full-text search, suitable for large-scale analysis

Structured Logs: JSON format for easy field-by-field search

Log Levels: Long-term retention only for ERROR and above recommended

Why Is Log Aggregation Necessary?#

In microservices environments, applications run across dozens or hundreds of containers. If each container generates its own log file, which server’s log should you look at when an incident occurs?

Distributed Tracing

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs operating microservices Prerequisites: Three Pillars of Observability After Reading: You’ll understand distributed tracing and be able to analyze request flows between services

TL;DR#

Key Summary:

Trace: Entire path of one request (composed of multiple Spans)

Span: Single unit of work (start/end time, metadata)

Context Propagation: Passing Trace ID between services

Sampling: Store only a portion of total traces (cost optimization)

Why Is Distributed Tracing Necessary?#

In microservices, a single request passes through multiple services. It’s hard to identify where delays occur.

OpenTelemetry

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs standardizing observability systems Prerequisites: Three Pillars of Observability, Distributed Tracing After Reading: You’ll understand OpenTelemetry and be able to apply it to your projects

TL;DR#

Key Summary:

OpenTelemetry (OTel): Vendor-neutral standard for Metrics, Logs, and Traces

Components: SDK (instrumentation) + Collector (collect/transform/export)

Advantages: No vendor lock-in, instrument once to support multiple backends

CNCF Project: Second most active project after Kubernetes

What Is OpenTelemetry?#

OpenTelemetry is a standard framework for generating, collecting, and exporting observability data.

Dashboard Design

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs designing Grafana dashboards Prerequisites: SRE Golden Signals After Reading: You’ll be able to design effective dashboards and quickly identify problems

TL;DR#

Key Summary:

Hierarchical Structure: Overview → Service → Detail order

5-Second Rule: Must be able to identify problem presence in 5 seconds

Golden Signals First: Latency, Traffic, Errors, Saturation

Remove Unnecessary Info: Exclude metrics that don’t lead to action

Dashboard Design Principles#

1. 5-Second Rule#

Within 5 seconds of viewing the dashboard, you should know: