Appendix on Advanced Beginner

Glossary

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Quick Navigation: A | C | E | F | G | H | I | L | M | O | P | R | S | T | U | W

A#

Alerting Rules#

Rules in Prometheus that define condition-based alerts. When conditions are met, alerts are sent to Alertmanager.

Alertmanager#

A component that receives alerts from Prometheus and groups, inhibits, and routes them.

C#

Cardinality#

The number of unique time series. Higher cardinality occurs with more label combinations.

Context Propagation#

A mechanism for passing Trace ID and Span ID between services in distributed systems.

FAQ

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

General#

Q: What’s the difference between Monitoring and Observability?#

Monitoring: Watching predefined metrics (“Has this value exceeded the threshold?”)

Observability: The ability to understand the internal state of a system from the outside (“Why did this problem occur?”)

Monitoring is part of Observability. Observability includes the ability to analyze unexpected problems.

Q: Which of the Three Pillars (Metrics, Logs, Traces) should I implement first?#

Recommended order:

Metrics - Understand system state, set up alerts
Logs - Analyze error causes
Traces - Analyze distributed system flows

For a single service, Metrics + Logs may be sufficient. For microservices, Traces are essential.

Alerting Actions Guide

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Situational guide for responding after receiving alerts.

Alert → Action Workflow#

graph LR
 A["Receive Alert"] --> B["Assess Situation"]
 B --> C["Evaluate Impact"]
 C --> D["Immediate Action"]
 D --> E["Root Cause Analysis"]
 E --> F["Permanent Fix"]
 F --> G["Post-Incident Review"]

Response by Situation#

1. Service Down (TargetDown)#

Alert

alert: TargetDown
expr: up == 0

Immediate Actions

# 1. Check service status
kubectl get pods -l app=order-service
docker ps -f name=order-service

# 2. Check logs
kubectl logs -l app=order-service --tail=100
docker logs order-service --tail=100

# 3. Attempt restart
kubectl rollout restart deployment/order-service
docker restart order-service

Root Cause Analysis

References

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Official Documentation#

Prometheus#

Resource	Link	Description
Prometheus Official Docs	https://prometheus.io/docs/	Configuration, PromQL, operations guide
PromQL Reference	https://prometheus.io/docs/prometheus/latest/querying/basics/	Query language details
Alerting Rules	https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/	Writing alerting rules

Grafana#

Resource	Link	Description
Grafana Official Docs	https://grafana.com/docs/grafana/latest/	Dashboard, panel configuration
Loki Docs	https://grafana.com/docs/loki/latest/	Log collection/querying
Tempo Docs	https://grafana.com/docs/tempo/latest/	Distributed tracing

OpenTelemetry#

Resource	Link	Description
OpenTelemetry Official	https://opentelemetry.io/docs/	Concepts, SDK, Collector
Java Instrumentation	https://opentelemetry.io/docs/languages/java/	Java auto/manual instrumentation
Semantic Conventions	https://opentelemetry.io/docs/concepts/semantic-conventions/	Standardized attribute names

Books#

Essential#

Book	Author	Content
Site Reliability Engineering	Google SRE Team	SRE principles, Golden Signals, SLO
Observability Engineering	Charity Majors, Liz Fong-Jones	Modern observability concepts
The SRE Workbook	Google SRE Team	Practical SRE application

Recommended#

Book	Author	Content
Prometheus: Up & Running	Brian Brazil	Detailed Prometheus guide
Distributed Tracing in Practice	Austin Parker et al.	Advanced distributed tracing
Database Reliability Engineering	Laine Campbell, Charity Majors	Database observability

Blogs & Articles#

Prometheus/Grafana#

Prometheus Best Practices - Official recommendations
Grafana Blog - New features, case studies
Robust Perception Blog - Brian Brazil’s Prometheus tips

SRE/Observability#

Google SRE Books (Free) - Full content freely available
Honeycomb Blog - Advanced observability
Charity Majors’ Blog - Insights from observability leader

Videos#

Conferences#

Video	Link	Content
PromCon	https://www.youtube.com/@PrometheusIo	Prometheus conference
GrafanaCon	https://www.youtube.com/@Grafana	Grafana conference
KubeCon	https://www.youtube.com/@caborgg	Kubernetes, observability sessions

Tutorials#

Prometheus Tutorial for Beginners - TechWorld with Nana
Grafana Tutorials - Grafana official

Online Courses#

Course	Platform	Description
Prometheus & Grafana	Udemy	Hands-on focused
Site Reliability Engineering	Coursera	Google’s SRE course
Observability with OpenTelemetry	Linux Foundation	OTel introduction

Community#

Slack#

Prometheus Users
Grafana Community
CNCF Slack - #prometheus, #opentelemetry

GitHub#

Project	Link
Prometheus	https://github.com/prometheus/prometheus
Grafana	https://github.com/grafana/grafana
Loki	https://github.com/grafana/loki
Tempo	https://github.com/grafana/tempo
OpenTelemetry	https://github.com/open-telemetry

Dashboards & Rules#

Grafana Dashboards#

ID	Name	Purpose
1860	Node Exporter Full	Server monitoring
3662	Prometheus Stats	Prometheus self-monitoring
4701	JVM Micrometer	Spring Boot JVM
7362	MySQL Overview	MySQL monitoring
7587	PostgreSQL	PostgreSQL monitoring
11074	Kafka Exporter	Kafka monitoring

Searchable at https://grafana.com/grafana/dashboards/