PromQL on Advanced Beginner

Basic Syntax

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers learning PromQL for the first time Prerequisites: Metrics Fundamentals What You’ll Learn: Write basic PromQL queries and retrieve desired metrics

TL;DR#

Key Summary:

Instant Vector: Current point-in-time value http_requests_total

Range Vector: Values over time range http_requests_total[5m]

Label Matching: Filter using =, !=, =~, !~

Operations: Arithmetic (+, -), Comparison (>, <), Logical (and, or)

Why Do We Need PromQL?#

Prometheus stores thousands of time series data points. But having data doesn’t automatically provide insights. To answer questions like “Is the system healthy right now?”, “How does it compare to yesterday?”, or “Which service has problems?”, you need a language to query and transform data. That’s PromQL.

Aggregation Operators

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers who know basic PromQL syntax Prerequisites: Basic Syntax What You’ll Learn: Aggregate multiple time series to calculate meaningful metrics

TL;DR#

Key Summary:

sum: Total - total request count

avg: Average - average response time

count: Count - number of active instances

topk/bottomk: Top/bottom N items

by/without: Specify grouping criteria

Aggregation Operators List#

Operator	Description	Returns
`sum`	Sum	Single value or per-group values
`avg`	Average	Single value or per-group values
`min`	Minimum	Single value or per-group values
`max`	Maximum	Single value or per-group values
`count`	Number of time series	Single value or per-group values
`stddev`	Standard deviation	Single value or per-group values
`stdvar`	Variance	Single value or per-group values
`topk`	Top K items	K time series
`bottomk`	Bottom K items	K time series
`count_values`	Count by value	One time series per value
`quantile`	Quantile	Single value or per-group values

sum (Total)#

Calculates the sum of all time series values.

rate and increase

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers working with Counter metrics Prerequisites: Metrics Fundamentals, Aggregation Operators What You’ll Learn: Calculate per-second rate and total increase from Counter metrics accurately

TL;DR#

Key Summary:

rate(): Average per-second increase rate → Used in dashboards and alerts

increase(): Total increase within time range → Used for period totals

irate(): Instantaneous rate of last two samples → Used for volatile metrics

Always apply these functions to Counters (raw values are meaningless)

Why rate/increase are Needed#

Counters are cumulative values. They record how many requests have come since the server started. But the number “150,000 requests” alone doesn’t tell us if the system is working well. If the server started 1 hour ago, it’s 40 requests/second, but if it started 1 year ago, it’s 0.005 requests/second.

histogram_quantile

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs who need response time analysis Prerequisites: Metrics Fundamentals, rate and increase What You’ll Learn: Calculate accurate percentiles from Histograms and monitor SLAs

TL;DR#

Key Summary:

histogram_quantile(φ, bucket): Calculate φ percentile (0 ≤ φ ≤ 1)

P50: histogram_quantile(0.5, ...) - Median

P95: histogram_quantile(0.95, ...) - 95% are at or below this value

P99: histogram_quantile(0.99, ...) - 99% are at or below this value

Must always use with rate() or increase()

Why Percentiles Matter#

Averages are distorted by extreme values. Percentiles better reflect actual user experience.

Recording Rules

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Operators and SREs experiencing dashboard performance issues Prerequisites: histogram_quantile What You’ll Learn: Improve query performance with Recording Rules and manage complex metrics

TL;DR#

Key Summary:

Recording Rules save query results as new metrics

Pre-compute complex queries to improve dashboard/alert performance

Naming: level:metric:operations pattern recommended

Define frequently used aggregations as Rules

Why Recording Rules are Needed#

Problem: Repeatedly Executing Complex Queries#

# Executed every 15 seconds on dashboard
histogram_quantile(0.99,
 sum by (service, le) (
 rate(http_request_duration_seconds_bucket[5m])
 )
)

Issues:

Alerting Rules

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs setting up monitoring alerts Prerequisites: Recording Rules What You’ll Learn: Write rules that reduce false positives and alert only on real issues

TL;DR#

Key Summary:

for: Fires when condition is met for specified duration (prevents false positives)

labels: Add metadata like severity, team

annotations: Include alert message, runbook URL

Use Recording Rules results for concise writing

Basic Syntax#

Alerting Rule Structure#

groups:
 - name: <group_name>
 rules:
 - alert: <alert_name>
 expr: <PromQL_condition>
 for: <duration>
 labels:
 <label_name>: <value>
 annotations:
 <annotation_name>: <value>

Basic Example#

groups:
 - name: availability
 rules:
 - alert: ServiceDown
 expr: up == 0
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "{{ $labels.instance }} is down"
 description: "{{ $labels.job }} has been down for more than 5 minutes."
 runbook_url: "https://wiki.example.com/runbook/service-down"

Core Components#

expr (Condition)#

The condition that triggers the alert.