Latency (Response Time)

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs looking to improve service response times Prerequisites: histogram_quantile After reading this: You’ll be able to accurately measure latency and set up SLA-based alerts

TL;DR#

Key Summary:

P50 (median): Typical user experience

P95: Most users’ experience

P99: Worst-case user experience (SLA baseline)

Percentiles better reflect actual experience than averages

Measure latency of successful/failed requests separately

Why Percentiles Matter#

Latency is quality directly felt by users. No matter how great the features, users leave if they have to wait 5 seconds. According to Amazon’s research, 100ms of added latency results in 1% decrease in sales.

Traffic (Throughput)

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs establishing capacity planning Prerequisites: rate and increase After reading this: You’ll be able to analyze traffic patterns and use them for capacity planning

TL;DR#

Key Summary:

RPS (Requests Per Second): Requests per second

Throughput: Throughput per second (bytes, messages, etc.)

Concurrent Connections: Active connections

Traffic changes are a leading indicator of other signals

Why Monitor Traffic?#

Traffic measures the demand entering the system. If Latency shows “how fast”, Traffic shows “how much”. To judge system health, you need both. No matter how fast the response, something is wrong if no requests come in. If requests spike but resources are insufficient, failure is imminent.

Errors (Error Rate)

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs improving service reliability Prerequisites: Aggregation Operators After reading this: You’ll be able to systematically classify errors and set up SLO-based monitoring

TL;DR#

Key Summary:

Error rate: Failed requests / Total requests

Not just HTTP 5xx: Include business logic failures

Error budget: Allowed error amount (based on SLO)

Error classification is important: Client vs server, temporary vs permanent

Why Error Monitoring Matters#

Errors are the most direct indicator of system health. Services still work with high Latency, but high error rates cause real harm to users. According to Google SRE team research, users who experience errors are 3x more likely to churn within 72 hours.

Saturation (Resource Usage)

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: SREs and infrastructure engineers managing system capacity Prerequisites: Prometheus Architecture After reading this: You’ll be able to detect resource bottlenecks early and establish capacity planning

TL;DR#

Key Summary:

Saturation: How “full” resources are (0-100%)

Key resources: CPU, memory, disk, network, connection pools

Approaching 100% causes latency spikes, errors

USE method: Utilization, Saturation, Errors

Why Monitor Saturation?#

Saturation is the only metric that can predict failures. Latency, Traffic, and Errors react after problems occur, but Saturation warns before problems occur.

By Service Type

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: SREs operating various service types Prerequisites: SRE Golden Signals After reading this: You’ll be able to apply the four signals tailored to service characteristics

TL;DR#

Key Summary:

Each service type has different key signals

Web API: Latency, Errors focused

Kafka: Traffic (Lag), Saturation focused

Database: Latency, Saturation focused

Batch jobs: Errors, Traffic (completion rate) focused

Key Signals by Service Type#

Service Type	Key Signals	Reason
Web API	Latency, Errors	Directly affects user experience
Kafka	Traffic (Lag), Saturation	Throughput is key
Database	Latency, Saturation	Query performance, connections
Cache (Redis)	Latency, Saturation	Response speed, memory
Batch jobs	Errors, Traffic	Completion rate, throughput
Load balancer	Traffic, Errors	Connection distribution

Web API / Microservices#

Key Metrics#

graph TD
 subgraph "Web API Key"
 L["Latency<br>P99 response time"]
 E["Errors<br>5xx ratio"]
 T["Traffic<br>RPS"]
 S["Saturation<br>Connections/threads"]
 end

 L --> |"SLA"| SLA["P99 < 500ms"]
 E --> |"SLO"| SLO["Error rate < 0.1%"]

PromQL Queries#

# Latency: P99 response time
histogram_quantile(0.99,
 sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

# Traffic: Requests per second
sum by (service) (rate(http_requests_total[5m]))

# Errors: Error rate
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (service) (rate(http_requests_total[5m]))

# Saturation: Concurrent requests
sum by (service) (http_server_requests_active)

Recording Rules#

groups:
 - name: web_api_golden_signals
 rules:
 - record: service:http_request_duration_seconds:p99
 expr: histogram_quantile(0.99, sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))

 - record: service:http_requests:rate5m
 expr: sum by (service) (rate(http_requests_total[5m]))

 - record: service:http_errors:ratio
 expr: sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (service) (rate(http_requests_total[5m]))

Kafka#

Key Metrics#

graph TD
 subgraph "Kafka Key"
 T["Traffic<br>Message throughput"]
 L["Latency<br>Consumer Lag"]
 S["Saturation<br>Disk, partitions"]
 E["Errors<br>Failed messages"]
 end

 L --> |"Important"| LAG["Lag < 10,000"]
 S --> |"Important"| DISK["Disk < 80%"]

PromQL Queries#

# Traffic: Messages per second
sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))

# Latency: Consumer Lag
sum by (consumer_group, topic) (kafka_consumer_group_lag)

# Saturation: Disk usage
kafka_log_log_size / kafka_log_log_max_size * 100

# Saturation: Under-replicated partitions
kafka_server_replicamanager_underreplicatedpartitions

# Errors: Failure rate
rate(kafka_producer_record_error_total[5m])

Alert Rules#

groups:
 - name: kafka_alerts
 rules:
 - alert: KafkaConsumerLagHigh
 expr: sum by (consumer_group, topic) (kafka_consumer_group_lag) > 10000
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "Kafka consumer lag high"

 - alert: KafkaUnderReplicated
 expr: kafka_server_replicamanager_underreplicatedpartitions > 0
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "Kafka under-replicated partitions"

 - alert: KafkaDiskHigh
 expr: kafka_log_log_size / kafka_log_log_max_size > 0.8
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "Kafka disk usage high"

Database (PostgreSQL/MySQL)#

Key Metrics#

graph TD
 subgraph "DB Key"
 L["Latency<br>Query time"]
 S["Saturation<br>Connections, buffer"]
 E["Errors<br>Deadlocks, failures"]
 T["Traffic<br>QPS"]
 end

 S --> |"Important"| CONN["Connections < 80%"]
 L --> |"Important"| SLOW["Slow queries"]

PromQL Queries (PostgreSQL)#

# Latency: Query time
rate(pg_stat_statements_total_time_seconds_sum[5m])
/ rate(pg_stat_statements_calls_total[5m])

# Traffic: Queries per second (QPS)
sum(rate(pg_stat_statements_calls_total[5m]))

# Saturation: Connection usage
sum(pg_stat_activity_count) / pg_settings_max_connections * 100

# Saturation: Buffer hit rate
pg_stat_bgwriter_buffers_alloc
/ (pg_stat_bgwriter_buffers_alloc + pg_stat_bgwriter_buffers_backend) * 100

# Errors: Deadlocks
rate(pg_stat_database_deadlocks_total[5m])

PromQL Queries (MySQL)#

# Traffic: QPS
rate(mysql_global_status_queries[5m])

# Latency: Slow queries
rate(mysql_global_status_slow_queries[5m])

# Saturation: Connection usage
mysql_global_status_threads_connected / mysql_global_variables_max_connections * 100

# Saturation: Buffer pool usage
mysql_global_status_innodb_buffer_pool_pages_data
/ mysql_global_status_innodb_buffer_pool_pages_total * 100

Cache (Redis)#

Key Metrics#

graph TD
 subgraph "Redis Key"
 L["Latency<br>Command time"]
 S["Saturation<br>Memory"]
 T["Traffic<br>Command count"]
 E["Errors<br>Rejects, evictions"]
 end

 S --> |"Important"| MEM["Memory < 80%"]
 L --> |"Important"| HIT["Hit rate > 90%"]

PromQL Queries#

# Traffic: Commands per second
rate(redis_commands_total[5m])

# Latency: Average command time
rate(redis_commands_duration_seconds_total[5m])
/ rate(redis_commands_total[5m])

# Saturation: Memory usage
redis_memory_used_bytes / redis_memory_max_bytes * 100

# Errors: Hit rate
rate(redis_keyspace_hits_total[5m])
/ (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))
* 100

# Saturation: Connection count
redis_connected_clients

Batch Jobs#

Key Metrics#

graph TD
 subgraph "Batch Key"
 E["Errors<br>Failure rate"]
 T["Traffic<br>Throughput"]
 L["Latency<br>Execution time"]
 end

 E --> |"Important"| FAIL["Failure rate < 1%"]
 T --> |"Important"| DONE["Completion rate 100%"]

PromQL Queries#

# Traffic: Completed jobs
increase(batch_jobs_completed_total[1h])

# Errors: Failure rate
increase(batch_jobs_failed_total[1h])
/ (increase(batch_jobs_completed_total[1h]) + increase(batch_jobs_failed_total[1h]))

# Latency: Average execution time
rate(batch_job_duration_seconds_sum[1h])
/ rate(batch_job_duration_seconds_count[1h])

# Traffic: Processed items
increase(batch_items_processed_total[1h])

Alert Rules#

 - alert: BatchJobFailed
 expr: increase(batch_jobs_failed_total[1h]) > 0
 labels:
 severity: warning
 annotations:
 summary: "Batch job failed"

 - alert: BatchJobNotRunning
 expr: time() - batch_job_last_success_timestamp > 86400
 labels:
 severity: critical
 annotations:
 summary: "Batch job hasn't run in 24 hours"

Load Balancer (Nginx/HAProxy)#

PromQL Queries (Nginx)#

# Traffic: Requests per second
sum(rate(nginx_http_requests_total[5m]))

# Errors: 5xx ratio
sum(rate(nginx_http_requests_total{status=~"5.."}[5m]))
/ sum(rate(nginx_http_requests_total[5m]))

# Saturation: Active connections
nginx_connections_active

# Latency: Upstream response time
histogram_quantile(0.99, rate(nginx_upstream_response_time_seconds_bucket[5m]))

Dashboard Templates#

Panel Layout by Service Type#

Service	Row 1 (Summary)	Row 2 (Detail)	Row 3 (Trend)
Web API	P99, RPS, Error rate	Status code distribution, By endpoint	Time series
Kafka	Lag, Throughput, Partitions	By consumer, By topic	Time series
DB	QPS, Connections, Slow queries	By query, By table	Time series
Redis	Commands, Hit rate, Memory	By command, By keyspace	Time series

Key Summary#

Service	Top Priority	2nd Priority	3rd Priority
Web API	Latency	Errors	Traffic
Kafka	Traffic (Lag)	Saturation	Errors
DB	Saturation	Latency	Traffic
Redis	Saturation	Latency	Traffic
Batch	Errors	Traffic	Latency

Next Steps#

Recommended Order	Document	What You’ll Learn
1	Environment Setup	Practice environment
2	Kafka Monitoring	Kafka details

SRE Golden Signals on Advanced Beginner

Latency (Response Time)

TL;DR#

Why Percentiles Matter#

Traffic (Throughput)

TL;DR#

Why Monitor Traffic?#

Errors (Error Rate)

TL;DR#

Why Error Monitoring Matters#

Saturation (Resource Usage)

TL;DR#

Why Monitor Saturation?#

By Service Type

TL;DR#

Key Signals by Service Type#

Web API / Microservices#

Key Metrics#

PromQL Queries#

Recording Rules#

Kafka#

Key Metrics#

PromQL Queries#

Alert Rules#

Database (PostgreSQL/MySQL)#

Key Metrics#

PromQL Queries (PostgreSQL)#

PromQL Queries (MySQL)#

Cache (Redis)#

Key Metrics#

PromQL Queries#

Batch Jobs#

Key Metrics#

PromQL Queries#

Alert Rules#

Load Balancer (Nginx/HAProxy)#

PromQL Queries (Nginx)#

Dashboard Templates#

Panel Layout by Service Type#

Key Summary#

Next Steps#