Troubleshooting on Advanced Beginner

Debugging High Latency

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Scenario: P99 response time exceeds SLA (500ms) Goal: Identify and resolve bottlenecks Duration: 15~30 minutes (depending on problem complexity) Success Criteria: P99 response time recovers below SLA threshold (500ms)

Problem Scenario#

Alert: HighP99Latency
Service: order-service
P99: 2.5s (Threshold: 500ms)
Duration: 10 minutes

Diagnostic Workflow#

graph TD
 A["1. Scope Analysis<br>Which service? Since when?"]
 B["2. Segment Analysis<br>Where is it slow?"]
 C["3. Resource Check<br>CPU/Memory/DB?"]
 D["4. Root Cause<br>Code? Query? External?"]
 E["5. Resolution"]

 A --> B --> C --> D --> E

Step 1: Scope Analysis#

Check Impact Scope#

# Which service is slow?
topk(5,
 histogram_quantile(0.99,
 sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
 )
)

# Since when did it become slow?
histogram_quantile(0.99,
 sum by (le) (rate(http_request_duration_seconds_bucket{service="order-service"}[5m]))
)
# → Time range: Last 1 hour

Check Specific Endpoints#

# P99 by endpoint
histogram_quantile(0.99,
 sum by (uri, le) (rate(http_request_duration_seconds_bucket{service="order-service"}[5m]))
)

Result: /orders POST endpoint is slow

Optimizing Metric Cardinality

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Scenario: Prometheus memory/storage spike, slow queries Goal: Optimize costs by reducing unnecessary time series Duration: 30 minutes~1 hour (depending on analysis and fix complexity) Success Criteria: Time series count reduced below target and memory usage stabilized

Problem Scenario#

Alert: PrometheusHighCardinality
Active Series: 2,500,000 (Threshold: 1,000,000)
Memory Usage: 32GB

What is Cardinality?#

Cardinality = Number of unique time series

http_requests_total{method="GET", status="200", path="/api/users"} # 1 series
http_requests_total{method="GET", status="200", path="/api/users/123"} # Another 1!
http_requests_total{method="GET", status="200", path="/api/users/456"} # Another 1!

Problem: If user_id is in path, time series created for each user

Managing Alert Fatigue

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Situation: Receiving dozens to hundreds of alerts daily, missing critical ones Goal: Only receive alerts that require actual action Time Required: 1-2 hours (analyzing and modifying alert rules) Success Criteria: Daily alert count reduced to a manageable level (e.g., 10 or fewer)

Before You Begin#

Required Environment#

Component	Version	Verification
Prometheus	2.40+	`prometheus --version`
Alertmanager	0.25+	`alertmanager --version`
amtool	0.25+	`amtool --version`

Required Permissions#

Write access to Prometheus configuration file (prometheus.yml)
Write access to Alertmanager configuration file (alertmanager.yml)
Permission to restart Prometheus/Alertmanager

Environment Check#

# Check Prometheus status
curl -s http://localhost:9090/-/healthy && echo "Prometheus OK"

# Check Alertmanager status
curl -s http://localhost:9093/-/healthy && echo "Alertmanager OK"

# Check amtool configuration
amtool config show --alertmanager.url=http://localhost:9093

Problem Scenario#

# Yesterday's alert summary
Critical: 15 (HighCPU 8, HighMemory 7)
Warning: 87 (SlowResponse 45, HighLatency 32, PodRestart 10)
Total: 102

# Actual incidents: 1
# Missed alerts: 1 (buried in HighCPU alerts)

When alert fatigue occurs: