실습 예제 on Advanced Beginner

환경 구성

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

소요 시간: 15분 선수 지식: Docker, Docker Compose 기본 이 문서를 읽으면: 로컬에 완전한 Observability 스택을 구축할 수 있습니다

전체 아키텍처#

graph TB
 APP["Spring Boot App"] --> |"metrics"| PROM["Prometheus"]
 APP --> |"logs"| LOKI["Loki"]
 APP --> |"traces"| TEMPO["Tempo"]

 PROM --> GF["Grafana"]
 LOKI --> GF
 TEMPO --> GF

 PROM --> AM["Alertmanager"]

Step 1: 디렉토리 구조 생성#

mkdir -p observability-stack/{prometheus/rules,grafana/{provisioning/datasources,provisioning/dashboards,dashboards},loki,promtail}
cd observability-stack

Step 2: Docker Compose 작성#

# docker-compose.yml
services:
 # 메트릭 수집
 prometheus:
 image: prom/prometheus:v2.50.0
 container_name: prometheus
 ports:
 - "9090:9090"
 volumes:
 - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
 - ./prometheus/rules:/etc/prometheus/rules
 - prometheus-data:/prometheus
 command:
 - '--config.file=/etc/prometheus/prometheus.yml'
 - '--storage.tsdb.path=/prometheus'
 - '--web.enable-lifecycle'
 - '--web.enable-remote-write-receiver'

 # 알림 관리
 alertmanager:
 image: prom/alertmanager:v0.26.0
 container_name: alertmanager
 ports:
 - "9093:9093"
 volumes:
 - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml

 # 시각화
 grafana:
 image: grafana/grafana:10.3.0
 container_name: grafana
 ports:
 - "3000:3000"
 environment:
 - GF_SECURITY_ADMIN_PASSWORD=admin
 - GF_AUTH_ANONYMOUS_ENABLED=true
 - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
 volumes:
 - ./grafana/provisioning:/etc/grafana/provisioning
 - ./grafana/dashboards:/var/lib/grafana/dashboards
 - grafana-data:/var/lib/grafana

 # 로그 수집
 loki:
 image: grafana/loki:2.9.0
 container_name: loki
 ports:
 - "3100:3100"
 volumes:
 - ./loki/loki.yml:/etc/loki/local-config.yaml
 - loki-data:/loki
 command: -config.file=/etc/loki/local-config.yaml

 # 로그 에이전트
 promtail:
 image: grafana/promtail:2.9.0
 container_name: promtail
 volumes:
 - ./promtail/promtail.yml:/etc/promtail/config.yml
 - /var/log:/var/log:ro
 - /var/lib/docker/containers:/var/lib/docker/containers:ro
 command: -config.file=/etc/promtail/config.yml

 # 분산 추적
 tempo:
 image: grafana/tempo:2.3.0
 container_name: tempo
 ports:
 - "4317:4317" # OTLP gRPC
 - "4318:4318" # OTLP HTTP
 - "3200:3200" # Tempo API
 volumes:
 - ./tempo/tempo.yml:/etc/tempo/config.yaml
 - tempo-data:/var/tempo
 command: -config.file=/etc/tempo/config.yaml

volumes:
 prometheus-data:
 grafana-data:
 loki-data:
 tempo-data:

Step 3: Prometheus 설정#

# prometheus/prometheus.yml
global:
 scrape_interval: 15s
 evaluation_interval: 15s

alerting:
 alertmanagers:
 - static_configs:
 - targets:
 - alertmanager:9093

rule_files:
 - /etc/prometheus/rules/*.yml

scrape_configs:
 - job_name: 'prometheus'
 static_configs:
 - targets: ['localhost:9090']

 - job_name: 'spring-app'
 metrics_path: '/actuator/prometheus'
 static_configs:
 - targets: ['host.docker.internal:8080']

Step 4: Loki 설정#

# loki/loki.yml
auth_enabled: false

server:
 http_listen_port: 3100

common:
 path_prefix: /loki
 storage:
 filesystem:
 chunks_directory: /loki/chunks
 rules_directory: /loki/rules
 replication_factor: 1
 ring:
 instance_addr: 127.0.0.1
 kvstore:
 store: inmemory

schema_config:
 configs:
 - from: 2020-10-24
 store: boltdb-shipper
 object_store: filesystem
 schema: v11
 index:
 prefix: index_
 period: 24h

limits_config:
 retention_period: 168h

Step 5: Promtail 설정#

# promtail/promtail.yml
server:
 http_listen_port: 9080

positions:
 filename: /tmp/positions.yaml

clients:
 - url: http://loki:3100/loki/api/v1/push

scrape_configs:
 - job_name: containers
 static_configs:
 - targets:
 - localhost
 labels:
 job: containerlogs
 __path__: /var/lib/docker/containers/*/*log

 pipeline_stages:
 - json:
 expressions:
 output: log
 stream: stream
 attrs:
 - json:
 expressions:
 tag:
 source: attrs
 - regex:
 expression: (?P<container_name>(?:[^|]*[^|]))
 source: tag
 - labels:
 container_name:
 stream:

Step 6: Tempo 설정#

# tempo/tempo.yml
server:
 http_listen_port: 3200

distributor:
 receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318

storage:
 trace:
 backend: local
 local:
 path: /var/tempo/traces
 wal:
 path: /var/tempo/wal

metrics_generator:
 registry:
 external_labels:
 source: tempo
 storage:
 path: /var/tempo/generator/wal
 remote_write:
 - url: http://prometheus:9090/api/v1/write
 send_exemplars: true

Step 7: Grafana 데이터소스 설정#

# grafana/provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
 - name: Prometheus
 type: prometheus
 access: proxy
 url: http://prometheus:9090
 isDefault: true

 - name: Loki
 type: loki
 access: proxy
 url: http://loki:3100

 - name: Tempo
 type: tempo
 access: proxy
 url: http://tempo:3200
 jsonData:
 tracesToLogsV2:
 datasourceUid: loki
 filterByTraceID: true
 tracesToMetrics:
 datasourceUid: prometheus
 serviceMap:
 datasourceUid: prometheus

Step 8: Alertmanager 설정#

mkdir -p alertmanager

# alertmanager/alertmanager.yml
global:
 resolve_timeout: 5m

route:
 receiver: 'default'
 group_by: ['alertname', 'job']
 group_wait: 30s
 group_interval: 5m
 repeat_interval: 4h

receivers:
 - name: 'default'
 webhook_configs:
 - url: 'http://localhost:5001/'
 send_resolved: true

Step 9: 스택 실행#

docker compose up -d

예상 출력:

Spring Boot 메트릭

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

소요 시간: 20분 선수 지식: Spring Boot, 환경 구성 이 문서를 읽으면: Spring Boot 애플리케이션에 Observability를 적용할 수 있습니다

Step 1/8: 의존성 추가 (2분)#

// build.gradle.kts
dependencies {
 // Actuator + Prometheus
 implementation("org.springframework.boot:spring-boot-starter-actuator")
 implementation("io.micrometer:micrometer-registry-prometheus")

 // Tracing
 implementation("io.micrometer:micrometer-tracing-bridge-otel")
 implementation("io.opentelemetry:opentelemetry-exporter-otlp")

 // Logging (Loki)
 implementation("com.github.loki4j:loki-logback-appender:1.4.2")
}

Step 2/8: application.yml 설정 (3분)#

spring:
 application:
 name: order-service

management:
 endpoints:
 web:
 exposure:
 include: health,info,prometheus,metrics
 endpoint:
 health:
 show-details: always
 metrics:
 tags:
 application: ${spring.application.name}
 distribution:
 percentiles-histogram:
 http.server.requests: true
 slo:
 http.server.requests: 50ms, 100ms, 200ms, 500ms, 1s
 tracing:
 sampling:
 probability: 1.0
 otlp:
 tracing:
 endpoint: http://localhost:4318/v1/traces

logging:
 pattern:
 level: "%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]"

Step 3/8: Loki 로그 설정 (3분)#

<!-- src/main/resources/logback-spring.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
 <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

 <springProperty scope="context" name="appName" source="spring.application.name"/>

 <!-- Console Appender -->
 <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
 <encoder>
 <pattern>%d{ISO8601} %5p [${appName},%X{traceId:-},%X{spanId:-}] --- [%t] %-40.40logger{39} : %m%n</pattern>
 </encoder>
 </appender>

 <!-- Loki Appender -->
 <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
 <http>
 <url>http://localhost:3100/loki/api/v1/push</url>
 </http>
 <format>
 <label>
 <pattern>app=${appName},host=${HOSTNAME},level=%level</pattern>
 </label>
 <message>
 <pattern>{"timestamp":"%d{ISO8601}","level":"%level","logger":"%logger","message":"%msg","traceId":"%X{traceId:-}","spanId":"%X{spanId:-}"}</pattern>
 </message>
 </format>
 </appender>

 <root level="INFO">
 <appender-ref ref="CONSOLE"/>
 <appender-ref ref="LOKI"/>
 </root>
</configuration>

중간 확인 (Step 1~3 완료)#

build.gradle.kts에 의존성 추가됨
application.yml에 management 설정 완료
logback-spring.xml 파일 생성됨
./gradlew bootRun 실행 후 http://localhost:8080/actuator/prometheus 접속 시 메트릭 표시

Step 4/8: 커스텀 메트릭 추가 (5분)#

@RestController
@RequiredArgsConstructor
@RequestMapping("/orders")
public class OrderController {
 private final MeterRegistry meterRegistry;
 private final OrderService orderService;

 // Counter: 주문 수
 private Counter orderCounter(String status) {
 return Counter.builder("orders_total")
 .tag("status", status)
 .description("Total number of orders")
 .register(meterRegistry);
 }

 // Gauge: 처리 중인 주문
 private final AtomicInteger ordersInProgress = new AtomicInteger(0);

 @PostConstruct
 void registerGauge() {
 Gauge.builder("orders_in_progress", ordersInProgress, AtomicInteger::get)
 .description("Orders currently being processed")
 .register(meterRegistry);
 }

 @PostMapping
 public ResponseEntity<Order> createOrder(@RequestBody OrderRequest request) {
 ordersInProgress.incrementAndGet();
 try {
 // Timer로 처리 시간 측정
 return Timer.builder("order_processing_duration_seconds")
 .tag("type", request.getType())
 .publishPercentileHistogram()
 .register(meterRegistry)
 .record(() -> {
 Order order = orderService.create(request);
 orderCounter("success").increment();
 return ResponseEntity.ok(order);
 });
 } catch (Exception e) {
 orderCounter("failed").increment();
 throw e;
 } finally {
 ordersInProgress.decrementAndGet();
 }
 }
}

Step 5/8: 수동 트레이싱 (3분)#

@Service
@RequiredArgsConstructor
@Slf4j
public class OrderService {
 private final Tracer tracer;
 private final PaymentClient paymentClient;
 private final InventoryClient inventoryClient;

 public Order create(OrderRequest request) {
 // 현재 Span에 태그 추가
 Span currentSpan = tracer.currentSpan();
 if (currentSpan != null) {
 currentSpan.tag("order.type", request.getType());
 currentSpan.tag("order.items", String.valueOf(request.getItems().size()));
 }

 // 새 Span 생성
 Span inventorySpan = tracer.nextSpan().name("checkInventory").start();
 try (Tracer.SpanInScope ws = tracer.withSpan(inventorySpan)) {
 inventoryClient.check(request.getItems());
 inventorySpan.event("Inventory checked");
 } finally {
 inventorySpan.end();
 }

 // 결제 처리
 Span paymentSpan = tracer.nextSpan().name("processPayment").start();
 try (Tracer.SpanInScope ws = tracer.withSpan(paymentSpan)) {
 paymentClient.process(request.getPayment());
 paymentSpan.event("Payment processed");
 } finally {
 paymentSpan.end();
 }

 log.info("Order created: {}", request.getId());
 return new Order(request);
 }
}

중간 확인 (Step 4~5 완료)#

OrderController에 Counter, Gauge, Timer 메트릭 추가됨
OrderService에 수동 Span 생성 코드 추가됨
컴파일 오류 없이 빌드 성공: ./gradlew build

Step 6/8: 메트릭 확인 (2분)#

Prometheus 엔드포인트#

curl http://localhost:8080/actuator/prometheus

예상 출력:

Kafka 모니터링

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

소요 시간: 20분 선수 지식: Kafka 기초, 환경 구성 이 문서를 읽으면: Kafka 클러스터와 Consumer Lag을 모니터링할 수 있습니다

Kafka 모니터링 핵심 지표#

카테고리	지표	의미
Consumer	Lag	처리 지연 (가장 중요)
Broker	Under-replicated Partitions	복제 문제
Producer	Record Error Rate	전송 실패율
Broker	Disk Usage	저장 공간

Step 1: Kafka + JMX Exporter 설정#

# docker-compose.yml에 추가
services:
 kafka:
 image: confluentinc/cp-kafka:7.5.0
 container_name: kafka
 ports:
 - "9092:9092"
 - "9101:9101" # JMX
 environment:
 KAFKA_NODE_ID: 1
 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
 KAFKA_PROCESS_ROLES: broker,controller
 KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093
 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
 CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk
 KAFKA_JMX_PORT: 9101
 KAFKA_JMX_HOSTNAME: localhost
 KAFKA_OPTS: -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx-exporter/kafka.yml
 volumes:
 - ./jmx-exporter:/opt/jmx-exporter

 kafka-exporter:
 image: danielqsj/kafka-exporter:latest
 container_name: kafka-exporter
 ports:
 - "9308:9308"
 command:
 - --kafka.server=kafka:9092
 - --topic.filter=.*
 - --group.filter=.*

Step 2: JMX Exporter 설정#

mkdir -p jmx-exporter
curl -L https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.19.0/jmx_prometheus_javaagent-0.19.0.jar -o jmx-exporter/jmx_prometheus_javaagent.jar

# jmx-exporter/kafka.yml
lowercaseOutputName: true
rules:
 # Broker 메트릭
 - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
 name: kafka_server_$1_$2
 type: GAUGE
 labels:
 clientId: "$3"
 topic: "$4"
 partition: "$5"

 - pattern: kafka.server<type=(.+), name=(.+)><>Value
 name: kafka_server_$1_$2
 type: GAUGE

 # Under-replicated Partitions
 - pattern: kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value
 name: kafka_server_replicamanager_underreplicatedpartitions
 type: GAUGE

 # Messages In Per Sec
 - pattern: kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=(.+)><>OneMinuteRate
 name: kafka_server_brokertopicmetrics_messagesin_total
 type: GAUGE
 labels:
 topic: "$1"

 # Bytes In Per Sec
 - pattern: kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec, topic=(.+)><>OneMinuteRate
 name: kafka_server_brokertopicmetrics_bytesin_total
 type: GAUGE
 labels:
 topic: "$1"

Step 3: Prometheus 설정 추가#

# prometheus/prometheus.yml에 추가
scrape_configs:
 - job_name: 'kafka'
 static_configs:
 - targets: ['kafka:7071']

 - job_name: 'kafka-exporter'
 static_configs:
 - targets: ['kafka-exporter:9308']

Step 4: 핵심 PromQL 쿼리#

Consumer Lag#

# 컨슈머 그룹별 Lag
sum by (consumergroup, topic) (kafka_consumergroup_lag)

# Lag이 10000 이상인 그룹
sum by (consumergroup, topic) (kafka_consumergroup_lag) > 10000

# Lag 추이
sum(kafka_consumergroup_lag) by (consumergroup)

Broker 상태#

# Under-replicated Partitions
kafka_server_replicamanager_underreplicatedpartitions

# ISR (In-Sync Replicas) 축소
kafka_server_replicamanager_isrshrinks_total

트래픽#

# 토픽별 초당 메시지 수
sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))

# 토픽별 초당 바이트
sum by (topic) (rate(kafka_server_brokertopicmetrics_bytesin_total[5m]))

Step 5: 알림 규칙#

# prometheus/rules/kafka-alerts.yml
groups:
 - name: kafka
 rules:
 - alert: KafkaConsumerLagHigh
 expr: sum by (consumergroup, topic) (kafka_consumergroup_lag) > 10000
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "High consumer lag: {{ $labels.consumergroup }}"
 description: "Lag is {{ $value }} on topic {{ $labels.topic }}"

 - alert: KafkaUnderReplicatedPartitions
 expr: kafka_server_replicamanager_underreplicatedpartitions > 0
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "Kafka under-replicated partitions"
 description: "{{ $value }} partitions are under-replicated"

 - alert: KafkaBrokerDown
 expr: count(kafka_server_kafkaserver_brokerstate) < 3
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "Kafka broker down"

Step 6: Recording Rules#

# prometheus/rules/kafka-recording.yml
groups:
 - name: kafka_recording
 rules:
 - record: topic:kafka_messages:rate5m
 expr: sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))

 - record: consumergroup:kafka_lag:sum
 expr: sum by (consumergroup) (kafka_consumergroup_lag)

 - record: :kafka_underreplicated:sum
 expr: sum(kafka_server_replicamanager_underreplicatedpartitions)

Step 7: Grafana 대시보드#

Row 1: Overview#

패널	쿼리	타입
Total Lag	`sum(kafka_consumergroup_lag)`	Stat
Messages/sec	`sum(rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))`	Stat
Under-replicated	`sum(kafka_server_replicamanager_underreplicatedpartitions)`	Stat

Row 2: Consumer Lag#

# Time Series: Lag 추이
sum by (consumergroup) (kafka_consumergroup_lag)

# Table: 상세
sum by (consumergroup, topic, partition) (kafka_consumergroup_lag)

Row 3: Traffic#

# 토픽별 메시지/초
sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))

Spring Kafka 메트릭#

# application.yml
management:
 metrics:
 enable:
 kafka: true

# Producer 메트릭
kafka_producer_record_send_total
kafka_producer_record_error_total

# Consumer 메트릭
kafka_consumer_records_consumed_total
kafka_consumer_fetch_manager_records_lag

확인 체크리스트#

kafka-exporter 메트릭 노출 확인
Consumer Lag 쿼리 동작 확인
Grafana 대시보드 구성
알림 규칙 테스트

다음 단계#

추천 순서	문서	배우는 것
1	풀스택 예제	통합 예제
2	서비스 유형별 적용	Kafka 황금 신호

풀스택 Observability

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

소요 시간: 30분 선수 지식: Spring Boot 메트릭 이 문서를 읽으면: 3요소(Metrics, Logs, Traces)를 연결하여 문제를 추적할 수 있습니다

목표#

“에러율 급증” → “로그에서 trace_id 확인” → “트레이스로 근본 원인 파악”

graph LR
 A["1. Metrics<br>에러율 5% 감지"]
 B["2. Logs<br>trace_id 확인"]
 C["3. Traces<br>병목 구간 발견"]
 D["4. 해결<br>DB 쿼리 최적화"]

 A --> B --> C --> D

시나리오: 주문 서비스 장애 분석#

시스템 구성#

graph LR
 USER["User"] --> GW["API Gateway"]
 GW --> ORDER["Order Service"]
 ORDER --> PAYMENT["Payment Service"]
 ORDER --> INVENTORY["Inventory Service"]
 PAYMENT --> DB["Payment DB"]

Step 1: 장애 감지 (Metrics)#

Grafana Alert 발동#

Alert: HighErrorRate
Service: order-service
Error Rate: 5.2%
Threshold: 1%

대시보드 확인#

# 에러율 급증 확인
sum(rate(http_server_requests_seconds_count{application="order-service",status=~"5.."}[5m]))
/ sum(rate(http_server_requests_seconds_count{application="order-service"}[5m]))

# 어떤 엔드포인트?
sum by (uri) (rate(http_server_requests_seconds_count{application="order-service",status=~"5.."}[5m]))

결과: /orders POST 엔드포인트에서 에러 발생