Advanced Beginner

Implicit/Given Debugging

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Learn how to diagnose and resolve issues when the compiler cannot find implicit values.

Estimated time: 10-15 minutes

TL;DR

Scala 2: Use -Xlog-implicits flag to trace the search process

Scala 3: Compiler messages are clearer and provide import suggestions

Common: Check if implicit values are in scope and types match exactly

What This Guide Solves#

Use this guide when you encounter these compile errors:

Scala 2

could not find implicit value for parameter ord: Ordering[MyClass]

Pod Troubleshooting

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Objective: Identify and resolve issues when Pods fail to start or terminate abnormally Estimated Time: 30 minutes

Scope of This Guide
Covers: Pod startup failures, CrashLoopBackOff, ImagePullBackOff, Readiness failures

Does not cover: Network connectivity issues (see Network Troubleshooting), performance degradation due to resource constraints (see Resource Optimization)

Before You Begin#

Verify the following prerequisites.

1. Verify kubectl Installation and Version#

kubectl version --client

Success output:

Client Version: v1.28.0

Version Compatibility
kubectl version must be within ±1 minor version of cluster version. Example: Cluster 1.27 → kubectl 1.26~1.28 supported

2. Verify Cluster Access#

kubectl cluster-info

Success output:

Slow Query Optimization

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Learn how to diagnose and improve slow search response times.

Duration: Approximately 15-30 minutes

Scope of This Guide
Covered: Query-level performance optimization, Profile API analysis, cache utilization

Not Covered: Cluster-level tuning (adding nodes, hardware upgrades) - see Performance Tuning

TL;DR

Analyze query execution plans with Profile API

Remove unnecessary fields: _source filtering, stored fields

Optimize query types: Use term instead of match, avoid leading wildcards

Leverage caching: Use filter context, enable request cache

Before You Begin#

Verify the following requirements:

Troubleshooting OutOfMemoryError

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Estimated Time: About 15 minutes

TL;DR

Driver OOM: Reduce collect() result size, increase spark.driver.memory

Executor OOM: Increase partition count (repartition), increase spark.executor.memory

Diagnose First: Check Spark UI to identify where OOM occurs

Problem Definition#

The following error occurs during Spark application execution:

java.lang.OutOfMemoryError: Java heap space

Or:

Container killed by YARN for exceeding memory limits

This guide explains step-by-step how to diagnose and resolve OOM errors.

Prerequisites#

Item	Requirement	How to Verify
Spark Version	2.4 or higher (3.x recommended)	`spark-submit --version`
Java Version	8, 11, or 17	`java -version`
Spark UI	Accessible	Open `http://localhost:4040` in browser
Permissions	Can modify Spark settings	Verify spark-submit execution permissions

Supported Environments: Linux, macOS, Windows (WSL2 recommended)

Layered Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 15 Jan 2026 00:00:00 +0000

Target Audience: Developers learning architecture patterns for the first time Prerequisites: Basic understanding of Spring Boot MVC patterns Estimated Time: About 15 minutes

The most basic and widely used architecture pattern. If you are learning architecture for the first time, start here. Layered architecture divides software horizontally so each layer has a clear role. It follows the simple but powerful rule that each layer can only call the layer below it.

Basic Syntax

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

Target Audience: Developers with experience in Java or other statically typed languages Prerequisites: Basic concepts of variables, functions, and classes What you’ll learn: Understand the difference between val/var, leverage the type system, and write clean code with string interpolation

TL;DR

Use val (immutable) by default, var (mutable) only when necessary

Scala has powerful type inference so you can omit type declarations most of the time

String interpolation (s"Hello, $name") makes string composition simple

Every value is an object — you can call methods like 1.toString

Why Learn Scala’s Basic Syntax?#

Scala is 100% compatible with Java while enabling you to write more concise and expressive code. You can write the same logic with less code than Java, and catch more errors at compile time.

Environment Setup

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Install JDK, sbt, and Scala CLI all at once with Coursier

sbt is Scala’s standard build tool for compilation, testing, and dependency management

Configure IDE with IntelliJ IDEA or VS Code + Metals

Project structure: build.sbt (configuration) + src/main/scala/ (source) + src/test/scala/ (tests)

Target Audience: Developers new to Scala programming

Prerequisites:

Basic terminal/command-line usage
Java or other JVM language experience (recommended)
IDE usage experience

This document provides detailed instructions for setting up a Scala development environment. To start Scala programming, you need JDK, sbt (Scala Build Tool), and an IDE. This guide walks through the installation and configuration process step by step.

Glossary

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

Key Scala terms organized alphabetically. For detailed explanations, refer to the Concepts section. Each term includes a definition and links to related documentation.

TL;DR - Top 5 Terms

Case Class: Immutable data class, auto-generates equals/copy

Option[A]: Replacement for null, either Some(value) or None

Pattern Matching: Structure analysis and data extraction (match expressions)

Trait: Interface that can include implementations, supports mixin inheritance

given/using (Scala 3): New syntax for implicit values/parameters

A#

ADT (Algebraic Data Type): Algebraic data types. Combination of sum types and product types defined with sealed trait and Case Class. Can be defined more simply with enum in Scala 3. → Used with Pattern Matching
Applicative: A Type Class that combines independent effects. Provides pure and ap operations. More powerful than Functor, weaker than Monad. → Functional Patterns
apply method: A special method that allows objects to be called like functions. obj(args) is interpreted as obj.apply(args). Commonly used as a factory method in Companion Object.

C#

Case Class: A special class for immutable data. Automatically generates equals, hashCode, copy, unapply, etc. Used with Pattern Matching. → Case Classes Details
Companion Object: A singleton object with the same name as a class. Can access the class’s private members. Implements factory pattern with apply method. → Classes and Objects
Context Bound: Syntax requiring the existence of a Type Class instance, like def f[A: Ordering]. → Implicits
Currying: A technique for transforming a function with multiple arguments into a chain of single-argument functions. Used with Higher-Order Function. → Functions and Methods

A-C Key Points

ADT: Type-safe data modeling with sealed trait + case class

Case Class: Optimized for immutable data, used with pattern matching

Companion Object: Place for factory methods (apply) and utility functions

E#

Either[L, R]: A type that holds one of two types. Usually Left is failure, Right is success. Similar to Option but can contain failure information. Can be chained with flatMap.
Extension Method: A technique for adding new methods to existing types. Use extension keyword in Scala 3. Used in Type Class implementations. → Scala 3 Feature Comparison
ExecutionContext: Context providing a thread pool for executing Future. → Concurrency

F#

flatMap: An operation that transforms values inside a container and flattens the result. Core operation of Monad. Can be elegantly expressed with For Comprehension.
For Comprehension: Syntactic sugar for elegantly expressing combinations of flatMap, map, and withFilter. Used with Option, Future, Either, etc. → For Comprehension Details
Functor: A Type Class with a map operation. Transforms values inside a container. Foundation of Applicative and Monad. → Functional Patterns
Future[T]: A type representing an asynchronous computation not yet completed. Requires ExecutionContext. Sequential execution possible with For Comprehension. → Concurrency

G#

Given (Scala 3): Keyword for defining Type Class instances. Replaces Scala 2’s implicit val. → Scala 2 vs 3 Comparison

H#

Higher-Order Function: A function that takes functions as arguments or returns a function. Examples: map, filter, fold. → Higher-Order Functions Details
Higher-Kinded Type: A type that takes type constructors as arguments. Form of F[_]. Essential for defining Functor and Monad. → Advanced Type System

I#

Immutable: Something whose state cannot be changed after creation. Scala recommends immutability. Use Case Class, val, and immutable collections.
Implicit (Scala 2): Keyword for defining implicit values, parameters, and conversions. Replaced by given/using in Scala 3. → Implicits Details
Intersection Type (&): A type satisfying multiple types. A & B. Opposite of Union Type. → Advanced Type System

E-I Key Points

Either/Option: Used instead of null, functional approach to error handling

flatMap: Core of Monad, elegantly expressed with for comprehension

given/using (Scala 3): Clear syntax replacing implicit

L#

Lazy val: A value whose initialization is deferred until first access. Useful for expensive initialization. → Basics

M#

Match Expression: An expression that branches based on value patterns. Powerful version of switch. Optimized for Case Class and Sealed traits. → Pattern Matching Details
Monad: A Type Class with flatMap and pure operations. Composes sequential effects. Option, Either, Future are Monads. → Functional Patterns

O#

Object: Keyword for defining singleton instances. See Companion Object. → Classes and Objects
Opaque Type (Scala 3): A type that appears different externally but is identical to its base type internally. Type safety without runtime overhead. → Scala 3 Feature Comparison
Option[A]: A type representing the presence (Some) or absence (None) of a value. Replacement for null. Safely handled with flatMap and For Comprehension. → Basics

L-O Key Points

lazy val: Defers expensive initialization until first use

Monad: flatMap + pure, core abstraction for sequential effect composition

Option: Type-safe handling with Some/None instead of null

P#

Partial Function: A function defined only for some inputs. Same form as Pattern Matching cases. Used with the collect method. → Functions and Methods
Pattern Matching: A technique for analyzing value structure and extracting data. Used with Case Class and Sealed traits. → Pattern Matching Details
Promise[T]: A type allowing direct completion of a Future. Used for wrapping callback-based APIs. → Concurrency

R#

Referential Transparency: The property that replacing an expression with its result value doesn’t change program meaning. Key characteristic of pure functions. Related to Immutable data. → Functional Patterns

S#

Sealed: A modifier restricting inheritance to the same file only. Used for Pattern Matching exhaustiveness checking. Essential for ADT definitions. → Pattern Matching
Singleton Object: A unique instance defined with the Object keyword. See Companion Object.
summon (Scala 3): A function to retrieve an implicit instance of a given type. Replaces implicit implicitly. → Type Classes

T#

Tail Recursion: Recursion where the last operation is a call to itself. Can be optimized without stack overflow. Verified with @tailrec annotation. → Functions and Methods
Trait: Similar to Java interfaces but can include implementations. Supports mixin inheritance. Forms ADT with Sealed. → Classes and Objects
Try[T]: A type containing the result of a computation that may throw exceptions. Either Success or Failure. Similar error handling pattern to Either and Option. → Basics
Type Class: A pattern for adding functionality to existing types. Implements ad-hoc polymorphism. Representative examples: Functor, Monad. → Type Classes Details
Type Inference: The compiler’s ability to automatically infer types. Scala’s powerful type inference reduces boilerplate. → Basics

P-T Key Points

Pattern Matching: Destructure data structures and branch with match expressions

Sealed: Restricts inheritance to same file, enables pattern matching exhaustiveness checking

Type Class: Add functionality to existing types, implement ad-hoc polymorphism

U#

Union Type (|): A type representing one of multiple types. Int | String. Opposite of Intersection Type. Scala 3 only. → Advanced Type System
Using (Scala 3): Keyword for declaring implicit parameters. Replaces Implicit. Pairs with Given. → Scala 2 vs 3 Comparison

V#

val: Keyword for declaring Immutable values. Compare with var. Default choice in Scala.
var: Keyword for declaring mutable variables. Less recommended than val. → Basics
Variance: Subtyping relationship of type parameters. Covariant (+A), contravariant (-A), invariant. Important for collection design. → Variance Details

Y#

yield: Keyword for generating values in For Comprehension. Transformed to map calls. → For Comprehension Details

U-Y Key Points

Union Type (|): Scala 3 only, concise type expression instead of Either

val/var: Prefer val (immutable), minimize var (mutable)

Variance: Define type relationships with covariant (+A), contravariant (-A)

Next Steps#

Concepts - Core Scala concepts
Examples - Learn by coding
Spark Integration - Big data processing
References - Books, courses, community
FAQ - Frequently asked questions

Glossary

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

DDD Glossary#

A compilation of Domain-Driven Design core terminology. For detailed explanations, see the Concepts section.

TL;DR

Strategic Design: Define domain boundaries and language with Bounded Context, Context Mapping, Ubiquitous Language

Tactical Design: Implement domain models with Entity, Value Object, Aggregate, Repository, Domain Event

Architecture Patterns: Structure systems with Layered, Hexagonal, CQRS, Event Sourcing

Strategic Design#

For details: Strategic Design

Bounded Context#

Definition: An explicit boundary within which a specific domain model applies and maintains consistency

Project Setup

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

Project Setup#

TL;DR

DDD layered architecture (Domain, Application, Infrastructure, Interfaces) package structure

Spring Boot 3.2.x + Spring Kafka + JPA based dependencies

AggregateRoot, DomainEvent, Entity base class implementations

Docker Compose for Kafka + PostgreSQL development environment

Target Audience and Prerequisites#

Item	Required Level
Target Audience	Backend developers looking to apply DDD patterns with Spring Boot
Java	Java 17+ syntax, Record, Stream API
Spring Boot	Spring Boot basics, DI, @Service, @Repository
Gradle	Kotlin DSL basic syntax
Docker	Experience running docker-compose up/down

Setting up the structure and dependencies for the DDD example project.

Strategic Design

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

Target Audience: Architects and senior developers who want to understand the big picture of DDD Prerequisites: Have read Quick Start or understand basic DDD concepts Estimated Time: About 30 minutes Key Question: “How should we divide the system, and how should each part collaborate?”

Summary
4 core concepts of Strategic Design: Subdomain (classify business areas) -> Ubiquitous Language (define common language) -> Bounded Context (set model boundaries) -> Context Mapping (define collaboration between boundaries)

Strategic design is high-level design that decides how to divide and integrate complex domains. It is the starting point of DDD and the core activity that determines the overall system architecture.

Basic Syntax

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers learning PromQL for the first time Prerequisites: Metrics Fundamentals What You’ll Learn: Write basic PromQL queries and retrieve desired metrics

TL;DR#

Key Summary:

Instant Vector: Current point-in-time value http_requests_total

Range Vector: Values over time range http_requests_total[5m]

Label Matching: Filter using =, !=, =~, !~

Operations: Arithmetic (+, -), Comparison (>, <), Logical (and, or)

Why Do We Need PromQL?#

Prometheus stores thousands of time series data points. But having data doesn’t automatically provide insights. To answer questions like “Is the system healthy right now?”, “How does it compare to yesterday?”, or “Which service has problems?”, you need a language to query and transform data. That’s PromQL.

Debugging High Latency

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Scenario: P99 response time exceeds SLA (500ms) Goal: Identify and resolve bottlenecks Duration: 15~30 minutes (depending on problem complexity) Success Criteria: P99 response time recovers below SLA threshold (500ms)

Problem Scenario#

Alert: HighP99Latency
Service: order-service
P99: 2.5s (Threshold: 500ms)
Duration: 10 minutes

Diagnostic Workflow#

graph TD
 A["1. Scope Analysis<br>Which service? Since when?"]
 B["2. Segment Analysis<br>Where is it slow?"]
 C["3. Resource Check<br>CPU/Memory/DB?"]
 D["4. Root Cause<br>Code? Query? External?"]
 E["5. Resolution"]

 A --> B --> C --> D --> E

Step 1: Scope Analysis#

Check Impact Scope#

# Which service is slow?
topk(5,
 histogram_quantile(0.99,
 sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
 )
)

# Since when did it become slow?
histogram_quantile(0.99,
 sum by (le) (rate(http_request_duration_seconds_bucket{service="order-service"}[5m]))
)
# → Time range: Last 1 hour

Check Specific Endpoints#

# P99 by endpoint
histogram_quantile(0.99,
 sum by (uri, le) (rate(http_request_duration_seconds_bucket{service="order-service"}[5m]))
)

Result: /orders POST endpoint is slow

Environment Setup

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Time Required: 15 minutes Prerequisites: Docker, Docker Compose basics What You’ll Learn: Build a complete observability stack locally

Overall Architecture#

graph TB
 APP["Spring Boot App"] --> |"metrics"| PROM["Prometheus"]
 APP --> |"logs"| LOKI["Loki"]
 APP --> |"traces"| TEMPO["Tempo"]

 PROM --> GF["Grafana"]
 LOKI --> GF
 TEMPO --> GF

 PROM --> AM["Alertmanager"]

Step 1: Create Directory Structure#

mkdir -p observability-stack/{prometheus/rules,grafana/{provisioning/datasources,provisioning/dashboards,dashboards},loki,promtail}
cd observability-stack

Step 2: Write Docker Compose#

# docker-compose.yml
services:
 # Metrics collection
 prometheus:
 image: prom/prometheus:v2.50.0
 container_name: prometheus
 ports:
 - "9090:9090"
 volumes:
 - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
 - ./prometheus/rules:/etc/prometheus/rules
 - prometheus-data:/prometheus
 command:
 - '--config.file=/etc/prometheus/prometheus.yml'
 - '--storage.tsdb.path=/prometheus'
 - '--web.enable-lifecycle'
 - '--web.enable-remote-write-receiver'

 # Alert management
 alertmanager:
 image: prom/alertmanager:v0.26.0
 container_name: alertmanager
 ports:
 - "9093:9093"
 volumes:
 - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml

 # Visualization
 grafana:
 image: grafana/grafana:10.3.0
 container_name: grafana
 ports:
 - "3000:3000"
 environment:
 - GF_SECURITY_ADMIN_PASSWORD=admin
 - GF_AUTH_ANONYMOUS_ENABLED=true
 - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
 volumes:
 - ./grafana/provisioning:/etc/grafana/provisioning
 - ./grafana/dashboards:/var/lib/grafana/dashboards
 - grafana-data:/var/lib/grafana

 # Log collection
 loki:
 image: grafana/loki:2.9.0
 container_name: loki
 ports:
 - "3100:3100"
 volumes:
 - ./loki/loki.yml:/etc/loki/local-config.yaml
 - loki-data:/loki
 command: -config.file=/etc/loki/local-config.yaml

 # Log agent
 promtail:
 image: grafana/promtail:2.9.0
 container_name: promtail
 volumes:
 - ./promtail/promtail.yml:/etc/promtail/config.yml
 - /var/log:/var/log:ro
 - /var/lib/docker/containers:/var/lib/docker/containers:ro
 command: -config.file=/etc/promtail/config.yml

 # Distributed tracing
 tempo:
 image: grafana/tempo:2.3.0
 container_name: tempo
 ports:
 - "4317:4317" # OTLP gRPC
 - "4318:4318" # OTLP HTTP
 - "3200:3200" # Tempo API
 volumes:
 - ./tempo/tempo.yml:/etc/tempo/config.yaml
 - tempo-data:/var/tempo
 command: -config.file=/etc/tempo/config.yaml

volumes:
 prometheus-data:
 grafana-data:
 loki-data:
 tempo-data:

Step 3: Prometheus Configuration#

# prometheus/prometheus.yml
global:
 scrape_interval: 15s
 evaluation_interval: 15s

alerting:
 alertmanagers:
 - static_configs:
 - targets:
 - alertmanager:9093

rule_files:
 - /etc/prometheus/rules/*.yml

scrape_configs:
 - job_name: 'prometheus'
 static_configs:
 - targets: ['localhost:9090']

 - job_name: 'spring-app'
 metrics_path: '/actuator/prometheus'
 static_configs:
 - targets: ['host.docker.internal:8080']

Step 4: Loki Configuration#

# loki/loki.yml
auth_enabled: false

server:
 http_listen_port: 3100

common:
 path_prefix: /loki
 storage:
 filesystem:
 chunks_directory: /loki/chunks
 rules_directory: /loki/rules
 replication_factor: 1
 ring:
 instance_addr: 127.0.0.1
 kvstore:
 store: inmemory

schema_config:
 configs:
 - from: 2020-10-24
 store: boltdb-shipper
 object_store: filesystem
 schema: v11
 index:
 prefix: index_
 period: 24h

limits_config:
 retention_period: 168h

Step 5: Promtail Configuration#

# promtail/promtail.yml
server:
 http_listen_port: 9080

positions:
 filename: /tmp/positions.yaml

clients:
 - url: http://loki:3100/loki/api/v1/push

scrape_configs:
 - job_name: containers
 static_configs:
 - targets:
 - localhost
 labels:
 job: containerlogs
 __path__: /var/lib/docker/containers/*/*log

 pipeline_stages:
 - json:
 expressions:
 output: log
 stream: stream
 attrs:
 - json:
 expressions:
 tag:
 source: attrs
 - regex:
 expression: (?P<container_name>(?:[^|]*[^|]))
 source: tag
 - labels:
 container_name:
 stream:

Step 6: Tempo Configuration#

# tempo/tempo.yml
server:
 http_listen_port: 3200

distributor:
 receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318

storage:
 trace:
 backend: local
 local:
 path: /var/tempo/traces
 wal:
 path: /var/tempo/wal

metrics_generator:
 registry:
 external_labels:
 source: tempo
 storage:
 path: /var/tempo/generator/wal
 remote_write:
 - url: http://prometheus:9090/api/v1/write
 send_exemplars: true

Step 7: Grafana Datasource Configuration#

# grafana/provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
 - name: Prometheus
 type: prometheus
 access: proxy
 url: http://prometheus:9090
 isDefault: true

 - name: Loki
 type: loki
 access: proxy
 url: http://loki:3100

 - name: Tempo
 type: tempo
 access: proxy
 url: http://tempo:3200
 jsonData:
 tracesToLogsV2:
 datasourceUid: loki
 filterByTraceID: true
 tracesToMetrics:
 datasourceUid: prometheus
 serviceMap:
 datasourceUid: prometheus

Step 8: Alertmanager Configuration#

mkdir -p alertmanager

# alertmanager/alertmanager.yml
global:
 resolve_timeout: 5m

route:
 receiver: 'default'
 group_by: ['alertname', 'job']
 group_wait: 30s
 group_interval: 5m
 repeat_interval: 4h

receivers:
 - name: 'default'
 webhook_configs:
 - url: 'http://localhost:5001/'
 send_resolved: true

Step 9: Start the Stack#

docker compose up -d

Expected Output:

Glossary

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Quick Navigation: A | C | E | F | G | H | I | L | M | O | P | R | S | T | U | W

A#

Alerting Rules#

Rules in Prometheus that define condition-based alerts. When conditions are met, alerts are sent to Alertmanager.

Alertmanager#

A component that receives alerts from Prometheus and groups, inhibits, and routes them.

C#

Cardinality#

The number of unique time series. Higher cardinality occurs with more label combinations.

Context Propagation#

A mechanism for passing Trace ID and Span ID between services in distributed systems.

Latency (Response Time)

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs looking to improve service response times Prerequisites: histogram_quantile After reading this: You’ll be able to accurately measure latency and set up SLA-based alerts

TL;DR#

Key Summary:

P50 (median): Typical user experience

P95: Most users’ experience

P99: Worst-case user experience (SLA baseline)

Percentiles better reflect actual experience than averages

Measure latency of successful/failed requests separately

Why Percentiles Matter#

Latency is quality directly felt by users. No matter how great the features, users leave if they have to wait 5 seconds. According to Amazon’s research, 100ms of added latency results in 1% decrease in sales.

Three Pillars of Observability

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers new to Observability concepts Prerequisites: Basic understanding of web application architecture After Reading: You’ll understand the role of each pillar and know when to use which

TL;DR#

Key Summary:

Metrics: “How much?” - Numerically measurable states (CPU 80%, response time 200ms)

Logs: “What happened?” - Detailed records of individual events

Traces: “From where to where?” - Tracking the entire path of a request

The 3 pillars are complementary, most effective when used together

Why Are All Three Pillars Necessary?#

Analogy: A Doctor’s Diagnostic Process

Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to understand Kubernetes structure Prerequisites: Basic Docker concepts After reading this: You will understand Kubernetes cluster components and their roles

TL;DR

Kubernetes cluster consists of Control Plane (brain) and Worker Nodes (muscle)

Control Plane manages cluster state, Worker Nodes execute actual workloads

kubectl commands communicate with the cluster through API Server

Overall Cluster Structure#

A Kubernetes cluster is broadly divided into two parts.

Environment Setup

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Developers who want to practice Kubernetes locally Prerequisites: Docker basics After reading this: You’ll be able to set up a local Kubernetes environment with Minikube or Kind

TL;DR

Minikube: Single node, recommended for beginners, rich addon support

Kind: Multi-node capable, suitable for CI/CD, fast cluster creation

Docker Desktop: Simplest setup for macOS/Windows

Local Kubernetes Options Comparison#

Tool	Pros	Cons	Best For
Minikube	Rich addons, extensive docs	Single node only	Learning, development
Kind	Fast, multi-node, CI-friendly	Fewer addons	CI/CD, testing
Docker Desktop	Easy installation	High resource usage	Quick start
k3d	Lightweight, fast	Different from production	Quick testing

Minikube Installation#

Install#

macOS

# Install via Homebrew
brew install minikube

# Check version
minikube version

Linux

# Download binary
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

# Install
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Check version
minikube version

Windows

# Install via Chocolatey
choco install minikube

# Or manual installation
# https://minikube.sigs.k8s.io/docs/start/

Start Cluster#

# Start with default settings
minikube start

# Specify resources
minikube start --cpus=4 --memory=8192

# Use specific Kubernetes version
minikube start --kubernetes-version=v1.29.0

# Specify driver
minikube start --driver=docker

Available drivers:

Glossary

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

This glossary organizes core terms needed for Kubernetes learning and operations.

A-C#

Cluster#

The entire Kubernetes environment composed of multiple nodes (machines). It consists of one Control Plane and multiple Worker Nodes.

ConfigMap#

A resource that stores configuration data as key-value pairs. Can be injected into Pods as environment variables or configuration files.

Container#

An isolated execution environment that packages an application and its dependencies. Docker containers are the most representative.

Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Spark consists of Driver (coordinator) + Executor (workers) + Cluster Manager (resource management)

All Transformations are represented as DAG, split into Job → Stage → Task on Action call

Memory uses Unified Memory model where Execution (computation) and Storage (cache) dynamically share

For Java/Spring developers, SparkSession is like Spring Container, Executor is like Thread Pool Worker

Target Audience: Developers with Java/Spring backend development experience

Prerequisites:

Java basic syntax and JVM memory structure understanding
Basic multithreading concepts (Thread, ExecutorService)
Basic distributed systems concepts (optional)

Understand how Spark applications run in a distributed environment. We’ll explain using concepts familiar to Java/Spring developers.

Consumer Lag Troubleshooting

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

A step-by-step guide to diagnosing and resolving Consumer Lag issues.

TL;DR

Consumer Lag: Number of unread messages (Producer rate > Consumer processing rate)

Diagnosis: Check Lag with kafka-consumer-groups.sh, analyze trends with monitoring tools

Resolution: Increase Consumer count, optimize processing logic, adjust Partition count, tune batch settings

What is Consumer Lag?#

Consumer Lag is the number of messages that the Consumer has not yet read from those published by the Producer. If Lag continuously increases, message processing delays occur, and in the worst case, messages may be deleted after exceeding the retention period.

Core Components

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Cluster: A group of servers providing high availability and distributed processing

Node: A single Elasticsearch server in the cluster (Master, Data, Coordinating roles)

Index: A logical collection of documents (similar to RDB table)

Document: A JSON data unit (similar to RDB row)

Shard: A horizontal partition of an index (consists of Primary/Replica)

Target Audience: Developers new to Elasticsearch Prerequisites: JSON basics, REST API concepts

Understand the roles and relationships of Elasticsearch’s core components: Cluster, Node, Index, Document, and Shard.

Core Components

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Producer: Publishes messages and selects Partitions to send to Broker

Consumer: Reads messages from Partitions as part of a Consumer Group and manages Offsets

Broker: Stores and replicates messages, providing high availability through Leader/Follower structure

Topic: A logical channel for categorizing messages

Partition: A unit that physically divides Topics to enable parallel processing

Target Audience: Developers new to Kafka or those learning basic concepts of distributed messaging systems

Prerequisites: Basic network communication concepts, REST API experience, Spring Boot fundamentals

Defining Aggregate Boundaries

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

Problem It Solves: Performance issues from making Aggregates too large, or consistency violations from making them too small Time Required: ~30 minutes Prerequisites: Assumes you have read the Aggregate Deep Dive document

Success Criteria
After completing this guide, you will be able to:

Identify invariants to determine Aggregate boundaries

Decide whether to include an Entity using 4 key questions

Recognize signals when boundaries are wrong

1. Identify Core Invariants#

An Invariant is a business rule that must always be true. It is the most important criterion for defining Aggregate boundaries.

Environment Setup

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Quickly set up Elasticsearch 8.11 + Kibana environment with Docker Compose

Complete integration setup with Spring Boot 3.2 + Spring Data Elasticsearch

Install Nori analyzer for Korean search

Total time required: approximately 15 minutes

Version Information

Elasticsearch / Kibana: 8.11.0

Spring Boot: 3.2.0

Spring Data Elasticsearch: 5.2.x (included in Spring Boot 3.2)

Java: 17+

Configure Elasticsearch + Kibana with Docker and set up a Spring Boot project.

Docker Environment Setup#

docker-compose.yml#

docker/elasticsearch/docker-compose.yml file:

Environment Setup

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

Reference guide for setting up Kafka with Spring Boot.

TL;DR

Kafka Execution: Run Kafka 3.6.1 in KRaft mode via Docker Compose

Dependencies: Add spring-kafka, spring-boot-starter-web

Basic Settings: Configure bootstrap-servers, Serializer/Deserializer, group-id

Production: Ensure stability with acks: all, enable.idempotence: true

Target Audience and Prerequisites#

Item	Description
Target Audience	Developers setting up Kafka environment in Spring Boot projects
Prerequisites	Basic Docker usage, Gradle or Maven build tools, Spring Boot configuration files (application.yml)
Required Tools	Docker Desktop or Docker Engine, JDK 17+, IDE (IntelliJ IDEA recommended)
Estimated Time	About 15 minutes

If you completed Quick Start, you’ve already set up the basic environment. This document is a reference guide for configuration details and production environment setup.

Glossary

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Index/Document/Field: Corresponds to Table/Row/Column in RDB

Shard/Replica: Basic units of data distribution and replication

Analyzer/Tokenizer: Breaks text into searchable tokens

Query/Filter Context: Search methods distinguished by scoring calculation

Sorted alphabetically, each term links to related concept documents

Quick reference for Elasticsearch core terms. For detailed explanations, see the Concepts section.

A-E#

Aggregation#

Feature for grouping search results and calculating statistics. Similar to SQL’s GROUP BY. Three types: Bucket/Metric/Pipeline. → Aggregations | Query DSL

Glossary

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

This glossary covers key Kafka terminology. For detailed explanations of each term, refer to the Concepts section. Terms are sorted alphabetically, with cross-references to related terms.

TL;DR

Core Components: Topic (logical channel), Partition (parallel processing unit), Broker (server), Producer (publisher), Consumer (receiver)

Reliability: ACK (delivery confirmation), ISR (synchronized replicas), Replication Factor (number of copies)

Consumer Management: Consumer Group (parallel processing), Offset (position), Commit (save), Rebalancing (redistribution)

Spring Kafka: KafkaTemplate (Producer), @KafkaListener (Consumer)

Metadata Management: KRaft (recommended for new clusters), Zookeeper (legacy)

A#

ACK (Acknowledgment)

Environment Setup

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Environment Setup#

Configure the environment for using Spark in Java/Spring Boot projects.

Pure Java Project#

The simplest configuration.

build.gradle#

plugins {
 id 'java'
 id 'application'
}

group = 'com.example'
version = '1.0.0'

java {
 sourceCompatibility = '17'
}

repositories {
 mavenCentral()
}

dependencies {
 // Spark Core
 implementation 'org.apache.spark:spark-core_2.13:3.5.1'

 // Spark SQL (DataFrame API)
 implementation 'org.apache.spark:spark-sql_2.13:3.5.1'

 // Kafka integration (optional)
 implementation 'org.apache.spark:spark-sql-kafka-0-10_2.13:3.5.1'

 // MLlib (optional)
 implementation 'org.apache.spark:spark-mllib_2.13:3.5.1'

 // Logging (conflict prevention)
 implementation 'org.slf4j:slf4j-simple:2.0.9'
}

// Logging conflict prevention
configurations.all {
 exclude group: 'org.slf4j', module: 'slf4j-log4j12'
 exclude group: 'log4j', module: 'log4j'
}

application {
 mainClass = 'com.example.SparkApp'
}

// Memory settings for execution
tasks.named('run') {
 jvmArgs = ['-Xmx2g']
}

Basic Structure#

src/
├── main/
│ ├── java/
│ │ └── com/example/
│ │ └── SparkApp.java
│ └── resources/
│ └── data/
│ └── sample.csv

SparkApp.java#

package com.example;

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

public class SparkApp {
 public static void main(String[] args) {
 SparkSession spark = SparkSession.builder()
 .appName("My Spark App")
 .master("local[*]")
 .config("spark.driver.memory", "2g")
 .getOrCreate();

 spark.sparkContext().setLogLevel("WARN");

 // Data processing
 Dataset<Row> df = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .csv("src/main/resources/data/sample.csv");

 df.show();

 spark.stop();
 }
}

Spring Boot Integration#

Configuration for using Spark with Spring Boot.

Glossary

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Glossary#

Key terminology and concepts used in Spark. Each term links to related documentation.

Core Concepts#

Action#

Operations that trigger actual computation on RDD/DataFrame and return results. Examples include count(), collect(), show(), write(). → Transformations and Actions

Application#

A Spark program submitted by the user. Consists of a Driver and Executors. → Architecture

Broadcast Variable#

A shared variable distributed to all nodes as read-only. Used for efficiently sharing small datasets. → Performance Tuning

Catalyst Optimizer#

Spark SQL’s query optimization engine. Transforms logical plans into optimized physical plans. → Spark SQL

Future Error Handling

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Learn how to safely handle and debug exceptions in asynchronous code.

Estimated time: 15-20 minutes

TL;DR

recover/recoverWith: Handle specific exceptions and return fallback values

transform: Transform both success and failure cases

onComplete: Use only for side effects (logging, etc.)

Never do this: Use Await.result to throw exceptions synchronously

What This Guide Solves#

Use this guide when:

Exceptions in Future silently disappear
You don’t know how to handle errors in asynchronous code
You need to decide between recover, recoverWith, and transform

Symptoms#

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

val future = Future {
 throw new RuntimeException("Something went wrong")
}
// Program exits normally, but where did the exception go?

What This Guide Does NOT Cover#

Basic Future concepts: See Concurrency Concepts
Cats Effect IO / ZIO error handling: Refer to respective library documentation
Akka actor system error handling: Refer to Akka documentation

Before You Begin#

Ensure you have the following environment ready:

Identifying Bounded Contexts

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Problem Solved: Unsure how to divide the system, or high inter-team dependencies due to incorrectly defined Context boundaries Time Required: ~25 min Prerequisites: Assumes you have read the Strategic Design document

Success Criteria
After completing this guide, you will be able to:

Identify Context boundaries using 3 signals

Validate boundaries through domain expert interviews

Recognize and fix symptoms of incorrect boundaries

1. Detecting Signals That Indicate Context Boundaries#

Check the following 3 signals to determine if Bounded Context separation is needed.

Memory Troubleshooting

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Learn how to diagnose and resolve OutOfMemoryError and GC issues.

Duration: Approximately 20-40 minutes (additional 10 minutes for GC log analysis)

Scope of This Guide
Covered: Heap memory settings, Circuit Breaker, Field Data optimization, GC tuning

Not Covered: Adding nodes, hardware upgrades - see Cluster Management

TL;DR

Heap memory: 50% or less of total memory, maximum 31GB

Circuit Breaker: Check settings to prevent memory overuse

Field data: Avoid aggregations on text fields, use doc_values

GC tuning: Use G1GC, analyze logs to identify issues

Before You Begin#

Verify the following requirements:

Resolving Data Skew

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Estimated Time: About 20 minutes

TL;DR

Diagnosis: Compare Task Duration Min/Max in Spark UI Stages tab (10x+ difference = skew)

Enable AQE: spark.sql.adaptive.skewJoin.enabled=true (Spark 3.0+)

Manual Fix: Distribute hot keys using Salting technique

Problem Definition#

A Spark job mostly completes quickly but some Tasks take much longer:

Stage 3: 199/200 tasks completed... (last 1 running for tens of minutes)

This is Data Skew - data is concentrated on specific keys, causing those partitions to be overloaded.

Resource Optimization

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Objective: Find appropriate CPU/memory settings to improve resource efficiency Estimated Time: 45 minutes

Scope of This Guide
Covers: Measuring resource usage, determining appropriate requests/limits values, resolving throttling/OOM

Does not cover: Auto-scaling (see Scaling), Pod startup issues (see Pod Troubleshooting)

Before You Begin#

Verify the following prerequisites.

1. Verify kubectl Installation and Version#

kubectl version --client

Success output:

Client Version: v1.28.0

2. Verify Metrics Server Installation#

kubectl top nodes

Success output:

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
node-1 250m 12% 1024Mi 50%

If you get an error, install Metrics Server.

Hexagonal Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 15 Jan 2026 00:00:00 +0000

Target Audience: Developers considering testability and external dependency replacement Prerequisites: Understanding the limitations of Layered Architecture Estimated Time: About 20 minutes

Also known as the Ports and Adapters pattern. An architecture that completely isolates the application core from the external world. The core idea of hexagonal architecture is to place business logic at the center and handle all external interactions through Ports and Adapters. This way, even if external technologies change, core business logic remains unaffected.

Basic Examples

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Model immutable data with case classes, utilize auto-generated equals and copy

Option for null safety, Either for detailed error handling

Pattern matching for type-based branching

Collection API (map, filter, groupBy) for declarative data processing

Type classes for extensible functionality implementation

Target Audience: Developers who have completed Scala environment setup and want to learn Scala through practical code

Prerequisites:

Completed Scala development environment setup (see Environment Setup)
Basic object-oriented programming concepts
Functional programming basics (recommended)

Comprehensive examples utilizing core Scala concepts. This document demonstrates commonly used features in real-world scenarios with example code, including case classes, pattern matching, collection processing, Option, Either, and type classes.

Control Structures

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Scala’s if, for, and match are all expressions that return values

for comprehensions concisely express collection transformations and monadic operations

match is more powerful than Java’s switch, supporting type matching and guard conditions

while is a statement, so avoid it in functional code

Target Audience: Developers with experience in other languages like Java/Python Prerequisites: Scala basic syntax (variables, types)

Scala’s control structures are expressions. That is, all control structures return values. This is fundamentally different from statement-based control structures in languages like Java or C. The expression-based approach makes code more concise and functional.

Scala 2 vs Scala 3 Version Comparison

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

An overview of the main differences between Scala 2 and Scala 3. Scala 3, released in 2020, provides more concise syntax, a more powerful type system, and improved implicit features.

TL;DR

New Projects: Scala 3 recommended (more concise syntax, improved type system)

Using Spark: Maintain Scala 2.12/2.13 (Spark doesn’t support Scala 3 yet)

Key Changes: implicit → given/using, indentation-based syntax, enum added

Compatibility: Scala 3 can use Scala 2.13 libraries

Migration: Gradual transition with -source:3.0-migration option

New Features (Scala 3)#

Major features added in Scala 3, organized by category.

Order Domain

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

Order Domain Implementation#

TL;DR

Order: Aggregate Root. Manages the order’s consistency boundary

OrderLine: Internal Entity. Can only be created/modified through Order

Money, ShippingAddress, OrderId: Value Objects. Immutable and compared by value

Invariants: At least 1 order line, max amount 100M won, quantity 1~999

Domain Events: OrderCreatedEvent, OrderConfirmedEvent, etc. published on state changes

Target Audience and Prerequisites#

Item	Required Level
Target Audience	Developers implementing DDD tactical patterns in code
DDD Basics	Understanding of Aggregate, Entity, Value Object, Domain Event concepts
Java	Experience with Record, Optional, Stream API
Prerequisite	Project Setup completed

Implementing the order domain with DDD patterns.

References

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

References#

Books, articles, and presentations for learning DDD.

TL;DR

Essential Reading: Blue Book (original), Red Book (implementation), DDD Distilled (intro)

Learning Order: Intro -> Fundamentals -> Advanced -> Practice (progressive learning)

Practice: Start with refactoring existing projects or EventStorming

Essential Reading#

Domain-Driven Design (Blue Book)#

Author: Eric Evans
Title: Domain-Driven Design: Tackling Complexity in the Heart of Software (2003)

The original text of DDD. Contains the principles and philosophy of strategic/tactical design patterns.

Tactical Design

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to implement DDD building blocks in code Prerequisites: Read Strategic Design or understand Bounded Context concept Time Required: About 40 minutes Key Question: “What patterns should be used to implement domain models?”

Summary
Tactical Design Building Blocks: Entity(distinguished by identity) + Value Object(distinguished by value) → Aggregate(consistency boundary) + Repository(persistence) + Domain Service(domain logic) + Domain Event(event communication)

Tactical design consists of patterns for concretely implementing domain models. If strategic design draws the “big picture,” tactical design provides “specific implementation methods.”

Aggregation Operators

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers who know basic PromQL syntax Prerequisites: Basic Syntax What You’ll Learn: Aggregate multiple time series to calculate meaningful metrics

TL;DR#

Key Summary:

sum: Total - total request count

avg: Average - average response time

count: Count - number of active instances

topk/bottomk: Top/bottom N items

by/without: Specify grouping criteria

Aggregation Operators List#

Operator	Description	Returns
`sum`	Sum	Single value or per-group values
`avg`	Average	Single value or per-group values
`min`	Minimum	Single value or per-group values
`max`	Maximum	Single value or per-group values
`count`	Number of time series	Single value or per-group values
`stddev`	Standard deviation	Single value or per-group values
`stdvar`	Variance	Single value or per-group values
`topk`	Top K items	K time series
`bottomk`	Bottom K items	K time series
`count_values`	Count by value	One time series per value
`quantile`	Quantile	Single value or per-group values

sum (Total)#

Calculates the sum of all time series values.

FAQ

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

General#

Q: What’s the difference between Monitoring and Observability?#

Monitoring: Watching predefined metrics (“Has this value exceeded the threshold?”)

Observability: The ability to understand the internal state of a system from the outside (“Why did this problem occur?”)

Monitoring is part of Observability. Observability includes the ability to analyze unexpected problems.

Q: Which of the Three Pillars (Metrics, Logs, Traces) should I implement first?#

Recommended order:

Metrics - Understand system state, set up alerts
Logs - Analyze error causes
Traces - Analyze distributed system flows

For a single service, Metrics + Logs may be sufficient. For microservices, Traces are essential.

Metrics Fundamentals

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers designing Prometheus metrics for the first time Prerequisites: Three Pillars of Observability After Reading: You’ll be able to select the appropriate metric type and implement it correctly

TL;DR#

Key Summary:

Counter: Only cumulative increase (request count, error count) → Calculate rate of change with rate()

Gauge: Current value (temperature, memory) → Use as-is or avg() for average

Histogram: Distribution measurement (response time) → Calculate percentiles with histogram_quantile()

Summary: Percentile calculation on client (rarely used)

Why Are Metric Types Important?#

Metric types are not just a technical choice. They reflect the nature of the data.

Optimizing Metric Cardinality

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Scenario: Prometheus memory/storage spike, slow queries Goal: Optimize costs by reducing unnecessary time series Duration: 30 minutes~1 hour (depending on analysis and fix complexity) Success Criteria: Time series count reduced below target and memory usage stabilized

Problem Scenario#

Alert: PrometheusHighCardinality
Active Series: 2,500,000 (Threshold: 1,000,000)
Memory Usage: 32GB

What is Cardinality?#

Cardinality = Number of unique time series

http_requests_total{method="GET", status="200", path="/api/users"} # 1 series
http_requests_total{method="GET", status="200", path="/api/users/123"} # Another 1!
http_requests_total{method="GET", status="200", path="/api/users/456"} # Another 1!

Problem: If user_id is in path, time series created for each user

Spring Boot Metrics

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Time Required: 20 minutes Prerequisites: Spring Boot, Environment Setup What You’ll Learn: Apply observability to Spring Boot applications

Step 1/8: Add Dependencies (2 min)#

// build.gradle.kts
dependencies {
 // Actuator + Prometheus
 implementation("org.springframework.boot:spring-boot-starter-actuator")
 implementation("io.micrometer:micrometer-registry-prometheus")

 // Tracing
 implementation("io.micrometer:micrometer-tracing-bridge-otel")
 implementation("io.opentelemetry:opentelemetry-exporter-otlp")

 // Logging (Loki)
 implementation("com.github.loki4j:loki-logback-appender:1.4.2")
}

Step 2/8: application.yml Configuration (3 min)#

spring:
 application:
 name: order-service

management:
 endpoints:
 web:
 exposure:
 include: health,info,prometheus,metrics
 endpoint:
 health:
 show-details: always
 metrics:
 tags:
 application: ${spring.application.name}
 distribution:
 percentiles-histogram:
 http.server.requests: true
 slo:
 http.server.requests: 50ms, 100ms, 200ms, 500ms, 1s
 tracing:
 sampling:
 probability: 1.0
 otlp:
 tracing:
 endpoint: http://localhost:4318/v1/traces

logging:
 pattern:
 level: "%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]"

Step 3/8: Loki Log Configuration (3 min)#

<!-- src/main/resources/logback-spring.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
 <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

 <springProperty scope="context" name="appName" source="spring.application.name"/>

 <!-- Console Appender -->
 <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
 <encoder>
 <pattern>%d{ISO8601} %5p [${appName},%X{traceId:-},%X{spanId:-}] --- [%t] %-40.40logger{39} : %m%n</pattern>
 </encoder>
 </appender>

 <!-- Loki Appender -->
 <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
 <http>
 <url>http://localhost:3100/loki/api/v1/push</url>
 </http>
 <format>
 <label>
 <pattern>app=${appName},host=${HOSTNAME},level=%level</pattern>
 </label>
 <message>
 <pattern>{"timestamp":"%d{ISO8601}","level":"%level","logger":"%logger","message":"%msg","traceId":"%X{traceId:-}","spanId":"%X{spanId:-}"}</pattern>
 </message>
 </format>
 </appender>

 <root level="INFO">
 <appender-ref ref="CONSOLE"/>
 <appender-ref ref="LOKI"/>
 </root>
</configuration>

Intermediate Verification (Steps 1~3 Complete)#

Dependencies added to build.gradle.kts
Management configuration complete in application.yml
logback-spring.xml file created
After ./gradlew bootRun, metrics displayed at http://localhost:8080/actuator/prometheus

Step 4/8: Add Custom Metrics (5 min)#

@RestController
@RequiredArgsConstructor
@RequestMapping("/orders")
public class OrderController {
 private final MeterRegistry meterRegistry;
 private final OrderService orderService;

 // Counter: Order count
 private Counter orderCounter(String status) {
 return Counter.builder("orders_total")
 .tag("status", status)
 .description("Total number of orders")
 .register(meterRegistry);
 }

 // Gauge: Orders in progress
 private final AtomicInteger ordersInProgress = new AtomicInteger(0);

 @PostConstruct
 void registerGauge() {
 Gauge.builder("orders_in_progress", ordersInProgress, AtomicInteger::get)
 .description("Orders currently being processed")
 .register(meterRegistry);
 }

 @PostMapping
 public ResponseEntity<Order> createOrder(@RequestBody OrderRequest request) {
 ordersInProgress.incrementAndGet();
 try {
 // Timer to measure processing time
 return Timer.builder("order_processing_duration_seconds")
 .tag("type", request.getType())
 .publishPercentileHistogram()
 .register(meterRegistry)
 .record(() -> {
 Order order = orderService.create(request);
 orderCounter("success").increment();
 return ResponseEntity.ok(order);
 });
 } catch (Exception e) {
 orderCounter("failed").increment();
 throw e;
 } finally {
 ordersInProgress.decrementAndGet();
 }
 }
}

Step 5/8: Manual Tracing (3 min)#

@Service
@RequiredArgsConstructor
@Slf4j
public class OrderService {
 private final Tracer tracer;
 private final PaymentClient paymentClient;
 private final InventoryClient inventoryClient;

 public Order create(OrderRequest request) {
 // Add tags to current Span
 Span currentSpan = tracer.currentSpan();
 if (currentSpan != null) {
 currentSpan.tag("order.type", request.getType());
 currentSpan.tag("order.items", String.valueOf(request.getItems().size()));
 }

 // Create new Span
 Span inventorySpan = tracer.nextSpan().name("checkInventory").start();
 try (Tracer.SpanInScope ws = tracer.withSpan(inventorySpan)) {
 inventoryClient.check(request.getItems());
 inventorySpan.event("Inventory checked");
 } finally {
 inventorySpan.end();
 }

 // Process payment
 Span paymentSpan = tracer.nextSpan().name("processPayment").start();
 try (Tracer.SpanInScope ws = tracer.withSpan(paymentSpan)) {
 paymentClient.process(request.getPayment());
 paymentSpan.event("Payment processed");
 } finally {
 paymentSpan.end();
 }

 log.info("Order created: {}", request.getId());
 return new Order(request);
 }
}

Intermediate Verification (Steps 4~5 Complete)#

Counter, Gauge, Timer metrics added to OrderController
Manual Span creation code added to OrderService
Build successful without compilation errors: ./gradlew build

Step 6/8: Verify Metrics (2 min)#

Prometheus Endpoint#

curl http://localhost:8080/actuator/prometheus

Expected Output:

Traffic (Throughput)

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs establishing capacity planning Prerequisites: rate and increase After reading this: You’ll be able to analyze traffic patterns and use them for capacity planning

TL;DR#

Key Summary:

RPS (Requests Per Second): Requests per second

Throughput: Throughput per second (bytes, messages, etc.)

Concurrent Connections: Active connections

Traffic changes are a leading indicator of other signals

Why Monitor Traffic?#

Traffic measures the demand entering the system. If Latency shows “how fast”, Traffic shows “how much”. To judge system health, you need both. No matter how fast the response, something is wrong if no requests come in. If requests spike but resources are insufficient, failure is imminent.

Basic Example

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Developers who want to practice Kubernetes core resources hands-on Prerequisites: Pod, Deployment, Service concepts After reading this: You’ll be able to create and manage Pod, Deployment, Service, and ConfigMap directly

TL;DR

Create Nginx Deployment and expose via Service

Inject configuration with ConfigMap

Practice rolling updates and rollbacks

Prerequisites#

You need:

Local Kubernetes cluster (Minikube or Kind)
kubectl

# Check cluster status
kubectl cluster-info

Exercise 1: Deployment and Service#

Create Deployment#

Deploy a web server.

FAQ

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

This page organizes frequently asked questions and answers in Kubernetes learning and operations.

Before Getting Started#

Q: What should I know before learning Kubernetes?#

A: Having the following knowledge will make learning easier:

Required: Docker basics (building images, running containers)
Required: Basic Linux commands (cd, ls, cat, grep)
Recommended: YAML syntax, networking fundamentals (IP, ports, DNS)

Q: How can I practice Kubernetes locally?#

A: There are three methods:

Tool	Features	Recommended For
Minikube	Easiest, extensive documentation	Beginners
Kind	Quick start, multi-node	CI/CD
Docker Desktop	Easy installation	macOS/Windows

If you’re a beginner, start with Minikube: minikube start

Pod

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to understand Kubernetes Pod concept Prerequisites: Basic Docker container concepts After reading this: You will understand what Pods are and why we use Pods instead of containers

TL;DR

Pod is the smallest deployable unit in Kubernetes

A Pod contains one or more containers that share network and storage

Most Pods contain only a single container

What is a Pod?#

A Pod is the smallest deployable unit that can be created and managed in Kubernetes. While containers are the smallest unit in Docker, Pods are the smallest unit in Kubernetes.

Basic Examples

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Implement Document CRUD and basic search using Spring Data Elasticsearch

Handle simple queries with Repository pattern, complex queries with ElasticsearchOperations

Choose field types based on purpose: Text (for search), Keyword (for filtering), Integer (for range queries)

Total time required: approximately 20 minutes

Implement Document CRUD and basic search using Spring Data Elasticsearch.

Project Structure#

src/main/java/com/example/
├── ElasticsearchApplication.java
├── config/
│ └── ElasticsearchConfig.java
├── domain/
│ └── Product.java
├── repository/
│ └── ProductRepository.java
├── service/
│ └── ProductService.java
└── controller/
 └── ProductController.java

Domain Class#

Product.java#

package com.example.domain;

import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;
import org.springframework.data.elasticsearch.annotations.Setting;

@Document(indexName = "products")
@Setting(settingPath = "elasticsearch/settings.json") // Optional
public class Product {

 @Id
 private String id;

 @Field(type = FieldType.Text, analyzer = "standard")
 private String name;

 @Field(type = FieldType.Keyword)
 private String category;

 @Field(type = FieldType.Integer)
 private Integer price;

 @Field(type = FieldType.Text)
 private String description;

 @Field(type = FieldType.Boolean)
 private Boolean inStock;

 // Default constructor
 public Product() {}

 // All-args constructor
 public Product(String id, String name, String category,
 Integer price, String description, Boolean inStock) {
 this.id = id;
 this.name = name;
 this.category = category;
 this.price = price;
 this.description = description;
 this.inStock = inStock;
 }

 // Getter/Setter
 public String getId() { return id; }
 public void setId(String id) { this.id = id; }

 public String getName() { return name; }
 public void setName(String name) { this.name = name; }

 public String getCategory() { return category; }
 public void setCategory(String category) { this.category = category; }

 public Integer getPrice() { return price; }
 public void setPrice(Integer price) { this.price = price; }

 public String getDescription() { return description; }
 public void setDescription(String description) { this.description = description; }

 public Boolean getInStock() { return inStock; }
 public void setInStock(Boolean inStock) { this.inStock = inStock; }
}

Annotation Explanation#

Annotation	Description
`@Document`	Elasticsearch index mapping
`@Id`	Document unique ID field
`@Field`	Field type and analyzer settings
`@Setting`	Index settings file path

Why This Design?#

Field type selection rationale:

Data Modeling

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Mapping: Schema defining document structure (similar to RDB table definitions)

text: For full-text search, tokenized by Analyzer

keyword: For exact value matching, sorting/aggregation

Analyzer: Converts text into searchable tokens (use Nori for Korean)

Denormalization: Include related data in one document since there’s no JOIN

Target Audience: Developers looking to use Elasticsearch search features Prerequisites: Core Components, basic JSON syntax

This document covers Mapping, Field Type, and Analyzer design for effectively storing and searching data in Elasticsearch.

FAQ

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

ES vs RDB: Use ES for full-text search/log analysis, RDB for transactions/JOINs

text vs keyword: text is for search (analyzed), keyword is for filter/sort (not analyzed)

Performance issues: Use Filter Context, return only needed fields, search_after pagination

Korean search: Nori analyzer installation required

Common errors: Most caused by disk shortage, memory shortage, or type mismatch

Frequently asked questions and answers.

Basic Concepts#

Elasticsearch vs RDB, when should I use what?#

Situation	Recommendation
Full-text search	Elasticsearch
Transaction integrity needed	RDB
Complex JOINs	RDB
Real-time aggregation/analysis	Elasticsearch
General CRUD	RDB
High-volume log storage	Elasticsearch

Common pattern: Use RDB as main store, Elasticsearch as search secondary store.

Implementing Producer/Consumer with Spring Kafka

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

This document provides a step-by-step guide to sending and receiving messages using Spring Kafka.

TL;DR

Producer: Synchronous/asynchronous message sending with KafkaTemplate, Partition assignment using Key

Consumer: Message reception with @KafkaListener, batch processing and pattern subscription support

Manual Commit: Explicit commit after processing completion with Acknowledgment

Error Handling: Retry and Dead Letter Topic configuration with @RetryableTopic

Before You Begin#

Item	Requirement
Target Audience	Backend developers who want to use Kafka in Spring Boot applications
Prerequisites	Basic Java syntax, Spring Boot fundamentals, Kafka basic concepts
Prior Completion	Quick Start example completed, Environment Setup configuration done
Estimated Time	About 30 minutes

Platform-specific Notes
Windows Users: Use gradlew.bat instead of ./gradlew in commands.

Message Flow

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Producer serializes messages and Partitioner determines target Partition

With Key, same Key goes to same Partition ensuring order guarantee

Broker stores messages in Partition, replicates to ISR, then returns ACK

Consumer fetches messages via Pull method, commits Offset after processing

Three delivery guarantee levels: At-Most-Once, At-Least-Once, Exactly-Once

Target Audience: Developers who understand Kafka’s basic components, those learning detailed message delivery operations

Prerequisites: Understanding of Producer, Consumer, Broker, Topic, Partition concepts from Core Components

Producer Performance Tuning

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

A step-by-step guide to increasing Producer throughput and reducing latency.

TL;DR

Throughput priority: Increase batch.size, set linger.ms, enable compression

Latency priority: linger.ms=0, acks=1, small batch size

Durability priority: acks=all, enable.idempotence=true, retry settings

Trade-offs: Throughput, latency, and durability are competing concerns

Three Axes of Performance Optimization#

Producer performance optimization is about balancing three factors:

flowchart TB
 T[Throughput] <--> L[Latency]
 L <--> D[Durability]
 D <--> T

 style T fill:#e1f5fe
 style L fill:#fff3e0
 style D fill:#e8f5e9

Factor	Description	Optimization Direction
Throughput	Messages sent per second	Increase batch size, enable compression
Latency	Time from send to acknowledgment	Reduce batch wait time
Durability	Message loss prevention	acks=all, wait for replication

Step 1: Measure Current Performance#

Measure current performance before optimizing. Without a baseline, you cannot know the improvement effect.

RDD Basics

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

RDD (Resilient Distributed Dataset) is a distributed immutable data collection, Spark’s basic data abstraction

Data processing through Transformation (lazy evaluation) and Action (immediate execution)

Automatic recovery on failure through Lineage

DataFrame/Dataset recommended now, but RDD useful when low-level control needed

Target Audience: Java/Spring developers, beginners learning Spark basics

Prerequisites:

Java Collections API (List, Map, etc.)
Lambda expressions and functional programming basics
Understanding of Architecture document

RDD is Spark’s most fundamental data abstraction. As the low-level API underlying DataFrame and Dataset, it’s essential for understanding how Spark works.

References

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

This section compiles additional resources for learning Kafka. From official documentation to books, online courses, communities, and tools, we’ve gathered resources for systematic learning.

TL;DR

Official Documentation: Apache Kafka, Spring Kafka, Confluent docs are essential

Beginner Books: “Kafka: The Definitive Guide”, “Kafka in Action” recommended

Online Courses: Confluent Developer (free), Udemy, Inflearn

Communities: Stack Overflow, Confluent Forum, OKKY

Tools: Kafka UI (management), Prometheus+Grafana (monitoring)

Official Documentation#

Apache Kafka

Apache Kafka official documentation provides the most accurate and up-to-date information. Apache Kafka Documentation covers Kafka’s complete architecture, configuration, and APIs. Kafka Quickstart is the official Quick Start guide for beginners, covering installation through basic message sending/receiving. Kafka APIs provides detailed references for Producer API, Consumer API, Streams API, Admin API, etc. Kafka Configuration includes a complete list and descriptions of all configuration options for Broker, Producer, Consumer, Topic, etc.

Basic Examples

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Basic Examples#

Example code utilizing Spark’s core features.

Data Loading#

Reading CSV Files#

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

public class CsvExample {
 public static void main(String[] args) {
 SparkSession spark = SparkSession.builder()
 .appName("CSV Example")
 .master("local[*]")
 .getOrCreate();

 // Basic read
 Dataset<Row> df = spark.read()
 .option("header", "true") // First row as header
 .option("inferSchema", "true") // Auto type inference
 .csv("data/employees.csv");

 // Detailed options
 Dataset<Row> dfDetailed = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .option("sep", ",") // Delimiter
 .option("quote", "\"") // Quote character
 .option("escape", "\\") // Escape character
 .option("nullValue", "NA") // Null representation
 .option("dateFormat", "yyyy-MM-dd")
 .option("timestampFormat", "yyyy-MM-dd HH:mm:ss")
 .csv("data/employees.csv");

 df.show();
 df.printSchema();

 spark.stop();
 }
}

Reading JSON Files#

// Single file
Dataset<Row> df = spark.read().json("data/users.json");

// Multiline JSON
Dataset<Row> dfMulti = spark.read()
 .option("multiLine", "true")
 .json("data/users_multiline.json");

// JSON Lines (one JSON object per line)
Dataset<Row> dfLines = spark.read().json("data/users.jsonl");

df.show();

Reading Parquet Files#

// Parquet (recommended format)
Dataset<Row> df = spark.read().parquet("data/users.parquet");

// Read only specific columns (column pruning)
Dataset<Row> selected = spark.read()
 .parquet("data/users.parquet")
 .select("id", "name");

Reading from Database Using JDBC#

Dataset<Row> df = spark.read()
 .format("jdbc")
 .option("url", "jdbc:mysql://localhost:3306/mydb")
 .option("dbtable", "employees")
 .option("user", "user")
 .option("password", "password")
 .option("driver", "com.mysql.cj.jdbc.Driver")
 .load();

// Using query
Dataset<Row> dfQuery = spark.read()
 .format("jdbc")
 .option("url", "jdbc:mysql://localhost:3306/mydb")
 .option("query", "SELECT * FROM employees WHERE age > 30")
 .option("user", "user")
 .option("password", "password")
 .load();

Data Transformation#

Column Operations#

import static org.apache.spark.sql.functions.*;

Dataset<Row> employees = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .csv("data/employees.csv");

// Add new column
Dataset<Row> withBonus = employees.withColumn(
 "bonus",
 col("salary").multiply(0.1)
);

// Add multiple columns
Dataset<Row> enhanced = employees
 .withColumn("bonus", col("salary").multiply(0.1))
 .withColumn("total_compensation", col("salary").plus(col("bonus")))
 .withColumn("hire_year", year(col("hire_date")))
 .withColumn("name_upper", upper(col("name")));

// Rename column
Dataset<Row> renamed = employees.withColumnRenamed("name", "employee_name");

// Drop columns
Dataset<Row> dropped = employees.drop("middle_name", "suffix");

// Cast column type
Dataset<Row> casted = employees.withColumn(
 "salary",
 col("salary").cast("double")
);

enhanced.show();

Filtering#

// Single condition
Dataset<Row> highEarners = employees.filter(col("salary").gt(60000));

// Multiple conditions
Dataset<Row> filtered = employees.filter(
 col("age").geq(30)
 .and(col("department").equalTo("Engineering"))
 .and(col("salary").between(50000, 80000))
);

// String conditions
Dataset<Row> smithFamily = employees.filter(col("name").startsWith("Smith"));
Dataset<Row> hasEmail = employees.filter(col("email").contains("@company.com"));
Dataset<Row> pattern = employees.filter(col("name").rlike("^[A-Z][a-z]+$"));

// NULL handling
Dataset<Row> withManager = employees.filter(col("manager_id").isNotNull());
Dataset<Row> noManager = employees.filter(col("manager_id").isNull());

// IN condition
Dataset<Row> depts = employees.filter(
 col("department").isin("Engineering", "Marketing", "Sales")
);

filtered.show();

Sorting#

// Single column
Dataset<Row> sorted = employees.orderBy("salary");
Dataset<Row> sortedDesc = employees.orderBy(col("salary").desc());

// Multiple columns
Dataset<Row> multiSort = employees.orderBy(
 col("department").asc(),
 col("salary").desc()
);

// NULL handling
Dataset<Row> nullsFirst = employees.orderBy(col("manager_id").asc_nulls_first());
Dataset<Row> nullsLast = employees.orderBy(col("salary").desc_nulls_last());

multiSort.show();

Aggregation#

Basic Aggregation#

// Overall aggregation
Dataset<Row> stats = employees.agg(
 count("*").alias("total_count"),
 countDistinct("department").alias("dept_count"),
 sum("salary").alias("total_salary"),
 avg("salary").alias("avg_salary"),
 max("salary").alias("max_salary"),
 min("salary").alias("min_salary"),
 stddev("salary").alias("stddev_salary")
);

stats.show();

Group Aggregation#

// Single group
Dataset<Row> byDept = employees
 .groupBy("department")
 .agg(
 count("*").alias("employee_count"),
 avg("salary").alias("avg_salary"),
 sum("salary").alias("total_salary")
 )
 .orderBy(col("total_salary").desc());

// Multiple groups
Dataset<Row> byDeptLevel = employees
 .groupBy("department", "level")
 .agg(
 count("*").alias("count"),
 avg("salary").alias("avg_salary")
 );

byDept.show();

Pivot#

// Pivot table
Dataset<Row> pivoted = employees
 .groupBy("department")
 .pivot("level", Arrays.asList("Junior", "Senior", "Lead"))
 .agg(avg("salary"));

pivoted.show();
// +----------+-------+-------+------+
// |department| Junior| Senior| Lead|
// +----------+-------+-------+------+
// | Sales| 45000 | 65000 | 85000|
// | Eng | 55000 | 75000 | 95000|
// +----------+-------+-------+------+

Joins#

Basic Join#

Dataset<Row> employees = spark.read().parquet("employees.parquet");
Dataset<Row> departments = spark.read().parquet("departments.parquet");

// Inner Join
Dataset<Row> joined = employees.join(
 departments,
 employees.col("department_id").equalTo(departments.col("id"))
);

// When column names are the same
Dataset<Row> simpleJoin = employees.join(departments, "department_id");

// Left Join
Dataset<Row> leftJoined = employees.join(
 departments,
 employees.col("department_id").equalTo(departments.col("id")),
 "left"
);

// Join types: inner, left, right, full, left_semi, left_anti, cross

Multi-Condition Join#

Dataset<Row> multiJoin = orders.join(
 products,
 orders.col("product_id").equalTo(products.col("id"))
 .and(orders.col("region").equalTo(products.col("region")))
);

Broadcast Join#

import static org.apache.spark.sql.functions.broadcast;

// Broadcast small table
Dataset<Row> optimized = employees.join(
 broadcast(departments),
 "department_id"
);

Using SQL#

// Register temporary views
employees.createOrReplaceTempView("employees");
departments.createOrReplaceTempView("departments");

// Execute SQL query
Dataset<Row> result = spark.sql("""
 SELECT
 e.name,
 e.salary,
 d.department_name,
 AVG(e.salary) OVER (PARTITION BY e.department_id) as dept_avg_salary
 FROM employees e
 JOIN departments d ON e.department_id = d.id
 WHERE e.age > 25
 ORDER BY e.salary DESC
 LIMIT 100
 """);

result.show();

// Using CTE
Dataset<Row> cteResult = spark.sql("""
 WITH dept_stats AS (
 SELECT
 department_id,
 AVG(salary) as avg_salary,
 COUNT(*) as emp_count
 FROM employees
 GROUP BY department_id
 )
 SELECT
 d.department_name,
 ds.avg_salary,
 ds.emp_count
 FROM dept_stats ds
 JOIN departments d ON ds.department_id = d.id
 ORDER BY ds.avg_salary DESC
 """);

cteResult.show();

Saving Data#

Saving to Files#

// Parquet (recommended)
result.write()
 .mode("overwrite")
 .parquet("output/result.parquet");

// Partitioning
result.write()
 .mode("overwrite")
 .partitionBy("year", "month")
 .parquet("output/partitioned");

// CSV
result.write()
 .mode("overwrite")
 .option("header", "true")
 .csv("output/result.csv");

// Save as single file
result.coalesce(1)
 .write()
 .mode("overwrite")
 .option("header", "true")
 .csv("output/single_file");

Save Modes#

Mode	Description
`overwrite`	Overwrite existing data
`append`	Append to existing data
`ignore`	Ignore if already exists
`error` (default)	Error if already exists

Complete Example: Sales Analysis#

public class SalesAnalysisExample {
 public static void main(String[] args) {
 SparkSession spark = SparkSession.builder()
 .appName("Sales Analysis")
 .master("local[*]")
 .getOrCreate();

 spark.sparkContext().setLogLevel("WARN");

 // Load data
 Dataset<Row> orders = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .csv("data/orders.csv");

 Dataset<Row> products = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .csv("data/products.csv");

 Dataset<Row> customers = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .csv("data/customers.csv");

 // 1. Join orders with products
 Dataset<Row> ordersWithProducts = orders
 .join(broadcast(products), "product_id")
 .withColumn("revenue", col("quantity").multiply(col("price")));

 // 2. Monthly revenue aggregation
 Dataset<Row> monthlyRevenue = ordersWithProducts
 .withColumn("month", date_format(col("order_date"), "yyyy-MM"))
 .groupBy("month")
 .agg(
 sum("revenue").alias("total_revenue"),
 count("*").alias("order_count"),
 countDistinct("customer_id").alias("unique_customers")
 )
 .orderBy("month");

 System.out.println("=== Monthly Revenue ===");
 monthlyRevenue.show();

 // 3. Top 5 products by category
 WindowSpec windowSpec = Window
 .partitionBy("category")
 .orderBy(col("total_revenue").desc());

 Dataset<Row> topProducts = ordersWithProducts
 .groupBy("category", "product_name")
 .agg(sum("revenue").alias("total_revenue"))
 .withColumn("rank", rank().over(windowSpec))
 .filter(col("rank").leq(5));

 System.out.println("=== Top 5 Products by Category ===");
 topProducts.show(20);

 // 4. Customer segment analysis
 Dataset<Row> customerStats = ordersWithProducts
 .join(customers, "customer_id")
 .groupBy("customer_id", "customer_name", "segment")
 .agg(
 sum("revenue").alias("total_spent"),
 count("*").alias("order_count"),
 avg("revenue").alias("avg_order_value")
 );

 Dataset<Row> segmentAnalysis = customerStats
 .groupBy("segment")
 .agg(
 count("*").alias("customer_count"),
 avg("total_spent").alias("avg_customer_value"),
 sum("total_spent").alias("segment_revenue")
 )
 .orderBy(col("segment_revenue").desc());

 System.out.println("=== Customer Segment Analysis ===");
 segmentAnalysis.show();

 // 5. Complex analysis with SQL
 ordersWithProducts.createOrReplaceTempView("orders_enriched");

 Dataset<Row> sqlAnalysis = spark.sql("""
 WITH monthly_category AS (
 SELECT
 DATE_FORMAT(order_date, 'yyyy-MM') as month,
 category,
 SUM(revenue) as revenue
 FROM orders_enriched
 GROUP BY DATE_FORMAT(order_date, 'yyyy-MM'), category
 ),
 category_growth AS (
 SELECT
 month,
 category,
 revenue,
 LAG(revenue) OVER (PARTITION BY category ORDER BY month) as prev_revenue
 FROM monthly_category
 )
 SELECT
 month,
 category,
 revenue,
 prev_revenue,
 ROUND((revenue - prev_revenue) / prev_revenue * 100, 2) as growth_rate
 FROM category_growth
 WHERE prev_revenue IS NOT NULL
 ORDER BY month, category
 """);

 System.out.println("=== Monthly Growth Rate by Category ===");
 sqlAnalysis.show(20);

 // 6. Save results
 monthlyRevenue.write()
 .mode("overwrite")
 .parquet("output/monthly_revenue");

 segmentAnalysis.write()
 .mode("overwrite")
 .parquet("output/segment_analysis");

 spark.stop();
 }
}

Real Public Dataset Examples#

Examples using public datasets commonly utilized in practice.

FAQ

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

FAQ#

Frequently asked questions and solutions to common problems.

General Questions#

Which Java versions does Spark support?#

Spark 3.5 supports Java 8, 11, and 17. Java 21 is not officially supported yet.

java -version # Check version

Can I use Spark with Java only, without Scala?#

Yes, you can. Spark fully supports the Java API. However, since the Spark runtime is written in Scala, Scala libraries are included in the dependencies.

What’s the difference between DataFrame and Dataset?#

DataFrame (Dataset<Row>): Has schema but no compile-time type checking
Dataset (Dataset<T>): Uses POJO types to provide compile-time type safety

In Java, DataFrame is an alias for Dataset<Row>.

Mapping Migration

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

This guide walks you through changing index mappings (field types, analyzers, etc.) without downtime.

Estimated time: About 20-40 minutes (additional time may be required depending on data size)

Scope of This Guide
Covers: Zero-downtime mapping changes using the Reindex API, Alias switching strategy, and validation methods

Does not cover: For large-scale index rebuilds, see Index Rebuild. For cluster-level changes, see Cluster Scaling.

TL;DR

Alias-based operations: Your application accesses indices through Aliases

Create new index: Create a new index with the updated mapping and Reindex

Switch Alias: After validation, switch the Alias to the new index

Rollback ready: You can roll back instantly as long as you don’t delete the old index

Before You Begin#

Verify the following prerequisites:

Type Error Debugging

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

This guide walks you through decoding and resolving complex type error messages.

Estimated time: About 15-20 minutes

TL;DR

Type Mismatch: Compare expected and actual types to find where the mismatch occurs

Bounds violation: Check upper/lower bound constraints on type parameters

Variance issues: Verify that the correct types appear in covariant/contravariant positions

splain plugin: Transforms complex type errors into a more readable format

Problems This Guide Solves#

Use this guide when you encounter the following compilation errors:

Designing Value Objects

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Problem Solved: High complexity from making everything an Entity, or low type safety due to Primitive Obsession Time Required: ~20 min Prerequisites: Assumes you have read the Tactical Design document

Success Criteria
After completing this guide, you will be able to:

Distinguish Entity from Value Object using 3 questions

Implement Value Objects using Java Records

Refactor Primitive Obsession to Value Objects

1. Distinguishing Entity vs Value Object#

1.1 Three Decision Questions#

Apply the following questions in order:

Managing Alert Fatigue

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Situation: Receiving dozens to hundreds of alerts daily, missing critical ones Goal: Only receive alerts that require actual action Time Required: 1-2 hours (analyzing and modifying alert rules) Success Criteria: Daily alert count reduced to a manageable level (e.g., 10 or fewer)

Before You Begin#

Required Environment#

Component	Version	Verification
Prometheus	2.40+	`prometheus --version`
Alertmanager	0.25+	`alertmanager --version`
amtool	0.25+	`amtool --version`

Required Permissions#

Write access to Prometheus configuration file (prometheus.yml)
Write access to Alertmanager configuration file (alertmanager.yml)
Permission to restart Prometheus/Alertmanager

Environment Check#

# Check Prometheus status
curl -s http://localhost:9090/-/healthy && echo "Prometheus OK"

# Check Alertmanager status
curl -s http://localhost:9093/-/healthy && echo "Alertmanager OK"

# Check amtool configuration
amtool config show --alertmanager.url=http://localhost:9093

Problem Scenario#

# Yesterday's alert summary
Critical: 15 (HighCPU 8, HighMemory 7)
Warning: 87 (SlowResponse 45, HighLatency 32, PodRestart 10)
Total: 102

# Actual incidents: 1
# Missed alerts: 1 (buried in HighCPU alerts)

When alert fatigue occurs:

Message Loss Prevention Guide

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

A step-by-step guide on how to prevent message loss and ensure reliable messaging in Kafka.

Time Required: Approximately 30 minutes (for configuration application)

TL;DR

Producer side: acks=all, enable.idempotence=true, proper retry settings

Broker side: min.insync.replicas=2, unclean.leader.election.enable=false

Consumer side: Manual commit, commit after processing, retry mechanism

End-to-End: Message trace ID, monitoring and alerting setup

Before You Begin#

You need the following environment to follow this guide.

Prerequisites#

Item	Minimum Version	Verification Command
Apache Kafka	2.8+	`kafka-topics.sh --version`
Java	17+	`java -version`
Spring Boot	3.0+	`./gradlew dependencies \| grep spring-boot`
Spring Kafka	3.0+	`./gradlew dependencies \| grep spring-kafka`

Environment Verification#

Verify that you can access the Kafka cluster:

Network Troubleshooting

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Objective: Identify and resolve issues when Service or Ingress fails to connect Estimated Time: 30 minutes

Scope of This Guide
Covers: Service connectivity issues, DNS issues, Ingress routing issues

Does not cover: Pod startup issues (see Pod Troubleshooting), external network firewall configuration

Before You Begin#

Verify the following prerequisites.

1. Verify kubectl Installation and Cluster Access#

kubectl cluster-info

Success output:

Kubernetes control plane is running at https://xxx.xxx.xxx.xxx
CoreDNS is running at https://xxx.xxx.xxx.xxx/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

2. Verify Pod Status#

Verify the target Pods you want to connect to are in Running status.

Optimizing Shuffle

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Estimated Time: About 20 minutes

TL;DR

Check Shuffle: Exchange node in df.explain() = shuffle occurs

Eliminate Unnecessary Shuffles: Perform multiple aggregations at once in the same group

Broadcast Join: Use broadcast() for small tables (tens of MB)

Shuffle Partition Count: Adjust spark.sql.shuffle.partitions (default 200)

Problem Definition#

Shuffle optimization is needed when you see these symptoms:

Symptom	Where to Check
Stage transitions take 10+ seconds	Spark UI → Jobs tab
Shuffle Read/Write is tens of GB or more	Spark UI → Stages tab
Timeout due to network I/O	Application logs
Many `Exchange` nodes in `explain()`	Execution plan output

Shuffle is the most expensive operation in Spark. The following operations cause shuffle:

Clean Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 15 Jan 2026 00:00:00 +0000

Target Audience: Developers of large-scale projects requiring strict dependency management Prerequisites: Understanding of Hexagonal Architecture’s Port/Adapter concept Estimated Time: About 25 minutes

Clean Architecture#

An architecture proposed by Uncle Bob (Robert C. Martin) in 2012. Strictly adhering to the Dependency Rule is the core principle.

Clean Architecture emerged from a persistent problem in software development. When business logic is tightly coupled to frameworks or databases, changing the technology stack or writing tests becomes very difficult. For example, if JPA annotations are directly attached to domain objects, testing the domain logic always requires a database connection. Clean Architecture aims to completely isolate business rules from technical details.

FAQ

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

Frequently asked questions and answers about Scala. Questions are categorized for quick access to the information you need.

TL;DR

Version Choice: Scala 3 for new projects, Scala 2.12/2.13 when using Spark

val vs var: Always prefer val (immutable)

Instead of null: Use Option, Some, None for safe null handling

IDE: IntelliJ IDEA + Scala plugin (most mature)

Performance: Equivalent to Java, watch out for collection chaining and lambda usage

General#

General questions about learning Scala.

Functions and Methods

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

In Scala, functions are first-class citizens that can be stored in variables, passed as arguments, and returned as values

Currying implements a pattern of applying configuration values once and reusing them

Methods (def) belong to classes, while function values (val) are suitable for passing/storing

@tailrec ensures tail recursion optimization

Target Audience: Java developers or those with OOP experience Prerequisites: Scala basic syntax, control structures

In Scala, functions are first-class citizens. You can store functions in variables, pass them as arguments, and return them as values. This characteristic is the essence of functional programming and makes code more concise and reusable.

Scala 2 vs Scala 3 Comparison

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Syntax: Scala 3 supports simpler syntax (indentation-based, no braces required)

Enums: Simple definition with enum keyword (replaces sealed trait + case object)

Implicits: Clearer with given/using keywords (replacing implicit)

New features: Union Types, Opaque Types, Intersection Types

Compatibility: Most Scala 2 code works in Scala 3

Target Audience: Developers familiar with Scala 2, developers choosing Scala 3

Prerequisites:

Scala basic syntax
Functional programming basics
Pattern matching

Compare the differences by implementing the same logic in Scala 2 and Scala 3. Most Scala 2 syntax works in Scala 3, but Scala 3 offers more concise and readable new syntax. This document compares code by topic to help you understand the differences.

Aggregate Deep Dive

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

Target Audience: Developers who understand Aggregate basics and need deep dive into design Prerequisites: Entity, Value Object concepts from Tactical Design Reading Time: About 25 minutes Key Question: “How do I determine Aggregate boundaries, and what is the appropriate size?”

Summary
Aggregate design core: Invariant-based boundary setting → Keep it small → Reference by ID → Use eventual consistency

Aggregate Deep Dive#

A deep dive into Aggregate design principles, transaction boundaries, and practical patterns.

Application Layer

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

Application Layer Implementation#

TL;DR

OrderService: Use Case orchestration. Calls domain logic, manages transactions, publishes events

Command objects: Encapsulate requests as immutable objects like CreateOrderCommand

DTO: Separate response representation from domain with OrderResponse, etc.

JPA Repository: Implementation of Domain Repository interface. Uses Mapper for Entity conversion

Event handler: Publishes to Kafka after commit using @TransactionalEventListener

Target Audience and Prerequisites#

Item	Required Level
Target Audience	Developers learning Application Service and Infrastructure layer implementation
Spring	Experience with @Service, @Transactional, @Repository annotations
JPA	Understanding of Entity, @Embedded, CascadeType
Prerequisites	Project Setup, Order Domain completed

Implementing Application Services that orchestrate use cases and the infrastructure layer.

FAQ

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

DDD Frequently Asked Questions (FAQ)#

Common questions and answers when applying DDD.

TL;DR

DDD is a methodology, not an architecture, and provides value when there is complex business logic

Entity is identified by ID, Value Object equality is determined by attribute values

Design Aggregate as the minimum unit that protects true invariants

Ubiquitous Language is the most important element when applying DDD

Can be applied gradually to legacy systems through ACL (Anti-Corruption Layer)

Basic Concepts#

Q: Is DDD an architecture?#

A: No. DDD is a methodology for handling complex domains.

Alerting Actions Guide

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Situational guide for responding after receiving alerts.

Alert → Action Workflow#

graph LR
 A["Receive Alert"] --> B["Assess Situation"]
 B --> C["Evaluate Impact"]
 C --> D["Immediate Action"]
 D --> E["Root Cause Analysis"]
 E --> F["Permanent Fix"]
 F --> G["Post-Incident Review"]

Response by Situation#

1. Service Down (TargetDown)#

Alert

alert: TargetDown
expr: up == 0

Immediate Actions

# 1. Check service status
kubectl get pods -l app=order-service
docker ps -f name=order-service

# 2. Check logs
kubectl logs -l app=order-service --tail=100
docker logs order-service --tail=100

# 3. Attempt restart
kubectl rollout restart deployment/order-service
docker restart order-service

Root Cause Analysis

Errors (Error Rate)

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs improving service reliability Prerequisites: Aggregation Operators After reading this: You’ll be able to systematically classify errors and set up SLO-based monitoring

TL;DR#

Key Summary:

Error rate: Failed requests / Total requests

Not just HTTP 5xx: Include business logic failures

Error budget: Allowed error amount (based on SLO)

Error classification is important: Client vs server, temporary vs permanent

Why Error Monitoring Matters#

Errors are the most direct indicator of system health. Services still work with high Latency, but high error rates cause real harm to users. According to Google SRE team research, users who experience errors are 3x more likely to churn within 72 hours.

Kafka Monitoring

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Time Required: 20 minutes Prerequisites: Kafka basics, Environment Setup What You’ll Learn: Monitor Kafka clusters and Consumer Lag

Core Kafka Monitoring Metrics#

Category	Metric	Meaning
Consumer	Lag	Processing delay (most important)
Broker	Under-replicated Partitions	Replication issues
Producer	Record Error Rate	Send failure rate
Broker	Disk Usage	Storage space

Step 1: Kafka + JMX Exporter Configuration#

# Add to docker-compose.yml
services:
 kafka:
 image: confluentinc/cp-kafka:7.5.0
 container_name: kafka
 ports:
 - "9092:9092"
 - "9101:9101" # JMX
 environment:
 KAFKA_NODE_ID: 1
 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
 KAFKA_PROCESS_ROLES: broker,controller
 KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093
 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
 CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk
 KAFKA_JMX_PORT: 9101
 KAFKA_JMX_HOSTNAME: localhost
 KAFKA_OPTS: -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx-exporter/kafka.yml
 volumes:
 - ./jmx-exporter:/opt/jmx-exporter

 kafka-exporter:
 image: danielqsj/kafka-exporter:latest
 container_name: kafka-exporter
 ports:
 - "9308:9308"
 command:
 - --kafka.server=kafka:9092
 - --topic.filter=.*
 - --group.filter=.*

Step 2: JMX Exporter Configuration#

mkdir -p jmx-exporter
curl -L https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.19.0/jmx_prometheus_javaagent-0.19.0.jar -o jmx-exporter/jmx_prometheus_javaagent.jar

# jmx-exporter/kafka.yml
lowercaseOutputName: true
rules:
 # Broker metrics
 - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
 name: kafka_server_$1_$2
 type: GAUGE
 labels:
 clientId: "$3"
 topic: "$4"
 partition: "$5"

 - pattern: kafka.server<type=(.+), name=(.+)><>Value
 name: kafka_server_$1_$2
 type: GAUGE

 # Under-replicated Partitions
 - pattern: kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value
 name: kafka_server_replicamanager_underreplicatedpartitions
 type: GAUGE

 # Messages In Per Sec
 - pattern: kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=(.+)><>OneMinuteRate
 name: kafka_server_brokertopicmetrics_messagesin_total
 type: GAUGE
 labels:
 topic: "$1"

 # Bytes In Per Sec
 - pattern: kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec, topic=(.+)><>OneMinuteRate
 name: kafka_server_brokertopicmetrics_bytesin_total
 type: GAUGE
 labels:
 topic: "$1"

Step 3: Add Prometheus Configuration#

# Add to prometheus/prometheus.yml
scrape_configs:
 - job_name: 'kafka'
 static_configs:
 - targets: ['kafka:7071']

 - job_name: 'kafka-exporter'
 static_configs:
 - targets: ['kafka-exporter:9308']

Step 4: Core PromQL Queries#

Consumer Lag#

# Lag by consumer group
sum by (consumergroup, topic) (kafka_consumergroup_lag)

# Groups with lag over 10000
sum by (consumergroup, topic) (kafka_consumergroup_lag) > 10000

# Lag trends
sum(kafka_consumergroup_lag) by (consumergroup)

Broker Status#

# Under-replicated Partitions
kafka_server_replicamanager_underreplicatedpartitions

# ISR (In-Sync Replicas) shrinkage
kafka_server_replicamanager_isrshrinks_total

Traffic#

# Messages per second by topic
sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))

# Bytes per second by topic
sum by (topic) (rate(kafka_server_brokertopicmetrics_bytesin_total[5m]))

Step 5: Alert Rules#

# prometheus/rules/kafka-alerts.yml
groups:
 - name: kafka
 rules:
 - alert: KafkaConsumerLagHigh
 expr: sum by (consumergroup, topic) (kafka_consumergroup_lag) > 10000
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "High consumer lag: {{ $labels.consumergroup }}"
 description: "Lag is {{ $value }} on topic {{ $labels.topic }}"

 - alert: KafkaUnderReplicatedPartitions
 expr: kafka_server_replicamanager_underreplicatedpartitions > 0
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "Kafka under-replicated partitions"
 description: "{{ $value }} partitions are under-replicated"

 - alert: KafkaBrokerDown
 expr: count(kafka_server_kafkaserver_brokerstate) < 3
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "Kafka broker down"

Step 6: Recording Rules#

# prometheus/rules/kafka-recording.yml
groups:
 - name: kafka_recording
 rules:
 - record: topic:kafka_messages:rate5m
 expr: sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))

 - record: consumergroup:kafka_lag:sum
 expr: sum by (consumergroup) (kafka_consumergroup_lag)

 - record: :kafka_underreplicated:sum
 expr: sum(kafka_server_replicamanager_underreplicatedpartitions)

Step 7: Grafana Dashboard#

Row 1: Overview#

Panel	Query	Type
Total Lag	`sum(kafka_consumergroup_lag)`	Stat
Messages/sec	`sum(rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))`	Stat
Under-replicated	`sum(kafka_server_replicamanager_underreplicatedpartitions)`	Stat

Row 2: Consumer Lag#

# Time Series: Lag trends
sum by (consumergroup) (kafka_consumergroup_lag)

# Table: Details
sum by (consumergroup, topic, partition) (kafka_consumergroup_lag)

Row 3: Traffic#

# Messages per second by topic
sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))

Spring Kafka Metrics#

# application.yml
management:
 metrics:
 enable:
 kafka: true

# Producer metrics
kafka_producer_record_send_total
kafka_producer_record_error_total

# Consumer metrics
kafka_consumer_records_consumed_total
kafka_consumer_fetch_manager_records_lag

Verification Checklist#

kafka-exporter metrics exposed
Consumer Lag query works
Grafana dashboard configured
Alert rules tested

Next Steps#

Recommended Order	Document	What You’ll Learn
1	Full-Stack Example	Integrated example
2	By Service Type	Kafka golden signals

Prometheus Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers who want to operate or deeply understand Prometheus Prerequisites: Metrics Fundamentals After Reading: You’ll understand Prometheus design philosophy and components, and be able to plan operational strategies

TL;DR#

Key Summary:

Pull Model: Prometheus fetches metrics from targets (not Push)

Time Series DB: Label-based multidimensional data model

Service Discovery: Auto-discover targets with Kubernetes, Consul, etc.

Single Server Design: Optimized for single server rather than horizontal scaling (extend with Federation)

Prometheus Overall Structure#

graph TB
 subgraph "Data Collection"
 T1["Target 1<br>/metrics"]
 T2["Target 2<br>/metrics"]
 T3["Target 3<br>/metrics"]
 PG["Pushgateway<br>(for batch jobs)"]
 end

 subgraph "Prometheus Server"
 R["Retrieval<br>(Scraper)"]
 TSDB["TSDB<br>(Time Series DB)"]
 HTTP["HTTP Server<br>(PromQL API)"]
 R --> TSDB
 TSDB --> HTTP
 end

 subgraph "Service Discovery"
 K8S["Kubernetes"]
 CONSUL["Consul"]
 FILE["File SD"]
 end

 subgraph "Alerting"
 AM["Alertmanager"]
 SLACK["Slack"]
 PD["PagerDuty"]
 end

 subgraph "Visualization"
 GF["Grafana"]
 end

 T1 --> |"pull"| R
 T2 --> |"pull"| R
 T3 --> |"pull"| R
 PG --> |"pull"| R

 K8S --> |"target list"| R
 CONSUL --> |"target list"| R
 FILE --> |"target list"| R

 TSDB --> |"alerting rules"| AM
 AM --> SLACK
 AM --> PD

 HTTP --> |"PromQL"| GF

Why Pull Model?#

Two Philosophies of Monitoring#

There are broadly two philosophies for collecting metrics.

rate and increase

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers working with Counter metrics Prerequisites: Metrics Fundamentals, Aggregation Operators What You’ll Learn: Calculate per-second rate and total increase from Counter metrics accurately

TL;DR#

Key Summary:

rate(): Average per-second increase rate → Used in dashboards and alerts

increase(): Total increase within time range → Used for period totals

irate(): Instantaneous rate of last two samples → Used for volatile metrics

Always apply these functions to Counters (raw values are meaningless)

Why rate/increase are Needed#

Counters are cumulative values. They record how many requests have come since the server started. But the number “150,000 requests” alone doesn’t tell us if the system is working well. If the server started 1 hour ago, it’s 40 requests/second, but if it started 1 year ago, it’s 0.005 requests/second.

Deployment

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to deploy applications in Kubernetes Prerequisites: Pod concepts After reading this: You will understand how to deploy and update applications with Deployments

TL;DR

Deployment manages declarative updates of Pods

Rolling updates enable zero-downtime deployment

Can rollback to previous version when problems occur

What is Deployment?#

Deployment is a resource that manages Pod deployment and updates. Using Deployment instead of directly managing Pods offers several benefits.

References

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

This page organizes official documentation and additional learning resources helpful for Kubernetes learning and operations.

Learning Roadmap by Stage#

If you’re unsure where to start, refer to the roadmap below.

flowchart TD
 START[Start] --> L1[Stage 1: Introduction]
 L1 --> L2[Stage 2: Basics]
 L2 --> L3[Stage 3: Advanced]
 L3 --> L4[Stage 4: Expert]

 L1 -.->|Recommended| R1[Kubernetes Basics Tutorial]
 L2 -.->|Recommended| R2[kubectl Cheat Sheet + This Guide]
 L3 -.->|Recommended| R3[Kubernetes in Action Book]
 L4 -.->|Recommended| R4[CKA/CKAD Certification]

Stage	Goal	Recommended Resources	Expected Duration
1. Introduction	First successful deployment	Kubernetes Basics, Play with K8s	1 week
2. Basics	Understanding core concepts	This guide, kubectl Cheat Sheet	2-4 weeks
3. Advanced	Operations capability	Kubernetes in Action, Prometheus	1-3 months
4. Expert	Certification/Architecture	CKA/CKAD, Kubernetes Patterns	3-6 months

Official Documentation#

Kubernetes Official#

Resource	Description	Link
Kubernetes Documentation	Official docs (Korean supported)	kubernetes.io/docs
Kubernetes Blog	New features, release news	kubernetes.io/blog
Kubernetes GitHub	Source code, issues	github.com/kubernetes

kubectl Reference#

Resource	Description	Link
kubectl Cheat Sheet	Collection of frequently used commands	kubectl cheatsheet
kubectl Reference	Complete command reference	kubectl reference

API Reference#

Resource	Description	Link
API Reference	Kubernetes API documentation	API Reference

Learning Resources#

Interactive Tutorials#

Resource	Description	Link
Kubernetes Basics	Official interactive tutorial	Learn Kubernetes Basics
Katacoda (Killercoda)	Browser-based practice environment	killercoda.com
Play with Kubernetes	Free online practice environment	labs.play-with-k8s.com

Books#

Book	Author	Features
Kubernetes in Action	Marko Lukša	Practice-oriented, detailed explanations
Kubernetes Patterns	Bilgin Ibryam	Design pattern focused
Cloud Native DevOps with Kubernetes	John Arundel	DevOps perspective

Certifications#

Certification	Target Audience	Description
CKA	Administrators	Validates Kubernetes cluster administration skills
CKAD	Developers	Validates Kubernetes application development skills
CKS	Security Specialists	Validates Kubernetes security skills

Tools#

Local Development#

Tool	Description	Link
Minikube	Local Kubernetes cluster	minikube.sigs.k8s.io
Kind	Docker-based local cluster	kind.sigs.k8s.io
k3d	Lightweight Kubernetes (k3s)	k3d.io

Package Management#

Tool	Description	Link
Helm	Kubernetes package manager	helm.sh
Kustomize	Configuration customization	kustomize.io

Monitoring#

Tool	Description	Link
Prometheus	Metrics collection	prometheus.io
Grafana	Visualization dashboard	grafana.com
Lens	Kubernetes IDE	k8slens.dev

CLI Tools#

Tool	Description	Link
kubectx/kubens	Context/namespace switching	github.com/ahmetb/kubectx
k9s	Terminal UI	k9scli.io
stern	Multi-pod logs	github.com/stern/stern

Managed Kubernetes#

Service	Cloud	Link
Amazon EKS	AWS	aws.amazon.com/eks
Google GKE	GCP	cloud.google.com/kubernetes-engine
Azure AKS	Azure	azure.microsoft.com/services/kubernetes-service

Community#

Channel	Description	Link
Kubernetes Slack	Official community	slack.kubernetes.io
CNCF	Cloud Native Foundation	cncf.io
Stack Overflow	Q&A	stackoverflow.com/questions/tagged/kubernetes

Versions and Releases#

Resource	Description	Link
Release Notes	Changes by version	kubernetes.io/releases
Deprecation Policy	Deprecation policy	Deprecation Policy
Version Skew Policy	Version compatibility	Version Skew

Spring Boot Deployment

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to deploy Spring Boot applications to Kubernetes Prerequisites: Deployment, Service, ConfigMap, Docker After reading this: You’ll be able to containerize and deploy Spring Boot apps to Kubernetes

TL;DR

Write Dockerfile → Build image → Deploy to Kubernetes

Separate environment-specific configuration with ConfigMap/Secret

Configure health checks with Actuator endpoints

Overall Flow#

flowchart LR
 A[Spring Boot App] --> B[Dockerfile]
 B --> C[Docker Build]
 C --> D[Push Image]
 D --> E[K8s Deployment]
 E --> F[Service]
 F --> G[Ingress]

1. Prepare Spring Boot Application#

Example Application Structure#

my-app/
├── src/main/
│ ├── java/com/example/
│ │ └── MyAppApplication.java
│ └── resources/
│ └── application.yml
├── Dockerfile
├── build.gradle.kts
└── kubernetes/
 ├── deployment.yaml
 ├── service.yaml
 ├── configmap.yaml
 └── secret.yaml

application.yml#

spring:
 application:
 name: my-app
 datasource:
 url: ${DATABASE_URL:jdbc:postgresql://localhost:5432/mydb}
 username: ${DATABASE_USERNAME:postgres}
 password: ${DATABASE_PASSWORD:postgres}

server:
 port: 8080

management:
 endpoints:
 web:
 exposure:
 include: health,info,prometheus
 endpoint:
 health:
 probes:
 enabled: true
 health:
 livenessState:
 enabled: true
 readinessState:
 enabled: true

Simple Controller#

@RestController
public class HelloController {

 @GetMapping("/")
 public String hello() {
 return "Hello from Kubernetes!";
 }

 @GetMapping("/hostname")
 public String hostname() {
 try {
 return "Pod: " + InetAddress.getLocalHost().getHostName();
 } catch (UnknownHostException e) {
 return "Unknown";
 }
 }
}

2. Build Docker Image#

Dockerfile (Multi-stage Build)#

# Build stage
FROM eclipse-temurin:17-jdk AS build
WORKDIR /app
COPY gradle gradle
COPY build.gradle.kts settings.gradle.kts gradlew ./
COPY src src
RUN ./gradlew bootJar --no-daemon

# Runtime stage
FROM eclipse-temurin:17-jre
WORKDIR /app

# Security: use non-root user
RUN addgroup --system appgroup && adduser --system appuser --ingroup appgroup
USER appuser

COPY --from=build /app/build/libs/*.jar app.jar

EXPOSE 8080

ENTRYPOINT ["java", "-jar", "app.jar"]

Build Image#

# Build Docker image
docker build -t my-app:1.0 .

# Test locally
docker run -p 8080:8080 my-app:1.0

# Use local image in Minikube
eval $(minikube docker-env)
docker build -t my-app:1.0 .

# Use local image in Kind
kind load docker-image my-app:1.0

3. Write Kubernetes Resources#

ConfigMap (Environment Configuration)#

# kubernetes/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
 name: my-app-config
data:
 SPRING_PROFILES_ACTIVE: "kubernetes"
 DATABASE_URL: "jdbc:postgresql://postgres-service:5432/mydb"
 JAVA_OPTS: "-Xms256m -Xmx512m"

Secret (Sensitive Information)#

# Create Secret (command)
kubectl create secret generic my-app-secret \
 --from-literal=DATABASE_USERNAME=myuser \
 --from-literal=DATABASE_PASSWORD=mypassword

Or with YAML:

Consumer Group & Offset

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Consumer Group is a logical grouping that enables parallel processing of Consumers with the same purpose

Core rule: One Partition can only be read by one Consumer within a Group

Offset is the message position number within a Partition, stored in the __consumer_offsets topic

Auto commit is convenient but risks data loss; manual commit provides precise control

When a Consumer fails, rebalancing automatically redistributes Partitions

Target audience: Developers building or operating Kafka Consumers, those learning about parallel processing and state management

Order System

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

Implement an event-driven order system closer to real-world applications. In this example, we process the entire flow from order creation to delivery completion using Kafka events, and learn about order guarantee using Message Keys and extension patterns utilizing multiple Consumer Groups.

TL;DR

Event-driven Architecture: Convert REST API requests to Kafka events for asynchronous processing

Message Key: Use orderId as Key to guarantee ordering of events for the same order

State Machine: State transitions in order: CREATED -> PAID -> SHIPPED -> DELIVERED

Extension Pattern: Multiple Consumer Groups independently subscribe to the same Topic

Target Audience and Prerequisites#

Item	Description
Target Audience	Backend developers looking to build event-driven systems
Prerequisites	Spring Boot fundamentals, Kafka basic concepts, Basic Examples completed
Required Environment	Kafka running via Docker, JDK 17+, Gradle
Estimated Time	About 45 minutes

All code examples on this page use the following common imports. The implementation centers around Spring Kafka’s KafkaTemplate and @KafkaListener, with JSON serialization handled through Jackson ObjectMapper.

Product Search System

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

Complete Example Project
If you want to run this code immediately, use the complete Spring Boot project:

📁 examples/elasticsearch/product-search/

Instant Elasticsearch execution with docker-compose (including Nori)

Automatic sample data initialization

TL;DR

Apply Korean morphological analysis with Nori analyzer to enable search for “삼성” and “전자” from “삼성전자”

Implement autocomplete functionality with Edge N-gram

Provide filtering and facets with Bool Query + Aggregation

Highlight search terms with highlighting

Total time required: approximately 30 minutes

Implement a production-level product search system with Korean morphological analysis, autocomplete, and filtering.

Query DSL

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Query Context: Full-text search that calculates relevance score

Filter Context: Only checks conditions without score, cached for speed

match/match_phrase: Used for full-text search (text fields)

term/terms/range: Used for exact value search (keyword fields)

bool: Combines queries with must/should/must_not/filter

Target Audience: Developers who understand Elasticsearch basics Prerequisites: Core Components, JSON basics

Learn how to write various search queries using Elasticsearch’s Query DSL (Domain Specific Language).

Basic Query Structure#

GET /products/_search
{
 "query": {
 // Search conditions
 },
 "from": 0, // Pagination offset
 "size": 10, // Number of documents to return
 "sort": [ // Sorting
 { "price": "asc" }
 ],
 "_source": ["name", "price"] // Fields to return
}

Query Context vs Filter Context#

Elasticsearch queries execute in two contexts.

References

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Official Documentation: Elastic official guide, Spring Data Elasticsearch documentation

Korean Resources: Real-world cases from Woowahan Brothers, Kakao, Naver D2 tech blogs

Cloud Services: Elastic Cloud (official), Amazon OpenSearch (AWS integration), Self-Managed (full control)

Learning Path: Official free training → Korean blogs → Elastic certification

Official documentation, recommended books, and community resource links.

Official Documentation#

Elasticsearch#

Kibana#

Spring Data Elasticsearch#

Recommended Books#

Beginner#

Book	Author	Features
Elasticsearch in Action (2nd Ed)	Madhusudhan Konda	Hands-on examples
Learning Elastic Stack 8.0	Pranav Shukla	Full ELK stack coverage

Advanced#

Book	Author	Features
Elasticsearch: The Definitive Guide	Clinton Gormley	Principle understanding (older version but useful for concepts)
Relevant Search	Doug Turnbull	Search relevance optimization

Online Courses#

Free#

Elastic Official Training - Free foundational courses
YouTube: Elastic Official Channel

Paid#

Elastic Certification Program
Udemy, Coursera Elasticsearch courses

Community#

Forums & Q&A#

Elastic Discuss - Official forum
Stack Overflow elasticsearch tag

Korean Community#

Korean Learning Resources#

Blogs & Articles#

Topic	Link	Features
ES Basics	Woowahan Brothers Tech Blog	Real-world application cases
Search Optimization	Kakao Tech Blog	Large-scale search experience
Log Analysis	Naver D2	ELK stack operation
Infrastructure	Daangn Tech Blog	Startup ES adoption story

Korean Books#

Book	Features
Elasticsearch Practical Guide	Korean, practice-focused
ElasticSearch from Fundamentals	Beginner-friendly

Tip: Combining English official docs with Korean blogs accelerates understanding.

DataFrame and Dataset

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

DataFrame and Dataset#

DataFrame and Dataset are Spark’s modern high-level APIs. They are easier to use than RDDs and provide automatic optimization through the Catalyst Optimizer.

Concept Overview#

DataFrame#

A DataFrame is a distributed data collection organized into named columns. It’s similar to a relational database table or a DataFrame in Python/R.

// DataFrame is an alias for Dataset<Row>
Dataset<Row> df = spark.read().json("employees.json");

Dataset#

A Dataset is a distributed data collection with a specific type. It provides compile-time type safety.

References

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

References#

Official documentation and additional resources for learning Apache Spark.

Official Documentation#

Apache Spark Official Site#

Spark Official Site — Downloads, news, release information
Spark 3.5 Documentation — Current stable version documentation
Spark Latest Documentation — Latest version documentation

Programming Guides#

RDD Programming Guide — Detailed RDD API explanation
Spark SQL, DataFrames and Datasets Guide — SQL and DataFrame API
Structured Streaming Programming Guide — Real-time stream processing
MLlib Guide — Machine learning library
GraphX Programming Guide — Graph processing

Operations Guides#

Cluster Overview — Cluster architecture
Tuning Guide — Performance tuning
Monitoring Guide — Monitoring
Configuration — Configuration options
Security — Security settings

Cluster Manager Guides#

API Documentation#

Java API#

Spark Java API (Javadoc) — Java API reference
Dataset — DataFrame class
SparkSession — Entry point class
functions — Built-in functions

Scala API#

Spark Scala API (Scaladoc)

Additional Learning Resources#

Online Courses#

Databricks Academy — Official training from Spark’s co-creator company
Coursera: Big Data Analysis with Scala and Spark — EPFL’s Scala/Spark course
edX: Big Data Analytics Using Spark — UC San Diego course

Blogs and Documentation#

Databricks Blog — Latest Spark technology and use cases
Spark By Examples — Java, Scala, Python examples
Baeldung Spark Tutorials — Spark tutorials for Java developers

Community#

Stack Overflow - apache-spark — Q&A
Spark Mailing Lists — Developer mailing lists
GitHub - apache/spark — Source code and issue tracker

Data Sources#

Kafka — Streaming data source
HDFS — Distributed file system
Parquet — Columnar format
Delta Lake — Storage with ACID transaction support

Cluster Environments#

Hadoop YARN — Resource management
Kubernetes — Container orchestration

Cloud Services#

AWS EMR — AWS managed Spark
Google Dataproc — GCP managed Spark
Azure HDInsight — Azure managed Hadoop/Spark
Databricks — Unified Data Analytics Platform

Version Release Notes#

Performance Benchmarks#

TPC-DS Benchmark — Decision support system benchmark
Spark SQL Performance Tests — Databricks performance testing tools

Recommended Books#

Beginner#

Learning Spark, 2nd Edition (O’Reilly) — Jules S. Damji et al.
Spark: The Definitive Guide (O’Reilly) — Bill Chambers, Matei Zaharia

Advanced#

High Performance Spark (O’Reilly) — Holden Karau, Rachel Warren
Spark in Action, 2nd Edition (Manning) — Jean-Georges Perrin

Index Rebuild

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

This guide walks you through efficiently rebuilding large indices.

Estimated time: About 30-60 minutes (may take several hours depending on data size)

Scope of This Guide
Covers: _reindex API, Snapshot/Restore, Logstash comparison, large-scale processing strategies, and performance optimization

Does not cover: For simple mapping changes, see Mapping Migration. For cluster scaling, see Cluster Scaling.

TL;DR

_reindex API: Simplest option, best for rebuilds within the same cluster

Snapshot/Restore: Best for cross-cluster migration and very large datasets

Logstash: Best for complex transformations or external source integration

Performance optimization: Use refresh_interval: -1, replica 0, and sliced scroll

Before You Begin#

Verify the following prerequisites:

Resolving sbt Dependency Conflicts

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

This guide walks you through diagnosing and resolving library version conflicts.

Estimated time: About 15-20 minutes

TL;DR

Symptom: Compilation succeeds but NoSuchMethodError or ClassNotFoundException occurs at runtime

Diagnosis: Use sbt dependencyTree to identify conflicting libraries

Resolution: Use exclude, dependencyOverrides, or force()

Prevention: Do not ignore eviction warnings

Problems This Guide Solves#

Use this guide in the following situations:

Compilation succeeds but NoSuchMethodError occurs at runtime
ClassNotFoundException occurs even though the library is clearly present
Eviction warnings appear during the sbt build
Multiple libraries require different versions of the same dependency

Symptoms#

# Compilation succeeds but error at runtime
$ sbt run
java.lang.NoSuchMethodError: 'void com.google.common.collect.ImmutableMap.forEach()'

# Or class not found
$ sbt run
java.lang.ClassNotFoundException: org.apache.commons.lang3.StringUtils

What This Guide Does Not Cover#

sbt basics: See the official sbt documentation
Maven/Gradle dependency management: See the respective build tool documentation
Scala binary compatibility: Compatibility between Scala versions is a separate topic

Before You Begin#

Verify the following environment is ready:

StatefulSet Hands-on

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

Target Audience: Developers who want to deploy stateful applications on Kubernetes Prerequisites: Deployment, Service, PersistentVolume concepts After reading this: You will be able to deploy MySQL with StatefulSet and verify data persistence

TL;DR

Deploy MySQL with StatefulSet

Access individual Pods via Headless Service

Persist data with PersistentVolumeClaim templates

Verify data preservation during scale up/down

Overall Flow#

flowchart LR
 A[Headless Service] --> B[StatefulSet]
 B --> C[mysql-0]
 B --> D[mysql-1]
 B --> E[mysql-2]
 C --> F[PVC-0]
 D --> G[PVC-1]
 E --> H[PVC-2]

Prerequisites#

You will need the following:

DDD Debate: Proponents vs Skeptics

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

DDD Debate: Proponents vs Skeptics#

Two senior developers debate Domain-Driven Design (DDD).

About This Document
This document records a debate between two senior developers with different perspectives on DDD. Rather than presenting a definitive answer, its purpose is to help you develop a balanced view of DDD through diverse perspectives.

Participants:

Proponent (Pro-DDD): A senior developer who advocates for DDD
Skeptic (Anti-DDD): A senior developer who is skeptical of DDD

Date: 2026-01-17

Designing Domain Events

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Problem Solved: Complex transactions due to tight coupling between Aggregates, or difficulty propagating changes Time Required: ~25 min Prerequisites: Assumes you have read the Aggregate Deep Dive document

Success Criteria
After completing this guide, you will be able to:

Identify situations that need domain events

Apply event naming conventions and payload design

Implement Spring event publishing/subscribing patterns

1. Identifying Situations That Need Domain Events#

1.1 Three Signals That Indicate Events Are Needed#

Check the following 3 signals to determine if domain events are needed.

Log Collection & Analysis

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Objective: Effectively collect and analyze logs to diagnose problems Estimated Time: 25 minutes

Scope of This Guide
Covers: kubectl logs usage, multi-container logs, log filtering, stern tool

Does not cover: Central logging system (ELK, Loki) setup, long-term log retention

Before You Begin#

Verify the following prerequisites.

1. Verify kubectl Installation and Cluster Access#

kubectl cluster-info

Success output:

Kubernetes control plane is running at https://xxx.xxx.xxx.xxx

2. Verify Log Access Permissions#

kubectl auth can-i get pods/log

Success output:

Topic Design and Partition Planning Guide

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

A guide on how to determine the appropriate partition count, replication factor, and retention policy when creating new Kafka topics.

Time Required: Approximately 15 minutes

TL;DR

Partition count: Target throughput / Per-consumer throughput, minimum 6 recommended

Replication factor: 3 for production, 1 for development

Retention period: Business requirements + reprocessing buffer time

Naming: Use {domain}.{entity}.{event-type} pattern

Before You Begin#

Prerequisites#

Item	Requirement	Verification Command
Kafka CLI	Installed and in PATH	`kafka-topics.sh --version`
Broker Access	Admin privileges	`kafka-acls.sh --list`
Cluster Info	Know Broker count	`kafka-broker-api-versions.sh`

Environment Verification#

Verify topic creation permissions and cluster status:

Onion Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 15 Jan 2026 00:00:00 +0000

Target Audience: Developers looking for an architecture that works well with DDD Prerequisites: Hexagonal Architecture and Dependency Inversion Principle Estimated Time: About 20 minutes

Onion Architecture#

An architecture proposed by Jeffrey Palermo in 2008. It places the domain model at the very center and wraps it in concentric layers, like an onion.

Onion architecture starts from the limitations of traditional layered architecture. In layered architecture, upper layers depend on lower layers, so changes in the database layer cascade into the service layer. In contrast, onion architecture is designed so that the domain model depends on nothing, and infrastructure depends on the domain. This allows domain logic to express pure business rules regardless of changes in databases, frameworks, or external services. This is precisely why it pairs so well with DDD (Domain-Driven Design).

Classes and Objects

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Classes: Write constructor parameters directly in the declaration to reduce boilerplate

object: Singleton instances are supported at the language level

Companion objects: Replace static members with same-named objects as classes

Traits: Enjoy the benefits of multiple inheritance while avoiding the diamond problem

Target Audience: Developers with Java/OOP experience Prerequisites: Scala basic syntax, functions and methods

Scala supports both object-oriented and functional programming. Classes, objects, traits, and other OOP features can be used harmoniously with the functional paradigm. In particular, Scala’s object supports the singleton pattern at the language level, and traits maximize code reuse while solving multiple inheritance problems.

References

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

A collection of reference materials helpful for learning Scala. Resources are categorized from official documentation to books, online courses, and communities.

TL;DR - Top Recommendations

Official Docs: Scala 3 Book - Step-by-step tutorial

Beginner Book: “Scala for the Impatient” - Optimal for quick learning

Functional Deep Dive: “Scala with Cats” (free online) - Master type classes

Online Practice: Scastie - Run code instantly in browser

Community: Scala Discord - Real-time Q&A

Official Documentation#

Official Scala site and documentation provide the most accurate and up-to-date information.

Spark Integration

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Scala is Spark’s native language: Latest features supported first, most concise API

DataFrame: SQL-style data processing, $"column" syntax for column references

Dataset[T]: Type-safe data processing with Case Classes, compile-time error detection

Performance optimization: Utilize broadcast joins, caching, Predicate Pushdown

Note: Spark 3.5 only supports Scala 2.12/2.13 (Scala 3 not supported)

Target Audience: Scala developers learning large-scale data processing, Spark beginners

Prerequisites:

Scala basic syntax and functional programming concepts
sbt build tool usage
SQL basics (recommended)

Learn how to use Apache Spark with Scala. Scala is Spark’s native language, providing the richest API. Since Spark itself is written in Scala, new features are added to the Scala API first, and you can maximize the benefits of type safety and functional programming.

Aggregate Practical Patterns

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

Target Audience: Developers who have learned Aggregate basics and want to learn practical patterns Prerequisites: Understanding core concepts from Aggregate Deep Dive Reading Time: About 30 minutes Key Question: “What patterns should I apply when implementing Aggregates?”

Summary
Practical patterns: Factory Method (creation) + Domain Event publishing + Optimistic Locking (concurrency) + Soft Delete (deletion)

Aggregate Practical Patterns#

Implementation patterns and decision guides for Aggregate design.

Common imports for examples in this page:

Event Sourcing Practice

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

TL;DR

Event Sourcing: Store events instead of state. Restore state by replaying events

Event Store: Persistent storage for events. Version-based optimistic concurrency control

Event-Sourced Aggregate: Apply/when pattern for event application and state changes

Snapshot: Periodically save state snapshots for performance optimization

Point-in-time Recovery: Replay events up to a specific version to query past state

Target Audience and Prerequisites#

Item	Required Level
Target Audience	Developers looking to implement the Event Sourcing pattern
DDD	Understanding of Aggregate and Domain Event concepts
Java	Switch Expression, Pattern Matching syntax
Prerequisites	Order Domain, Application Layer completed

Implement the Event Sourcing pattern for an order domain. Store events instead of state, and restore state by replaying events.

Full-Stack Observability

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Time Required: 30 minutes Prerequisites: Spring Boot Metrics What You’ll Learn: Track issues by connecting the three pillars (Metrics, Logs, Traces)

Goal#

“Error rate spike” → “Find trace_id in logs” → “Identify root cause with traces”

graph LR
 A["1. Metrics<br>Detect 5% error rate"]
 B["2. Logs<br>Find trace_id"]
 C["3. Traces<br>Discover bottleneck"]
 D["4. Resolution<br>Optimize DB query"]

 A --> B --> C --> D

Scenario: Order Service Failure Analysis#

System Architecture#

graph LR
 USER["User"] --> GW["API Gateway"]
 GW --> ORDER["Order Service"]
 ORDER --> PAYMENT["Payment Service"]
 ORDER --> INVENTORY["Inventory Service"]
 PAYMENT --> DB["Payment DB"]

Step 1: Failure Detection (Metrics)#

Grafana Alert Triggered#

Alert: HighErrorRate
Service: order-service
Error Rate: 5.2%
Threshold: 1%

Check Dashboard#

# Confirm error rate spike
sum(rate(http_server_requests_seconds_count{application="order-service",status=~"5.."}[5m]))
/ sum(rate(http_server_requests_seconds_count{application="order-service"}[5m]))

# Which endpoint?
sum by (uri) (rate(http_server_requests_seconds_count{application="order-service",status=~"5.."}[5m]))

Result: Errors occurring on /orders POST endpoint

histogram_quantile

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs who need response time analysis Prerequisites: Metrics Fundamentals, rate and increase What You’ll Learn: Calculate accurate percentiles from Histograms and monitor SLAs

TL;DR#

Key Summary:

histogram_quantile(φ, bucket): Calculate φ percentile (0 ≤ φ ≤ 1)

P50: histogram_quantile(0.5, ...) - Median

P95: histogram_quantile(0.95, ...) - 95% are at or below this value

P99: histogram_quantile(0.99, ...) - 99% are at or below this value

Must always use with rate() or increase()

Why Percentiles Matter#

Averages are distorted by extreme values. Percentiles better reflect actual user experience.

References

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Official Documentation#

Prometheus#

Resource	Link	Description
Prometheus Official Docs	https://prometheus.io/docs/	Configuration, PromQL, operations guide
PromQL Reference	https://prometheus.io/docs/prometheus/latest/querying/basics/	Query language details
Alerting Rules	https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/	Writing alerting rules

Grafana#

Resource	Link	Description
Grafana Official Docs	https://grafana.com/docs/grafana/latest/	Dashboard, panel configuration
Loki Docs	https://grafana.com/docs/loki/latest/	Log collection/querying
Tempo Docs	https://grafana.com/docs/tempo/latest/	Distributed tracing

OpenTelemetry#

Resource	Link	Description
OpenTelemetry Official	https://opentelemetry.io/docs/	Concepts, SDK, Collector
Java Instrumentation	https://opentelemetry.io/docs/languages/java/	Java auto/manual instrumentation
Semantic Conventions	https://opentelemetry.io/docs/concepts/semantic-conventions/	Standardized attribute names

Books#

Essential#

Book	Author	Content
Site Reliability Engineering	Google SRE Team	SRE principles, Golden Signals, SLO
Observability Engineering	Charity Majors, Liz Fong-Jones	Modern observability concepts
The SRE Workbook	Google SRE Team	Practical SRE application

Recommended#

Book	Author	Content
Prometheus: Up & Running	Brian Brazil	Detailed Prometheus guide
Distributed Tracing in Practice	Austin Parker et al.	Advanced distributed tracing
Database Reliability Engineering	Laine Campbell, Charity Majors	Database observability

Blogs & Articles#

Prometheus/Grafana#

Prometheus Best Practices - Official recommendations
Grafana Blog - New features, case studies
Robust Perception Blog - Brian Brazil’s Prometheus tips

SRE/Observability#

Google SRE Books (Free) - Full content freely available
Honeycomb Blog - Advanced observability
Charity Majors’ Blog - Insights from observability leader

Videos#

Conferences#

Video	Link	Content
PromCon	https://www.youtube.com/@PrometheusIo	Prometheus conference
GrafanaCon	https://www.youtube.com/@Grafana	Grafana conference
KubeCon	https://www.youtube.com/@caborgg	Kubernetes, observability sessions

Tutorials#

Prometheus Tutorial for Beginners - TechWorld with Nana
Grafana Tutorials - Grafana official

Online Courses#

Course	Platform	Description
Prometheus & Grafana	Udemy	Hands-on focused
Site Reliability Engineering	Coursera	Google’s SRE course
Observability with OpenTelemetry	Linux Foundation	OTel introduction

Community#

Slack#

Prometheus Users
Grafana Community
CNCF Slack - #prometheus, #opentelemetry

GitHub#

Project	Link
Prometheus	https://github.com/prometheus/prometheus
Grafana	https://github.com/grafana/grafana
Loki	https://github.com/grafana/loki
Tempo	https://github.com/grafana/tempo
OpenTelemetry	https://github.com/open-telemetry

Dashboards & Rules#

Grafana Dashboards#

ID	Name	Purpose
1860	Node Exporter Full	Server monitoring
3662	Prometheus Stats	Prometheus self-monitoring
4701	JVM Micrometer	Spring Boot JVM
7362	MySQL Overview	MySQL monitoring
7587	PostgreSQL	PostgreSQL monitoring
11074	Kafka Exporter	Kafka monitoring

Searchable at https://grafana.com/grafana/dashboards/

Saturation (Resource Usage)

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: SREs and infrastructure engineers managing system capacity Prerequisites: Prometheus Architecture After reading this: You’ll be able to detect resource bottlenecks early and establish capacity planning

TL;DR#

Key Summary:

Saturation: How “full” resources are (0-100%)

Key resources: CPU, memory, disk, network, connection pools

Approaching 100% causes latency spikes, errors

USE method: Utilization, Saturation, Errors

Why Monitor Saturation?#

Saturation is the only metric that can predict failures. Latency, Traffic, and Errors react after problems occur, but Saturation warns before problems occur.

Service

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to configure network access in Kubernetes Prerequisites: Pod, Deployment concepts After reading this: You will understand how to reliably access Pods with Services

TL;DR

Service provides a stable network endpoint for a set of Pods

Pod IPs change but Service IP (ClusterIP) is fixed

Three types exist: ClusterIP, NodePort, LoadBalancer

Why Service is Needed#

Pod IPs change whenever Pods are recreated. How can we communicate reliably in this situation?

Consumer Advanced Operations

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

session.timeout.ms controls failure detection; max.poll.interval.ms limits processing time

Use Cooperative Sticky Assignor to minimize rebalancing impact (Kafka 2.4+)

Static Group Membership prevents rebalancing on restarts

Consumer Lag is the most important monitoring metric; trend observation is key

Use kafka-consumer-groups.sh for manual Offset reset (Consumer must be stopped)

Target audience: Developers and operators running Kafka Consumers in production

Prerequisites: Basic concepts from Consumer Group & Offset, ISR and Leader concepts from Replication

Frequently Asked Questions

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

This section compiles frequently asked questions and answers about Kafka. Organized by topic from basic concepts to configuration, error handling, performance tuning, operations, and Spring Kafka.

TL;DR

Basic Concepts: Kafka is a distributed event streaming platform, not a message queue; order is guaranteed per Partition

Configuration: Use acks=all for safety, auto.offset.reset=earliest recommended, manual commit to prevent message loss

Error Handling: Use @RetryableTopic for retries then move to Dead Letter Topic

Performance: Producer - batch/compression; Consumer - increase instances and adjust fetch settings

Operations: Recommend replication.factor=3, min.insync.replicas=2

Spring Kafka: Simple implementation with KafkaTemplate and @KafkaListener

Basic Concepts#

Q: Is Kafka a message queue?

Log Analysis System

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

Complete Example Project
If you want to run this code immediately, use the complete Spring Boot project:

📁 examples/elasticsearch/log-analysis/

Instant Elasticsearch + Kibana execution with docker-compose

Includes sample log generation API

TL;DR

Send application logs directly to Elasticsearch using Logback Appender

Ensure traceability by automatically including request ID and user ID in logs with MDC

Analyze error rates and response time percentiles with Aggregations

Automatically manage log lifecycle (Hot→Warm→Cold→Delete) with ILM

Total time required: approximately 40 minutes

Implement a system to collect, store, and analyze application logs using Elasticsearch.

Microservices Integration

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

Implement event-driven communication using Kafka in a microservices environment.

TL;DR

Event Chaining: Event propagation in order: Order -> Payment -> Shipment -> Notification

Correlation ID: Propagate correlation ID for distributed tracing

Saga Pattern: Handle distributed transactions with compensation transactions

Idempotency: Safely handle duplicate messages

Target Audience and Prerequisites#

Item	Description
Target Audience	Developers building event-driven communication in microservices architecture
Prerequisites	Kafka basics, Spring Boot, understanding of Order System example
Required Environment	Docker, JDK 17+, environment capable of running multiple services
Estimated Time	About 60 minutes

Scenario: Order Processing System#

In this example, the Order Service, Payment Service, Shipment Service, and Notification Service exchange events through Kafka. When the Order Service creates an order, it publishes an event to the orders Topic. The Payment Service receives this and processes payment, publishing results to the payments Topic. The Shipment Service receives payment completed events and creates shipments. The Notification Service subscribes to all Topics and sends notifications to customers.

Replication

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Replication Factor (RF) is the number of Partition replicas; RF=3 is recommended for production

Leader handles all reads/writes, Followers only replicate and promote on failure

ISR (In-Sync Replicas) is the set of replicas synchronized with the Leader

Use min.insync.replicas=2 + acks=all combination for data safety

KRaft mode (Kafka 3.3+) enables cluster operation without Zookeeper

Target Audience: Developers and operators managing Kafka clusters or designing high-availability systems

Prerequisites: Understanding of Broker and Partition concepts from Core Components

Search Relevance

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 08 Jan 2026 00:00:00 +0000

Prerequisites
Before reading this document, understand these concepts first:

Query DSL - match, bool query basics

Data Modeling - Analyzer operation principles

Learn relevance tuning methods including Score, BM25, and Boosting to improve search result quality.

The core value of a search engine is enabling users to find what they want on the first page. No matter how quickly results are returned, if what users want is on page 10, it’s meaningless. Search Relevance is the key concept for solving this problem.

Monitoring Setup

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Spark Monitoring Setup Guide#

Monitoring configuration guide for stable operation of Spark applications in production environments.

Monitoring Architecture#

flowchart TB
 subgraph Spark["Spark Cluster"]
 Driver[Driver]
 E1[Executor 1]
 E2[Executor 2]
 E3[Executor N]
 end

 subgraph Metrics["Metrics Collection"]
 SparkUI[Spark UI :4040]
 History[History Server :18080]
 Prometheus[Prometheus]
 Graphite[Graphite]
 end

 subgraph Viz["Visualization/Alerting"]
 Grafana[Grafana Dashboard]
 AlertManager[Alert Manager]
 end

 Driver --> SparkUI
 Driver --> History
 E1 --> Prometheus
 E2 --> Prometheus
 E3 --> Prometheus
 Prometheus --> Grafana
 Grafana --> AlertManager

Spark UI Configuration#

Enable Basic Spark UI#

SparkSession spark = SparkSession.builder()
 .appName("Monitored Application")
 // Spark UI settings
 .config("spark.ui.enabled", "true")
 .config("spark.ui.port", "4040")
 .config("spark.ui.retainedJobs", "1000")
 .config("spark.ui.retainedStages", "1000")
 .config("spark.ui.retainedTasks", "10000")
 // Shuffle/storage metrics retention
 .config("spark.sql.ui.retainedExecutions", "200")
 .getOrCreate();

History Server Setup#

View logs even after application termination.

Spark SQL

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Spark SQL#

Spark SQL is a Spark module for structured data processing. It supports both SQL queries and the DataFrame API, using the same execution engine (Catalyst Optimizer).

Benefits of Spark SQL#

Familiar SQL syntax: Leverage existing SQL knowledge directly
Optimization: Catalyst Optimizer automatically optimizes queries
Various data sources: Integration with JDBC, Parquet, JSON, Hive, etc.
DataFrame interoperability: SQL results returned as DataFrames

Deep Dive into Catalyst Optimizer#

Catalyst is Spark SQL’s query optimization engine. It analyzes user queries and optimizes them to generate efficient execution plans.

Cluster Scaling

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

This guide walks you through safely scaling an Elasticsearch cluster.

Estimated time: About 30-60 minutes (additional time for node addition and shard rebalancing)

Scope of This Guide
Covers: Adding data nodes, role-based node separation, Hot-Warm-Cold architecture, and post-scaling validation

Does not cover: For query-level performance optimization, see Slow Query Optimization. For memory issues, see Memory Troubleshooting.

TL;DR

Horizontal scaling: Add data nodes and rebalance shards

Role separation: Separate master, data, coordinating, and ingest nodes for stability

Hot-Warm-Cold: Tier nodes by data access frequency for cost optimization

Post-scaling validation: Always verify cluster status, shard distribution, and performance metrics

Before You Begin#

Verify the following prerequisites:

Performance Profiling

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

This guide walks you through finding and optimizing performance bottlenecks in Scala applications.

Estimated time: About 20-25 minutes

TL;DR

CPU profiling: Identify hotspots with JFR (Java Flight Recorder)

Memory analysis: Analyze heap dumps with VisualVM or MAT

Collection selection: Choosing the right collection for the job can make a significant performance difference

Optimization techniques: @specialized, @tailrec, avoiding boxing

Problems This Guide Solves#

Use this guide in the following situations:

Your application’s response time has suddenly become slow
Memory usage is continuously increasing (suspected memory leak)
You need to decide which collection to use from a performance perspective
GC pauses are occurring frequently

What This Guide Does Not Cover#

JVM tuning (GC options, etc.): See the official JVM documentation
Distributed system performance optimization: This is a separate topic
Cats Effect / ZIO performance optimization: See the respective library documentation

Before You Begin#

Verify the following environment is ready:

RBAC Configuration Lab

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

Target Audience: Developers/operators who want to configure access control for Kubernetes clusters Prerequisites: Namespace, Pod, Deployment concepts After reading this: You will be able to configure per-namespace access control using Role, RoleBinding, and ServiceAccount

TL;DR

Create a limited-permission Role for the development team

Create a ServiceAccount and bind it to the Role

Test permissions with kubectl auth can-i

Separate access permissions by namespace

RBAC Structure#

flowchart TD
 SA[ServiceAccount] -->|Binding| RB[RoleBinding]
 R[Role] -->|Reference| RB
 RB -->|Grant permissions| NS[Namespace Resources]

 SA2[ServiceAccount] -->|Binding| CRB[ClusterRoleBinding]
 CR[ClusterRole] -->|Reference| CRB
 CRB -->|Grant permissions| CL[Cluster-wide Resources]

Resource	Scope	Description
Role	Namespace	Permissions for resources within a specific Namespace
ClusterRole	Cluster	Permissions for cluster-wide resources
RoleBinding	Namespace	Bind Role/ClusterRole to subjects
ClusterRoleBinding	Cluster	Bind ClusterRole to subjects

Prerequisites#

You will need the following:

CQRS

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 15 Jan 2026 00:00:00 +0000

Target Audience: Developers designing systems with complex query requirements or performance optimization needs Prerequisites: Event-Driven Architecture or basic understanding of event-driven architecture concepts Estimated Time: About 35 minutes Key Question: “When should you separate read and write models?”

Summary
CQRS core: Separate Command (state changes, uses domain model) and Query (reads, uses optimized read model) to optimize each according to its requirements

Analogy: A Library
You can compare CQRS to library operations:

Case Classes

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Case classes are special classes for immutable data modeling

apply, unapply, copy, equals, hashCode, and toString are automatically generated

Define ADTs (Algebraic Data Types) with sealed trait + case classes

Show their true value when used with pattern matching

Target Audience: Developers who have learned Scala basic syntax Prerequisites: Classes and objects, basic understanding of type system

Case classes are special classes for immutable data modeling. You can define data classes without boilerplate code. The compiler automatically generates useful methods like equals, hashCode, toString, and copy, so you can define data-centric classes very concisely. They show their true value especially when used with pattern matching.

Practical Projects

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

REST API Server: Build functional web server with http4s + Circe + Cats Effect

Data Pipeline: Memory-efficient stream processing with FS2

CLI Tool: Type-safe command-line parser with scopt

Error Handling: Collect multiple validation errors simultaneously with Cats Validated

All examples emphasize functional style with immutability, referential transparency, and type safety

Target Audience: Developers building real services with Scala

Prerequisites:

Scala basic syntax and functional programming concepts
sbt build tool usage
REST API and HTTP basics (Project 1)
(Optional) Basic knowledge of Cats Effect/IO monad

Examples of building real services with Scala. Implement REST API servers and data pipelines. These examples demonstrate how to write type-safe, maintainable code by leveraging Scala’s functional programming characteristics.

By Service Type

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: SREs operating various service types Prerequisites: SRE Golden Signals After reading this: You’ll be able to apply the four signals tailored to service characteristics

TL;DR#

Key Summary:

Each service type has different key signals

Web API: Latency, Errors focused

Kafka: Traffic (Lag), Saturation focused

Database: Latency, Saturation focused

Batch jobs: Errors, Traffic (completion rate) focused

Key Signals by Service Type#

Service Type	Key Signals	Reason
Web API	Latency, Errors	Directly affects user experience
Kafka	Traffic (Lag), Saturation	Throughput is key
Database	Latency, Saturation	Query performance, connections
Cache (Redis)	Latency, Saturation	Response speed, memory
Batch jobs	Errors, Traffic	Completion rate, throughput
Load balancer	Traffic, Errors	Connection distribution

Web API / Microservices#

Key Metrics#

graph TD
 subgraph "Web API Key"
 L["Latency<br>P99 response time"]
 E["Errors<br>5xx ratio"]
 T["Traffic<br>RPS"]
 S["Saturation<br>Connections/threads"]
 end

 L --> |"SLA"| SLA["P99 < 500ms"]
 E --> |"SLO"| SLO["Error rate < 0.1%"]

PromQL Queries#

# Latency: P99 response time
histogram_quantile(0.99,
 sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

# Traffic: Requests per second
sum by (service) (rate(http_requests_total[5m]))

# Errors: Error rate
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (service) (rate(http_requests_total[5m]))

# Saturation: Concurrent requests
sum by (service) (http_server_requests_active)

Recording Rules#

groups:
 - name: web_api_golden_signals
 rules:
 - record: service:http_request_duration_seconds:p99
 expr: histogram_quantile(0.99, sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))

 - record: service:http_requests:rate5m
 expr: sum by (service) (rate(http_requests_total[5m]))

 - record: service:http_errors:ratio
 expr: sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (service) (rate(http_requests_total[5m]))

Kafka#

Key Metrics#

graph TD
 subgraph "Kafka Key"
 T["Traffic<br>Message throughput"]
 L["Latency<br>Consumer Lag"]
 S["Saturation<br>Disk, partitions"]
 E["Errors<br>Failed messages"]
 end

 L --> |"Important"| LAG["Lag < 10,000"]
 S --> |"Important"| DISK["Disk < 80%"]

PromQL Queries#

# Traffic: Messages per second
sum by (topic) (rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))

# Latency: Consumer Lag
sum by (consumer_group, topic) (kafka_consumer_group_lag)

# Saturation: Disk usage
kafka_log_log_size / kafka_log_log_max_size * 100

# Saturation: Under-replicated partitions
kafka_server_replicamanager_underreplicatedpartitions

# Errors: Failure rate
rate(kafka_producer_record_error_total[5m])

Alert Rules#

groups:
 - name: kafka_alerts
 rules:
 - alert: KafkaConsumerLagHigh
 expr: sum by (consumer_group, topic) (kafka_consumer_group_lag) > 10000
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "Kafka consumer lag high"

 - alert: KafkaUnderReplicated
 expr: kafka_server_replicamanager_underreplicatedpartitions > 0
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "Kafka under-replicated partitions"

 - alert: KafkaDiskHigh
 expr: kafka_log_log_size / kafka_log_log_max_size > 0.8
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "Kafka disk usage high"

Database (PostgreSQL/MySQL)#

Key Metrics#

graph TD
 subgraph "DB Key"
 L["Latency<br>Query time"]
 S["Saturation<br>Connections, buffer"]
 E["Errors<br>Deadlocks, failures"]
 T["Traffic<br>QPS"]
 end

 S --> |"Important"| CONN["Connections < 80%"]
 L --> |"Important"| SLOW["Slow queries"]

PromQL Queries (PostgreSQL)#

# Latency: Query time
rate(pg_stat_statements_total_time_seconds_sum[5m])
/ rate(pg_stat_statements_calls_total[5m])

# Traffic: Queries per second (QPS)
sum(rate(pg_stat_statements_calls_total[5m]))

# Saturation: Connection usage
sum(pg_stat_activity_count) / pg_settings_max_connections * 100

# Saturation: Buffer hit rate
pg_stat_bgwriter_buffers_alloc
/ (pg_stat_bgwriter_buffers_alloc + pg_stat_bgwriter_buffers_backend) * 100

# Errors: Deadlocks
rate(pg_stat_database_deadlocks_total[5m])

PromQL Queries (MySQL)#

# Traffic: QPS
rate(mysql_global_status_queries[5m])

# Latency: Slow queries
rate(mysql_global_status_slow_queries[5m])

# Saturation: Connection usage
mysql_global_status_threads_connected / mysql_global_variables_max_connections * 100

# Saturation: Buffer pool usage
mysql_global_status_innodb_buffer_pool_pages_data
/ mysql_global_status_innodb_buffer_pool_pages_total * 100

Cache (Redis)#

Key Metrics#

graph TD
 subgraph "Redis Key"
 L["Latency<br>Command time"]
 S["Saturation<br>Memory"]
 T["Traffic<br>Command count"]
 E["Errors<br>Rejects, evictions"]
 end

 S --> |"Important"| MEM["Memory < 80%"]
 L --> |"Important"| HIT["Hit rate > 90%"]

PromQL Queries#

# Traffic: Commands per second
rate(redis_commands_total[5m])

# Latency: Average command time
rate(redis_commands_duration_seconds_total[5m])
/ rate(redis_commands_total[5m])

# Saturation: Memory usage
redis_memory_used_bytes / redis_memory_max_bytes * 100

# Errors: Hit rate
rate(redis_keyspace_hits_total[5m])
/ (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))
* 100

# Saturation: Connection count
redis_connected_clients

Batch Jobs#

Key Metrics#

graph TD
 subgraph "Batch Key"
 E["Errors<br>Failure rate"]
 T["Traffic<br>Throughput"]
 L["Latency<br>Execution time"]
 end

 E --> |"Important"| FAIL["Failure rate < 1%"]
 T --> |"Important"| DONE["Completion rate 100%"]

PromQL Queries#

# Traffic: Completed jobs
increase(batch_jobs_completed_total[1h])

# Errors: Failure rate
increase(batch_jobs_failed_total[1h])
/ (increase(batch_jobs_completed_total[1h]) + increase(batch_jobs_failed_total[1h]))

# Latency: Average execution time
rate(batch_job_duration_seconds_sum[1h])
/ rate(batch_job_duration_seconds_count[1h])

# Traffic: Processed items
increase(batch_items_processed_total[1h])

Alert Rules#

 - alert: BatchJobFailed
 expr: increase(batch_jobs_failed_total[1h]) > 0
 labels:
 severity: warning
 annotations:
 summary: "Batch job failed"

 - alert: BatchJobNotRunning
 expr: time() - batch_job_last_success_timestamp > 86400
 labels:
 severity: critical
 annotations:
 summary: "Batch job hasn't run in 24 hours"

Load Balancer (Nginx/HAProxy)#

PromQL Queries (Nginx)#

# Traffic: Requests per second
sum(rate(nginx_http_requests_total[5m]))

# Errors: 5xx ratio
sum(rate(nginx_http_requests_total{status=~"5.."}[5m]))
/ sum(rate(nginx_http_requests_total[5m]))

# Saturation: Active connections
nginx_connections_active

# Latency: Upstream response time
histogram_quantile(0.99, rate(nginx_upstream_response_time_seconds_bucket[5m]))

Dashboard Templates#

Panel Layout by Service Type#

Service	Row 1 (Summary)	Row 2 (Detail)	Row 3 (Trend)
Web API	P99, RPS, Error rate	Status code distribution, By endpoint	Time series
Kafka	Lag, Throughput, Partitions	By consumer, By topic	Time series
DB	QPS, Connections, Slow queries	By query, By table	Time series
Redis	Commands, Hit rate, Memory	By command, By keyspace	Time series

Key Summary#

Service	Top Priority	2nd Priority	3rd Priority
Web API	Latency	Errors	Traffic
Kafka	Traffic (Lag)	Saturation	Errors
DB	Saturation	Latency	Traffic
Redis	Saturation	Latency	Traffic
Batch	Errors	Traffic	Latency

Next Steps#

Recommended Order	Document	What You’ll Learn
1	Environment Setup	Practice environment
2	Kafka Monitoring	Kafka details

Recording Rules

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Operators and SREs experiencing dashboard performance issues Prerequisites: histogram_quantile What You’ll Learn: Improve query performance with Recording Rules and manage complex metrics

TL;DR#

Key Summary:

Recording Rules save query results as new metrics

Pre-compute complex queries to improve dashboard/alert performance

Naming: level:metric:operations pattern recommended

Define frequently used aggregations as Rules

Why Recording Rules are Needed#

Problem: Repeatedly Executing Complex Queries#

# Executed every 15 seconds on dashboard
histogram_quantile(0.99,
 sum by (service, le) (
 rate(http_request_duration_seconds_bucket[5m])
 )
)

Issues:

ConfigMap and Secret

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to manage application configuration in Kubernetes Prerequisites: Pod concepts After reading this: You will understand how to separate and manage configuration and sensitive information with ConfigMap and Secret

TL;DR

ConfigMap stores general configuration (DB host, log level, etc.)

Secret stores sensitive information (passwords, API keys, etc.)

Both can be injected into Pods as environment variables or files

Overall Flow#

Let’s first examine the flow of ConfigMap and Secret injection into Pods.

Advanced Concepts

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

acks=0 (fast/may lose), acks=1 (Leader only), acks=all (all ISR, recommended)

Use acks=all + min.insync.replicas=2 combination for data safety

Message Key sends to same Partition for ordering; be cautious when changing Partition count

Retention: time-based deletion (default 7 days), size-based deletion, Log Compaction

Idempotent Producer (default in Kafka 3.0+) prevents duplicates on network errors

Target Audience: Developers who want to understand Kafka’s advanced concepts and optimize production settings

Prerequisites: Topic, Partition, Broker concepts from Message Flow, ISR, Leader, Follower concepts from Replication

Aggregations

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 08 Jan 2026 00:00:00 +0000

Prerequisites
Before reading this document, understand these concepts first:

Query DSL - Basic query structure

Data Modeling - keyword vs text types

Learn how to analyze data and extract statistics using Elasticsearch Aggregations.

While Elasticsearch is a search engine, it’s also a powerful real-time analytics engine. When you search for “laptop” on an e-commerce site, the brand filters, price range filters, and rating distributions that appear on the left side are all results of Aggregations. Implementing these features with RDBMS would require complex subqueries and GROUP BY combinations, and performance issues would be severe with large datasets.

Spring Boot Integration

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Spring Boot and Spark Integration Guide#

Spark and Spring Boot integration patterns for Java/Spring developers.

Architecture Pattern#

flowchart TB
 subgraph SpringBoot["Spring Boot Application"]
 Controller[REST Controller]
 Service[Business Service]
 SparkService[Spark Service]
 Config[Spark Configuration]
 end

 subgraph Spark["Spark Context"]
 SparkSession[SparkSession Bean]
 Jobs[Batch Jobs]
 end

 subgraph Storage["Data Storage"]
 DB[(Database)]
 HDFS[(HDFS/S3)]
 Kafka[Kafka]
 end

 Controller --> Service
 Service --> SparkService
 SparkService --> SparkSession
 SparkSession --> Jobs
 Jobs --> HDFS
 Jobs --> DB
 Jobs --> Kafka
 Config --> SparkSession

Gradle Configuration#

// build.gradle.kts
plugins {
 java
 id("org.springframework.boot") version "3.2.5"
 id("io.spring.dependency-management") version "1.1.4"
}

java {
 toolchain {
 languageVersion.set(JavaLanguageVersion.of(17))
 }
}

repositories {
 mavenCentral()
}

val sparkVersion = "3.5.1"
val scalaVersion = "2.13"

dependencies {
 // Spring Boot
 implementation("org.springframework.boot:spring-boot-starter-web")
 implementation("org.springframework.boot:spring-boot-starter-actuator")

 // Spark (Log4j exclusion required)
 implementation("org.apache.spark:spark-core_$scalaVersion:$sparkVersion") {
 exclude(group = "org.slf4j", module = "slf4j-log4j12")
 exclude(group = "log4j", module = "log4j")
 exclude(group = "org.apache.logging.log4j")
 }
 implementation("org.apache.spark:spark-sql_$scalaVersion:$sparkVersion") {
 exclude(group = "org.slf4j", module = "slf4j-log4j12")
 }

 // Spark JDBC (optional)
 implementation("org.apache.spark:spark-hive_$scalaVersion:$sparkVersion")

 // Jackson for JSON (version unification)
 implementation("com.fasterxml.jackson.module:jackson-module-scala_$scalaVersion:2.15.3")

 // Logging (Spring Boot standard)
 implementation("org.springframework.boot:spring-boot-starter-logging")

 // Test
 testImplementation("org.springframework.boot:spring-boot-starter-test")
}

// Dependency conflict resolution
configurations.all {
 exclude(group = "org.slf4j", module = "slf4j-log4j12")
 exclude(group = "log4j", module = "log4j")
 resolutionStrategy {
 force("com.fasterxml.jackson.core:jackson-databind:2.15.3")
 force("com.fasterxml.jackson.core:jackson-core:2.15.3")
 }
}

SparkSession Bean Configuration#

Basic Configuration#

package com.example.spark.config;

import org.apache.spark.SparkConf;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import jakarta.annotation.PreDestroy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Configuration
public class SparkConfig {
 private static final Logger logger = LoggerFactory.getLogger(SparkConfig.class);

 @Value("${spark.app.name:spring-spark-app}")
 private String appName;

 @Value("${spark.master:local[*]}")
 private String masterUrl;

 @Value("${spark.driver.memory:2g}")
 private String driverMemory;

 @Value("${spark.executor.memory:2g}")
 private String executorMemory;

 @Value("${spark.sql.shuffle.partitions:200}")
 private int shufflePartitions;

 private SparkSession sparkSession;

 @Bean
 @Profile("!test") // Use separate configuration for tests
 public SparkSession sparkSession() {
 logger.info("Initializing SparkSession: app={}, master={}", appName, masterUrl);

 SparkConf conf = new SparkConf()
 .setAppName(appName)
 .setMaster(masterUrl)
 .set("spark.driver.memory", driverMemory)
 .set("spark.executor.memory", executorMemory)
 .set("spark.sql.shuffle.partitions", String.valueOf(shufflePartitions))
 // Adaptive Query Execution
 .set("spark.sql.adaptive.enabled", "true")
 .set("spark.sql.adaptive.coalescePartitions.enabled", "true")
 // Serialization optimization
 .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
 .set("spark.kryoserializer.buffer.max", "1024m")
 // UI settings
 .set("spark.ui.enabled", "true")
 .set("spark.ui.port", "4040");

 this.sparkSession = SparkSession.builder()
 .config(conf)
 .getOrCreate();

 // Adjust Spark log level
 sparkSession.sparkContext().setLogLevel("WARN");

 logger.info("SparkSession initialized successfully");
 return sparkSession;
 }

 @Bean
 @Profile("test")
 public SparkSession testSparkSession() {
 logger.info("Initializing Test SparkSession");

 this.sparkSession = SparkSession.builder()
 .appName("test-spark")
 .master("local[2]")
 .config("spark.sql.shuffle.partitions", "2")
 .config("spark.ui.enabled", "false")
 .config("spark.driver.bindAddress", "127.0.0.1")
 .getOrCreate();

 return sparkSession;
 }

 @PreDestroy
 public void cleanup() {
 if (sparkSession != null && !sparkSession.sparkContext().isStopped()) {
 logger.info("Stopping SparkSession");
 sparkSession.stop();
 }
 }
}

application.yml Configuration#

# application.yml
spring:
 application:
 name: spark-spring-app

# Spark settings
spark:
 app:
 name: ${spring.application.name}
 master: local[*]
 driver:
 memory: 2g
 executor:
 memory: 2g
 sql:
 shuffle:
 partitions: 100

# Profile-specific settings
---
spring:
 config:
 activate:
 on-profile: production

spark:
 master: spark://spark-master:7077
 driver:
 memory: 4g
 executor:
 memory: 8g
 sql:
 shuffle:
 partitions: 200

---
spring:
 config:
 activate:
 on-profile: test

spark:
 master: local[2]
 driver:
 memory: 512m
 sql:
 shuffle:
 partitions: 2

Service Layer Patterns#

Basic Spark Service#

package com.example.spark.service;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.stereotype.Service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import static org.apache.spark.sql.functions.*;

@Service
public class DataAnalysisService {
 private static final Logger logger = LoggerFactory.getLogger(DataAnalysisService.class);

 private final SparkSession spark;

 public DataAnalysisService(SparkSession spark) {
 this.spark = spark;
 }

 /**
 * Generate sales summary report
 */
 public SalesSummary generateSalesSummary(String dataPath, String startDate, String endDate) {
 logger.info("Generating sales summary: {} ~ {}", startDate, endDate);

 Dataset<Row> sales = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .parquet(dataPath);

 Dataset<Row> filtered = sales
 .filter(col("sale_date").between(startDate, endDate));

 Row summary = filtered.agg(
 count("*").alias("total_orders"),
 sum("amount").alias("total_revenue"),
 avg("amount").alias("avg_order_value"),
 countDistinct("customer_id").alias("unique_customers")
 ).first();

 return new SalesSummary(
 summary.getLong(0),
 summary.getDouble(1),
 summary.getDouble(2),
 summary.getLong(3)
 );
 }

 /**
 * Revenue aggregation by category
 */
 public List<CategoryRevenue> getCategoryRevenue(String dataPath) {
 Dataset<Row> result = spark.read()
 .parquet(dataPath)
 .groupBy("category")
 .agg(
 sum("amount").alias("revenue"),
 count("*").alias("order_count")
 )
 .orderBy(col("revenue").desc());

 return result.collectAsList().stream()
 .map(row -> new CategoryRevenue(
 row.getString(0),
 row.getDouble(1),
 row.getLong(2)
 ))
 .toList();
 }

 // DTO classes
 public record SalesSummary(
 long totalOrders,
 double totalRevenue,
 double avgOrderValue,
 long uniqueCustomers
 ) {}

 public record CategoryRevenue(
 String category,
 double revenue,
 long orderCount
 ) {}
}

Asynchronous Batch Processing#

package com.example.spark.service;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;
import org.springframework.scheduling.annotation.Async;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.LocalDate;
import java.time.format.DateTimeFormatter;
import java.util.concurrent.CompletableFuture;

import static org.apache.spark.sql.functions.*;

@Service
public class BatchProcessingService {
 private static final Logger logger = LoggerFactory.getLogger(BatchProcessingService.class);

 private final SparkSession spark;

 public BatchProcessingService(SparkSession spark) {
 this.spark = spark;
 }

 /**
 * Daily ETL job (scheduled)
 */
 @Scheduled(cron = "0 0 2 * * *") // Daily at 2 AM
 public void dailyETL() {
 String yesterday = LocalDate.now().minusDays(1)
 .format(DateTimeFormatter.ISO_DATE);

 logger.info("Starting daily ETL: {}", yesterday);

 try {
 // 1. Read raw data
 Dataset<Row> rawData = spark.read()
 .parquet("s3://data-lake/raw/events/date=" + yesterday);

 // 2. Data cleaning
 Dataset<Row> cleaned = rawData
 .filter(col("event_type").isNotNull())
 .dropDuplicates("event_id")
 .withColumn("processed_at", current_timestamp());

 // 3. Create aggregation table
 Dataset<Row> aggregated = cleaned
 .groupBy("user_id", "event_type")
 .agg(
 count("*").alias("event_count"),
 sum("value").alias("total_value")
 );

 // 4. Save
 aggregated.write()
 .mode(SaveMode.Overwrite)
 .partitionBy("event_type")
 .parquet("s3://data-lake/processed/daily/" + yesterday);

 logger.info("Daily ETL completed: {} records processed", cleaned.count());

 } catch (Exception e) {
 logger.error("Daily ETL failed: {}", e.getMessage(), e);
 throw new RuntimeException("ETL failed", e);
 }
 }

 /**
 * Asynchronous large dataset processing
 */
 @Async
 public CompletableFuture<ProcessingResult> processLargeDataset(String inputPath, String outputPath) {
 logger.info("Starting async processing: {}", inputPath);
 long startTime = System.currentTimeMillis();

 try {
 Dataset<Row> data = spark.read().parquet(inputPath);

 Dataset<Row> processed = data
 .repartition(200) // Parallel processing optimization
 .transform(this::applyBusinessLogic);

 processed.write()
 .mode(SaveMode.Overwrite)
 .parquet(outputPath);

 long duration = System.currentTimeMillis() - startTime;
 long recordCount = processed.count();

 logger.info("Async processing completed: {} records, {}ms", recordCount, duration);

 return CompletableFuture.completedFuture(
 new ProcessingResult(true, recordCount, duration, null)
 );

 } catch (Exception e) {
 logger.error("Async processing failed: {}", e.getMessage(), e);
 return CompletableFuture.completedFuture(
 new ProcessingResult(false, 0, 0, e.getMessage())
 );
 }
 }

 private Dataset<Row> applyBusinessLogic(Dataset<Row> df) {
 return df
 .filter(col("status").equalTo("ACTIVE"))
 .withColumn("score", col("value").multiply(0.8).plus(col("bonus")));
 }

 public record ProcessingResult(
 boolean success,
 long recordCount,
 long durationMs,
 String errorMessage
 ) {}
}

REST API Integration#

Analytics API Controller#

package com.example.spark.controller;

import com.example.spark.service.DataAnalysisService;
import com.example.spark.service.DataAnalysisService.SalesSummary;
import com.example.spark.service.DataAnalysisService.CategoryRevenue;
import com.example.spark.service.BatchProcessingService;
import com.example.spark.service.BatchProcessingService.ProcessingResult;

import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.util.List;
import java.util.concurrent.CompletableFuture;

@RestController
@RequestMapping("/api/v1/analytics")
public class AnalyticsController {

 private final DataAnalysisService analysisService;
 private final BatchProcessingService batchService;

 public AnalyticsController(
 DataAnalysisService analysisService,
 BatchProcessingService batchService) {
 this.analysisService = analysisService;
 this.batchService = batchService;
 }

 /**
 * Get sales summary
 * GET /api/v1/analytics/sales/summary?start=2024-01-01&end=2024-01-31
 */
 @GetMapping("/sales/summary")
 public ResponseEntity<SalesSummary> getSalesSummary(
 @RequestParam String start,
 @RequestParam String end) {

 SalesSummary summary = analysisService.generateSalesSummary(
 "data/sales.parquet", start, end);

 return ResponseEntity.ok(summary);
 }

 /**
 * Get revenue by category
 * GET /api/v1/analytics/category/revenue
 */
 @GetMapping("/category/revenue")
 public ResponseEntity<List<CategoryRevenue>> getCategoryRevenue() {
 List<CategoryRevenue> revenues = analysisService.getCategoryRevenue(
 "data/sales.parquet");

 return ResponseEntity.ok(revenues);
 }

 /**
 * Start async batch processing
 * POST /api/v1/analytics/batch/process
 */
 @PostMapping("/batch/process")
 public ResponseEntity<String> startBatchProcess(
 @RequestBody BatchRequest request) {

 CompletableFuture<ProcessingResult> future = batchService
 .processLargeDataset(request.inputPath(), request.outputPath());

 // Return job ID (integrate with job tracking system in real implementation)
 String jobId = "job-" + System.currentTimeMillis();

 return ResponseEntity.accepted()
 .body("{\"jobId\": \"" + jobId + "\", \"status\": \"PROCESSING\"}");
 }

 public record BatchRequest(String inputPath, String outputPath) {}
}

Writing Tests#

Spark Integration Tests#

package com.example.spark.service;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
import org.junit.jupiter.api.*;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.ActiveProfiles;

import java.util.Arrays;
import java.util.List;

import static org.assertj.core.api.Assertions.assertThat;

@SpringBootTest
@ActiveProfiles("test")
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class DataAnalysisServiceTest {

 @Autowired
 private SparkSession spark;

 @Autowired
 private DataAnalysisService analysisService;

 private String testDataPath;

 @BeforeAll
 void setupTestData() {
 // Test data schema
 StructType schema = new StructType(new StructField[]{
 DataTypes.createStructField("order_id", DataTypes.StringType, false),
 DataTypes.createStructField("category", DataTypes.StringType, false),
 DataTypes.createStructField("amount", DataTypes.DoubleType, false),
 DataTypes.createStructField("customer_id", DataTypes.StringType, false),
 DataTypes.createStructField("sale_date", DataTypes.StringType, false)
 });

 // Generate test data
 List<Row> testData = Arrays.asList(
 RowFactory.create("O001", "Electronics", 150.0, "C001", "2024-01-15"),
 RowFactory.create("O002", "Electronics", 200.0, "C002", "2024-01-15"),
 RowFactory.create("O003", "Clothing", 50.0, "C001", "2024-01-16"),
 RowFactory.create("O004", "Books", 25.0, "C003", "2024-01-17"),
 RowFactory.create("O005", "Electronics", 300.0, "C002", "2024-01-18")
 );

 Dataset<Row> testDf = spark.createDataFrame(testData, schema);

 testDataPath = "target/test-data/sales.parquet";
 testDf.write()
 .mode("overwrite")
 .parquet(testDataPath);
 }

 @Test
 @DisplayName("Sales summary should be calculated correctly")
 void shouldCalculateSalesSummary() {
 // when
 var summary = analysisService.generateSalesSummary(
 testDataPath, "2024-01-01", "2024-01-31");

 // then
 assertThat(summary.totalOrders()).isEqualTo(5);
 assertThat(summary.totalRevenue()).isEqualTo(725.0);
 assertThat(summary.avgOrderValue()).isEqualTo(145.0);
 assertThat(summary.uniqueCustomers()).isEqualTo(3);
 }

 @Test
 @DisplayName("Category revenues should be sorted descending")
 void shouldReturnCategoryRevenuesSorted() {
 // when
 var revenues = analysisService.getCategoryRevenue(testDataPath);

 // then
 assertThat(revenues).hasSize(3);
 assertThat(revenues.get(0).category()).isEqualTo("Electronics");
 assertThat(revenues.get(0).revenue()).isEqualTo(650.0);
 }

 @AfterAll
 void cleanup() {
 // Clean up test data
 new java.io.File(testDataPath).delete();
 }
}

Java vs Scala Comparison#

Aspect	Java	Scala
Type declaration	`Dataset<Row>`	`DataFrame` (type alias)
Lambda expressions	`df.filter(row -> row.getInt(0) > 10)`	`df.filter(row => row.getInt(0) > 10)`
Column reference	`col("name")`	`$"name"`
String interpolation	`"value: " + value`	`s"value: $value"`
Case Class	`record` (Java 17+)	`case class`
Pattern matching	switch expression (Java 21+)	`match` expression
Spring integration	Native support	Annotation compatible
IDE support	IntelliJ, VS Code	IntelliJ + Scala plugin

Same Logic Comparison#

Java:

Transformations and Actions

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Transformations and Actions#

All Spark operations are classified into two categories: Transformations and Actions. Understanding this distinction is fundamental to Spark programming.

Core Concepts#

Transformation#

Transformations are operations that create new RDDs/DataFrames from existing ones.

Lazy Evaluation: Not executed immediately when called
Immutability: Creates new data without modifying original
DAG Construction: Adds operations to the execution plan (DAG)

// These lines don't execute - they just add to the DAG
Dataset<Row> df = spark.read().csv("data.csv");
Dataset<Row> filtered = df.filter(col("age").gt(30)); // Transformation
Dataset<Row> selected = filtered.select("name", "age"); // Transformation

// Nothing has executed yet!

Action#

Actions are operations that trigger actual computation and return results.

CronJob Lab

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

Target Audience: Developers who want to schedule periodic tasks on Kubernetes Prerequisites: Pod, Job concepts After reading this: You will be able to create periodic backup tasks with CronJob, and manage execution history and concurrency policies

TL;DR

Automate periodic data backups with CronJob

Manage execution history with successfulJobsHistoryLimit

Control concurrent execution with concurrencyPolicy

Configure failure alerts

CronJob Execution Flow#

flowchart TD
 CRON[CronJob Controller] -->|On schedule| JOB[Create Job]
 JOB --> POD[Run Pod]
 POD -->|Success| SUCCESS[Completion Record]
 POD -->|Failure| RETRY{Retry?}
 RETRY -->|Within backoffLimit| POD
 RETRY -->|Exceeded| FAIL[Failure Record]

 SUCCESS --> HIST[History Management<br>successfulJobsHistoryLimit]
 FAIL --> HIST2[History Management<br>failedJobsHistoryLimit]

Prerequisites#

You will need the following:

Domain Model Patterns

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 17 Feb 2026 00:00:00 +0000

Target Audience: Backend developers who understand Tactical Design building blocks Prerequisites: Understanding of Entity, Value Object, Repository, Domain Service concepts Estimated Time: About 25-30 minutes Key Question: “How do you structure domain logic that is difficult to express with core building blocks alone?”

Summary
Core building blocks (Entity, Value Object, Repository, Domain Service, Factory) are the skeleton of a domain model. The patterns covered in this document are tools for systematically expressing business rules on top of that skeleton: Specification (condition checking) + Policy (calculation/decision) + Module (code organization) + Builder (complex creation) + Null Object (null safety)

You can implement a domain model with just the core building blocks of tactical design. However, real businesses have more complex requirements. Condition checks like “can this order be confirmed?”, policy decisions like “what percentage discount applies to VIP customers?”, and code organization spanning multiple modules — let us explore the auxiliary patterns that cleanly solve these problems.

Event-Driven Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 15 Jan 2026 00:00:00 +0000

Target Audience: Developers who understand domain modeling and transaction concepts Prerequisites: Understanding of Onion Architecture or Aggregate boundary concepts Estimated Time: About 30 minutes Key Question: “When and why should you use domain events?”

Event-Driven Architecture#

Analogy: Company Announcement System
You can think of domain events as a company announcement system:

Event publishing: HR sends out a “New employee has joined” announcement. HR does not know who will see this announcement, nor does it need to.

Event subscription: The IT team sees the announcement and creates an account, the General Affairs team orders business cards, and the Training team schedules onboarding. Each department acts independently.

Loose coupling: The system works even if HR does not know the IT team exists. Even if a “Welcome Gift team” is added later, HR does not need any changes.

Key point: The publisher only announces “something happened,” and subscribers each do what they need to do.

Pattern Matching

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Pattern matching is a powerful feature for analyzing value structures and extracting data

Supports various patterns: literals, variables, types, tuples, case classes, sequences, and more

Guards (if) add conditions, @ binds the entire value

Compiler checks exhaustiveness for sealed types

Target Audience: Developers familiar with basic Scala syntax Prerequisites: Case classes, basic type system

Pattern matching is one of Scala’s most powerful features. It elegantly handles value structure analysis, data extraction, and conditional branching. Unlike Java’s switch statement, Scala’s match is an expression that returns a value and provides much richer functionality including type matching, destructuring, guard conditions, and more.

Alerting Rules

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs setting up monitoring alerts Prerequisites: Recording Rules What You’ll Learn: Write rules that reduce false positives and alert only on real issues

TL;DR#

Key Summary:

for: Fires when condition is met for specified duration (prevents false positives)

labels: Add metadata like severity, team

annotations: Include alert message, runbook URL

Use Recording Rules results for concise writing

Basic Syntax#

Alerting Rule Structure#

groups:
 - name: <group_name>
 rules:
 - alert: <alert_name>
 expr: <PromQL_condition>
 for: <duration>
 labels:
 <label_name>: <value>
 annotations:
 <annotation_name>: <value>

Basic Example#

groups:
 - name: availability
 rules:
 - alert: ServiceDown
 expr: up == 0
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "{{ $labels.instance }} is down"
 description: "{{ $labels.job }} has been down for more than 5 minutes."
 runbook_url: "https://wiki.example.com/runbook/service-down"

Core Components#

expr (Condition)#

The condition that triggers the alert.

Volume and Storage

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to persist data in Kubernetes Prerequisites: Pod concepts After reading this: You will understand Volume, PersistentVolume, and PersistentVolumeClaim concepts and usage

TL;DR

Pods are ephemeral by default, data disappears on termination

Volumes allow data sharing between containers within a Pod

PersistentVolume (PV) and PersistentVolumeClaim (PVC) maintain data independently of Pod lifecycle

Why Volumes are Needed?#

Container filesystems are ephemeral by default.

Problem	Volume Solution
Data loss on container restart	Preserve data with Volume
Cannot share files between containers in same Pod	Mount shared Volume
All data lost on Pod termination	Persist with PersistentVolume

Volume Types Overview#

Kubernetes provides various Volume types.

Transactions and Exactly-Once

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

At-Most-Once: may lose/no duplicates, At-Least-Once: no loss/may duplicate

Exactly-Once: requires Idempotent Producer + Transactional API + read_committed

Idempotent Producer is enabled by default in Kafka 3.0+, prevents duplicates for single Partition

Kafka transactions guarantee atomic writes across multiple Partitions

Use Outbox pattern when DB + Kafka atomic processing is needed

Target Audience: Developers building systems where message delivery guarantees are critical

Prerequisites: acks and Idempotent Producer concepts from Advanced Concepts

Delta Lake Integration

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 08 Jan 2026 00:00:00 +0000

Add ACID transactions, schema management, and time travel capabilities to your data lake using Delta Lake.

What is Delta Lake?#

Problems with Traditional Data Lakes#

Problem	Description
No ACID Support	Data corruption possible during concurrent writes
Schema Inconsistency	Different schemas across files
Small Files	Performance degradation
No Rollback	Difficult to recover from bad data

Delta Lake Solution#

flowchart LR
 subgraph Before["Parquet (Traditional)"]
 P1[part-001.parquet]
 P2[part-002.parquet]
 P3[part-003.parquet]
 end

 subgraph After["Delta Lake"]
 D1[part-001.parquet]
 D2[part-002.parquet]
 D3[_delta_log/]
 D3 --> D4[00000.json<br>Transaction Log]
 end

Feature	Description
ACID Transactions	Atomic writes, concurrency control
Schema Evolution	Safe column addition/modification
Time Travel	Query/restore past versions
Compaction	Auto merge small files
Z-Order	Query optimization clustering

Environment Setup#

build.sbt#

ThisBuild / scalaVersion := "2.13.12"

lazy val root = (project in file("."))
 .settings(
 name := "spark-delta-example",
 libraryDependencies ++= Seq(
 "org.apache.spark" %% "spark-core" % "3.5.1",
 "org.apache.spark" %% "spark-sql" % "3.5.1",
 "io.delta" %% "delta-spark" % "3.1.0"
 )
 )

SparkSession Configuration#

import org.apache.spark.sql.SparkSession
import io.delta.sql.DeltaSparkSessionExtension

val spark = SparkSession.builder()
 .appName("Delta Lake Example")
 .master("local[*]")
 .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
 .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
 .getOrCreate()

Basic CRUD Operations#

Create: Create Delta Table#

import io.delta.tables._
import org.apache.spark.sql.functions._

case class Order(
 orderId: String,
 customerId: String,
 product: String,
 quantity: Int,
 price: Double,
 orderDate: String
)

val orders = Seq(
 Order("O001", "C1", "Laptop", 1, 1200.0, "2024-01-15"),
 Order("O002", "C2", "Phone", 2, 800.0, "2024-01-15"),
 Order("O003", "C1", "Tablet", 1, 500.0, "2024-01-16")
).toDF()

// Save as Delta table
orders.write
 .format("delta")
 .mode("overwrite")
 .save("/data/orders")

Read: Query Data#

// DataFrame API
val df = spark.read.format("delta").load("/data/orders")
df.show()

// SQL
spark.sql("SELECT * FROM delta.`/data/orders`").show()

Update: Modify Data#

import io.delta.tables.DeltaTable

val deltaTable = DeltaTable.forPath(spark, "/data/orders")

// Conditional update
deltaTable.update(
 condition = expr("orderId = 'O002'"),
 set = Map(
 "quantity" -> lit(3),
 "price" -> lit(750.0)
 )
)

// SQL update
spark.sql("""
 UPDATE orders
 SET quantity = 3, price = 750.0
 WHERE orderId = 'O002'
""")

Delete: Remove Data#

// Conditional delete
deltaTable.delete(expr("customerId = 'C2'"))

// SQL delete
spark.sql("DELETE FROM orders WHERE customerId = 'C2'")

Merge (Upsert)#

val newOrders = Seq(
 Order("O002", "C2", "Phone", 5, 700.0, "2024-01-17"), // Update existing
 Order("O004", "C3", "Monitor", 2, 300.0, "2024-01-17") // Insert new
).toDF()

deltaTable.as("target")
 .merge(
 newOrders.as("source"),
 "target.orderId = source.orderId"
 )
 .whenMatched
 .updateAll()
 .whenNotMatched
 .insertAll()
 .execute()

Time Travel#

Query by Version#

// Query specific version
val version0 = spark.read
 .format("delta")
 .option("versionAsOf", 0)
 .load("/data/orders")

// Query as of timestamp
val asOf = spark.read
 .format("delta")
 .option("timestampAsOf", "2024-01-16 10:00:00")
 .load("/data/orders")

// SQL query
spark.sql("""
 SELECT * FROM orders VERSION AS OF 0
""").show()

spark.sql("""
 SELECT * FROM orders TIMESTAMP AS OF '2024-01-16 10:00:00'
""").show()

View Version History#

val history = deltaTable.history()
history.select("version", "timestamp", "operation", "operationParameters").show(false)

// Result:
// +-------+--------------------+---------+---------------------------+
// |version|timestamp |operation|operationParameters |
// +-------+--------------------+---------+---------------------------+
// |3 |2024-01-17 15:30:00|MERGE |{predicate -> ...} |
// |2 |2024-01-17 14:00:00|UPDATE |{predicate -> orderId = ...}|
// |1 |2024-01-16 10:00:00|WRITE |{mode -> Append} |
// |0 |2024-01-15 09:00:00|WRITE |{mode -> Overwrite} |
// +-------+--------------------+---------+---------------------------+

Restore to Previous Version#

// Restore to version
deltaTable.restoreToVersion(1)

// Restore to timestamp
deltaTable.restoreToTimestamp("2024-01-16 10:00:00")

// SQL restore
spark.sql("RESTORE orders TO VERSION AS OF 1")

Schema Evolution#

Add Columns#

// Data with new column
val ordersWithStatus = Seq(
 ("O005", "C4", "Keyboard", 1, 100.0, "2024-01-18", "CONFIRMED")
).toDF("orderId", "customerId", "product", "quantity", "price", "orderDate", "status")

// Auto merge schema
ordersWithStatus.write
 .format("delta")
 .mode("append")
 .option("mergeSchema", "true")
 .save("/data/orders")

Optimization#

Compaction (File Merging)#

// Merge small files
deltaTable.optimize().executeCompaction()

// Optimize specific partition
deltaTable.optimize()
 .where("orderDate = '2024-01-15'")
 .executeCompaction()

// SQL
spark.sql("OPTIMIZE orders")

Z-Order (Data Clustering)#

// Cluster by frequently filtered columns
deltaTable.optimize()
 .executeZOrderBy("customerId", "product")

// SQL
spark.sql("OPTIMIZE orders ZORDER BY (customerId, product)")

Vacuum (Clean Old Files)#

// Delete versions older than 7 days
deltaTable.vacuum(168) // 168 hours = 7 days

// SQL
spark.sql("VACUUM orders RETAIN 168 HOURS")

// Warning: Versions before retention period cannot time travel

Change Data Feed (CDC)#

Enable CDC#

// Enable when creating table
spark.sql("""
 CREATE TABLE orders_cdc (
 orderId STRING,
 customerId STRING,
 product STRING,
 quantity INT,
 price DOUBLE
 )
 USING DELTA
 TBLPROPERTIES (delta.enableChangeDataFeed = true)
""")

// Enable on existing table
spark.sql("""
 ALTER TABLE orders
 SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
""")

Query Changes#

// Query changes by version range
val changes = spark.read
 .format("delta")
 .option("readChangeFeed", "true")
 .option("startingVersion", 2)
 .option("endingVersion", 5)
 .table("orders")

changes.show()
// +-------+----------+-------+--------+-----+------------------+---------------+
// |orderId|customerId|product|quantity|price|_change_type |_commit_version|
// +-------+----------+-------+--------+-----+------------------+---------------+
// |O002 |C2 |Phone |5 |700.0|update_postimage |3 |
// |O002 |C2 |Phone |3 |750.0|update_preimage |3 |
// |O004 |C3 |Monitor|2 |300.0|insert |3 |
// +-------+----------+-------+--------+-----+------------------+---------------+

Practical Example: ETL Pipeline#

Bronze → Silver → Gold Architecture#

object DeltaLakePipeline extends App {
 val spark = SparkSession.builder()
 .appName("Delta Lake Pipeline")
 .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
 .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
 .getOrCreate()

 import spark.implicits._

 // Bronze: Store raw data as-is
 def ingestToBronze(): Unit = {
 val rawData = spark.read
 .option("header", "true")
 .csv("/data/raw/orders/*.csv")

 rawData.write
 .format("delta")
 .mode("append")
 .option("mergeSchema", "true")
 .save("/data/bronze/orders")

 println("Bronze layer ingestion complete")
 }

 // Silver: Clean and validate
 def transformToSilver(): Unit = {
 val bronze = spark.read.format("delta").load("/data/bronze/orders")

 val silver = bronze
 // Data cleaning
 .filter($"orderId".isNotNull && $"price" > 0)
 // Type conversion
 .withColumn("price", $"price".cast("double"))
 .withColumn("quantity", $"quantity".cast("int"))
 .withColumn("orderDate", to_date($"orderDate"))
 // Deduplicate
 .dropDuplicates("orderId")
 // Derived columns
 .withColumn("totalAmount", $"price" * $"quantity")
 .withColumn("processedAt", current_timestamp())

 // Incremental processing with Merge
 val silverTable = DeltaTable.forPath(spark, "/data/silver/orders")

 silverTable.as("target")
 .merge(silver.as("source"), "target.orderId = source.orderId")
 .whenMatched.updateAll()
 .whenNotMatched.insertAll()
 .execute()

 println("Silver layer transformation complete")
 }

 // Gold: Business aggregations
 def aggregateToGold(): Unit = {
 val silver = spark.read.format("delta").load("/data/silver/orders")

 // Daily sales aggregation
 val dailySales = silver
 .groupBy($"orderDate")
 .agg(
 count("*").as("orderCount"),
 sum("totalAmount").as("totalRevenue"),
 avg("totalAmount").as("avgOrderValue"),
 countDistinct("customerId").as("uniqueCustomers")
 )
 .withColumn("updatedAt", current_timestamp())

 dailySales.write
 .format("delta")
 .mode("overwrite")
 .save("/data/gold/daily_sales")

 // Customer metrics
 val customerMetrics = silver
 .groupBy($"customerId")
 .agg(
 count("*").as("totalOrders"),
 sum("totalAmount").as("lifetimeValue"),
 min("orderDate").as("firstOrderDate"),
 max("orderDate").as("lastOrderDate")
 )
 .withColumn("updatedAt", current_timestamp())

 customerMetrics.write
 .format("delta")
 .mode("overwrite")
 .save("/data/gold/customer_metrics")

 println("Gold layer aggregation complete")
 }

 // Run pipeline
 def runPipeline(): Unit = {
 ingestToBronze()
 transformToSilver()
 aggregateToGold()

 // Optimize
 DeltaTable.forPath(spark, "/data/silver/orders").optimize().executeCompaction()
 DeltaTable.forPath(spark, "/data/silver/orders").vacuum(168)

 println("Pipeline complete")
 }

 runPipeline()
 spark.stop()
}

Considerations#

Item	Note
Vacuum	Versions before retention period cannot time travel
Concurrent Writes	Conflicts possible when writing to same partition
Schema Changes	Type changes require overwriteSchema
File Size	Recommend target file size setting (128MB~1GB)

Next Steps#

Structured Streaming - Real-time processing
Performance Tuning - Spark optimization
Kafka Integration - Stream source

Indexing Strategy

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 08 Jan 2026 00:00:00 +0000

Prerequisites
Before reading this document, understand these concepts first:

Core Components - Shard, Segment concepts

Data Modeling - Mapping, Analyzer basics

Learn Bulk indexing, Refresh, and Index Lifecycle Management for efficiently storing large volumes of data.

Elasticsearch indexing goes beyond simply storing data. When a document is indexed, it goes through a complex process including text analysis, inverted index creation, and segment management. Understanding this process reveals why Bulk indexing is 10x faster than single-document indexing, why documents aren’t immediately searchable after indexing, and why Refresh should be disabled during bulk indexing. The right indexing strategy directly impacts system performance and operational costs.

ETL Pipeline

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Production ETL Pipeline Examples#

Complete ETL (Extract-Transform-Load) pipeline examples ready for production environments.

Pipeline Architecture#

flowchart LR
 subgraph Extract["Extract"]
 S3[(S3/HDFS)]
 DB[(Database)]
 API[REST API]
 end

 subgraph Transform["Transform"]
 Clean[Data Cleaning]
 Enrich[Data Enrichment]
 Agg[Aggregation]
 end

 subgraph Load["Load"]
 DW[(Data Warehouse)]
 Lake[(Data Lake)]
 Cache[(Redis Cache)]
 end

 S3 --> Clean
 DB --> Clean
 API --> Clean
 Clean --> Enrich
 Enrich --> Agg
 Agg --> DW
 Agg --> Lake
 Agg --> Cache

Project Structure#

etl-pipeline/
├── src/main/java/com/example/etl/
│ ├── EtlApplication.java
│ ├── config/
│ │ └── SparkConfig.java
│ ├── job/
│ │ ├── AbstractEtlJob.java
│ │ ├── SalesEtlJob.java
│ │ └── CustomerEtlJob.java
│ ├── transformer/
│ │ ├── DataCleaner.java
│ │ └── DataEnricher.java
│ └── util/
│ ├── SchemaUtils.java
│ └── DateUtils.java
├── src/main/resources/
│ ├── application.yml
│ └── schemas/
│ └── sales-schema.json
└── build.gradle.kts

Basic ETL Structure#

Abstract ETL Job Class#

package com.example.etl.job;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.LocalDate;
import java.time.format.DateTimeFormatter;

/**
 * Abstract class defining the basic structure of ETL jobs
 */
public abstract class AbstractEtlJob {
 protected final Logger logger = LoggerFactory.getLogger(getClass());
 protected final SparkSession spark;
 protected final LocalDate processDate;

 protected AbstractEtlJob(SparkSession spark, LocalDate processDate) {
 this.spark = spark;
 this.processDate = processDate;
 }

 /**
 * Execute ETL job (Template Method pattern)
 */
 public final EtlResult execute() {
 String jobName = getJobName();
 long startTime = System.currentTimeMillis();
 logger.info("ETL job started: {} (processing date: {})", jobName, processDate);

 try {
 // 1. Extract
 logger.info("[Extract] Starting data extraction");
 Dataset<Row> rawData = extract();
 long extractCount = rawData.count();
 logger.info("[Extract] Complete: {} records", extractCount);

 // 2. Transform
 logger.info("[Transform] Starting data transformation");
 Dataset<Row> transformedData = transform(rawData);
 long transformCount = transformedData.count();
 logger.info("[Transform] Complete: {} records", transformCount);

 // 3. Validate
 logger.info("[Validate] Starting data validation");
 ValidationResult validation = validate(transformedData);
 if (!validation.isValid()) {
 throw new EtlValidationException(validation.getErrors());
 }
 logger.info("[Validate] Complete: validation passed");

 // 4. Load
 logger.info("[Load] Starting data loading");
 load(transformedData);
 logger.info("[Load] Complete");

 long duration = System.currentTimeMillis() - startTime;
 logger.info("ETL job completed: {} (duration: {}ms)", jobName, duration);

 return EtlResult.success(extractCount, transformCount, duration);

 } catch (Exception e) {
 long duration = System.currentTimeMillis() - startTime;
 logger.error("ETL job failed: {} - {}", jobName, e.getMessage(), e);
 return EtlResult.failure(e.getMessage(), duration);
 }
 }

 // Implemented by subclasses
 protected abstract String getJobName();
 protected abstract Dataset<Row> extract();
 protected abstract Dataset<Row> transform(Dataset<Row> data);
 protected abstract ValidationResult validate(Dataset<Row> data);
 protected abstract void load(Dataset<Row> data);

 // Common utility methods
 protected String getPartitionPath() {
 return processDate.format(DateTimeFormatter.ofPattern("year=yyyy/month=MM/day=dd"));
 }
}

ETL Result Class#

package com.example.etl.job;

public record EtlResult(
 boolean success,
 long inputRecords,
 long outputRecords,
 long durationMs,
 String errorMessage
) {
 public static EtlResult success(long input, long output, long duration) {
 return new EtlResult(true, input, output, duration, null);
 }

 public static EtlResult failure(String error, long duration) {
 return new EtlResult(false, 0, 0, duration, error);
 }
}

Sales Data ETL Example#

Full Implementation#

package com.example.etl.job;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.*;

import java.time.LocalDate;
import java.util.ArrayList;
import java.util.List;

import static org.apache.spark.sql.functions.*;

/**
 * Sales Data ETL Pipeline
 * - Source: S3 JSON files
 * - Transform: Cleaning, derived columns, aggregation
 * - Load: Parquet partitioned table
 */
public class SalesEtlJob extends AbstractEtlJob {

 private final String inputPath;
 private final String outputPath;
 private final StructType inputSchema;

 public SalesEtlJob(SparkSession spark, LocalDate processDate,
 String inputPath, String outputPath) {
 super(spark, processDate);
 this.inputPath = inputPath;
 this.outputPath = outputPath;
 this.inputSchema = createInputSchema();
 }

 @Override
 protected String getJobName() {
 return "SalesETL";
 }

 private StructType createInputSchema() {
 return new StructType()
 .add("order_id", DataTypes.StringType, false)
 .add("customer_id", DataTypes.StringType, false)
 .add("product_id", DataTypes.StringType, false)
 .add("quantity", DataTypes.IntegerType, false)
 .add("unit_price", DataTypes.DoubleType, false)
 .add("discount", DataTypes.DoubleType, true)
 .add("order_timestamp", DataTypes.TimestampType, false)
 .add("shipping_address", DataTypes.StringType, true)
 .add("payment_method", DataTypes.StringType, true);
 }

 @Override
 protected Dataset<Row> extract() {
 String datePath = processDate.format(
 java.time.format.DateTimeFormatter.ofPattern("yyyy/MM/dd")
 );

 return spark.read()
 .schema(inputSchema)
 .option("mode", "PERMISSIVE") // Allow corrupt records
 .option("columnNameOfCorruptRecord", "_corrupt_record")
 .json(inputPath + "/" + datePath + "/*.json");
 }

 @Override
 protected Dataset<Row> transform(Dataset<Row> data) {
 return data
 // 1. Remove corrupt records
 .filter(col("_corrupt_record").isNull())
 .drop("_corrupt_record")

 // 2. Handle NULL values
 .withColumn("discount",
 when(col("discount").isNull(), lit(0.0))
 .otherwise(col("discount")))
 .withColumn("shipping_address",
 when(col("shipping_address").isNull(), lit("Unknown"))
 .otherwise(col("shipping_address")))

 // 3. Create derived columns
 .withColumn("gross_amount",
 col("quantity").multiply(col("unit_price")))
 .withColumn("discount_amount",
 col("gross_amount").multiply(col("discount")))
 .withColumn("net_amount",
 col("gross_amount").minus(col("discount_amount")))

 // 4. Time derived columns
 .withColumn("order_date",
 to_date(col("order_timestamp")))
 .withColumn("order_hour",
 hour(col("order_timestamp")))
 .withColumn("order_dayofweek",
 dayofweek(col("order_timestamp")))

 // 5. Extract region (from address)
 .withColumn("region",
 when(col("shipping_address").contains("New York"), "Northeast")
 .when(col("shipping_address").contains("Los Angeles"), "West")
 .when(col("shipping_address").contains("Chicago"), "Midwest")
 .when(col("shipping_address").contains("Houston"), "South")
 .otherwise("Other"))

 // 6. Add processing metadata
 .withColumn("etl_timestamp", current_timestamp())
 .withColumn("etl_batch_id",
 lit(processDate.format(
 java.time.format.DateTimeFormatter.BASIC_ISO_DATE
 )));
 }

 @Override
 protected ValidationResult validate(Dataset<Row> data) {
 List<String> errors = new ArrayList<>();

 // 1. Check required columns exist
 String[] requiredColumns = {"order_id", "customer_id", "net_amount"};
 for (String col : requiredColumns) {
 if (!java.util.Arrays.asList(data.columns()).contains(col)) {
 errors.add("Missing required column: " + col);
 }
 }

 // 2. Data quality checks
 long totalCount = data.count();
 if (totalCount == 0) {
 errors.add("No data to process");
 }

 // Check for negative amounts
 long negativeAmounts = data.filter(col("net_amount").lt(0)).count();
 if (negativeAmounts > 0) {
 double ratio = (double) negativeAmounts / totalCount * 100;
 if (ratio > 1.0) { // Error if exceeds 1%
 errors.add(String.format("Negative amount ratio exceeded: %.2f%%", ratio));
 } else {
 logger.warn("Negative amount records found: {} ({}%)", negativeAmounts, ratio);
 }
 }

 // Check for duplicate orders
 long uniqueOrders = data.select("order_id").distinct().count();
 if (uniqueOrders < totalCount) {
 long duplicates = totalCount - uniqueOrders;
 errors.add("Duplicate orders found: " + duplicates);
 }

 // 3. Schema validation
 StructType actualSchema = data.schema();
 if (!actualSchema.fieldNames().length >= 10) {
 errors.add("Schema mismatch from expected");
 }

 return new ValidationResult(errors.isEmpty(), errors);
 }

 @Override
 protected void load(Dataset<Row> data) {
 // Save partitioned by date
 data.write()
 .mode(SaveMode.Overwrite)
 .partitionBy("order_date")
 .option("compression", "snappy")
 .parquet(outputPath + "/sales_fact");

 // Also create daily summary table
 Dataset<Row> dailySummary = data
 .groupBy("order_date", "region")
 .agg(
 count("*").alias("order_count"),
 sum("net_amount").alias("total_revenue"),
 avg("net_amount").alias("avg_order_value"),
 countDistinct("customer_id").alias("unique_customers")
 );

 dailySummary.write()
 .mode(SaveMode.Overwrite)
 .parquet(outputPath + "/daily_summary/" + getPartitionPath());
 }
}

Validation Result Class#

package com.example.etl.job;

import java.util.List;

public record ValidationResult(
 boolean valid,
 List<String> errors
) {
 public boolean isValid() {
 return valid;
 }

 public List<String> getErrors() {
 return errors;
 }
}

Data Cleaning Utilities#

Reusable Data Cleaner#

package com.example.etl.transformer;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.types.DataTypes;

import java.util.Arrays;
import java.util.List;

import static org.apache.spark.sql.functions.*;

/**
 * Reusable data cleaning transformers
 */
public class DataCleaner {

 /**
 * Remove duplicates (based on specified keys)
 */
 public Dataset<Row> removeDuplicates(Dataset<Row> df, String... keyColumns) {
 return df.dropDuplicates(keyColumns);
 }

 /**
 * Handle NULL values (replace with defaults)
 */
 public Dataset<Row> fillNulls(Dataset<Row> df,
 java.util.Map<String, Object> defaults) {
 Dataset<Row> result = df;
 for (var entry : defaults.entrySet()) {
 String colName = entry.getKey();
 Object defaultValue = entry.getValue();

 if (Arrays.asList(df.columns()).contains(colName)) {
 result = result.withColumn(colName,
 when(col(colName).isNull(), lit(defaultValue))
 .otherwise(col(colName)));
 }
 }
 return result;
 }

 /**
 * Normalize strings (trim whitespace, lowercase)
 */
 public Dataset<Row> normalizeStrings(Dataset<Row> df, String... columns) {
 Dataset<Row> result = df;
 for (String colName : columns) {
 result = result.withColumn(colName,
 lower(trim(col(colName))));
 }
 return result;
 }

 /**
 * Filter outliers (IQR method)
 */
 public Dataset<Row> removeOutliers(Dataset<Row> df, String column,
 double lowerPercentile,
 double upperPercentile) {
 // Calculate percentiles
 double[] percentiles = df.stat().approxQuantile(
 column,
 new double[]{lowerPercentile, upperPercentile},
 0.01
 );

 double lower = percentiles[0];
 double upper = percentiles[1];

 return df.filter(
 col(column).geq(lower).and(col(column).leq(upper))
 );
 }

 /**
 * Standardize date format
 */
 public Dataset<Row> standardizeDates(Dataset<Row> df,
 String column,
 String inputFormat) {
 return df.withColumn(column,
 to_date(col(column), inputFormat));
 }

 /**
 * Validate email format
 */
 public Dataset<Row> validateEmails(Dataset<Row> df, String column) {
 String emailPattern = "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$";
 return df.withColumn(column + "_valid",
 col(column).rlike(emailPattern));
 }

 /**
 * Normalize phone numbers (US format)
 */
 public Dataset<Row> normalizePhoneNumbers(Dataset<Row> df, String column) {
 return df.withColumn(column,
 regexp_replace(col(column), "[^0-9]", ""))
 .withColumn(column,
 when(col(column).startsWith("1"),
 substring(col(column), 2, 100))
 .otherwise(col(column)));
 }
}

Incremental ETL#

CDC-Based Incremental Processing#

package com.example.etl.job;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;

import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;

import static org.apache.spark.sql.functions.*;

/**
 * Incremental ETL Job (Delta approach)
 */
public class IncrementalEtlJob {
 private final SparkSession spark;
 private final String watermarkPath;

 public IncrementalEtlJob(SparkSession spark, String watermarkPath) {
 this.spark = spark;
 this.watermarkPath = watermarkPath;
 }

 /**
 * Get last processing timestamp
 */
 public LocalDateTime getLastWatermark() {
 try {
 Dataset<Row> watermark = spark.read().json(watermarkPath);
 String lastTs = watermark.first().getString(0);
 return LocalDateTime.parse(lastTs, DateTimeFormatter.ISO_DATE_TIME);
 } catch (Exception e) {
 // Return default if watermark file doesn't exist
 return LocalDateTime.of(1970, 1, 1, 0, 0);
 }
 }

 /**
 * Update watermark
 */
 public void updateWatermark(LocalDateTime newWatermark) {
 spark.createDataFrame(
 java.util.List.of(new Watermark(
 newWatermark.format(DateTimeFormatter.ISO_DATE_TIME)
 )),
 Watermark.class
 ).write().mode(SaveMode.Overwrite).json(watermarkPath);
 }

 /**
 * Extract incremental data
 */
 public Dataset<Row> extractIncremental(String sourcePath,
 LocalDateTime fromTs,
 LocalDateTime toTs) {
 return spark.read()
 .parquet(sourcePath)
 .filter(col("updated_at").gt(lit(fromTs.toString()))
 .and(col("updated_at").leq(lit(toTs.toString()))));
 }

 /**
 * Upsert (Insert or Update)
 */
 public void upsertToTarget(Dataset<Row> updates,
 String targetPath,
 String... keyColumns) {
 // Read existing data
 Dataset<Row> existing;
 try {
 existing = spark.read().parquet(targetPath);
 } catch (Exception e) {
 // Create new if target table doesn't exist
 updates.write()
 .mode(SaveMode.Overwrite)
 .parquet(targetPath);
 return;
 }

 // Remove duplicate keys (new data takes priority)
 Dataset<Row> merged = existing
 .join(updates.select(keyColumns), scala.collection.JavaConverters
 .asScalaBuffer(java.util.Arrays.asList(keyColumns)).toSeq(), "left_anti")
 .union(updates);

 // Save result
 merged.write()
 .mode(SaveMode.Overwrite)
 .parquet(targetPath);
 }

 // DTO
 public static class Watermark implements java.io.Serializable {
 private String timestamp;

 public Watermark() {}

 public Watermark(String timestamp) {
 this.timestamp = timestamp;
 }

 public String getTimestamp() { return timestamp; }
 public void setTimestamp(String timestamp) { this.timestamp = timestamp; }
 }
}

Error Handling and Retry#

Robust ETL Runner#

package com.example.etl;

import com.example.etl.job.AbstractEtlJob;
import com.example.etl.job.EtlResult;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.LocalDate;
import java.util.concurrent.TimeUnit;

/**
 * ETL runner with retry logic
 */
public class EtlRunner {
 private static final Logger logger = LoggerFactory.getLogger(EtlRunner.class);

 private final int maxRetries;
 private final long retryDelayMs;

 public EtlRunner(int maxRetries, long retryDelayMs) {
 this.maxRetries = maxRetries;
 this.retryDelayMs = retryDelayMs;
 }

 public EtlResult runWithRetry(AbstractEtlJob job) {
 int attempt = 0;
 EtlResult lastResult = null;

 while (attempt < maxRetries) {
 attempt++;
 logger.info("ETL execution attempt {}/{}", attempt, maxRetries);

 lastResult = job.execute();

 if (lastResult.success()) {
 logger.info("ETL success: input={}, output={}, time={}ms",
 lastResult.inputRecords(),
 lastResult.outputRecords(),
 lastResult.durationMs());
 return lastResult;
 }

 if (attempt < maxRetries) {
 logger.warn("ETL failed, retrying in {}ms: {}",
 retryDelayMs, lastResult.errorMessage());
 try {
 TimeUnit.MILLISECONDS.sleep(retryDelayMs);
 } catch (InterruptedException e) {
 Thread.currentThread().interrupt();
 break;
 }
 }
 }

 logger.error("ETL final failure ({} attempts): {}",
 attempt, lastResult != null ? lastResult.errorMessage() : "Unknown");
 return lastResult;
 }

 /**
 * Run ETL for date range
 */
 public void runForDateRange(java.util.function.Function<LocalDate, AbstractEtlJob> jobFactory,
 LocalDate startDate,
 LocalDate endDate) {
 LocalDate current = startDate;

 while (!current.isAfter(endDate)) {
 logger.info("=== Processing date: {} ===", current);

 AbstractEtlJob job = jobFactory.apply(current);
 EtlResult result = runWithRetry(job);

 if (!result.success()) {
 logger.error("Failed processing date {}, aborting", current);
 throw new RuntimeException("ETL failed: " + current);
 }

 current = current.plusDays(1);
 }

 logger.info("Complete date range processing: {} ~ {}", startDate, endDate);
 }
}

Scheduling (Spring)#

package com.example.etl;

import com.example.etl.job.SalesEtlJob;
import org.apache.spark.sql.SparkSession;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import java.time.LocalDate;

@Component
public class EtlScheduler {

 private final SparkSession spark;
 private final EtlRunner runner;
 private final String inputPath;
 private final String outputPath;

 public EtlScheduler(SparkSession spark) {
 this.spark = spark;
 this.runner = new EtlRunner(3, 60000); // 3 retries, 1 minute wait
 this.inputPath = System.getenv("ETL_INPUT_PATH");
 this.outputPath = System.getenv("ETL_OUTPUT_PATH");
 }

 @Scheduled(cron = "0 30 1 * * *") // Daily at 1:30 AM
 public void runDailyEtl() {
 LocalDate yesterday = LocalDate.now().minusDays(1);
 SalesEtlJob job = new SalesEtlJob(spark, yesterday, inputPath, outputPath);
 runner.runWithRetry(job);
 }
}

Basic Examples - DataFrame basic operations
Monitoring - Pipeline monitoring
Performance Tuning - Large-scale processing optimization

Partitioning and Shuffle

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Partitioning and Shuffle#

Partitioning is key to Spark performance. Understanding how data is distributed and optimizing it is crucial for large-scale data processing.

What is a Partition?#

A Partition is a logical division unit of RDD/DataFrame data. Each partition is processed on one node in the cluster.

DataFrame (100 million rows)
├── Partition 0 (10 million rows) → Executor 1, Task 0
├── Partition 1 (10 million rows) → Executor 2, Task 1
├── Partition 2 (10 million rows) → Executor 1, Task 2
├── ...
└── Partition 9 (10 million rows) → Executor 4, Task 9

Why Partitions Matter#

Partition Count	Impact
Too few	Low parallelism, possible memory issues, underutilized nodes
Too many	Scheduling overhead, small task inefficiency, increased shuffle cost
Optimal	Balanced load distribution, efficient resource usage

Checking and Adjusting Partition Count#

Checking Current Partition Count#

Dataset<Row> df = spark.read().parquet("data.parquet");
int partitions = df.rdd().getNumPartitions();
System.out.println("Partition count: " + partitions);

// Count data per partition
df.mapPartitions(
 iter -> {
 int count = 0;
 while (iter.hasNext()) { iter.next(); count++; }
 return Collections.singletonList(count).iterator();
 },
 Encoders.INT()
).collectAsList().forEach(System.out::println);

Guidelines for Determining Partition Count#

Recommended partition size: 100MB ~ 200MB
Minimum partition count: Total cluster cores × 2-4
Maximum partition count: Data size (GB) × 2-4

Example:

Collections

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Scala uses immutable collections by default (thread-safe, predictable)

List is fast for prepending, Vector is fast for random access

Built-in support for higher-order functions like map, filter, fold

Option handles missing values type-safely instead of null

Target Audience: Developers familiar with Java collections Prerequisites: Basic Scala syntax, functions and methods

Scala’s collection library provides rich data structures optimized for functional programming. Immutable collections are the default, and a consistent API allows working with various data structures. Unlike Java collections, Scala collections have built-in support for higher-order functions like map, filter, and fold.

Testing Strategy

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

Target Audience: Developers establishing testing strategies for DDD-based projects Prerequisites: Tactical Design building blocks, basic JUnit/Mockito usage Time Required: About 25 minutes Key Question: “How should tests be organized in DDD architecture?”

Summary
Testing Strategy Core: Unit Tests(domain model focused, most numerous) → Integration Tests(Repository, external integrations) → E2E Tests(critical scenarios only)

Systems applying Domain-Driven Design require a different testing strategy than typical CRUD applications. Since business logic is concentrated in domain models, tests should also be designed around the domain layer. This document explores testing strategies suitable for DDD architecture and specific implementation methods.

Networking

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to understand Kubernetes network structure Prerequisites: Service concepts, basic networking knowledge (IP, ports, DNS) After reading this: You will understand Kubernetes network model and Ingress

TL;DR

All Pods have unique IPs and can communicate directly with other Pods

Service provides stable access point for a set of Pods

Ingress routes HTTP/HTTPS traffic from outside the cluster

Kubernetes Network Model#

Kubernetes networking follows three basic principles.

Producer Tuning

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Adjust batch efficiency with batch.size and linger.ms (operates as OR condition)

Just linger.ms=5 can increase throughput by ~2.7x

compression.type: recommend snappy (general), lz4 (high performance), zstd (high compression)

Idempotent Producer (Kafka 3.0+ default) prevents duplicates and guarantees order

BufferExhaustedException occurs when buffer.memory is insufficient

Target Audience: Developers optimizing Producer performance, operators needing high-volume message processing

Prerequisites: acks, Idempotent Producer concepts from Advanced Concepts, Topic, Partition concepts from Message Flow

Understanding the core settings to optimize Producer performance. This document is written for Kafka 3.6.x and code examples are validated in Spring Boot 3.2.x, Spring Kafka 3.1.x, and Java 17 environments.

Cluster Management

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 08 Jan 2026 00:00:00 +0000

Learn how to configure nodes, allocate shards, and monitor status in an Elasticsearch cluster.

Elasticsearch was designed from the ground up as a distributed system. Even when starting with a single node, it operates internally as a cluster. Thanks to this distributed architecture, you can scale horizontally by simply adding nodes as data grows, and the service can continue even when some nodes experience failures.

However, distributed systems introduce different complexities than single servers. You need to understand role distribution among nodes, physical shard placement, master election processes, and behavior during network partitions to operate stably. In particular, incorrect Master node configuration can bring down the entire cluster, and unbalanced shard placement can concentrate load on specific nodes. This document covers core concepts and practical know-how for stable cluster operation.

Caching and Persistence

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Caching and Persistence#

Learn how to leverage Spark’s in-memory computing capabilities to cache intermediate results for reuse.

What is Caching?#

Caching is storing RDD/DataFrame in memory or disk for reuse in subsequent operations.

Why Caching is Needed#

// Without caching: Inefficient
Dataset<Row> processed = df
 .filter(col("status").equalTo("ACTIVE"))
 .groupBy("category")
 .agg(sum("amount").alias("total"));

// Same data processed 3 times (full recomputation each time)
long count = processed.count(); // Job 1: Full computation
processed.show(); // Job 2: Full recomputation
processed.write().parquet("output"); // Job 3: Full recomputation

// With caching: Efficient
processed.cache(); // Register cache
long count = processed.count(); // Job 1: Compute + cache
processed.show(); // Read from cache (fast)
processed.write().parquet("output"); // Read from cache (fast)
processed.unpersist(); // Release cache

Basic Usage#

cache()#

Dataset<Row> df = spark.read().parquet("large-data.parquet");

// cache() call - Cache with default storage level
df.cache();

// Actual caching happens on first Action
df.count(); // Data processing + cache to memory

// Subsequent Actions read from cache
df.filter(col("x").gt(10)).count();
df.groupBy("category").count().show();

// Release cache
df.unpersist();

persist()#

Allows specifying storage level directly.

Higher-Order Functions

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Higher-order functions are functions that take functions as arguments or return functions

They separate “what” to do from “how” to do it, improving code reusability

Process collections declaratively with map, filter, fold, etc.

Implement advanced abstractions with currying, closures, and partial functions

Target Audience: Developers interested in functional programming Prerequisites: Scala functions and methods, collection basics

Higher-order functions are functions that take functions as arguments or return functions. As a core concept of functional programming, they increase code reusability and raise the level of abstraction.

Anti-Patterns and Pitfalls

d8lzz1gpw@mozmail.com (kimbenji) — Tue, 13 Jan 2026 00:00:00 +0000

Target Audience: Developers who are applying or considering adopting DDD Prerequisites: Basic concepts from Quick Start and Tactical Design Reading Time: About 20 minutes Key Question: “What are the common mistakes when applying DDD, and how can they be avoided?”

Warning
The anti-patterns introduced in this document frequently occur in real projects. Check if your codebase exhibits these symptoms.

DDD (Domain-Driven Design) is a powerful design methodology, but if not applied correctly, it can only increase complexity. This document examines common DDD anti-patterns encountered in practice and their solutions. Understanding the symptoms of each anti-pattern and catching them early to correct is crucial.

Resource Management

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to efficiently manage resources in Kubernetes Prerequisites: Pod, Deployment concepts After reading this: You will understand the difference between requests and limits, and how to configure appropriate resources

TL;DR

requests: Minimum guaranteed resources used for scheduling

limits: Maximum usable resources

CPU excess causes throttling, memory excess causes OOMKilled

Why Resource Configuration is Needed?#

Running Pods without resource configuration causes several problems.

Problem	After Resource Configuration
One Pod monopolizes node resources	Limit max usage with limits
Resources not considered during scheduling	Schedule based on requests
Random Pod termination on memory shortage	Priority based on QoS class
Unpredictable resource usage	Explicit resource allocation

requests and limits#

Kubernetes provides two resource control methods.

Consumer Tuning

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Use fetch.min.bytes/fetch.max.wait.ms to balance batch efficiency and latency

max.poll.records and max.poll.interval.ms are key to preventing rebalancing

session.timeout.ms should be at least 3x heartbeat.interval.ms

Throughput optimization: increase fetch.min.size, increase max.poll.records

Latency optimization: decrease fetch.min.size, decrease fetch.max.wait

Target Audience: Developers optimizing Consumer performance, operators seeking operational stability

Prerequisites: Consumer operation principles from Consumer Group & Offset

Understand Consumer performance optimization and stable operation settings.

Consumer Internal Structure#

Consumer operates by fetching messages from Broker, delivering to the application, and committing Offset after processing. When Fetcher retrieves data from Broker, it’s delivered to the application through poll(), and Offset is committed after processing.

Performance Tuning

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 08 Jan 2026 00:00:00 +0000

Learn how to optimize Elasticsearch search and indexing performance.

While Elasticsearch provides fast search with default settings, production environments require tuning based on data scale and usage patterns. Default values are “generally acceptable,” not “optimal for all situations.”

Performance tuning can be divided into four areas: Index Design, Query Optimization, Caching, and System Settings. Each area affects the others. For example, poor shard design cannot be solved by query optimization, and caching strategy depends on query patterns. Therefore, it’s effective to first identify where bottlenecks occur, then focus on improving that specific area. This document covers key tuning points and practical considerations for each area.

Structured Streaming

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Structured Streaming#

Structured Streaming is Spark’s stream processing engine. It processes real-time data using the same DataFrame/Dataset API as batch processing.

What is Structured Streaming?#

Structured Streaming treats stream data as an infinitely appending table. When new data arrives, it performs incremental processing.

Core Concepts#

Input Stream (Infinite Table)
 ↓
[Data Arrival] → [Incremental Processing] → [Result Update]
 ↓
Output (Continuously Updated Result Table)

Batch vs Streaming Code Comparison#

// Batch processing (static data)
Dataset<Row> batchDf = spark.read()
 .json("input/data.json");

Dataset<Row> result = batchDf
 .groupBy("category")
 .count();

result.write().parquet("output");

// Stream processing (dynamic data) - almost identical!
Dataset<Row> streamDf = spark.readStream() // Changed to readStream
 .json("input/");

Dataset<Row> result = streamDf // Same processing logic
 .groupBy("category")
 .count();

result.writeStream() // Changed to writeStream
 .outputMode("complete")
 .format("console")
 .start()
 .awaitTermination();

Basic Usage#

Reading Streams#

SparkSession spark = SparkSession.builder()
 .appName("Structured Streaming")
 .master("local[*]")
 .getOrCreate();

// File source (directory monitoring)
Dataset<Row> fileStream = spark.readStream()
 .schema(schema) // Schema required for streams
 .json("input/");

// Kafka source
Dataset<Row> kafkaStream = spark.readStream()
 .format("kafka")
 .option("kafka.bootstrap.servers", "localhost:9092")
 .option("subscribe", "topic-name")
 .load();

// Socket source (for testing)
Dataset<Row> socketStream = spark.readStream()
 .format("socket")
 .option("host", "localhost")
 .option("port", 9999)
 .load();

// Rate source (for testing - generates specified rows per second)
Dataset<Row> rateStream = spark.readStream()
 .format("rate")
 .option("rowsPerSecond", 10)
 .load();

Processing Streams#

Apply the same operations as regular DataFrames to stream DataFrames:

Generics

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Generics allow you to write type-safe and reusable code

Upper bound (A <: B): A is a subtype of B

Lower bound (A >: B): A is a supertype of B

Context bound (A : Ordering): Requires a type class instance

Target Audience: Developers familiar with Java generics Prerequisites: Classes, traits, basic type system

Generics allow you to write type-safe and reusable code. Through type parameters, you can define classes and methods that work with various types, while ensuring type safety at compile time.

Scaling

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to configure auto-scaling in Kubernetes Prerequisites: Deployment, resource management concepts After reading this: You will understand auto-scaling methods using HPA and VPA

TL;DR

HPA (Horizontal Pod Autoscaler) automatically adjusts Pod count

VPA (Vertical Pod Autoscaler) automatically adjusts Pod resource requests

In most cases, consider HPA first

Scaling Methods Comparison#

Kubernetes provides two scaling methods.

Method	Description	Suitable For
Horizontal scaling (HPA)	Adjust Pod count	Stateless applications
Vertical scaling (VPA)	Adjust Pod resources	Stateful, resource optimization

flowchart LR
 subgraph Horizontal[Horizontal Scaling]
 H1[Pod] --> H2[Pod]
 H2 --> H3[Pod]
 end
 subgraph Vertical[Vertical Scaling]
 V1[Pod<br>256Mi] --> V2[Pod<br>512Mi]
 end

HPA (Horizontal Pod Autoscaler)#

HPA automatically adjusts Pod count based on metrics (CPU, memory, custom).

Advanced Error Handling

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Error types: Deserialization errors (skip), Transient errors (retry), Permanent errors (DLT)

Use @RetryableTopic for declarative retry and DLT handling (Spring Kafka 2.7+)

DefaultErrorHandler + DeadLetterPublishingRecoverer for DLT publishing

Handle deserialization errors with ErrorHandlingDeserializer

Set up alerts when DLT messages arrive for quick response

Target Audience: Developers implementing Consumer error handling strategies

Prerequisites: Consumer operation principles from Consumer Group & Offset, Spring Kafka basics

Understand Kafka Consumer error handling strategies and Dead Letter Topic patterns.

High Availability

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 08 Jan 2026 00:00:00 +0000

Learn about Elasticsearch cluster Replica, Snapshot, and failure response strategies.

High Availability Concepts#

HA (High Availability) Goals#

Metric	Description	Target
Availability	Service uptime	99.9% (8.76 hours downtime/year)
Durability	Data loss prevention	99.999999% (9-nines)
Recovery Time	Failure to recovery	< 30 minutes

HA Components#

flowchart TB
 A[High Availability] --> B[Replica Shard]
 A --> C[Snapshot & Restore]
 A --> D[Cross-Cluster Replication]
 A --> E[Cluster Design]

Replica Shard#

Role#

flowchart LR
 subgraph Node1
 P0[Primary 0]
 end
 subgraph Node2
 R0[Replica 0]
 end
 subgraph Node3
 P1[Primary 1]
 end

 P0 -->|Replication| R0
 Client -->|Write| P0
 Client -->|Read| R0

Data Redundancy: Replica is promoted when Primary fails
Read Performance: Search requests distributed

Replica Configuration#

PUT /products
{
 "settings": {
 "number_of_shards": 3,
 "number_of_replicas": 1
 }
}

Dynamic Change#

PUT /products/_settings
{
 "number_of_replicas": 2
}

Recommended Settings#

Environment	number_of_replicas
Development	0
Small Production	1
Large/Critical Data	2

Auto-Expand Replicas#

Automatically adjust based on node count:

MLlib

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

MLlib (Machine Learning Library)#

MLlib is Spark’s distributed machine learning library. It enables training and prediction of machine learning models on large-scale datasets.

MLlib Overview#

Two APIs#

API	Package	Data Type	Status
spark.ml	`org.apache.spark.ml`	DataFrame	Currently recommended
spark.mllib	`org.apache.spark.mllib`	RDD	Maintenance mode

This guide covers the spark.ml (DataFrame-based) API.

Key Components#

Transformer: Data transformation (transform without fit)
Estimator: Trainable model (fit creates Transformer)
Pipeline: Connect multiple stages
Evaluator: Evaluate model performance
CrossValidator/TrainValidationSplit: Hyperparameter tuning

Basic Workflow#

import org.apache.spark.ml.classification.LogisticRegression;
import org.apache.spark.ml.classification.LogisticRegressionModel;
import org.apache.spark.ml.feature.VectorAssembler;

SparkSession spark = SparkSession.builder()
 .appName("MLlib Example")
 .master("local[*]")
 .getOrCreate();

// 1. Load data
Dataset<Row> data = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .csv("data.csv");

// 2. Create feature vector
VectorAssembler assembler = new VectorAssembler()
 .setInputCols(new String[]{"feature1", "feature2", "feature3"})
 .setOutputCol("features");

Dataset<Row> assembled = assembler.transform(data);

// 3. Train/test split
Dataset<Row>[] splits = assembled.randomSplit(new double[]{0.8, 0.2}, 42);
Dataset<Row> training = splits[0];
Dataset<Row> test = splits[1];

// 4. Train model
LogisticRegression lr = new LogisticRegression()
 .setFeaturesCol("features")
 .setLabelCol("label")
 .setMaxIter(100)
 .setRegParam(0.1);

LogisticRegressionModel model = lr.fit(training);

// 5. Predict
Dataset<Row> predictions = model.transform(test);
predictions.select("label", "prediction", "probability").show(10);

// 6. Save model
model.write().overwrite().save("models/logistic-regression");

// 7. Load model
LogisticRegressionModel loadedModel = LogisticRegressionModel.load("models/logistic-regression");

Feature Transformation#

VectorAssembler#

Combine multiple columns into a single feature vector:

For Comprehension

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

For comprehension is syntactic sugar for flatMap, map, and withFilter

Allows writing nested flatMap calls in a readable, declarative form

Works with various monadic types including Option, Either, Future, and List

Without yield, it only executes side effects (converted to foreach)

Target Audience: Developers who understand higher-order functions Prerequisites: map, flatMap, and filter higher-order functions

For Comprehension is syntactic sugar that elegantly expresses flatMap, map, and withFilter. It allows you to write nested flatMap and map calls in a readable, declarative form, and works with various monadic types such as Option, Either, Future, and List.

Health Checks

d8lzz1gpw@mozmail.com (kimbenji) — Sun, 11 Jan 2026 00:00:00 +0000

Target Audience: Backend developers who want to monitor application status in Kubernetes Prerequisites: Pod, Deployment concepts After reading this: You will understand the differences between Liveness, Readiness, and Startup Probes and how to configure them

TL;DR

Liveness Probe: Check if container is alive, restart on failure

Readiness Probe: Check if ready to receive traffic, remove from service on failure

Startup Probe: Check startup completion, other Probes disabled until completion

Why Health Checks are Needed?#

A Running container doesn’t mean the application is healthy.

Monitoring Basics

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Consumer Lag is the most important metric; take immediate action on increasing trends

Broker key metrics: UnderReplicatedPartitions, ActiveControllerCount, OfflinePartitionsCount

Producer metrics: record-send-rate, record-error-rate, request-latency-avg

Prometheus + Grafana recommended for visualization and alerting

On Lag spike: Check Consumer status → Processing speed → Rebalancing → Scale out

Target Audience: Operators and developers managing and monitoring Kafka clusters

Prerequisites: Offset and Lag concepts from Consumer Group & Offset

Understand key metrics for Kafka clusters and applications.

Vector Search (kNN)

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Vector Search: Semantic-based search that finds similar content even with different keywords

dense_vector: Field type for storing vectors, specify similarity calculation method

kNN Query: Search method to find k nearest neighbor documents

Hybrid Search: Combine kNN + keyword search for best results

Embedding Model: Converts text/images to vectors (multilingual models recommended for Korean)

Target Audience: Developers implementing semantic search or recommendation systems Prerequisites: Core Components, Query DSL, Basic ML concepts

Learn how to implement semantic search and similar image search using Elasticsearch’s vector search (kNN).

Performance Tuning

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Performance Tuning#

Learn strategies and specific configuration methods for optimizing Spark application performance.

Tuning Principles#

Tuning Order#

Code optimization — Improve algorithms and logic
Data structure optimization — Partitioning, schema optimization
Resource configuration — Memory, CPU, partition count
Detailed settings — Spark configuration parameters

Measurement Tools#

Spark UI: Job, Stage, Task analysis
Event Log: Detailed execution history
Metrics: Executor metric monitoring

Code-Level Optimization#

1. Minimize Shuffle#

// Bad: Unnecessary shuffles
Dataset<Row> result = df
 .repartition(100) // Shuffle
 .groupBy("key") // Shuffle
 .agg(sum("value"))
 .orderBy("key"); // Shuffle

// Good: Only necessary shuffles
Dataset<Row> result = df
 .groupBy("key")
 .agg(sum("value"));
// Only add orderBy when truly needed

2. Early Filtering#

// Bad: Filter after join
Dataset<Row> joined = large.join(small, "key")
 .filter(col("status").equalTo("ACTIVE"));

// Good: Filter before join
Dataset<Row> filtered = large.filter(col("status").equalTo("ACTIVE"));
Dataset<Row> joined = filtered.join(small, "key");

3. Column Selection#

// Bad: Keep all columns
Dataset<Row> result = df.join(other, "id")
 .groupBy("category")
 .agg(sum("value"));

// Good: Select only needed columns
Dataset<Row> result = df.select("id", "category", "value")
 .join(other.select("id"), "id")
 .groupBy("category")
 .agg(sum("value"));

4. Broadcast Join#

import static org.apache.spark.sql.functions.broadcast;

// Broadcast small tables (tens of MB)
Dataset<Row> result = largeTable.join(
 broadcast(smallTable),
 "key"
);

// Set auto broadcast threshold
spark.conf().set("spark.sql.autoBroadcastJoinThreshold", "100MB");

5. Leverage Caching#

// Cache repeatedly used data
Dataset<Row> processed = df.filter(...).groupBy(...).agg(...);
processed.cache();

// Use multiple times
process1(processed);
process2(processed);
process3(processed);

processed.unpersist();

Data Structure Optimization#

1. File Formats#

Format	Compression	Column Pruning	Use Case
Parquet	High	Supported	Analytical queries (recommended)
ORC	High	Supported	Hive compatibility
Avro	Medium	Not supported	Streaming, schema evolution
JSON	None	Not supported	When compatibility needed
CSV	None	Not supported	Simple data exchange

// Save as Parquet (recommended)
df.write()
 .mode("overwrite")
 .option("compression", "snappy") // snappy, gzip, zstd
 .parquet("output/data");

2. Partitioning Strategy#

// Partition matching query patterns
df.write()
 .partitionBy("year", "month")
 .parquet("output/data");

// Partition pruning on query
Dataset<Row> jan2024 = spark.read()
 .parquet("output/data")
 .filter(col("year").equalTo(2024).and(col("month").equalTo(1)));
// Only reads year=2024/month=1 partition

3. Bucketing#

// Bucket by join key
df.write()
 .bucketBy(100, "user_id")
 .sortBy("timestamp")
 .saveAsTable("events");

// Join tables with same bucketing has no shuffle

4. Small File Problem Resolution#

// Adjust partitions before writing
df.coalesce(10) // Merge into 10 files
 .write()
 .parquet("output");

// Or specify max records per file
df.write()
 .option("maxRecordsPerFile", 1000000)
 .parquet("output");

Resource Configuration#

Memory Structure#

Executor Memory (spark.executor.memory)
├── Execution Memory: Shuffle, join, sort
├── Storage Memory: Cache, broadcast
└── User Memory: UDFs, user objects

Default ratio (spark.memory.fraction = 0.6):
├── Execution + Storage: 60%
└── User: 40%

Recommended Settings#

# Executor settings
spark.executor.instances=10 # Number of Executors
spark.executor.memory=8g # Executor memory
spark.executor.cores=4 # Cores per Executor

# Driver settings
spark.driver.memory=4g # Driver memory
spark.driver.cores=2 # Driver cores

# Parallelism
spark.default.parallelism=200 # Default RDD partition count
spark.sql.shuffle.partitions=200 # SQL shuffle partition count

Cluster Sizing Guide#

Total cores = spark.executor.instances × spark.executor.cores
Recommended partitions = Total cores × 2-4

Example: 50 Executors × 4 cores = 200 cores
→ Partition count: 400-800

Executor Memory Calculation#

spark.executor.memory = Container memory × 0.9 - Overhead

Example: 10GB container
- Overhead (10%): 1GB
- spark.executor.memory: ~8GB

Shuffle Optimization#

// Shuffle partition count (most important)
spark.conf().set("spark.sql.shuffle.partitions", "400");

// Shuffle buffer size
spark.conf().set("spark.shuffle.file.buffer", "64k");

// Shuffle compression
spark.conf().set("spark.shuffle.compress", "true");

// Shuffle spill threshold
spark.conf().set("spark.shuffle.spill.compress", "true");

AQE (Adaptive Query Execution)#

Automatic optimization in Spark 3.0+:

Namespace

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

Overall Analogy: Building Wings in an Apartment Complex#

Namespaces are easy to understand when compared to building wings in an apartment complex:

Apartment Complex Analogy	Kubernetes Namespace	Role
Wing 101, Wing 102 separation	Namespace	Logically isolate resources
Per-wing maintenance budget	ResourceQuota	Limit total resources per namespace
Per-unit electricity/water limit	LimitRange	Default resources per Pod/container
Management office restricted area	kube-system	Dedicated space for system components
Public bulletin board area	kube-public	Space accessible to all users
Per-wing access card	RBAC + Namespace	Separate access permissions

In this way, a Namespace is like “dividing management areas by building wing within a single apartment complex.”

Implicit / Given

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Implicit features allow the compiler to automatically pass values or convert types

Scala 2: implicit / Scala 3: given/using/extension

Primarily used for type classes, extension methods, and context passing

Avoid implicit definitions for overly generic types (String, Int)

Target Audience: Developers familiar with higher-order functions and generics Prerequisites: For comprehensions, type parameters

Implicit features are one of Scala’s most powerful capabilities. They allow the compiler to automatically pass values or convert types, reducing boilerplate code and increasing expressiveness. This covers both Scala 2’s implicit and Scala 3’s given/using.

Log Aggregation

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs designing log systems Prerequisites: Three Pillars of Observability After Reading: You’ll be able to select a log collection system and design effective logs

TL;DR#

Key Summary:

Loki: Label-based, lightweight, excellent Grafana integration

ELK: Powerful full-text search, suitable for large-scale analysis

Structured Logs: JSON format for easy field-by-field search

Log Levels: Long-term retention only for ERROR and above recommended

Why Is Log Aggregation Necessary?#

In microservices environments, applications run across dozens or hundreds of containers. If each container generates its own log file, which server’s log should you look at when an incident occurs?

Security

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Kafka security has 3 layers: SSL/TLS (encryption) -> SASL (authentication) -> ACL (authorization)

SSL/TLS encrypts data in transit, mutual TLS for client authentication

SASL mechanisms: SCRAM-SHA-256 recommended (production), PLAIN only with SSL

ACL provides fine-grained Topic/Group access control per Principal

Set allow.everyone.if.no.acl.found=false for default deny policy

Target Audience: Operators and security administrators configuring Kafka cluster security

Prerequisites: Understanding of Broker, Producer, Consumer concepts from Core Components, basic security concepts (encryption, authentication, authorization)

Korean Search Optimization

d8lzz1gpw@mozmail.com (kimbenji) — Thu, 08 Jan 2026 00:00:00 +0000

Prerequisites
Before reading this document, understand these concepts first:

Data Modeling - Analyzer, text vs keyword types

Query DSL - match, multi_match queries

This document covers how to optimize Korean language search in Elasticsearch. We’ll implement the Nori analyzer, autocomplete, and initial consonant (chosung) search.

Korean has fundamentally different linguistic characteristics from English, making proper search impossible with the default English analyzer. Searching “Samsung Electronics” (삼성전자) won’t find “Samsung Electronics Inc.” (삼성전자가) or “Samsung Electronics Co.” (삼성전자를), and it can’t extract “purchase” (구매) from “purchased” (구매했습니다). When users can’t find results they expect to find, they lose trust in the search service.

Deployment and Cluster Management

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Deployment and Cluster Management#

Learn how to deploy and manage Spark applications in various cluster environments.

Cluster Manager Types#

Type	Characteristics	Suitable Environment
Local	Single JVM	Development/Testing
Standalone	Spark built-in	Small dedicated clusters
YARN	Hadoop ecosystem integration	Existing Hadoop environments
Kubernetes	Container-based	Cloud native
Mesos	General-purpose scheduler	Mixed workloads

Local Mode#

Used for development and testing.

SparkSession spark = SparkSession.builder()
 .appName("Local App")
 .master("local") // Single thread
 .master("local[4]") // 4 threads
 .master("local[*]") // All cores
 .getOrCreate();

Standalone Cluster#

A simple cluster manager built into Spark.

StatefulSet

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

Overall Analogy: A Restaurant with Reserved Seats#

StatefulSets are easy to understand when compared to a restaurant with reserved seats:

Restaurant Analogy	Kubernetes StatefulSet	Role
Reserved seat numbers (Seat 1, 2, 3)	Pod names (app-0, app-1, app-2)	Stable and unique identifiers
Personal locker for each guest	PersistentVolumeClaim	Dedicated persistent storage per Pod
Seated/unseated in order	Sequential creation/deletion	Created from 0, deleted in reverse
Name plate reservation	Stable network ID	Same hostname preserved after Pod restart
Open seating restaurant	Deployment	Sit anywhere, seats may change

In this way, a StatefulSet is like “having reserved seats and personal lockers so you always use the same seat and the same belongings.”

Type Classes

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Type classes are a pattern for adding new functionality without modifying existing types

Three components: type class (trait), instances (given), API (extension)

You can define multiple instances for the same type and choose based on context

Show, Eq, Ordering, and Monoid are representative type classes

Target Audience: Developers who understand Implicit/Given Prerequisites: Implicit features, generics

Type classes are a pattern for adding new functionality to existing types. They enable polymorphism without inheritance and allow you to define new behavior without modifying existing types. Originating from Haskell, this pattern combines with Scala’s implicit features to provide powerful abstractions.

Distributed Tracing

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs operating microservices Prerequisites: Three Pillars of Observability After Reading: You’ll understand distributed tracing and be able to analyze request flows between services

TL;DR#

Key Summary:

Trace: Entire path of one request (composed of multiple Spans)

Span: Single unit of work (start/end time, metadata)

Context Propagation: Passing Trace ID between services

Sampling: Store only a portion of total traces (cost optimization)

Why Is Distributed Tracing Necessary?#

In microservices, a single request passes through multiple services. It’s hard to identify where delays occur.

Ecosystem

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Kafka Connect: Data movement between external systems and Kafka, build pipelines with configuration alone, no coding required

Schema Registry: Centralized message schema management, compatibility validation prevents runtime errors

Kafka Streams: Real-time stream processing library, no separate cluster required

Debezium: CDC (Change Data Capture) streams database changes to Kafka

Schema compatibility: Supports BACKWARD (default), FORWARD, FULL, NONE policies

Target Audience: Developers and data engineers building data pipelines using the Kafka ecosystem

Spark Connect

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Spark Connect (Spark 3.4+)#

Spark Connect is a new client-server architecture introduced in Spark 3.4. It enables connecting to remote Spark clusters with a thin client.

Traditional Approach vs Spark Connect#

flowchart TB
 subgraph Traditional["Traditional Approach"]
 App1[Application]
 Driver1[Driver + SparkContext]
 Exec1[Executors]
 App1 --> Driver1
 Driver1 --> Exec1
 end

 subgraph Connect["Spark Connect"]
 App2[Thin Client]
 Server[Spark Connect Server]
 Driver2[Driver]
 Exec2[Executors]
 App2 -->|gRPC| Server
 Server --> Driver2
 Driver2 --> Exec2
 end

Key Differences#

Aspect	Traditional Approach	Spark Connect
Architecture	Monolithic (Driver embedded)	Client-server separation
Client size	Hundreds of MB (full Spark)	Few MB (gRPC client)
Language support	JVM required	Language independent (gRPC)
Resource isolation	Driver resources needed on client	Resources used only on server
Upgrades	Client redeployment required	Only server upgrade needed
Stability	Job stops on client crash	Server runs independently stable

Spark Connect Advantages#

1. Lightweight Client#

<!-- Traditional approach: spark-core + spark-sql (~200MB) -->
<dependency>
 <groupId>org.apache.spark</groupId>
 <artifactId>spark-sql_2.13</artifactId>
 <version>3.5.1</version>
</dependency>

<!-- Spark Connect: lightweight client (~10MB) -->
<dependency>
 <groupId>org.apache.spark</groupId>
 <artifactId>spark-connect-client-jvm_2.13</artifactId>
 <version>3.5.1</version>
</dependency>

2. Isolated Execution Environment#

Client memory exhaustion doesn’t affect Spark jobs
Multiple clients can share the same cluster
Server continues work on client failure

3. Language Independence#

gRPC-based connectivity from various languages
Supports Python, Scala, Java, Go, etc.
More languages planned for future support

Server Setup#

Starting Spark Connect Server#

# Start standalone server
./sbin/start-connect-server.sh \
 --packages org.apache.spark:spark-connect_2.13:3.5.1 \
 --conf spark.connect.grpc.binding.port=15002

# Or start with spark-submit
spark-submit \
 --class org.apache.spark.sql.connect.service.SparkConnectServer \
 --packages org.apache.spark:spark-connect_2.13:3.5.1 \
 --conf spark.connect.grpc.binding.port=15002 \
 local:///dev/null

Running Server with Docker#

# docker-compose.yml
version: '3.8'
services:
 spark-connect:
 image: apache/spark:3.5.1
 command: >
 /opt/spark/sbin/start-connect-server.sh
 --packages org.apache.spark:spark-connect_2.13:3.5.1
 ports:
 - "15002:15002" # gRPC port
 - "4040:4040" # Spark UI
 environment:
 - SPARK_CONNECT_GRPC_BINDING_PORT=15002
 volumes:
 - ./data:/opt/spark/data

Server Configuration Options#

# spark-defaults.conf
spark.connect.grpc.binding.port=15002
spark.connect.grpc.arrow.maxBatchSize=4194304
spark.connect.grpc.maxInboundMessageSize=134217728
spark.connect.extensions.relation.classes=
spark.connect.extensions.expression.classes=
spark.connect.extensions.command.classes=

Java Client Usage#

Dependency Setup#

// build.gradle.kts
dependencies {
 // Spark Connect client (lightweight)
 implementation("org.apache.spark:spark-connect-client-jvm_2.13:3.5.1")
}

Connection and Usage#

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import static org.apache.spark.sql.functions.*;

public class SparkConnectExample {
 public static void main(String[] args) {
 // Connect to Spark Connect server
 SparkSession spark = SparkSession.builder()
 .remote("sc://localhost:15002") // Spark Connect URL
 .build();

 try {
 // Use existing DataFrame API identically
 Dataset<Row> df = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .csv("data/sales.csv");

 Dataset<Row> result = df
 .filter(col("amount").gt(100))
 .groupBy("category")
 .agg(
 sum("amount").alias("total"),
 count("*").alias("count")
 )
 .orderBy(col("total").desc());

 result.show();

 // Collect results
 List<Row> rows = result.collectAsList();
 for (Row row : rows) {
 System.out.println(row.getString(0) + ": " + row.getDouble(1));
 }

 } finally {
 spark.stop();
 }
 }
}

Connection URL Format#

// Basic connection
SparkSession.builder().remote("sc://hostname:port")

// Token authentication
SparkSession.builder().remote("sc://hostname:port;token=<auth_token>")

// User ID specification
SparkSession.builder().remote("sc://hostname:port;user_id=my_user")

// SSL/TLS
SparkSession.builder().remote("sc://hostname:port;use_ssl=true")

Supported Features#

Fully Supported#

Feature	Status	Notes
DataFrame API	✅ Supported	select, filter, groupBy, etc.
SQL queries	✅ Supported	spark.sql()
UDF (User-defined functions)	✅ Supported	Must be registered on server
File read/write	✅ Supported	CSV, JSON, Parquet
Aggregate functions	✅ Supported	sum, avg, count, etc.
Window functions	✅ Supported	rank, row_number, etc.
Joins	✅ Supported	inner, left, right, etc.

Limitations#

Feature	Status	Alternative
RDD API	❌ Not supported	Use DataFrame API
Streaming	⚠️ Partial support	Execute on server side
MLlib	⚠️ Partial support	Basic features only
GraphX	❌ Not supported	Use GraphFrame
Local file access	❌ Not supported	Server path or cloud storage

Spring Boot Integration#

package com.example.config;

import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import jakarta.annotation.PreDestroy;

@Configuration
public class SparkConnectConfig {

 @Value("${spark.connect.url:sc://localhost:15002}")
 private String sparkConnectUrl;

 private SparkSession spark;

 @Bean
 @Profile("!local") // Use in non-local environments
 public SparkSession sparkSession() {
 this.spark = SparkSession.builder()
 .remote(sparkConnectUrl)
 .build();
 return spark;
 }

 @Bean
 @Profile("local") // For local development
 public SparkSession localSparkSession() {
 this.spark = SparkSession.builder()
 .appName("local-dev")
 .master("local[*]")
 .getOrCreate();
 return spark;
 }

 @PreDestroy
 public void cleanup() {
 if (spark != null) {
 spark.stop();
 }
 }
}

# application.yml
spring:
 profiles:
 active: local

---
spring:
 config:
 activate:
 on-profile: production

spark:
 connect:
 url: sc://spark-connect.internal:15002;token=${SPARK_TOKEN}

Best Practices#

1. Connection Pooling#

@Component
public class SparkConnectionPool {
 private final SparkSession spark;

 public SparkConnectionPool(SparkSession spark) {
 this.spark = spark;
 }

 // SparkSession is thread-safe so it can be shared
 public SparkSession getSession() {
 return spark;
 }
}

2. Large Result Handling#

// ❌ Collect all - memory risk
List<Row> allRows = df.collectAsList();

// ✅ Collect limited results only
List<Row> topRows = df.limit(1000).collectAsList();

// ✅ Save to file on server then download
df.write().parquet("s3://bucket/results/");

3. Error Handling#

import io.grpc.StatusRuntimeException;

try {
 Dataset<Row> result = spark.sql("SELECT * FROM table");
 result.show();
} catch (StatusRuntimeException e) {
 if (e.getStatus().getCode() == Status.Code.UNAVAILABLE) {
 logger.error("Spark Connect server connection failed");
 // Reconnection logic
 } else {
 throw e;
 }
} catch (AnalysisException e) {
 logger.error("SQL analysis error: {}", e.getMessage());
}

Migration Guide#

Migrating from Existing Code to Spark Connect#

// Before: Traditional approach
SparkSession spark = SparkSession.builder()
 .appName("MyApp")
 .master("spark://master:7077")
 .config("spark.executor.memory", "4g")
 .getOrCreate();

// After: Spark Connect
SparkSession spark = SparkSession.builder()
 .remote("sc://spark-connect:15002")
 .build();

// DataFrame API is used identically
Dataset<Row> df = spark.read().parquet("data.parquet");

Considerations#

No local file access: Use cloud storage instead of client local files
RDD API not supported: Convert to DataFrame/Dataset API
UDF server registration: User-defined functions must be pre-registered on server
Network latency: Some overhead from remote connection

Architecture - Spark cluster structure
Spring Boot Integration - Spring integration
Deployment - Cluster deployment methods

RBAC

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

Overall Analogy: Building Access Card System#

RBAC is easy to understand when compared to a building access card system:

Building Access Analogy	Kubernetes RBAC	Role
Access permission set (3rd floor server room access)	Role	Define allowed operations within a specific Namespace
Master key for all buildings	ClusterRole	Define allowed operations across the entire cluster
Grant permission to employee badge	RoleBinding	Assign a Role to a user
Grant access to all branches	ClusterRoleBinding	Assign a ClusterRole to a user
Employee badge	ServiceAccount	Authentication identity used by Pods
“Access only necessary areas” principle	Principle of Least Privilege	Grant only required permissions

In this way, RBAC is like systematically managing “who can do what and where.”

Covariance / Contravariance

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Covariant (+A): Producer role, List[Dog] <: List[Animal]

Contravariant (-A): Consumer role, Printer[Animal] <: Printer[Dog]

Invariant (A): When both read/write are needed (default)

Function is contravariant in input, covariant in output: Function1[-A, +B]

Target Audience: Developers who understand generics Prerequisites: Type parameters, inheritance hierarchy

Variance defines the subtyping relationship of type parameters. It determines how generic types behave in the inheritance hierarchy and is a core concept for writing type-safe generic code.

OpenTelemetry

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs standardizing observability systems Prerequisites: Three Pillars of Observability, Distributed Tracing After Reading: You’ll understand OpenTelemetry and be able to apply it to your projects

TL;DR#

Key Summary:

OpenTelemetry (OTel): Vendor-neutral standard for Metrics, Logs, and Traces

Components: SDK (instrumentation) + Collector (collect/transform/export)

Advantages: No vendor lock-in, instrument once to support multiple backends

CNCF Project: Second most active project after Kubernetes

What Is OpenTelemetry?#

OpenTelemetry is a standard framework for generating, collecting, and exporting observability data.

Jobs and CronJobs

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

Overall Analogy: Part-Time Work and Scheduled Cleaning#

Jobs and CronJobs are easy to understand when compared to part-time work and scheduled cleaning:

Analogy	Kubernetes	Role
One-time part-time work	Job	A task that runs once and finishes
Scheduled cleaning routine	CronJob	A task that runs periodically on a schedule
Hire 3 workers	completions: 3	Number of successful completions required
Deploy 2 workers at once	parallelism: 2	Number of Pods to run simultaneously
Maximum retry attempts	backoffLimit	Retry limit on failure
Work deadline	activeDeadlineSeconds	Overall time limit for the task

In this way, a Job is “a task that finishes once the assigned work is done,” and a CronJob is “a scheduled task that runs repeatedly at set times.”

Advanced Type System

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Union Types (|): One of several types, Int | String

Intersection Types (&): Satisfies all types, A & B

Opaque Types: Type safety without runtime overhead

Match Types: Type-level pattern matching

Scala 3-only features for more powerful type expression

Target Audience: Scala developers who understand generics and variance Prerequisites: Type parameters, type bounds, variance

Scala 3 provides an even more powerful and expressive type system. This document covers Scala 3’s new type features.

Dashboard Design

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 12 Jan 2026 00:00:00 +0000

Target Audience: Developers and SREs designing Grafana dashboards Prerequisites: SRE Golden Signals After Reading: You’ll be able to design effective dashboards and quickly identify problems

TL;DR#

Key Summary:

Hierarchical Structure: Overview → Service → Detail order

5-Second Rule: Must be able to identify problem presence in 5 seconds

Golden Signals First: Latency, Traffic, Errors, Saturation

Remove Unnecessary Info: Exclude metrics that don’t lead to action

Dashboard Design Principles#

1. 5-Second Rule#

Within 5 seconds of viewing the dashboard, you should know:

NetworkPolicy

d8lzz1gpw@mozmail.com (kimbenji) — Mon, 23 Mar 2026 00:00:00 +0000

Overall Analogy: Apartment Complex Firewall#

NetworkPolicy is easy to understand when compared to an apartment complex access control system:

Apartment Complex Analogy	Kubernetes NetworkPolicy	Role
Per-wing access control rules	NetworkPolicy	Define traffic allow/block between Pods
“Only Wing 101 residents allowed”	podSelector	Select target Pods for the policy
“Delivery person only to lobby”	Ingress rules	Control incoming traffic
“Packages sent through security”	Egress rules	Control outgoing traffic
Security system (CCTV, key cards)	CNI plugin	Network engine that enforces policies
No rules means free access	Default Allow All	All traffic allowed when no NetworkPolicy is set

In this way, NetworkPolicy is like firewall rules that control “which Pods can communicate with which Pods.”

Macros and Metaprogramming

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

inline: Inlines code at compile time to eliminate runtime overhead

inline if/match: Conditional compilation, type-specific optimization

compiletime package: Compile-time operations like error, constValue, summonInline

Macros (${ }): Compile-time code generation using scala.quoted API

Scala 3 macros are more type-safe and concise than Scala 2

Target Audience: Advanced developers familiar with type-level programming Prerequisites: Generics, type classes, advanced type system

Metaprogramming enables code generation and validation at compile time. Scala 3 provides inline and a new macro system. By leveraging these features, you can reduce boilerplate code, perform optimizations at compile time, and enable type-safe code generation.

Concurrency

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Future: Type representing asynchronous computation results

ExecutionContext: Manages thread pools, passed as implicit parameter

map/flatMap: Compose Futures, can use for comprehension

Promise: Write-only type that can complete Future directly

Advanced libraries: Cats Effect, ZIO, Akka

Target Audience: Developers who understand asynchronous programming basics Prerequisites: Higher-order functions, for comprehension, implicit parameters

Scala supports asynchronous programming through Future. This document covers Future, Promise, and ExecutionContext. Through asynchronous programming, you can perform other tasks during I/O wait time, greatly improving application throughput.

Functional Programming Patterns

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 14 Jan 2026 00:00:00 +0000

TL;DR

Referential Transparency: No change in program meaning when replacing function calls with results

Functor: map operation, transforms values while preserving structure

Monad: flatMap operation, chains sequential effects

Option/Either/Try: Type-safe representation of operations that may fail

More powerful abstractions possible with functional libraries like Cats and ZIO

Target Audience: Developers who understand higher-order functions and For Comprehension Prerequisites: map, flatMap, filter, generics

This document covers core functional programming patterns used in Scala. Functional programming is a paradigm that minimizes side effects and uses pure functions and immutable data to write predictable and easily testable code.