Concepts on Advanced Beginner

Core Components

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Producer: Publishes messages and selects Partitions to send to Broker

Consumer: Reads messages from Partitions as part of a Consumer Group and manages Offsets

Broker: Stores and replicates messages, providing high availability through Leader/Follower structure

Topic: A logical channel for categorizing messages

Partition: A unit that physically divides Topics to enable parallel processing

Target Audience: Developers new to Kafka or those learning basic concepts of distributed messaging systems

Prerequisites: Basic network communication concepts, REST API experience, Spring Boot fundamentals

Message Flow

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Producer serializes messages and Partitioner determines target Partition

With Key, same Key goes to same Partition ensuring order guarantee

Broker stores messages in Partition, replicates to ISR, then returns ACK

Consumer fetches messages via Pull method, commits Offset after processing

Three delivery guarantee levels: At-Most-Once, At-Least-Once, Exactly-Once

Target Audience: Developers who understand Kafka’s basic components, those learning detailed message delivery operations

Prerequisites: Understanding of Producer, Consumer, Broker, Topic, Partition concepts from Core Components

Consumer Group & Offset

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Consumer Group is a logical grouping that enables parallel processing of Consumers with the same purpose

Core rule: One Partition can only be read by one Consumer within a Group

Offset is the message position number within a Partition, stored in the __consumer_offsets topic

Auto commit is convenient but risks data loss; manual commit provides precise control

When a Consumer fails, rebalancing automatically redistributes Partitions

Target audience: Developers building or operating Kafka Consumers, those learning about parallel processing and state management

Consumer Advanced Operations

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

session.timeout.ms controls failure detection; max.poll.interval.ms limits processing time

Use Cooperative Sticky Assignor to minimize rebalancing impact (Kafka 2.4+)

Static Group Membership prevents rebalancing on restarts

Consumer Lag is the most important monitoring metric; trend observation is key

Use kafka-consumer-groups.sh for manual Offset reset (Consumer must be stopped)

Target audience: Developers and operators running Kafka Consumers in production

Prerequisites: Basic concepts from Consumer Group & Offset, ISR and Leader concepts from Replication

Replication

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Replication Factor (RF) is the number of Partition replicas; RF=3 is recommended for production

Leader handles all reads/writes, Followers only replicate and promote on failure

ISR (In-Sync Replicas) is the set of replicas synchronized with the Leader

Use min.insync.replicas=2 + acks=all combination for data safety

KRaft mode (Kafka 3.3+) enables cluster operation without Zookeeper

Target Audience: Developers and operators managing Kafka clusters or designing high-availability systems

Prerequisites: Understanding of Broker and Partition concepts from Core Components

Advanced Concepts

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

acks=0 (fast/may lose), acks=1 (Leader only), acks=all (all ISR, recommended)

Use acks=all + min.insync.replicas=2 combination for data safety

Message Key sends to same Partition for ordering; be cautious when changing Partition count

Retention: time-based deletion (default 7 days), size-based deletion, Log Compaction

Idempotent Producer (default in Kafka 3.0+) prevents duplicates on network errors

Target Audience: Developers who want to understand Kafka’s advanced concepts and optimize production settings

Prerequisites: Topic, Partition, Broker concepts from Message Flow, ISR, Leader, Follower concepts from Replication

Transactions and Exactly-Once

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

At-Most-Once: may lose/no duplicates, At-Least-Once: no loss/may duplicate

Exactly-Once: requires Idempotent Producer + Transactional API + read_committed

Idempotent Producer is enabled by default in Kafka 3.0+, prevents duplicates for single Partition

Kafka transactions guarantee atomic writes across multiple Partitions

Use Outbox pattern when DB + Kafka atomic processing is needed

Target Audience: Developers building systems where message delivery guarantees are critical

Prerequisites: acks and Idempotent Producer concepts from Advanced Concepts

Producer Tuning

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Adjust batch efficiency with batch.size and linger.ms (operates as OR condition)

Just linger.ms=5 can increase throughput by ~2.7x

compression.type: recommend snappy (general), lz4 (high performance), zstd (high compression)

Idempotent Producer (Kafka 3.0+ default) prevents duplicates and guarantees order

BufferExhaustedException occurs when buffer.memory is insufficient

Target Audience: Developers optimizing Producer performance, operators needing high-volume message processing

Prerequisites: acks, Idempotent Producer concepts from Advanced Concepts, Topic, Partition concepts from Message Flow

Understanding the core settings to optimize Producer performance. This document is written for Kafka 3.6.x and code examples are validated in Spring Boot 3.2.x, Spring Kafka 3.1.x, and Java 17 environments.

Consumer Tuning

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Use fetch.min.bytes/fetch.max.wait.ms to balance batch efficiency and latency

max.poll.records and max.poll.interval.ms are key to preventing rebalancing

session.timeout.ms should be at least 3x heartbeat.interval.ms

Throughput optimization: increase fetch.min.size, increase max.poll.records

Latency optimization: decrease fetch.min.size, decrease fetch.max.wait

Target Audience: Developers optimizing Consumer performance, operators seeking operational stability

Prerequisites: Consumer operation principles from Consumer Group & Offset

Understand Consumer performance optimization and stable operation settings.

Consumer Internal Structure#

Consumer operates by fetching messages from Broker, delivering to the application, and committing Offset after processing. When Fetcher retrieves data from Broker, it’s delivered to the application through poll(), and Offset is committed after processing.

Advanced Error Handling

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Error types: Deserialization errors (skip), Transient errors (retry), Permanent errors (DLT)

Use @RetryableTopic for declarative retry and DLT handling (Spring Kafka 2.7+)

DefaultErrorHandler + DeadLetterPublishingRecoverer for DLT publishing

Handle deserialization errors with ErrorHandlingDeserializer

Set up alerts when DLT messages arrive for quick response

Target Audience: Developers implementing Consumer error handling strategies

Prerequisites: Consumer operation principles from Consumer Group & Offset, Spring Kafka basics

Understand Kafka Consumer error handling strategies and Dead Letter Topic patterns.

Monitoring Basics

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Consumer Lag is the most important metric; take immediate action on increasing trends

Broker key metrics: UnderReplicatedPartitions, ActiveControllerCount, OfflinePartitionsCount

Producer metrics: record-send-rate, record-error-rate, request-latency-avg

Prometheus + Grafana recommended for visualization and alerting

On Lag spike: Check Consumer status → Processing speed → Rebalancing → Scale out

Target Audience: Operators and developers managing and monitoring Kafka clusters

Prerequisites: Offset and Lag concepts from Consumer Group & Offset

Understand key metrics for Kafka clusters and applications.

Security

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Kafka security has 3 layers: SSL/TLS (encryption) -> SASL (authentication) -> ACL (authorization)

SSL/TLS encrypts data in transit, mutual TLS for client authentication

SASL mechanisms: SCRAM-SHA-256 recommended (production), PLAIN only with SSL

ACL provides fine-grained Topic/Group access control per Principal

Set allow.everyone.if.no.acl.found=false for default deny policy

Target Audience: Operators and security administrators configuring Kafka cluster security

Prerequisites: Understanding of Broker, Producer, Consumer concepts from Core Components, basic security concepts (encryption, authentication, authorization)

Ecosystem

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Kafka Connect: Data movement between external systems and Kafka, build pipelines with configuration alone, no coding required

Schema Registry: Centralized message schema management, compatibility validation prevents runtime errors

Kafka Streams: Real-time stream processing library, no separate cluster required

Debezium: CDC (Change Data Capture) streams database changes to Kafka

Schema compatibility: Supports BACKWARD (default), FORWARD, FULL, NONE policies

Target Audience: Developers and data engineers building data pipelines using the Kafka ecosystem