How-To Guides on Advanced Beginner

Consumer Lag Troubleshooting

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

A step-by-step guide to diagnosing and resolving Consumer Lag issues.

TL;DR

Consumer Lag: Number of unread messages (Producer rate > Consumer processing rate)

Diagnosis: Check Lag with kafka-consumer-groups.sh, analyze trends with monitoring tools

Resolution: Increase Consumer count, optimize processing logic, adjust Partition count, tune batch settings

What is Consumer Lag?#

Consumer Lag is the number of messages that the Consumer has not yet read from those published by the Producer. If Lag continuously increases, message processing delays occur, and in the worst case, messages may be deleted after exceeding the retention period.

Producer Performance Tuning

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

A step-by-step guide to increasing Producer throughput and reducing latency.

TL;DR

Throughput priority: Increase batch.size, set linger.ms, enable compression

Latency priority: linger.ms=0, acks=1, small batch size

Durability priority: acks=all, enable.idempotence=true, retry settings

Trade-offs: Throughput, latency, and durability are competing concerns

Three Axes of Performance Optimization#

Producer performance optimization is about balancing three factors:

flowchart TB
 T[Throughput] <--> L[Latency]
 L <--> D[Durability]
 D <--> T

 style T fill:#e1f5fe
 style L fill:#fff3e0
 style D fill:#e8f5e9

Factor	Description	Optimization Direction
Throughput	Messages sent per second	Increase batch size, enable compression
Latency	Time from send to acknowledgment	Reduce batch wait time
Durability	Message loss prevention	acks=all, wait for replication

Step 1: Measure Current Performance#

Measure current performance before optimizing. Without a baseline, you cannot know the improvement effect.

Message Loss Prevention Guide

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

A step-by-step guide on how to prevent message loss and ensure reliable messaging in Kafka.

Time Required: Approximately 30 minutes (for configuration application)

TL;DR

Producer side: acks=all, enable.idempotence=true, proper retry settings

Broker side: min.insync.replicas=2, unclean.leader.election.enable=false

Consumer side: Manual commit, commit after processing, retry mechanism

End-to-End: Message trace ID, monitoring and alerting setup

Before You Begin#

You need the following environment to follow this guide.

Prerequisites#

Item	Minimum Version	Verification Command
Apache Kafka	2.8+	`kafka-topics.sh --version`
Java	17+	`java -version`
Spring Boot	3.0+	`./gradlew dependencies \| grep spring-boot`
Spring Kafka	3.0+	`./gradlew dependencies \| grep spring-kafka`

Environment Verification#

Verify that you can access the Kafka cluster:

Topic Design and Partition Planning Guide

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

A guide on how to determine the appropriate partition count, replication factor, and retention policy when creating new Kafka topics.

Time Required: Approximately 15 minutes

TL;DR

Partition count: Target throughput / Per-consumer throughput, minimum 6 recommended

Replication factor: 3 for production, 1 for development

Retention period: Business requirements + reprocessing buffer time

Naming: Use {domain}.{entity}.{event-type} pattern

Before You Begin#

Prerequisites#

Item	Requirement	Verification Command
Kafka CLI	Installed and in PATH	`kafka-topics.sh --version`
Broker Access	Admin privileges	`kafka-acls.sh --list`
Cluster Info	Know Broker count	`kafka-broker-api-versions.sh`

Environment Verification#

Verify topic creation permissions and cluster status: