Kafka

Why Kafka

When every system talks directly to the others, every new integration is one more wire, and the number of wires grows fast. Kafka puts a central event log in the middle: producers write once, and whoever needs it consumes when and as many times as they want.

“A high-performance distributed event streaming platform”:

• High throughput and low latency: millions of messages per second in a cluster.
• Horizontally scalable: more brokers, more partitions.
• Storage: messages don’t disappear when read; they stay for the configured time.
• High availability, through replication across machines.
• Connectivity: ready-made connectors and libraries for almost everything.

The log: the core idea

Each partition of a topic is a log: a sequence of messages that only grows at the end. Each message gets a sequential number, the offset. Consumers don’t delete anything; each one only keeps track of how far it has read.

That is why two consumers can be at different points of the same log, a new consumer can read everything from the beginning and, after fixing a bug, you can rewind the offset and reprocess.

Lab: local Kafka in two minutes

With Docker, a real broker (already in KRaft mode) starts with one command:

docker run -d --name kafka -p 9092:9092 apache/kafka:4.2.2

The command-line scripts live in /opt/kafka/bin inside the container:

# create a topic with 3 partitions
docker exec -it kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3

# terminal 1: produce messages in key:value format
docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic orders --property parse.key=true --property key.separator=:

# terminal 2: consume from the beginning, showing key and partition
docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders --from-beginning --property print.key=true --property print.partition=true

Send order-42:created, order-7:created and order-42:paid: both events for order 42 always land on the same partition.

Cluster architecture

• Broker: a Kafka server. It stores on disk the partitions it is responsible for and serves producers and consumers.
• Cluster: a set of brokers. In production, at least 3, to tolerate losing one.
• KRaft: the consensus protocol the brokers themselves (or dedicated nodes, the controllers) use to keep the cluster metadata: which topics exist, who leads each partition, which brokers are alive. It replaced ZooKeeper, which was removed in Kafka 4.0.

Topics, partitions and keys

A topic is a named channel (orders, payments). It is split into partitions, spread across brokers, which distribute the load and increase throughput. Order is only guaranteed within each partition.

Each message has:

• Key: decides the partition (by hash). Same key, same partition, guaranteed order among them. Without a key, messages are spread across partitions.
• Value: the content (JSON, Avro, Protobuf).
• Headers: metadata (trace id, schema version).
• Timestamp.

How many partitions?

The number of partitions is the parallelism ceiling of a consumer group. You can increase it later, but not decrease it, and increasing it changes which partition each key goes to, breaking ordering during the transition. Start with room to grow (e.g. 6 or 12 for a topic with meaningful traffic).

Retention and compaction

• By time or size (retention.ms, 7 days by default): old messages are deleted, whether they were read or not.
• Compaction (cleanup.policy=compact): keeps only the latest message for each key. It turns the topic into a table with the current state (e.g. each customer’s latest address).

Producers

The producer chooses how much acknowledgement it waits for with acks:

Since 3.0, the producer is also idempotent by default: network retries don’t create duplicate or out-of-order messages in the partition. For throughput, it groups messages into batches (linger.ms, batch.size) and can compress them (compression.type=zstd).

var props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());

try (var producer = new KafkaProducer<String, String>(props)) {
    // the same key guarantees the order of this order's events
    producer.send(new ProducerRecord<>("orders", "order-42", "{\"status\":\"PAID\"}"), (meta, error) -> {
        if (error != null) log.error("failed to publish", error);
    });
}

Consumers

Consumer groups

Consumers with the same group.id split the partitions among themselves: each partition is read by one consumer in the group. Different groups read everything, independently.

Consequence: more consumers than partitions in a group doesn’t increase throughput; the extra ones sit idle, on standby.

Offsets and commit

The consumer reads in a loop (poll) and commits how far it has processed. The committed offset is stored in Kafka itself, and it is where the group resumes after a restart.

props.put("group.id", "billing");
props.put("enable.auto.commit", "false"); // commit only after processing

try (var consumer = new KafkaConsumer<String, String>(props)) {
    consumer.subscribe(List.of("orders"));
    while (true) {
        var records = consumer.poll(Duration.ofMillis(500));
        for (var record : records) {
            process(record.key(), record.value());
        }
        consumer.commitSync();
    }
}

With Spring Boot, the loop, the commit and the threads are handled by the framework:

@KafkaListener(topics = "orders", groupId = "billing")
void onOrder(ConsumerRecord<String, String> record) {
    process(record.key(), record.value());
}

Rebalance

When a consumer joins, leaves or hangs, the group redistributes the partitions. Since Kafka 4.0, a new rebalance protocol (KIP-848) is production-ready: coordinated by the broker, it moves only the partitions that need to move instead of stopping the whole group. It is already enabled on the brokers, but on the client the default is still the classic protocol: to use the new one, set group.protocol=consumer.

Lag

Lag is the distance between the last message in the topic and the offset committed by the group: how much work is piling up. It is a consumer’s most important metric.

docker exec -it kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group billing

Replication

• Replication factor: how many copies of each partition exist in the cluster, counting the main one. The usual value is 3.
• Leader: each partition has a leader, which receives the writes. The other copies (followers) replicate from it.
• ISR (in-sync replicas): the replicas that are caught up with the leader. If the leader dies, an ISR replica takes over, and the topic stays available.
• min.insync.replicas=2 with acks=all: a write is only acknowledged if at least two copies have it. With a factor of 3, the cluster can lose a broker without losing data or stopping.

Delivery guarantees

In practice, use at least once and make the consumer idempotent: processing the same message twice must not cause a duplicate effect (e.g. store the id of events already processed). See the glossary.

Queues in Kafka: share groups

Topics and classic queues solve different problems:

Since Kafka 4.2 (February 2026), share groups bring queue semantics into Kafka. Each message is delivered to a consumer with a temporary lock, and the consumer answers:

• ACCEPT: processed, don’t deliver again.
• RELEASE: temporary failure, hand it back for another consumer to try.
• REJECT: can’t be processed, discard it.

If the lock expires without an answer, the message goes back to the queue, and after a few attempts (5 by default) it is discarded, which keeps a broken message from looping forever. In Java, the client is KafkaShareConsumer.

Ecosystem

• Kafka Connect: moves data between Kafka and other systems without code, using connectors. Combined with Debezium, it does CDC (change data capture): every change to a database table becomes an event.
• Kafka Streams: a Java library to process streams (filter, aggregate, join topics) inside your application.
• Schema Registry (Confluent, outside the Apache project): stores schemas (Avro, Protobuf, JSON Schema) and stops a producer from publishing something that breaks consumers.
• Apache Flink: large-scale stream processing, with state and time windows.

Production patterns

• Outbox: to publish an event without risking inconsistency with the database, write it to a table in the same transaction and let another process publish it. See the Transactional Outbox note.
• Retry and dead letter topic: a failing message must not block the partition. Send it to a retry topic with a delay and, after N attempts, to a dead letter topic, with an alert.
• Idempotent consumers: assume every message can arrive twice.
• Monitor the lag of every consumer group, not just broker health.
• Versioned contracts: evolve schemas in a compatible way; never change the meaning of an existing field.

When to use it

See also: System design, CQRS and Observability.

Videos

• What is Apache Kafka? (Tim Berglund, Confluent): the concepts with great analogies (the video embedded at the top). It is from 2019, so it still talks about ZooKeeper, which has since been replaced by KRaft.
• Apache Kafka in 6 minutes (James Cutajar): partitions, offsets and replication with an example, very to the point.

References

• Introduction to Apache Kafka: events, topics, partitions and the main concepts (official documentation).
• Quickstart: the official walkthrough, with Docker and with the binaries.
• Kafka 4.2 announcement: share groups production-ready.
• Kafka share groups explained: acknowledgements, locks and delivery limits (Conduktor).