How does Kafka guarantee ordering, and how do you choose the number of partitions?

Kafka guarantees ordering within a partition only - never across partitions. Messages with the same key always land in the same partition, so per-key ordering is achieved by keying on the entity that needs it, such as user ID or order ID. Partition count is the unit of consumer parallelism: you cannot have more useful consumers in a group than partitions. Size it from target throughput divided by per-consumer throughput, add headroom, and remember you can increase partitions later but never decrease them - and increasing them changes the key-to-partition mapping, breaking ordering for in-flight keys. Ordering is per key, achieved by partitioning: The requirement is almost never 'global ordering' - it is 'events for this user must be in order'. Key by user_id and every event for that user goes to one partition, one consumer, in order. Global ordering would require a single partition, which caps throughput at one consumer. Per-key ordering with many partitions is the design; global ordering is a red flag. Sizing partitions 1 · Start from throughput: Target 50,000 msg/s and one consumer handles 5,000 msg/s → you need at least 10 partitions. Add headroom for growth and for uneven key distribution: 20–30 is reasonable here.. 2 · Respect the parallelism cap: Consumers in a group cannot exceed partitions - extras idle. Partition count is a hard ceiling on how fast a group can drain, so size for peak, not average.. 3 · Do not over-partition: Each partition costs file handles, memory, and replication overhead on every broker, and more partitions mean longer leader elections and slower rebalances. Thousands per broker degrades the cluster.. 4 · Plan for the one-way door: You can add partitions but never remove them, and adding them changes hash(key) % N - so a key that used to

StackScholar

Messaging, Queues & StreamsAdvanced15 min read1 questions

Kafka Architecture and Delivery Semantics

How Kafka actually stores and delivers data, why partitions are the unit of both parallelism and ordering, and the operational realities - rebalances, lag and hot partitions.

Covers: Topics, partitions, offsets, consumer groups, replication and ISR, ordering guarantees, rebalancing, consumer lag

Kafka is the default answer to “we need an event backbone”, and it is asked about constantly. The whole model reduces to one idea: a topic is a set of partitions, and a partition is an append-only file. Every guarantee and every limitation follows from that.

Topics, partitions and consumer groups

Tap each node. Partitions are the unit of ordering, parallelism, and therefore of every scaling decision you will make.

ServiceQueueComputeTap a box for detail

Flow

ProducersPartition 0
ProducersPartition 1
ProducersPartition 2
Partition 0Consumer group A
Partition 1Consumer group A
Partition 2Consumer group A
Partition 0Consumer group B
Partition 1Consumer group B
Partition 2Consumer group B

Filter

0/1 mastered

AdvancedkafkapartitionsorderingAsked at Confluent, LinkedIn

30-second answer

Kafka guarantees ordering within a partition only - never across partitions. Messages with the same key always land in the same partition, so per-key ordering is achieved by keying on the entity that needs it, such as user ID or order ID. Partition count is the unit of consumer parallelism: you cannot have more useful consumers in a group than partitions. Size it from target throughput divided by per-consumer throughput, add headroom, and remember you can increase partitions later but never decrease them - and increasing them changes the key-to-partition mapping, breaking ordering for in-flight keys.

javaThe key is the design decision

// No key: round-robin across partitions. Maximum spread, zero ordering.
producer.send(new ProducerRecord<>("orders", null, payload));

// Keyed: hash(key) % numPartitions -> same key always to the same partition.
producer.send(new ProducerRecord<>("orders", order.userId(), payload));

// Consequences of keying by userId:
//   ✅ All events for a user are ordered
//   ✅ One consumer sees a user's full history, so local state works
//   ⚠️  A very active user creates a hot partition
//   ⚠️  Adding partitions later remaps keys and breaks ordering across the change

Sizing partitions

1 · Start from throughput
Target 50,000 msg/s and one consumer handles 5,000 msg/s → you need at least 10 partitions. Add headroom for growth and for uneven key distribution: 20–30 is reasonable here.
2 · Respect the parallelism cap
Consumers in a group cannot exceed partitions - extras idle. Partition count is a hard ceiling on how fast a group can drain, so size for peak, not average.
3 · Do not over-partition
Each partition costs file handles, memory, and replication overhead on every broker, and more partitions mean longer leader elections and slower rebalances. Thousands per broker degrades the cluster.
4 · Plan for the one-way door
You can add partitions but never remove them, and adding them changes hash(key) % N - so a key that used to go to partition 3 now goes to partition 7, and its ordering guarantee is broken across the boundary. Over-provision modestly at creation rather than resizing later.

Durability: replication and ISR

Setting	Meaning	Recommended
`replication.factor`	Copies of each partition	3
`acks`	How many replicas must acknowledge a write	`all` for durability, `1` for throughput
`min.insync.replicas`	Minimum in-sync replicas for an `acks=all` write to succeed	2 (with RF=3)
`unclean.leader.election`	Allow an out-of-sync replica to become leader	`false` - true means silent data loss
`enable.idempotence`	Producer deduplicates its own retries	`true`

Rebalancing: the operational pain

When a consumer joins or leaves a group, partitions are reassigned. Under the classic eager protocol, every consumer stops during the reassignment - a stop-the-world pause proportional to group size. A consumer that is slow to process can miss max.poll.interval.ms, be considered dead, and trigger a rebalance, which slows everyone further and can cascade into a rebalance loop.

Use cooperative sticky assignment so only the moving partitions pause, not the whole group.
Set max.poll.records low enough that a batch always completes within max.poll.interval.ms.
Use static group membership so a rolling restart does not trigger a full reassignment for each pod.
Do slow work outside the poll loop, or increase the interval deliberately rather than by accident.

Say these points

Ordering is per partition; key by entity to get per-key ordering.
Partition count caps consumer parallelism and cannot be reduced.
Adding partitions remaps keys and breaks ordering across the change.
RF=3 with acks=all and min.insync.replicas=2 is the durability sweet spot.
Watch per-partition consumer lag in time units; use cooperative rebalancing.

Avoid these mistakes

Claiming Kafka gives global ordering.
Over-partitioning and degrading the cluster.
`min.insync.replicas` equal to the replication factor, making a single restart fatal.
Leaving unclean leader election enabled.
Long processing inside the poll loop, triggering rebalance loops.

Expect these follow-ups

One partition is 10× hotter than the rest. What do you do?
How would you migrate a topic to more partitions without breaking ordering?
What exactly does Kafka's exactly-once processing cover, and what does it not?

Hands-on challenge

Build it - this is what you talk about in a deep-dive round.

Design a Kafka topic layout for an e-commerce platform

Events: page views (500k/s), add-to-cart (10k/s), orders (1k/s), payments (1k/s). Consumers include analytics, fraud detection, inventory and the data warehouse.

Requirements

Define topics, partition counts and partition keys for each event type, with the throughput arithmetic.
State exactly which ordering guarantee each consumer needs and how the keying provides it.
Choose replication and acks settings per topic, justifying differences between page views and payments.
Set retention per topic and explain the storage implication.
Design the consumer-lag alerting, expressed in time, per consumer group.

Stretch goals

Handle a single merchant generating 20% of all add-to-cart events.
Design the migration to double partitions on the orders topic without breaking per-order ordering.

Kafka Architecture and Delivery Semantics

Topics, partitions and consumer groups