Dainius Jocas

Don't Change the Partition Count for Kafka Topics!

A well-intentioned infrastructure change silently corrupted our search index. Discover how increasing a Kafka topic's partition count can break your entire data pipeline.

Don't Change the Partition Count for Kafka Topics!
#1about 5 minutes

An overview of the data indexing pipeline architecture

The system moves data from a MySQL primary data store to an Elasticsearch search server using a Kafka and Kafka Connect pipeline.

#2about 1 minute

Using Kafka partition offset for optimistic concurrency control

The system leverages the Kafka partition offset as the document version number in Elasticsearch to enable parallel indexing without data consistency issues.

#3about 2 minutes

Investigating a mysterious data deletion failure in production

A bug report about Elasticsearch failing to delete documents, which serves stale data, could not be reproduced in local or testing environments.

#4about 5 minutes

Discovering the offset and version number mismatch

Manual inspection reveals that the document version in Elasticsearch is significantly higher than the new message offset in the Kafka topic for the same key.

#5about 4 minutes

How changing partition count breaks message ordering guarantees

Increasing the Kafka topic's partition count changes the key hashing algorithm, causing new messages for the same key to land in different partitions with lower offsets.

#6about 4 minutes

The solution and key lessons for managing Kafka topics

The fix required a full data re-ingestion into a new Kafka topic, highlighting the lesson to never increase partition count when message ordering is critical.

Related jobs
Jobs that call for the skills explored in this talk.

Job ad Dark

SabIna compys

SabIna compys
Vienna, Austria

Remote
€20-100K
Intermediate
JavaScript
.NET
+1

Matching moments

Exploring Kafka's core concepts of events, topics, and partitions
01:45 MIN

Exploring Kafka's core concepts of events, topics, and partitions

Let's Get Started With Apache Kafka® for Python Developers

Recapping Kafka's capabilities for real-time data feeds
01:17 MIN

Recapping Kafka's capabilities for real-time data feeds

Let's Get Started With Apache Kafka® for Python Developers

Navigating the Kafka ecosystem and the power of community
03:30 MIN

Navigating the Kafka ecosystem and the power of community

Let's Get Started With Apache Kafka® for Python Developers

Answering questions on Kafka use cases, careers, and learning
22:41 MIN

Answering questions on Kafka use cases, careers, and learning

Let's Get Started With Apache Kafka® for Python Developers

Common challenges of running Kafka at scale
05:28 MIN

Common challenges of running Kafka at scale

Tips, Techniques, and Common Pitfalls Debugging Kafka

A traditional approach to streaming with Kafka and Debezium
04:23 MIN

A traditional approach to streaming with Kafka and Debezium

Python-Based Data Streaming Pipelines Within Minutes

Decoupling microservices with event streams
03:41 MIN

Decoupling microservices with event streams

From event streaming to event sourcing 101

Managing data consistency with change data capture
01:34 MIN

Managing data consistency with change data capture

Software Engineering Social Connection: Yubo’s lean approach to scaling an 80M-user infrastructure

Featured Partners

Related Videos

See all videos
Tips, Techniques, and Common Pitfalls Debugging Kafka54:29

Tips, Techniques, and Common Pitfalls Debugging Kafka

DeveloperSteve

Practical Change Data Streaming Use Cases With Debezium And Quarkus52:15

Practical Change Data Streaming Use Cases With Debezium And Quarkus

Alex Soto

How to Benchmark Your Apache Kafka35:27

How to Benchmark Your Apache Kafka

Kirill Kulikov

Let's Get Started With Apache Kafka® for Python Developers38:50

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

Kafka Streams Microservices45:48

Kafka Streams Microservices

Denis Washington & Olli Salonen

Distributed search under the hood46:58

Distributed search under the hood

Alexander Reelsen

From event streaming to event sourcing 10127:33

From event streaming to event sourcing 101

Gerard Klijs

Application Modernization and Rabbits33:40

Application Modernization and Rabbits

Niklas Heidloff

Related Articles

View all articles
CH
Chris Heilmann
Dev Digest 134 - Where pixels sing?
News and ArticlesWeAreDevelopers LIVE Data and Security Day is on Wednesday, 25/09/2024. Learn about OPC UA Updates, Best Practices for Using GitHub Secrets, Passwordless Web 1.5, Emerging AI Security Risks, Data Privacy in LLMs and get a chance to t...
Dev Digest 134 - Where pixels sing?
CH
Chris Heilmann
Dev Digest 109 -Egg-citing things…
As we are heading into the Easter break, here are some things to spend some time on. There's resources on improving the performance of your code and you hear from the winners of CODE100 Amsterdam what it was like to be on stage. Also, hang tight as t...
Dev Digest 109 -Egg-citing things…
CH
Chris Heilmann
WeAreDevelopers LIVE days are changing - get ready to take part
Starting with this week's Web Dev Day edition of WeAreDevelopers LIVE Days, we changed the the way we run these online conferences. The main differences are:Shorter talks (half an hour tops)More interaction in Q&AA tips and tricks "Did you know" sect...
WeAreDevelopers LIVE days are changing - get ready to take part
CH
Chris Heilmann
Dev Digest 138 - Are you secure about this?
Hello there! This is the 2nd "out of the can" edition of 3 as I am on vacation in Greece eating lovely things on the beach. So, fewer news, but lots of great resources. Many around the topic of security. Enjoy! News and ArticlesGoogle Pixel phones t...
Dev Digest 138 - Are you secure about this?

From learning to earning

Jobs that call for the skills explored in this talk.

Java Engineer

Java Engineer

ING DiBa
Vienna, Austria

Intermediate
Java
Spring
Java EE
Hibernate