Hello, Habr! A positive informational background on the topic of event handling through Kafka Streams has formed around me. This tool attracts with a lot of video reports and articles on Habré, detailed documentation, understandable API and beautiful architecture. Some of my friends and colleagues use it to develop their systems. But what happens in real life when these systems go into production?
In this article, I will omit the introduction to Kafka Streams, assuming that the reader is already familiar with it, and I will talk about our experience of living with this library on the example of a fairly loaded system.
Briefly about the project
An internal team together with partners are working on the Ad Exchange, which helps resell advertising traffic. We have already described the specifics of such tools in an article on Habré . As the number of partners among SSP and DSP grows, the load on the exchange servers increases. And to increase the value of the exchange itself, we must collect detailed analytics from this traffic. This is where we tried to use Kafka Streams.
Implementing a new tool in your production is always a risk. We were aware of it, but put up with it, because in general, Kafka Streams should conceptually be well suited for calculating aggregates. Although the first impression was great, I will talk about the problems that led to.
, , . , - Kafka Streams. .
. , , - . , " ". , : , , ..
Kafka Streams: groupBy aggregate . , : changelog-. Kafka log compaction. ?
, , Kafka Streams repartition-, , changelog-. , " " UUID. ? , , repartition-. Kafka. , ?
, 100 . +600 (300 300 ). . Kafka Streams. , .
, , . Kafka Streams. - . , "" , Kafka Streams . / repartition-. , , . , .. , . - ( , ).
, . . . , Kafka Streams, . state-store RocksDB ( ), .
, , , - , . . , , - , , all(), . , . . , RocksDB Postgres.
, . , , HTTP? - Kafka Streams: , - . - . , - . . . ? Kafka Streams , .
Kafka Streams
. , , , . - Kafka, Kafka Streams DEAD , . , Kafka Spring @KafkaListener. , .
Kafka Streams : . , - Kafka Streams. . , . - . , - , . - Kafka Streams . : , Kafka Streams. " " , , .
Kafka Streams , watchdog: Kafka Streams, , .
, Kafka Streams Spring, StreamsBuilderFactoryBean, CleanupConfig. , RocksDB. , , changelog-. , .
KStream-KStream Join
: . Kafka . . , production.
Kafka Streams . , , DevOps . , .
, Kafka, , - : " ? ". . Kafka Streams.
, - (co-partitioning), , , , . ?
, : , , . production, , . , , , . Kafka Streams . , . Kafka Streams , -, .
, , . , Kafka Streams, , Kafka Streams . : Matthias J. Sax , , Kafka Streams application.id. , , , .
, . . , , - KSQL.
: , .
P.S. . VK, FB, Instagram Telegram-, Maxilect.