Kafka is a distributed message broker widely used as a general purpose bus for big data. Kafka allows you to both implement real-time processing of a large number of events and build a batch pipeline for delivering logs.
Why are we using Kafka? In short, unification. And if in a little more detail - dozens of suppliers, terabytes of logs every day, online and offline pipelines - it is extremely difficult to cope with this without a single high-performance data bus.
, Kafka, pipeline. , exactly once . , - Kafka, . , Kafka , - .
:
2017 . Kafka 0.8.2. . .
- , Rambler, . : «, Rambler. ». Rambler – . Bing .
, Google, -, .
, , , Rambler? : « Rambler ?». Rambler .
Rambler Group . , : Rambler, , , , - . 40 000 000 .
, , - . . , Rambler , .
, .
, , ?
- . , , . . , Ford . , - .
- .
- CTR.
- .
- .
- .
, .
, .
worker. , . , , Hadoop.
HDFS Spark Hive. , — key-value Aerospike.
, :
- -, . – ( ), ().
- -, worker . - , , Hadoop. Worker — .
- -, real-time pipeline . - . , HDFS, HDFS , Aerospike. , , — real-time .
? , , Kafka. Kafka – . , RabbitMQ, , . Kafka .
Kafka - HDFS, Gobblin.
Kafka LinkedIn. Map-Reduce , Hadoop Kafka HDFS.
real time.
Spark Streaming. Spark, Hadoop (LLAP). Kafka Spark Streaming , key-value .
, HDFS, . Kafka , Aerospike.
Aerospike .
?
- -, , Kafka exactly once.
- -, , - Kafka 4 , .
- -, , , Kafka .
, Kafka.
Kafka ( – ) ZooKeeper . . Kafka producers, Kafka consumers.
Kafka . topic. .
array list. , , , .
, .
Kafka, , Kafka , .
Kafka . : master-slave. Master Kafka .
Consumers, producers , . . (master ). a .
, — slave- ( ). , .
, , . . , , . Kafka - slave . , .
Kafka 2 : clean unclean. , unclean – - , , clean – , . . — .
slave , , , unclean, , , clean.
— .
, : , . Kafka , , , Kafka , .
, ?
. , - . Kafka.
. Kafka , .
, , . . ( ).
. -1, Kafka . Kafka — Kafka. , Kafka in sync. insync 1, .
insync.replicas = 2, Kafka , .
. , , . , , - overhead , . . Kafka, .
, , . - , Kafka , .
. , , — . insync.replicas = 2, replication factor = 3. insync , , Kafka.
Kafka .
, . - . :
Latest offset – .
Earliest offset – .
Kafka , - , , . earliest offset . , earliest latest offset, , .
consumer - , . . earliest offset offset - . consumer offset latest offset ( ).
, . , , consumer .
, consumer Kafka — .
– . .
? Consumer Kafka. Kafka : . , Kafka .
? : consumer , Kafka , , , . .
. , consumer , Kafka , . Kafka , . . . .
– . : consumer Kafka, , : «Kafka, , ». , at least once .
At least once – , , .
? : , , , consumer Kafka, - kill -9
consumer , , . .
. pipeline at least once .
exactly once. ?
exactly once – Kafka. . , : - , . .
, . – HDFS, .
LinkedIn. Gobblin exactly once HDFS.
, consumer 2 - . production . move (mv) Hadoop .
onsumer .
, , . consumer’, . , . .
— fetch , fetch’. , . exactly once .
:
- Producer .
- Kafka .
- HDFS exactly once. Exactly once – , .
. , .
-, . pipeline – : ( ) Hadoop.
( 1-2 — ), pipeline .
, , , .
pipeline, , . , .
-, . Exactly once, , .
, consumers ( real-time ): Kafka 2-3 .
consumers , : Kafka, Kafka, etc.
: Kafka — . Kafka .
( ), . , pipeline.
, Kafka 4 .
– . – . – .
, ( ) - .
, , — Kafka , - . : Kafka , .
? :
- -, Kafka .
- -, Kafka .
, . , .
, 4 — Kafka .
? - , – . Kafka , . .
, , , , .
— . : 4 , ?
, . :
, , , .
Kafka ? - , Kafka : « , - ». :
. .
, , 3, :
Kafka — , .
, Kafka 0.8.2, , . .
, , , .
. uplink 1 Gb. 125 . 150 . — .
, Kafka RAID 5. ? — . RAID 5 Kafka . , RAID 5 , Kafka . Kafka RAID 10.
:
- , .
- .
- .
- Kafka — .
.
?
, — at least once.
. consumer Kafka — Camus, Gobblin.
history. - . , .
- , Camus . , . - , .
. :
:
Kafka . 4 , , , consumer . Hadoop , , .
. ?
Kafka 5 , 3 . 5 . .
Kafka 18 . 5 . 5 .
, , , , .
.
:
, Kafka , .
, , , :
, 4 - . , . , Kafka leader reelection, .
-, . , , , .
-, . . . , , :
- , in sync
:
? , — . , , . . .
, -, . , , . , Kafka real-time . Kafka.
consumer’:
10 , . Kafka , 10 , : «!».
. , . , .
:
exactly once Kafka, . pipeline. producer', consumer' Kafka. , exactly once . , .
. — , .
It is also worth monitoring many additional metrics. The most important is the response time of Kafka to requests, both for the producer and the consumer. And it is undoubtedly worth monitoring the classic metrics like cpu, memory, etc.
For questions about the report, you can contact Artyom Vybornov