Why Kafka is so fast

. , , - . - , - CQRS. (, IoT) , .

, Apache Kafka .

, «» , . latency, (throughput) (jitter). «» : .

Apache Kafka , , latency jitter , , «at least once delivery». - — , - — , Kafka , , Kafka .

, Kafka LinkedIn — . Latency , . LinkedIn, , , , , . Kafka (near real-time systems), soft real-time systems.

: , « », «», «». , , (deadline), , . , . , , « ». , , , .

, Kafka , . . .

Kafka , , I/O , , . , ; ( ) . I/O SATA 7200 / - I/O. , read-ahead ( ) write-behind ( ), . - I/O I/O - , .

I/O I/O. , . (bottleneck) Kafka , . , , Kafka ( , ) . , .

, , . , , JSON, 5 7. , (client-side operation), , I/O .

MQ, ( I/O), Kafka . (offsets) (consumer group). Kafka __consumer_offsets. append-only ( ). ( compaction Kafka), .

, . , — point-to-point, point-to-multipoint. , pub-sub point-to-multipoint, . point-to-multipoint MQ- . read/write amplification. , publisher . fan-out, , amplification. , , I/O , , .

Kafka «», ( Kafka). , , . , . , .

Kafka, : Kafka fsync . ACK (), I/O-. , . Kafka , . Kafka — (disc-backed in-memory queue), / .

, , , , . , , , , . Kafka . , ( , ) — , ( - ). , I/O fsync, Kafka , .

, ( ) , () . , . , .

Kafka . , . , , , checksum . , . . , , . , , , . ( Kafka 2.4.0.)

Zero-copy

— . Kafka , , , , . , .

Kafka Linux UNIX Java’s NIO, transferTo() java.nio.channels.FileChannel. - . , NIO, , - :

File.read(fileDesc, buf, len);
Socket.send(socket, buf, len);

:

, , . .

:

1. read() . , DMA (Direct Memory Access, ). , .

   
   
   

2. read() . .

   
   
   

3. send() , — , . DMA, . send() .

   
   
   

4. / send() , .

   
   
   

, . , , . , . , .

, zero-copy . :

fileDesc.transferTo(offset, len, socket);

zero-copy :

. , transferTo() DMA. . , NIC DMA.

, CPU. .

, zero-copy. Linux 2.4 , , gather. .

transferTo() DMA, . gather . NIC . DMA. CPU .

zero-copy - zero-copy. , Kafka , JMV JNI .

, — (GC). , Kafka 32 RAM , 28-32 , GC. ( ), GC , . . GC, , , .

, GC , Kafka . GC, Shenandoah ZGC . , JVM, , .

- I/O — , . , Kafka . . , Kafka, . MQ: RabbitMQ , , .

Kafka. ; ( ) . Kafka . , Kafka , .

, , . ( - , ). , , — , ( ) . , , . , , , , . - - . . , -, .

, () . Kafka , . ( «», , ). , . , , . , ( ), .

:

1. . , - . , , . - , . , Kafka , .
2. . , , . ( , , , ; ). , . , , .

, Kafka, , , , , , .

, Kafka ( ) : , , — - , . latency. , Apache Pulsar — , latency, Kafka . Kafka , - . , , , : Kafka .

Kafka . . - , , , zero-copy I/O Kafka , . , , «at least once delivery».

Kafka , , . , , , , , . , , , .

: Kafka, Stripe. . , .