Hey!
I've often seen headers like "Apache Kafka vs RabbitMQ vs NATS", but what if there is something that does not suit the ready-made solutions? You can adjust, or you can invent something of your own. I went the second way. In this article, I would like to talk about my implementation of a message broker. If you are interested, welcome to cat.
1.1 Saving orders
Some time ago I worked in a very large online store. There was a common task: not to "miss" any order. That is, to make it so that it would be possible to save the order to several places, and if at least one of them is available, save the order.
1: .
2: , , .
3: ( ).
:( .
, . , .
.
1.2
, " ". , , , , , . , , , .
, .
2.
, , - .
- .
0: - ().
1: (). " " , .
2: , , ().
3: , ().
4: , , ().
*` ` - . , .
.
Kafka - . Kafka - , - .
RabbitMQ - - , . - .
NATS Streaming - . NATS Streaming . NATS Streaming . , PostgreSQL NATS Streaming . , , , . , , NATS Streaming .
.
, .
3. ( )
3.1
Cluster - , .
Queue - . . , . ID .
Handler - . , , , .
ExternalCluster - . . .
3.2
. . . . , .
4 :
,
, ,
. , - .
. , .
3.3
. ( ) (s3) ( ).
. . .
: () , .
. :
" ". " + ID". . , ID . . ( 1000) .
FIFO
. ( , )
3.4
( )
:
N Capella Queue
, , ,
( )
:
Capella Queue M (M < K)
:
+ ID.
M
- , .
, . .
( )
:
,
,
( )
:
Events are stored in a queue on the local cluster. You need to define a globally unique link source + external ID for the message. Thanks to copy handlers, messages will be available both on the device and in the DC
4. Immediate plans
Create a tutorial describing the main cases.
Screw on security.
Tighten the use of SSL certificates by the service.
Add segmentation - the ability to read and transfer data by segments.
Updating parameters for queues, clusters, and handlers.
Functionality to control that the message has been replicated to other clusters.
Metrics.
The code
On github