Clever Geek Handbook

Hybrids win or holivars are expensive

The motivation for writing this article was the fact that there is an increase in the appearance of marketing materials about Apache Kafka on habr.com. And also the fact that the articles give the impression that they are written by people who are a little far from real use is of course only an impression, but for some reason, most of the articles necessarily contain a comparison of Apache Kafka with RabbitMQ, and not in favor of the latter. What's most interesting is that by reading such articles, managers without a technical background begin to spend money on internal research so that leading developers and technical directors choose one of the solutions. Since I am very greedy / homely, and also since I am a supporter of the thesis "Truth is NOT born in a dispute", I suggest you familiarize yourself with another approach - almost without comparing different brokers.







No comparison anywhere



In general, in the right way, I should have made an article in a format Kafka+RabbitMQ+Nats+ActiveMQ+Mosquito+etc



, but it seems to me that for you dear readers it will be overkill, although usually all of the above services (and not only) are present in my architectural solutions. And I'm not talking about this yet about AzureServiceBus / AmazonServiceBus - which also participate in "hybrids" in large project programs. Therefore, for now, let's dwell on the Kafka + RabbitMQ bundle, and then you will understand why: by analogy, you can connect any service with its protocol. Because:







comparing Apache Kafka and RabbitMQ you are comparing 2 (two) brands, or rather 2 commercial companies - Confluent and vmWare, and a little Apache Software Foundation (but this is not a company)

that is, formally, when comparing, we must compare the business models of companies that are the main drivers of the development of our today's experimental subjects. Since Habr is still not an economic research portal, so for a start we must remember not the brands, but the descriptions that stand behind these brands (the way our current participants call themselves).







  • RabbitMQ is a multi-protocol and extensible message broker
  • Apache Kafka is a distributed event streaming platform
  • Confluent Platform - an event streaming platform with the ability to create high-performance data pipelines for analytics and integration in business scenarios


Confluent — Apache Kafka Confluent Apache Kafka. SchemeRegistry



, RestProxy



, kSQL



, , Kafka-Connect



.







— , RabbitMQ "", Kafka - ( ).







— , .







  • RabbitMQ — . ( Erlang)
  • Kafka — ( Scala/Java)








  • RabbitMQ . , .
  • Kafka , .


,



: , , - , , —



,



->



->









— . 14 , , "" ( ), .







  • ODBC
  • AMQP
  • MSMQ
  • XMPP
  • IP over Avian Carriers


(python, C#, java) 1 — One-S-Connectors



(https://code.google.com/archive/p/one-c-connectors/source/default/source). ( 1 1 " 1-" — ).

( 2006 ) , / -. . ODBC Kafka/NATs/ModBus.







— ( )



, — 1-, .









Kombu — , Apache Kafka https://github.com/celery/kombu/issues/301 - " ", Python https://github.com/confluentinc/confluent-kafka-python

— , : Java, GoLang, RUST, etc. NATs ActiveMQ JMS — : Java ,









? , , " " — RabbitMQ ( /deps



) RabbitMQ, Confluent Apache Kafka .







PostgreSQL — CREATE EXTENSION hypopg



, Pivotal/vmWare

— " " — 84Codes



https://github.com/84codes. — 84Codes , / CloudAMQP CloudKarafka.







, , 2 :







  • vmWare , RabbitMQ — . , GitHub.
  • Confuent Enterprise Enterprise-Kafka-Connect, GUI .


- https://github.com/jcustenborder/kafka-connect-rabbitmq, , Java Maven Archetype https://github.com/jcustenborder/kafka-connect-archtype — , Confluent , Kafka .

Kafka



, Java, Enterprise . RabbitMQ



, (Erlang ), 84Codes



. Erlang — , OpenStack.







—



. , , ITILv4, 3







  • ProtocolLock VendorLock — , , - — : .
  • , — .
  • — 3



    — TDD, BDD, CICD, ScallableAgile DevOps (DocOps, DevSecOps)



    — . TimeToMarket.


, , Docker-Compose. — () — , . — Kafka+RabbitMQ 84Codes



( — https://www.84codes.com/).









, . , , , Apache Kafka exactly-ones



. — , ->



Kafka ( Topic



) — Offsets



.







exactly-ones

exactly-ones — " 1", Exactly once — , .







. :







  • Zookeper
  • KafkaBroker
  • RabbitMQ
  • KafkaConnect








  • Python AMQP 0.9
  • # AMQP 1.0
  • C# Kafka


: Apache Kafka — ( ) Java, , librdkafka — C++ - ,. Kafka , " ": , https://github.com/edenhill/librdkafka/pulse/monthly, wmWare https://github.com/rabbitmq

:







RabbitMQ-Kafka-Sinc-Connector



— Confluent Github.









2 — - -.







RabbitMQ Kafka



— 







docker-compose -f dockers/infra.yml up -d


, , , Kafka-UI RabbitMQ-Sinc, Kafka RabbitMQ







    image: provectuslabs/kafka-ui:latest
    ports:
      - 8080:8080
    depends_on:
      - kafka-broker
      - zookeeper
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:29092
      KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
      KAFKA_CLUSTERS_0_JMXPORT: 9101


Java







    <parent>
        <groupId>com.github.jcustenborder.kafka.connect</groupId>
        <artifactId>kafka-connect-parent</artifactId>
        <version>1.0.0</version>
    </parent>


pom.xml — , https://github.com/jcustenborder/kafka-connect-parent, Java-Kafka-Adapter







c RMQ Java — https://www.rabbitmq.com/java-client.html 







            <groupId>com.rabbitmq</groupId>
            <artifactId>amqp-client</artifactId>
            <version>${rabbitmq.version}</version>


— , , :







  • java — -1-build-connect-jar.bat



  • — 00-build-connect-image.sh











  • — 01-start-infra.sh





— Docker PWD Windows Linux — . — sh













RabbitMQ :







:







  • 9092 — Kafka
  • 8080 — Apache Kafka UI
  • 5672 — AMQP 0.9 AMQP 1.0
  • 15672 — RabbitMQ
  • 28082 — curl





RabbitMQ Docker:







  • — enabled-rmq-plugins 





[
    rabbitmq_management, 
    rabbitmq_amqp1_0, 
    rabbitmq_mqtt, 
    rabbitmq_federation, 
    rabbitmq_federation_management,
    rabbitmq_shovel,
    rabbitmq_shovel_management,
    rabbitmq_prometheus
].


  • , — rmq_definitions.json 





     "bindings":[
        {
           "source":"orders-send",
           "vhost":"/",
           "destination":"orders-amqp-10-consumer",
           "destination_type":"queue",
           "routing_key":"",
           "arguments":{

      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    










docker-compose -f dockers/infra.yml restart protocol-connect-sync

docker-compose -f applications.yml build
docker-compose -f applications.yml up














:







  • -



    2 


        producer = conn.Producer(serializer='json')
        producer.publish({'client': '', 'count': 10, 'good': ''},
                      exchange=order_exchange,
                      declare=[kafka_queue, amqp10_queue])
        time.sleep(2)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    


RUN python -m pip install \
    kombu \
    librabbitmq


AMQP 0.9 — librabbitmq https://github.com/alanxz/rabbitmq-c







  • AMQP 1.0 — , . .


            Attach recvAttach = new Attach()
            {
                Source = new Source()
                {
                    Address = "orders-amqp-10-consumer",
                    Durable = 1,
                },
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    



            ReceiverLink receiver = 
                new ReceiverLink(session,"netcore_amqp_10_consumer", recvAttach, null);

            Console.WriteLine("Receiver connected to broker.");

            while (true) {
                Message message = receiver.Receive();
                if (message == null)
                {
                    Console.WriteLine("Client exiting.");
                    break;
                }
                Console.WriteLine("Received " 
                  + System.Text.Encoding.UTF8.GetString((byte[])message.Body)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    








  <ItemGroup>
    <PackageReference Include="AMQPNetLite.Core" Version="2.4.1" />
  </ItemGroup>


https://github.com/Azure/amqpnetlite Microsoft . AMQP 1.0 https://docs.microsoft.com/ru-ru/azure/service-bus-messaging/service-bus-amqp-overview













  • Kafka — . Exactly once.


                AutoOffsetReset = AutoOffsetReset.Earliest
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    


                c.Subscribe("orders-from-amqp");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    


                    while (true)
                    {
                        try
                        {
                            var cr = c.Consume(cts.Token);


:







  • 5








  • 3








  • Kafka-Ui








  • RabbitMQ








Java ?



— , , Kafka-Connect-Base







[submodule "dockers/rabbitmq-kafka-sink"]
    path = dockers/rabbitmq-kafka-sink
    url = https://github.com/aliczin/kafka-connect-rabbitmq


, Kafka-Connect — .







:









public class RabbitMQSourceTask extends SourceTask {
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    




        this.channel.basicConsume(queue, this.consumer);
        log.info("Setting channel.basicQos({}, {});", this.config.prefetchCount, this.config.prefetchGlobal);
        this.channel.basicQos(this.config.prefetchCount, this.config.prefetchGlobal);


  • .


  @Override
  public List<SourceRecord> poll() throws InterruptedException {
    List<SourceRecord> batch = new ArrayList<>(4096);

    while (!this.records.drain(batch)) {


AMQP 0.9 . Java . J2EE.







  private static final Logger log = LoggerFactory.getLogger(MessageConverter.class);
  static final String FIELD_ENVELOPE_DELIVERYTAG = "deliveryTag";
  static final String FIELD_ENVELOPE_ISREDELIVER = "isRedeliver";
  static final String FIELD_ENVELOPE_EXCHANGE = "exchange";
  static final String FIELD_ENVELOPE_ROUTINGKEY = "routingKey";

  static final Schema SCHEMA_ENVELOPE = SchemaBuilder.struct()
      .name("com.github.jcustenborder.kafka.connect.rabbitmq.Envelope")
      .doc("Encapsulates a group of parameters used for AMQP's Basic methods. See " +
          "`Envelope <https://www.rabbitmq.com/releases/rabbitmq-java-client/current-javadoc/com/rabbitmq/client/Envelope.html>`_")
      .field(FIELD_ENVELOPE_DELIVERYTAG, SchemaBuilder.int64().doc("The delivery tag included in this parameter envelope. See `Envelope.getDeliveryTag() <https://www.rabbitmq.com/releases/rabbitmq-java-client/current-javadoc/com/rabbitmq/client/Envelope.html#getDeliveryTag-->`_").build())
      .field(FIELD_ENVELOPE_ISREDELIVER, SchemaBuilder.bool().doc("The redelivery flag included in this parameter envelope. See `Envelope.isRedeliver() <https://www.rabbitmq.com/releases/rabbitmq-java-client/current-javadoc/com/rabbitmq/client/Envelope.html#isRedeliver-->`_").build())
      .field(FIELD_ENVELOPE_EXCHANGE, SchemaBuilder.string().optional().doc("The name of the exchange included in this parameter envelope. See `Envelope.getExchange() <https://www.rabbitmq.com/releases/rabbitmq-java-client/current-javadoc/com/rabbitmq/client/Envelope.html#getExchange-->`_"))
      .field(FIELD_ENVELOPE_ROUTINGKEY, SchemaBuilder.string().optional().doc("The routing key included in this parameter envelope. See `Envelope.getRoutingKey() <https://www.rabbitmq.com/releases/rabbitmq-java-client/current-javadoc/com/rabbitmq/client/Envelope.html#getRoutingKey-->`_").build())
      .build();


… , — . .









Github.







— https://github.com/aliczin/hybrid-eventing. Creative Commons Attribution 4.0 International.







— DevOps . , — , .







" " () — 







orderEventsApp->Amqp09: send order
Amqp09->Amqp10: fanout\n copy event
Amqp09->KafkaQ: fanout\n copy event
KafkaQ->KafkaConnect: consume\n on message
KafkaConnect->KafkaConnect: transform\n message
KafkaConnect->Kafka: publish to topic








— 







Amqp10->orderEventSubApp: subcribe\n for event
orderJournalApp->Kafka: read kafka journal


















  • Apache Kafka Java , librdkafka — KafkaAPI



    . Java .







  • , RabbitMQ/Kafka/Nats/ActiveMQ — -.







  • Docker, .

















:







  • Mosquito — SCADA ModBus/OPC-UA. — " " — https://github.com/mainflux/mainflux







  • ActiveMQ — Java , Erlang, — RabbitMQ AMQP 1.0 -> ActiveMQ



    RabbitMQ, JMS.







  • NATs — OpenFaaS



    , " " Amazon Lambda



    . — : https://github.com/nats-io/nats-kafka — OpenFaaS 1- — 2.5 https://youtu.be/8sF-oGGVa9M









(/ — : ) /, - , , . " " 







:  
          Produser/Consumer
        
           
              

: vmWare   Stream  RabbitMQ 
     vmWare     
         
         
            RabbitMQ
              

:   1-   ActiveMQ    1
          1   
            Kafka API
        ActivemeMQ2Kafka
              1

etc


, — — : https://github.com/fclairamb/ftpserver/pull/34 — FTP , S3.







— : : .







  • . DevOps k8s, OpenShift, etc — , - .
  • — PRODUCTION-READY .




( ) , - :







HTTP, AMQP 0.9, AMQP 1.0, Apache Kafka 23, MQTT, WebSockets, <SOAP>

. 1 — . Google 1+RabbitMQ 1+Kafka 1+OpenFaas



— RabbitMQ Kafka " 1" . 1 — , . Java/C#/Python/C++/Rust/etc.







https://shd101wyy.github.io/markdown-preview-enhanced Visual Studio Code — .







Well, as a final point, I would like to note that the choice of the JDK ecosystem Cunfluent Inc



as a development platform Kafka-Connect



looks all the same strange. I wouldn't be surprised if their competitors do the same, but on GoLang, NodeJS (something like Kafka-Beats-Hub



)













I make pretty pictures in GraphViz format using the clever project Docker2GraphViz - it helps to keep the outline and technical documentation in Markdown format up to date







set CURPATH=%~dp0
set DOCKER_DIR=%CURPATH%\dockers

docker run --rm -it --name dcv -v %DOCKER_DIR%\:/input pmsipilot/docker-compose-viz render -m image --force --output-file=infra-topology.png infra.yml
docker run --rm -it --name dcv -v %CURPATH%\:/input pmsipilot/docker-compose-viz render -m image --force --output-file=apps-topology.png applications.yml

copy /b/v/y dockers\infra-topology.png content\assets\infra-topology.png
copy /b/v/y apps-topology.png content\assets\apps-topology.png


More articles:


All Articles