Knight's move: how to receive messages in Kafka via Nginx

Hello, Habr! I am Alexey Kashavkin, a cloud operations engineer at G-Core Labs, and have been administering OpenStack for the past five years. Today I will talk about a crutch for non-standard use of technologies or how to put Kafka brokers behind Nginx. If for some reason you have no other way to receive messages - this note is just for you.

Why is it needed

, Kafka, . — — PoC- , Kafka c Elasticsearch. (SSL/TLS), PEM- Kafka . , Puppet. Nginx. , , Kafka - , . — , - - .

, - , legacy-, , - HTTP, .

, . Nginx:

1. echo

   
   
   
   
   

2. kafka-log

   
   
   
   
   

, . c , c :

load_module /etc/nginx/modules/ngx_http_echo_module.so;
load_module /etc/nginx/modules/ngx_http_kafka_log_module.so;

http {
    
  kafka_log_kafka_brokers kafka1:9092,kafka2:9092,kafka3:9092;
    
  server {
    listen	443 ssl; 

    ssl_certificate 	/etc/nginx/certs/your_cert.crt;
    ssl_certificate_key /etc/nginx/certs/your_cert_key.key;
          
    if ($request_uri ~* ^/(.*)$) {
      set $request_key $1;
    }
        
    location / {
      kafka_log kafka:$request_key $request_body;
      echo_read_request_body;
    }
  }
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

:

1. :

   
   
   
   
   

   • ngx_http_echo_module.so proxy_pass
     
     
     
     , fastcgi_pass
     
     
     
     , uwsgi_pass



scgi_pass
     
     
     
     ;

     
     
     
     
     

   • ngx_http_kafka_log_module.so — Nginx Kafka.

     
     
     
     
     

2. kafka_log_kafka_brokers — Kafka.

   
   
   
   
   

3. http c server, $request_uri
   
   
   
   . , “/”



$request_uri
   
   
   
   , rewrite
   
   
   
   — return
   
   
   
   echo_read_request_body. $request_key
   
   
   
   . — /topic_name, — topic_name.

   
   
   
   
   

4. location :

   
   
   
   
   

   • kafka_log — Kafka ;

     
     
     
     
     

   • echo_read_request_body— , $request_body
     
     
     
     .

     
     
     
     
     

— Kafka, URL https://example.com/topic_name

P.S. , . Kafka.