Callisto. Why did we come up with a Selenium Grid replacement

Habré has already written more than once that Selenium Grid has problems that cannot be solved in a simple way (for example: one , two , three ). In this article, we'll share our experience and share how we at Wrike managed to build a stable framework for Selenium tests.

TLDR: We wrote our own open source solution and completely replaced Selenium Grid with it.

We have already talked about how we scaled our Selenium farm using Google Cloud Engine and Kubernetes. We got rid of the queues for launching tests, but the QA department regularly received complaints about the instability of the test infrastructure.

Choosing a path

Selenium Grid GKE , . GKE Preemptible VMs. , , . . , , Selenium Grid, . , .

Selenoid Aerokube, , Selenium Grid (Selenium Hub Selenium Node). , , Kubernetes, : .

, Kubernetes, Zalnium, jsonwire-grid Moon. Zalenium jsonwire-grid Selenium Node, . Zalenium . 

Moon , . : , Moon , ( ). , - — Selenium Grid, , .

, . , , .

«»?

Selenium Grid . , . Selenium , W3C Webdriver. 

, HTTP API ( W3C Webdriver) ( -). , Google Chrome ChromeDriver, Mozilla Firefox — geckodriver . . W3C — -.

, . , capabilities , .

2 — / . Callisto — open source , . / , Kubernetes, Nginx. .

Callisto

Callisto : Nginx, Callisto pod’ -. 

Nginx 2 :

1. / Callisto.

   
   
   
   
   

2. -.

   
   
   
   
   

Callisto :

1. pod -.

   
   
   
   
   

2. - .

   
   
   
   
   

:

1. .

   
   
   
   
   

2. pod .

   
   
   
   
   

Docker- Selenoid.

pod’ . - session_id, -, pod’. , — Callisto. , Callisto stateless-.

: , - , Callisto session_id -, ip- pod’, . session_id — pod_ip .

, W3C Webdriver session_id URL-, Nginx. Nginx , session_id, ip- pod’ session_id pod’ session_id.

: pod’ Callisto .

:

Callisto

Web UI. Web UI Selenoid UI. 

Callisto :

• Selenium-.

  
  
  
  
  

• VNC.

  
  
  
  
  

• - .

  
  
  
  
  

.

: Selenoid UI. , .

. Callisto Prometheus: Selenium-.

Callisto, Selenoid, . . , . 

, - pod’ -. Kubernetes- , pod’.

pod’ , ~10 ( 2- , Kubernetes- ).

Callisto

Callisto — 2000 . , . , . , - .

499- Nginx. Nginx 499- . : Kubernetes- ( ) , . “Server disconnected error”.

Nginx , , , .

, worker-shutdown-timeout. 10 ( -) 120 .

: ingress-nginx 0.26.0 - 10 240 .

. Kubernetes- pod’ , session_id.

— session_id, , Nginx .

: ( GKE 1.12):

1000+ , . session_id, .

: pod’ , .

Kubernetes-API. 100 Kubernetes-API , , . , Callisto Kubernetes-API . . 

pod’ . / pod’ ~500 . 1000 .

GKE 1.12 1.14. 1.14 pod’ 10 2000 .

504- Nginx. - , Nginx 504- . , Preemptible VMs. Google VM, pod’ VM , .

Nginx ( ) : .

Callisto

, .

, , GKE 30-40% . , Callisto , pod’ , . Selenium Grid pod' , . pod' , .. Selenium Node. 

Selenium Grid Callisto:

:

• Callisto

  
  
  
  
  

• Helm chart

  
  
  
  
  

Callisto, . , -.

.