You (probably) need liveness and readiness probes

One of the most common questions that I get asked as a consultant is: "What is the difference between liveness and readiness samples?" The next most common question is: "Which ones do my application need?"





Anyone who has tried this question behind Duck Duck Go knows that it is not easy to find an answer to it on the internet. In this article, I hope I can help you answer these questions yourself. I will share my opinion on how best to use liveness and readiness of probes in applications deployed in Red Hat OpenShift . And I'm not suggesting a rigorous algorithm, but rather a general framework that you can use to make your own architectural decisions.





To make the abstractions more specific, I suggest looking at four general application examples. For each of them, we will find out whether we need to configure the liveness and readiness probes, and if necessary, how. Before moving on to examples, let's take a closer look at these two different types of samples.





: Kubernetes “startup” probe, OpenShift 4.5 clusters. startup probe, liveness readiness probes. startup probes.





Liveness readiness probes

Liveness () readiness () , OpenShift. , .





liveness , OpenShift’, . readiness , OpenShift’ , . , .





liveness , readiness , OpenShift , , , . , . ( - -, , , legacy-.)





. , , OpenShift.





liveness ?

Liveness OpenShift’, ( ), ( ). , OpenShift , . , OpenShift .





liveness ( ) . “” “” : “ ?”.





liveness ?

liveness , OpenShift PID 1. PID 1 , . , PID 1





PID 1 liveness , OpenShift ( ), . - . PID 1 , , , OpenShift .





, PID 1, , , liveness . init , tini dumb-init, , . , liveness , , .





readiness ?

OpenShift readiness ( ) , . , , ( ), , readiness . OpenShift , . OpenShift , , ( ) 502 , “connection refused”.





liveness , readiness ( ) . : “ ?”.





readiness ?

readiness , OpenShift , PID 1 . .





(, 502 OpenShift) , . readiness , . , , . , , , .





readiness . OpenShift' , .





, , .





: - , . - : OpenShift . liveness readiness OpenShift Container-as-a-Service, .





1: (Nginx)

Figure:  1: An example of server implementation for serving Nginx statics
. 1: Nginx

, 1 - , Nginx, . , : , 200 HTTP .





liveness ?

, , . , , liveness . Nginx , (, , SELinux Nginx, ).





readiness ?

Nginx readiness . , readiness , . Nginx , , , , best practice.





readiness

, , Deployment



. , . , template.





apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: application-nginx
  name: application-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: application-nginx
  template:
    metadata:
      labels:
        app: application-nginx
    spec:
      #  Will appear below as it changes
      
      



:





    spec:
      containers:
      - image: quay.io/<username>/nginx:latest
        name: application-nginx
        imagePullPolicy: Always
        ports:
        - containerPort: 8443
          protocol: TCP
        readinessProbe:
          httpGet:
            scheme: HTTPS
            path: /index.html
            port: 8443
          initialDelaySeconds: 10
          periodSeconds: 5
      
      



Figure:  Step 2: Implementing an Nginx Server for Serving Statics with Configured Readiness Checks
. 2: Nginx readiness

2: ( REST API)

HTTP -, “”. “” readiness , . , liveness . , , , . , , liveness . 





. 3 . .





Figure:  3: Implementing a job server without liveness checks.
. 3: liveness .

, liveness , .





liveness ? 

, . , , - . , PID 1.





PID 1, , . liveness OpenShift PID 1 . .





, . , deadlock, , , .





deadlock, /tmp/jobs.update



. shell ( exec liveness ) , , . liveness /usr/bin/my-application-jobs --alive



.





liveness ( , YAML- Deployment’a



, ):





    spec:
      containers:
      - image: quay.io/<username>/my-application-jobs:latest
        name: my-application-jobs
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - "/usr/bin/my-application-jobs --alive"
          initialDelaySeconds: 10
          periodSeconds: 5
      
      



readiness ?

readiness . , readiness OpenShift , . , . readiness . . 4 liveness .





Figure:  4: Implementing a Job Server with Liveness Checks
. 4: liveness

3: API

Figure:  5: SSR application without any checks
. 5: SSR -

SSR: HTML- , . Spring Boot, PHP, Ruby on Rails, Django, Node.js, .





liveness ?

, liveness , , . , liveness , , .





liveness exec, shell, , - . . , PID , , :





        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - "[ -f /run/my-application-web.pid ] && ps -A | grep my-application-web"
          initialDelaySeconds: 10
          periodSeconds: 5
      
      



readiness ?

readiness . readiness OpenShift , , . , , , , , , .





, OpenShift , . readiness , , OpenShift, :





        readinessProbe:
          httpGet:
            scheme: HTTPS
            path: /healthz
            port: 8443
          initialDelaySeconds: 10
          periodSeconds: 5
      
      



, YAML :





apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: backend-service
  name: backend-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend-service
  template:
    metadata:
      labels:
        app: backend-service
    spec:
      containers:
      - image: quay.io/<username>/backend-service:latest
        name: backend-service
        imagePullPolicy: Always
        ports:
        - containerPort: 8443
          protocol: TCP
        readinessProbe:
          httpGet:
            scheme: HTTPS
            path: /healthz
            port: 8443
          initialDelaySeconds: 10
          periodSeconds: 5
      
      



6 SSR liveness readiness .





Figure:  6. SSR application with customized liveness and readiness probes.
. 6. SSR liveness readiness .

4:

, . , , . .





Figure:  7: Realistic example application in OpenShift for learning the use of probes.
. 7: OpenShift .

, :





  • : REST API . , , , REST API.





  • Nginx: : (, JavaScript CSS). TLS- (Transport Layer Security) , URL. .





  • : , . , . , , .





:





  • : . - , .





  • . « - » (FIFO) . , .





:





  • Nginx TLS- . , HTTP. . volume space. , .





  • . , . , .





, . HTTP 8080 HTTPS 8443 readiness . liveness , , . , Kubelet', «»:





# Pod One - Application Server and Nginx
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-application-web
  name: my-application-web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-application-web
  template:
    metadata:
      labels:
        app: my-application-web
    spec:
      containers:
      - image: quay.io/<username>/my-application-nginx:latest
        name: my-application-nginx
        imagePullPolicy: Always
        ports:
        - containerPort: 8443
          protocol: TCP
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - "[ -f /run/nginx.pid ] && ps -A | grep nginx"
          initialDelaySeconds: 10
          periodSeconds: 5
        readinessProbe:
          httpGet:
            scheme: HTTPS
            path: /index.html
            port: 8443
          initialDelaySeconds: 10
          periodSeconds: 5
      - image: quay.io/<username>/my-application-app-server:latest
        name: my-application-app-server
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - "/usr/bin/my-application-web --alive"
          initialDelaySeconds: 10
          periodSeconds: 5
        readinessProbe:
          httpGet:
            scheme: HTTP
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5

# Pod Two - Jobs Server
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-application-jobs
  name: my-application-jobs
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-application-jobs
  template:
    metadata:
      labels:
        app: my-application-jobs
    spec:
      containers:
      - image: quay.io/<username>/my-application-jobs:latest
        name: my-application-jobs
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - "/usr/bin/my-application-jobs --alive"
          initialDelaySeconds: 10
          periodSeconds: 5
      
      



8 .





Figure:  8: Complete sample applications with both probes configured.
. 8: .

liveness readiness ?

, , . HTTP- , , , , , , OpenShift . , , .





HTTP endpoint, , liveness readiness endpoint. endpoint , , endpoint .





Liveness readiness OpenShift.  . liveness OpenShift, . readiness OpenShift .





, “” , . , , .





. , liveness , readiness . , .





, . - (SOA), . , readiness ? . / , , .





, . - ! . .








All Articles