Clever Geek Handbook

Validate Kubernetes YAML for Best Practices and Policies

Approx. transl. : With the growth in the number of YAML configurations for K8s environments, the need for their automated verification becomes more and more urgent. The author of this review not only selected the existing solutions for this task, but also looked at how they work using the example of Deployment. It turned out very informative for those who are interested in this topic.





TL; DR : This article compares six static tools for validating and evaluating Kubernetes YAML files against best practices and requirements.



Kubernetes workloads are typically defined in the form of YAML documents. One of the problems with YAML is the difficulty of specifying constraints or relationships between manifest files.



What if we need to make sure that all images deployed to the cluster are from a trusted registry?



Deployment', PodDisruptionBudgets?



. , production- .



YAML- Kubernetes :



  • API-. YAML- API- Kubernetes.
  • . , ..
  • . , , Rego Javascript.


:



  1. kubeval;
  2. kube-score;
  3. config-lint;
  4. copper;
  5. conftest;
  6. Polaris.


, !



Deployment'



, , .



: ?



apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http-echo
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
        image: hashicorp/http-echo
        args: ["-text", "hello-world"]
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo
spec:
  ports:
  - port: 5678
    protocol: TCP
    targetPort: 5678
  selector:
    app: http-echo


(base-valid.yaml)



YAML .



base-valid.yaml Git-.

-, — «Hello World» 5678. :



kubectl apply -f hello-world.yaml


— :



kubectl port-forward svc/http-echo 8080:5678


http://localhost:8080 , . ? .



1. Kubeval



kubeval , Kubernetes REST API. , API , YAML. .



kubeval .



0.15.0.

«» , :



$ kubeval base-valid.yaml
PASS - base-valid.yaml contains a valid Deployment (http-echo)
PASS - base-valid.yaml contains a valid Service (http-echo)


kubeval exit- 0. :



$ echo $?
0


kubeval :



apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
        image: hashicorp/http-echo
        args: ["-text", "hello-world"]
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo
spec:
  ports:
  - port: 5678
    protocol: TCP
    targetPort: 5678
  selector:
    app: http-echo


(kubeval-invalid.yaml)



? :



$ kubeval kubeval-invalid.yaml
WARN - kubeval-invalid.yaml contains an invalid Deployment (http-echo) - selector: selector is required
PASS - kubeval-invalid.yaml contains a valid Service (http-echo)

#   
$ echo $?
1


.



Deployment', API apps/v1, , pod'. , kubeval .



, , kubectl apply -f ?



, :



$ kubectl apply -f kubeval-invalid.yaml
error: error validating "kubeval-invalid.yaml": error validating data: ValidationError(Deployment.spec):
missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors,
turn validation off with --validate=false


, kubeval. , :



apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  selector:          # !!!
    matchLabels:     # !!!
      app: http-echo # !!!
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
        image: hashicorp/http-echo
        args: ["-text", "hello-world"]
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo
spec:
  ports:
  - port: 5678
    protocol: TCP
    targetPort: 5678
  selector:
    app: http-echo


(base-valid.yaml)



kubeval , .



, : .



kubeval Kubernetes API. Kubernetes. --kubernetes-version:



$ kubeval --kubernetes-version 1.16.1 base-valid.yaml


, Major.Minor.Patch.



, , JSON- GitHub, kubeval . kubeval , --schema-location.



YAML-, kubeval stdin.



, Kubeval CI. , kubeval :



  1. ;
  2. JSON;
  3. Test Anything Protocol (TAP).


, .



kubeval — Custom Resource Definitions (CRDs). kubeval .



Kubeval — ; , , , .



, latest . kubeval . YAML .



, YAML latest? YAML- ?



2. Kube-score



Kube-score YAML . , :



  • root'.
  • pod'.
  • request' limit' .


: OK, WARNING CRITICAL.



Kube-score .



kube-score 1.7.0.

base-valid.yaml:



$ kube-score score base-valid.yaml

apps/v1/Deployment http-echo
[CRITICAL] Container Image Tag
  · http-echo -> Image with latest tag
      Using a fixed tag is recommended to avoid accidental upgrades
[CRITICAL] Pod NetworkPolicy
  · The pod does not have a matching network policy
      Create a NetworkPolicy that targets this pod
[CRITICAL] Pod Probes
  · Container is missing a readinessProbe
      A readinessProbe should be used to indicate when the service is ready to receive traffic.
      Without it, the Pod is risking to receive traffic before it has booted. It is also used during
      rollouts, and can prevent downtime if a new version of the application is failing.
      More information: https://github.com/zegl/kube-score/blob/master/README_PROBES.md
[CRITICAL] Container Security Context
  · http-echo -> Container has no configured security context
      Set securityContext to run the container in a more secure context.
[CRITICAL] Container Resources
  · http-echo -> CPU limit is not set
      Resource limits are recommended to avoid resource DDOS. Set resources.limits.cpu
  · http-echo -> Memory limit is not set
      Resource limits are recommended to avoid resource DDOS. Set resources.limits.memory
  · http-echo -> CPU request is not set
      Resource requests are recommended to make sure that the application can start and run without
      crashing. Set resources.requests.cpu
  · http-echo -> Memory request is not set
      Resource requests are recommended to make sure that the application can start and run without crashing.
      Set resources.requests.memory
[CRITICAL] Deployment has PodDisruptionBudget
  · No matching PodDisruptionBudget was found
      It is recommended to define a PodDisruptionBudget to avoid unexpected downtime during Kubernetes
      maintenance operations, such as when draining a node.
[WARNING] Deployment has host PodAntiAffinity
  · Deployment does not have a host podAntiAffinity set
      It is recommended to set a podAntiAffinity that stops multiple pods from a deployment from
      being scheduled on the same node. This increases availability in case the node becomes unavailable.


YAML kubeval, kube-score :



  • .
  • request' limit' CPU .
  • Pod disruption budgets.
  • (anti-affinity) .
  • root'.


— , , Deployment .



kube-score WARNING CRITICAL, .



CI --output-format ci ( OK):



$ kube-score score base-valid.yaml --output-format ci

[OK] http-echo apps/v1/Deployment
[OK] http-echo apps/v1/Deployment
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) CPU limit is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Memory limit is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) CPU request is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Memory request is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Image with latest tag
[OK] http-echo apps/v1/Deployment
[CRITICAL] http-echo apps/v1/Deployment: The pod does not have a matching network policy
[CRITICAL] http-echo apps/v1/Deployment: Container is missing a readinessProbe
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Container has no configured security context
[CRITICAL] http-echo apps/v1/Deployment: No matching PodDisruptionBudget was found
[WARNING] http-echo apps/v1/Deployment: Deployment does not have a host podAntiAffinity set
[OK] http-echo v1/Service
[OK] http-echo v1/Service
[OK] http-echo v1/Service
[OK] http-echo v1/Service


kubeval, kube-score , CRITICAL. WARNING.



, API ( kubeval). 'hardcode' kube-score: Kubernetes . , K8s.



, issue .

kube-score .



kube-score — , , ? , .



Kube-score : .



, , : config-lint, copper, conftest polaris.



3. Config-lint



Config-lint — YAML, JSON, Terraform, CSV Kubernetes.



.



— 1.5.0.

Config-lint Kubernetes.



. YAML-, « » (rulesets), :



version: 1
description: Rules for Kubernetes spec files
type: Kubernetes
files:
  - "*.yaml"
rules:
   #


(rule.yaml)



:



  • type , config-lint. K8s Kubernetes.
  • files .
  • rules .


, , Deployment' my-company.com/myapp:1.0. config-lint, , :



- id: MY_DEPLOYMENT_IMAGE_TAG
  severity: FAILURE
  message: Deployment must use a valid image tag
  resource: Deployment
  assertions:
    - every:
        key: spec.template.spec.containers
        expressions:
          - key: image
            op: starts-with
            value: "my-company.com/"


(rule-trusted-repo.yaml)



:



  • id — ;
  • severity — FAILURE, WARNING NON_COMPLIANT;
  • message — ;
  • resource — , ;
  • assertions — , .


assertion every , Deployment' (key: spec.templates.spec.containers) (.., my-company.com/).



ruleset :



version: 1
description: Rules for Kubernetes spec files
type: Kubernetes
files:
  - "*.yaml"
rules:

 - id: DEPLOYMENT_IMAGE_REPOSITORY # !!!
    severity: FAILURE
    message: Deployment must use a valid image repository
    resource: Deployment
    assertions:
      - every:
          key: spec.template.spec.containers
          expressions:
            - key: image
              op: starts-with
              value: "my-company.com/"


(ruleset.yaml)



, check_image_repo.yaml. base-valid.yaml:



$ config-lint -rules check_image_repo.yaml base-valid.yaml

[
  {
  "AssertionMessage": "Every expression fails: And expression fails: image does not start with my-company.com/",
  "Category": "",
  "CreatedAt": "2020-06-04T01:29:25Z",
  "Filename": "test-data/base-valid.yaml",
  "LineNumber": 0,
  "ResourceID": "http-echo",
  "ResourceType": "Deployment",
  "RuleID": "DEPLOYMENT_IMAGE_REPOSITORY",
  "RuleMessage": "Deployment must use a valid image repository",
  "Status": "FAILURE"
  }
]


. :



apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http-echo
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
         image: my-company.com/http-echo:1.0 # !!!
         args: ["-text", "hello-world"]
         ports:
         - containerPort: 5678


(image-valid-mycompany.yaml)



. :



$ config-lint -rules check_image_repo.yaml image-valid-mycompany.yaml
[]


Config-lint — , YAML- Kubernetes YAML DSL.



, ? YAML ? , ?



4. Copper



Copper V2 — ( config-lint).



, YAML . JavaScript. Copper , Kubernetes .



Copper .



2.0.1 — .

config-lint, Copper . . , deployment' my-company.com.



check_image_repo.js :



$$.forEach(function($){
    if ($.kind === 'Deployment') {
        $.spec.template.spec.containers.forEach(function(container) {
            var image = new DockerImage(container.image);
            if (image.registry.lastIndexOf('my-company.com/') != 0) {
                errors.add_error('no_company_repo',"Image " + $.metadata.name + " is not from my-company.com repo", 1)
            }
        });
    }
});


, base-valid.yaml, copper validate:



$ copper validate --in=base-valid.yaml --validator=check_image_tag.js

Check no_company_repo failed with severity 1 due to Image http-echo is not from my-company.com repo
Validation failed


, copper — , Ingress pod', .



Copper :



  • DockerImage :

    • name — ,
    • tag — ,
    • registry — ,
    • registry_url — (https://) ,
    • fqin — .
  • findByName (kind) (name) .
  • findByLabels (kind) (labels).


.



YAML- $$ ( , jQuery).



Copper : JavaScript , , ..



, Copper ES5 JavaScript, ES6.

.



, JavaScript , , conftest.



5. Conftest



Conftest — . / Kubernetes. Rego.



conftest , .



0.18.2.

config-lint copper, conftest - . . , , .



conftest-checks, — check_image_registry.rego :



package main

deny[msg] {

  input.kind == "Deployment"
  image := input.spec.template.spec.containers[_].image
  not startswith(image, "my-company.com/")
  msg := sprintf("image '%v' doesn't come from my-company.com repository", [image])
}


base-valid.yaml conftest:



$ conftest test --policy ./conftest-checks base-valid.yaml

FAIL - base-valid.yaml - image 'hashicorp/http-echo' doesn't come from my-company.com repository
1 tests, 1 passed, 0 warnings, 1 failure


, .



Rego deny. . deny , conftest , .



conftest JSON, TAP — , CI. --output.



conftest --trace. , conftest .



conftest OCI- (Open Container Initiative) .



push pull . Docker conftest push.



Docker:



$ docker run -it --rm -p 5000:5000 registry


conftest-checks :



$ conftest push 127.0.0.1:5000/amitsaha/opa-bundle-example:latest


, :



2020/06/10 14:25:43 pushed bundle with digest: sha256:e9765f201364c1a8a182ca637bc88201db3417bacc091e7ef8211f6c2fd2609c


conftest pull. , :



$ cd $(mktemp -d)
$ conftest pull 127.0.0.1:5000/amitsaha/opa-bundle-example:latest


policy, :



$ tree
.
└── policy
  └── check_image_registry.rego


:



$ conftest test --update 127.0.0.1:5000/amitsaha/opa-bundle-example:latest base-valid.yaml
..
FAIL - base-valid.yaml - image 'hashicorp/http-echo' doesn't come from my-company.com repository
2 tests, 1 passed, 0 warnings, 1 failure


, DockerHub . , , Azure Container Registry (ACR) .



— , Open Policy Agent (OPA), conftest OPA.



conftest .



6. Polaris



, , — Polaris. ( — . .)



Polaris . , Kubernetes.



, , ( kube-score). , ( config-lint, copper conftest).



, Polaris : .



Polaris .



1.0.3.

polaris base-valid.yaml :



$ polaris audit --audit-path base-valid.yaml


JSON . :



{
  "PolarisOutputVersion": "1.0",
  "AuditTime": "0001-01-01T00:00:00Z",
  "SourceType": "Path",
  "SourceName": "test-data/base-valid.yaml",
  "DisplayName": "test-data/base-valid.yaml",
  "ClusterInfo": {
    "Version": "unknown",
    "Nodes": 0,
    "Pods": 2,
    "Namespaces": 0,
    "Controllers": 2
  },
  "Results": [
    /*   */
  ]
}


.



kube-score, Polaris , :



  • pod'.
  • .
  • root'.
  • request' limit' CPU.


: warning danger. , .



, --format score. Polaris 1 100 — score (.. ):



$ polaris audit --audit-path test-data/base-valid.yaml --format score
68


100, . exit- polaris audit, , 0.



polaris audit :



  • --set-exit-code-below-score 1-100. exit- 4, . , (, 75), alert, .
  • --set-exit-code-on-danger , 3, danger- .


, , . YAML, JSON Schema.



YAML- , checkImageRepo:



checkImageRepo:
  successMessage: Image registry is valid
  failureMessage: Image registry is not valid
  category: Images
  target: Container
  schema:
    '$schema': http://json-schema.org/draft-07/schema
    type: object
    properties:
      image:
        type: string
        pattern: ^my-company.com/.+$


:



  • successMessage — , ;
  • failureMessage — ;
  • category — : Images, Health Checks, Security, Networking Resources;
  • target—- , (spec) . : Container, Pod Controller;
  • schema JSON schema. pattern .


Polaris:



checks:
  checkImageRepo: danger
customChecks:
  checkImageRepo:
    successMessage: Image registry is valid
    failureMessage: Image registry is not valid
    category: Images
    target: Container
    schema:
      '$schema': http://json-schema.org/draft-07/schema
      type: object
      properties:
        image:
          type: string
          pattern: ^my-company.com/.+$


(polaris-conf.yaml)



:



  • checks . , , danger.
  • checkImageRepo customChecks.


custom_check.yaml. polaris audit YAML-, .



base-valid.yaml:



$ polaris audit --config custom_check.yaml --audit-path base-valid.yaml


polaris audit , , .



my-company.com/http-echo:1.0, Polaris . , image-valid-mycompany.yaml.



: ? ! . :



checks:
  cpuRequestsMissing: warning
  cpuLimitsMissing: warning
  # Other inbuilt checks..
  # ..
  # custom checks
  checkImageRepo: danger # !!!
customChecks:
  checkImageRepo:        # !!!
    successMessage: Image registry is valid
    failureMessage: Image registry is not valid
    category: Images
    target: Container
    schema:
      '$schema': http://json-schema.org/draft-07/schema
      type: object
      properties:
        image:
          type: string
          pattern: ^my-company.com/.+$


(config_with_custom_check.yaml)



.



base-valid.yaml, , :



$ polaris audit --config config_with_custom_check.yaml --audit-path base-valid.yaml


Polaris , .



, , Rego JavaScript, , .



Polaris .





YAML- Kubernetes, , .



, Kubernetes, , kubeval . , API Kubernetes.



, . kube-score Polaris.



, , copper, config-lint conftest.



Conftest config-lint YAML , copper , .



, , , , Polaris, , ? .



:
kubeval YAML- API CRD
kube-score YAML API Kubernetes
copper JavaScript- YAML- .
config-lint - , YAML. (, Terraform) . assertions
conftest Rego ( ). OCI bundles . Rego. Docker Hub
Polaris YAML- . JSON Schema , JSON Schema,


Kubernetes, . pull request' .



P.S.



:





More articles:


All Articles