Building Kubernetes clusters using Kubernetes itself

Do you think I'm out of my mind? I have already experienced this reaction when I first suggested deploying Kubernetes clusters using Kubernetes.

But I am convinced that there is no more effective tool for automating cloud infrastructure than Kubernetes itself. With one central K8s cluster, we can create hundreds of other controlled K8s clusters. In this article, I will show you how to do this.

. SAP Concur AWS EKS, Google GKE, Azure AKS Kubernetes .

Kubernetes . , AWS EKS :

$ eksctl create cluster

, Kubernetes,  — «production-ready» «production-ready» -, SAP Concur Kubernetes, .

• .  AWS, . , IP- , AWS, SSM .

  
  
  
  
  

• EKS .  AWS EKS .

  
  
  
  
  

• .  . :) , Istio, Logging Integration, Autoscaler .

  
  
  
  
  

• .  ( EKS ) . , . (, !)

  
  
  
  
  

( ). , , , - , , .

Argo, Argo Events Argo Workflows. Kubernetes CRD YAML, Kubernetes.

:

Argo Workflows

Argo Workflows — , Kubernetes. Argo Workflows Kubernetes CRD.

.  K8s YAML, , .

, Argo Workflows.

1.

BATS. BATS :

#!/usr/bin/env bats
@test “More than 100 available IP addresses in subnet MySubnet” {
AvailableIpAddressCount=$(aws ec2 describe-subnets --subnet-ids MySubnet | jq -r ‘.Subnets[0].AvailableIpAddressCount’)

 [ “${AvailableIpAddressCount}” -gt 100 ]
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

BATS (avail-ip-addresses.bats



) BATS Argo Workflows :

— name: preflight-tests
  templateRef: 
    name: argo-templates
    template: generic-template
  arguments:
    parameters:
    — name: command
      value: “{{item}}”
  withItems:
  — bats /tests/preflight/accnt-name-export.bats”
  — bats /tests/preflight/avail-ip-addresses.bats”
  — bats /tests/preflight/dhcp.bats”
  — bats /tests/preflight/subnet-export.bats”
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

2. EKS

EKS . , eksctl



, CloudFormation Terraform. EKS Argo Workflows CloudFormation (eks-controlplane.yaml



eks-nodegroup.yaml



) .

— name: eks-controlplane
  dependencies: [“preflight-tests”]
  templateRef: 
    name: argo-templates
    template: generic-template
 arguments:
   parameters:
   — name: command
     value: |
       aws cloudformation deploy \
       --stack-name {{workflow.parameters.CLUSTER_NAME}} \
       --template-file /eks-core/eks-controlplane.yaml \
       --capabilities CAPABILITY_IAM
- name: eks-nodegroup
  dependencies: [“eks-controlplane”]
  templateRef: 
    name: argo-templates
    template: generic-template
  arguments:
    parameters:
    — name: command
      value: |
        aws cloudformation deploy \
        --stack-name {{workflow.parameters.CLUSTER_NAME}}-nodegroup \
        --template-file /eks-core/eks-nodegroup.yaml \
        --capabilities CAPABILITY_IAM
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

3.

kubectl



, helm



, kustomize



. , metrics-server



helm



kubectl



, metrics-server



, Argo Workflows .

— name: metrics-server
  dependencies: [“eks-nodegroup”]
  templateRef: 
    name: argo-templates
    template: generic-template
  when: “‘{{workflow.parameters.METRICS-SERVER}}’ != none”
  arguments:
    parameters:
    — name: command
      value: |
        helm template /addons/{{workflow.parameters.METRICS-SERVER}}/ \
        --name “metrics-server” \
        --namespace “kube-system” \
        --set global.registry={{workflow.parameters.CONTAINER_HUB}} | \
        kubectl apply -f -
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

4.

BATS- DETIK, K8s.

#!/usr/bin/env bats
load “lib/utils”
load “lib/detik”
DETIK_CLIENT_NAME=”kubectl”
DETIK_CLIENT_NAMESPACE="kube-system"
@test “verify the deployment metrics-server” {
 
 run verify “there are 2 pods named ‘metrics-server’”
 [ “$status” -eq 0 ]
 
 run verify “there is 1 service named ‘metrics-server’”
 [ “$status” -eq 0 ]
 
 run try “at most 5 times every 30s to find 2 pods named ‘metrics-server’ with ‘status’ being ‘running’”
 [ “$status” -eq 0 ]
 
 run try “at most 5 times every 30s to get pods named ‘metrics-server’ and verify that ‘status’ is ‘running’”
 [ “$status” -eq 0 ]
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

BATS DETIK (metrics-server.bats



), metrics-server



, Argo Workflows :

— name: test-metrics-server
  dependencies: [“metrics-server”]
  templateRef:
    name: worker-containers
    template: addons-tests-template
  when: “‘{{workflow.parameters.METRICS-SERVER}}’ != none”
  arguments:
    parameters:
    — name: command
      value: |
        bats /addons/test/metrics-server.bats
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

, . Sonobuoy, Popeye Fairwinds Polaris? Argo Workflows!

, AWS EKS metrics-server



. , . !

 — .

Argo Workflows (WorkflowTemplates). . , , . ( ) . Argo Events.

Argo Events

Argo Events — - Kubernetes, K8s, Argo Workflows, , -, S3, , , Google Cloud Pub/Sub, SNS, SQS .

API- (Argo Events) JSON. , (WorkflowTemplates) API. Kubernetes (  ) :

• , ? API .

  
  
  
  
  

• «» EKS? API eks-core (control-plane nodegroup).

  
  
  
  
  

• EKS? API .

  
  
  
  
  

• ? API .

  
  
  
  
  

Argo

Argo Events Argo Workflows « », .

:

• 
  
  
  
  

• 
  
  
  
  

• (. : , Argo )

  
  
  
  
  

• 
  
  
  
  

• S3

  
  
  
  
  

• 
  
  
  
  

• 
  
  
  
  

. , Argo Events Workflows.  — . .

---

« Kubernetes». - «, k8s». 8. , .