![](https://habrastorage.org/getpro/habr/upload_files/93b/fe0/453/93bfe04537534e28df12b572ae529d8e.png)
Do you think I'm out of my mind? I have already experienced this reaction when I first suggested deploying Kubernetes clusters using Kubernetes.
But I am convinced that there is no more effective tool for automating cloud infrastructure than Kubernetes itself. With one central K8s cluster, we can create hundreds of other controlled K8s clusters. In this article, I will show you how to do this.
. SAP Concur AWS EKS, Google GKE, Azure AKS Kubernetes .
Kubernetes . , AWS EKS :
$ eksctl create cluster
, Kubernetes, â «production-ready» «production-ready» -, SAP Concur Kubernetes, .
. AWS, . , IP- , AWS, SSM .
EKS . AWS EKS .
. . :) , Istio, Logging Integration, Autoscaler .
. ( EKS ) . , . (, !)
( ). , , , - , , .
Argo, Argo Events Argo Workflows. Kubernetes CRD YAML, Kubernetes.
:
![K8s cluster ready for production. Built with Argo Workflows K8s cluster ready for production. Built with Argo Workflows](https://habrastorage.org/getpro/habr/upload_files/86d/f10/47e/86df1047e95859d3dfd9367a0eb780a8.png)
Argo Workflows
Argo Workflows â , Kubernetes. Argo Workflows Kubernetes CRD.
. K8s YAML, , .
, Argo Workflows.
1.
![Preliminary tests are run in parallel, retrying in case of failures Preliminary tests are run in parallel, retrying in case of failures](https://habrastorage.org/getpro/habr/upload_files/fcc/3a7/043/fcc3a7043bef2bd8f628a4f260c3c30f.png)
BATS. BATS :
#!/usr/bin/env bats
@test âMore than 100 available IP addresses in subnet MySubnetâ {
AvailableIpAddressCount=$(aws ec2 describe-subnets --subnet-ids MySubnet | jq -r â.Subnets[0].AvailableIpAddressCountâ)
[ â${AvailableIpAddressCount}â -gt 100 ]
}
BATS (avail-ip-addresses.bats
) BATS Argo Workflows :
â name: preflight-tests templateRef: name: argo-templates template: generic-template arguments: parameters: â name: command value: â{{item}}â withItems: â bats /tests/preflight/accnt-name-export.batsâ â bats /tests/preflight/avail-ip-addresses.batsâ â bats /tests/preflight/dhcp.batsâ â bats /tests/preflight/subnet-export.batsâ
2. EKS
![EKS control plane and node group with dependencies EKS control plane and node group with dependencies](https://habrastorage.org/getpro/habr/upload_files/9a8/1b0/df5/9a81b0df5cdce5aecc05ef2aa1b14e4e.png)
EKS . , eksctl
, CloudFormation Terraform. EKS Argo Workflows CloudFormation (eks-controlplane.yaml
eks-nodegroup.yaml
) .
â name: eks-controlplane dependencies: [âpreflight-testsâ] templateRef: name: argo-templates template: generic-template arguments: parameters: â name: command value: | aws cloudformation deploy \ --stack-name {{workflow.parameters.CLUSTER_NAME}} \ --template-file /eks-core/eks-controlplane.yaml \ --capabilities CAPABILITY_IAM - name: eks-nodegroup dependencies: [âeks-controlplaneâ] templateRef: name: argo-templates template: generic-template arguments: parameters: â name: command value: | aws cloudformation deploy \ --stack-name {{workflow.parameters.CLUSTER_NAME}}-nodegroup \ --template-file /eks-core/eks-nodegroup.yaml \ --capabilities CAPABILITY_IAM
3.
![Installing add-ons with dependencies in parallel Installing add-ons with dependencies in parallel](https://habrastorage.org/getpro/habr/upload_files/683/2de/190/6832de190b427cbc200d86687f9319b5.png)
kubectl
, helm
, kustomize
. , metrics-server
helm
kubectl
, metrics-server
, Argo Workflows .
â name: metrics-server
dependencies: [âeks-nodegroupâ]
templateRef:
name: argo-templates
template: generic-template
when: ââ{{workflow.parameters.METRICS-SERVER}}â != noneâ
arguments:
parameters:
â name: command
value: |
helm template /addons/{{workflow.parameters.METRICS-SERVER}}/ \
--name âmetrics-serverâ \
--namespace âkube-systemâ \
--set global.registry={{workflow.parameters.CONTAINER_HUB}} | \
kubectl apply -f -
4.
![Parallel cluster validation with retry on failure Parallel cluster validation with retry on failure](https://habrastorage.org/getpro/habr/upload_files/2d3/8cd/ab2/2d38cdab2c05f29a9ba13e06ca2cf34f.png)
BATS- DETIK, K8s.
#!/usr/bin/env bats
load âlib/utilsâ
load âlib/detikâ
DETIK_CLIENT_NAME=âkubectlâ
DETIK_CLIENT_NAMESPACE="kube-system"
@test âverify the deployment metrics-serverâ {
run verify âthere are 2 pods named âmetrics-serverââ
[ â$statusâ -eq 0 ]
run verify âthere is 1 service named âmetrics-serverââ
[ â$statusâ -eq 0 ]
run try âat most 5 times every 30s to find 2 pods named âmetrics-serverâ with âstatusâ being ârunningââ
[ â$statusâ -eq 0 ]
run try âat most 5 times every 30s to get pods named âmetrics-serverâ and verify that âstatusâ is ârunningââ
[ â$statusâ -eq 0 ]
}
BATS DETIK (metrics-server.bats
), metrics-server
, Argo Workflows :
â name: test-metrics-server
dependencies: [âmetrics-serverâ]
templateRef:
name: worker-containers
template: addons-tests-template
when: ââ{{workflow.parameters.METRICS-SERVER}}â != noneâ
arguments:
parameters:
â name: command
value: |
bats /addons/test/metrics-server.bats
, . Sonobuoy, Popeye Fairwinds Polaris? Argo Workflows!
, AWS EKS metrics-server
. , . !
â .
Argo Workflows (WorkflowTemplates). . , , . ( ) . Argo Events.
Argo Events
Argo Events â - Kubernetes, K8s, Argo Workflows, , -, S3, , , Google Cloud Pub/Sub, SNS, SQS .
API- (Argo Events) JSON. , (WorkflowTemplates) API. Kubernetes ( ) :
, ? API .
«» EKS? API eks-core (control-plane nodegroup).
EKS? API .
? API .
Argo
Argo Events Argo Workflows « », .
:
-
-
(. : , Argo )
-
S3
. , Argo Events Workflows. â . .
« Kubernetes». - «, k8s». 8. , .