Finding the optimal values for limiting Kubernetes resources is not an easy task, because you need to find a middle ground between too tight and not enough constraints.
In this article, which is a continuation of the series on resource management in Kubernetes , you will learn how to choose the right Kubernetes resource limits: from discovering containers without any restrictions to determining the optimal parameters that you should set in your cluster.
Prometheus is one of the most popular solutions for monitoring Kubernetes clusters. Therefore, each step in this tutorial contains examples of PromQL queries .
Discovery of containers without limiting resources
The first step in choosing the correct limits is to detect containers without any limits.
. pods QoS. - . - .
CPU Limit namespace
sum by (namespace)(count by (namespace,pod,container)(kube_pod_container_info{container!=""}) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="cpu"}))
Memory Limit namespace
sum by (namespace)(count by (namespace,pod,container)(kube_pod_container_info{container!=""}) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"}))
, . ? ! , .
-10 CPU Limits, CPU
topk(10,sum by (namespace,pod,container)(rate(container_cpu_usage_seconds_total{container!=""}[5m])) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="cpu"}))
-10 Memory Limits,
topk(10,sum by (namespace,pod,container)(container_memory_usage_bytes{container!=""}) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"}))
CPU Limits
, , :
(sum by (namespace,pod,container)(rate(container_cpu_usage_seconds_total{container!=""}[5m])) / sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="cpu"})) > 0.8
Memory Limits
, .
, .
, , :
(sum by (namespace,pod,container)(container_memory_usage_bytes{container!=""}) / sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"})) > 0.8
?
— . :
. , .
CPU Limit :
max by (namespace,owner_name,container)((rate(container_cpu_usage_seconds_total{container!="POD",container!=""}[5m])) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))
Memory Limit :
max by (namespace,owner_name,container)((container_memory_usage_bytes{container!="POD",container!=""}) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))
99 . 1% . , , .
CPU Limit :
quantile by (namespace,owner_name,container)(0.99,(rate(container_cpu_usage_seconds_total{container!="POD",container!=""}[5m])) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))
Memory Limit :
quantile by (namespace,owner_name,container)(0.99,(container_memory_usage_bytes{container!="POD",container!=""}) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))
?
, pods Requests pods. , .
, Kubernetes .
, , . , pods, - .
?
:
100 * sum(kube_pod_container_resource_limits{container!="",resource="memory"} ) / sum(kube_node_status_capacity_memory_bytes)
:
100 * sum(kube_pod_container_resource_limits{container!="",resource="cpu"} ) / sum(kube_node_status_capacity_cpu_cores)
, . 100% — , .
, , 125%, , 150% .
. , CPU Requests - 2 CPU Limit - 8. 4 , , .
:
sum by (node)(kube_pod_container_resource_limits{container!=””,resource=”memory”} ) / sum by (node)(kube_node_status_capacity_memory_bytes)
:
sum by (node)(kube_pod_container_resource_limits{container!=””,resource=”cpu”} ) / sum by (node)(kube_node_status_capacity_cpu_cores)
, Kubernetes Limits and Requests, , Kubernetes.
, Kubernetes.