In this article, I will share my experience of how the recent accident at OVH affected our monitoring infrastructure, how we solved the problem and what lessons we learned from this.
About monitoring with us
, «» . :
1. Blackbox- . — endpoint’ , . , health page JSON-, , . , — , / .
: HTTP/HTTPS. , , JSON ( status page- ). (, ).
( , OVH).
2. Kubernetes- , . Prometheus + Grafana + Alertmanager, . , (, , Kubernetes Deckhouse), , — (, ).
3. , Kubernetes. , (bare metal) , , . Okmeter ( — - ). OVH.
, , , — Okmeter. ( Kubernetes-), blackbox- ( , - ).
? ? — ?
«» Dead man’s switch (DMS). , «»:
OK, , « », (Prometheus, Okmeter ..) -.
, OK , .
, , OK , - ERROR . .
, (10 ): (ERROR) DMS.
:
— DMS, 3:20 .
Okmeter , , . , - . , , (blackbox Kubernetes). .
( 8:14) , , Okmeter , .
, Okmeter. - OVH:
SBG-2 — ;
SBG-1 — .
, - OVH . , , .
10 , , Okmeter — - .
, :
;
, ;
.
DevOps-, CTO . , , Okmeter.
, Okmeter . , ? :
3 . , , S1 ( ) S9 ( ). S1 — blackbox-, . , Okmeter ( . ). S2, (S3 ..).
S1 S2 Okmeter, , . .
, Okmeter, .
Okmeter
: S1-S2
, Okmeter? , — , Okmeter, — , 1 2020 .
:
.
( ).
.
(, ).
.
.
: S3
S3 : , , .
. , ZooKeeper.
Bash
Okmeter, , . : Ansible- , . 10 . - Bash.
:
shell-, bare metal-. ( ) , Okmeter: , severity, .. , . , .
, API — flint (flant integration).
Ansible- , , . Ansible- , , , .
, — , .
S1-S2 — S3. ( ) .
3000 .
:
, . :
, Okmeter , «» , , .
, : , . : , (DRP) . - , , .
:
, , . , OVH? …
«» Okmeter, : , .. ( ), «» , .
, : . , .
P.S.
: