Work on bugs. True story of moving to Sentry on the scale of a large grocery company. Grigory Koshelev

Errors are an interesting thing, because even the title of the report itself has a double bottom. This is work on the mistakes that we make in our software, programs, services. And also working on errors is creating a pipeline to handle all this, so that everything will be fine in the end.

Sentry is a tool for monitoring exceptions, errors in your applications.

– , Sentry. . : , .

2019-2020-.

, pipeline .

60 000 3 , 150 000 , 6 .

pipeline . , , , - . - Elastic. . . , .

, Elastic. . . , . Hercules. open source, GitHub .

– , Hercules .

Pipeline . Elastic Adapter , Elastic.

, Kafka. Kafka – , . daemon, Kafka Elastic.

, Elastic . Kibana, ??? ??? ( ).

pipeline . ?

pipeline , . daemons, , . . , , , . 15 .

? ?

– , , - , , , . .

Elastic, . Kibana , - Web, , Elastic, , . . .

?

• -, , , Exception - stacktrace.
• , exc_stacktrace_hash, .
• Web , . . , . , - , . , .

hash, .

. , – Web, . . serverless-, . , .

pipeline , ?

, , Elastic Adapter. .

- , . . . .

.

, , . . , .

email. .

, – Web, .

?

• -, . . , , . - .
• -, , , .

? , , , Web. , . - , - .

. ? staging-, , - . , . staging-, , . , .

. .

, post mortem, - . . , . – 2 . – . , – 1 300 . , , . , . - , . , . , , - . , , :

2 . , . 24 . . . , .

, - , . . .

. . - . – . 2 . . 8 000 . , . , . - , . , . -, , , - . - . , , , - . . . . . , . . . , - .

https://habr.com/ru/company/ru_mts/blog/308044/

? . , .

Habr. ? . .

, . . , , . , , . ?

, . , . , 12 .

, , , open space. . . , , .

.

, , , , , 10-15 % . , , , . . .

, . staging, . , .

, post mortem – . , .

Sentry

, . . , - . , , Sentry, .

, Sentry, - . , Sentry – , . , . , , – , Sentry cloud’ Sentry. ? .

, , , . , Sentry. . , , . , . Sentry .

, - , , , , .

.

. , . pipeline Sentry , .

, . 13.51

, - , . . . - , , - . .

, - . , , -. Sentry, - . , - . - stacktrace. , Sentry . , . . .

, . . , . , . . , .

, . , , . , . 5 .

Sentry. , pipeline . , Sentry SDK.

, exception Sentry. . – , Sentry- fire-and-forget. ? , Sentry. , . - - - , , .

, , , , Sentry . .

– vendor lock. Sentry SDK, , , Sentry . - , , . - – .

? - – . , . . . . .

, , . , . , .

SDK, Sentry. , , Sentry. .

, , . . pipeline, Elastic.

Sentry.

, , , . ? . .

– . , . , Sentry, , .

, 3 , . :

• Stacktrace,
• Exception, . . exception, ,
• Message, . . message, .

stacktrace - , . . 2 stacktraces , , , . .

, message. , messages ; messages , , .

. message, . ? - . – . - , . - . , . .

? , , , .

fingerprint. , , - , , .

? - hash, , .

, , , - . fingerprint . .

, , , .

, . . , , .

, , 500 – , - .

, , «-», , , . , Sentry .

1 000 Sentry, , , . . , Sentry , , , - .

: , ,

, .

pipeline? - ERROR FATAL.

, , , , .

– , - , , . . . , , . , . , .

. – + retry.

https://github.com/vostok/clusterclient.core/issues/7

? , . , . - URL. URL . - . - - . . , . . .

, , retry . . , Sentry . . , .

. , , .

. , , . 403 .

, . – . ?

– «» . «». , . , , token . error.

. , - - . . - .

, ? , - . , , , .

? , . . - . , - , - error.

. , error.

, , . API-, . backend. backend , , storage service.

storage- . , .

, , , . .

3 , , , , .

.

()

, .

– «» . , , , retry, . . , - . , , ?

– . – . , , , . . .

, , , . , . . , . , , - ? .

? , , . .

. USE- RED-. , , , . - ( ) , , , , - .

, – . , , . , .

– .

. , .

- , . .

?

. ?

, , ERROR WARNING , . , . , . . – .

. . , WARNING, . , . , , , , .

. . . , , , , . , , , . - , - , . – . . . . , . , . , . . , . , , , . . . . , , .

? , .

:

, . , , . . , ?

, - error, , , , . , - . .

, , errors - . . error. , . , - . - . - . - -. - . . 450 . , , , . . ERROR WARNING, .

, . , . , , , . , , , . , : « ? ?».

. , , , , , . . , , - , , .

, , , , . , - , .

, . . . , , . , , . , , . : «, ». : « ». .

, . , , , .

. , - , , - , , , , - .

, , - . , error budget. , .

.

?

, . Sentry, , . , , , .

, , .

, . 100 000 . , events . 300 . , , , , , 10. . .

. happy end. , :

• , .

  
  
  
  
  
  
  

• - . , - -.

  
  
  
  
  
  
  

• , .

  
  
  
  
  
  
  

, legacy, 10 – ! , - . .

, ?

150 000 . pipeline Sentry – 100 000. , : INFO, WARNING, ERROR. . Sentry events ERROR FATAL 75 .

, , . team plan. Sentry – 26 . 50 000 . , . .

, 6 000 000 . + 820 . , . , .

. 75 . ? , , , , 200 000 000.

, , – 6 000 000. , , , . , .

. 75 – . .

, . , - , .

, - 3 300 000 . pipeline, , Sentry .

, - , .

. , , , , .

. . , , , .

, 10 . , 10 . , . , . , - - . , .

– , . 3 . , , . 3 , .

, . .

– , . - , , . Sentry. , , - , – - . .

, , . , -. .

? , .

Sentry . 4 , .

:

• Slack,
• ,
• Telegram,
• Pushover.

, .

, Slack Pushover. Slack - , Pushover , , .

: « , , ?». , , events, Sentry. ? -, . - , . , , , . - , graceful degradation , ?

, , . - staging, , , . , - Slack, , - - .

. , . : « ?». Sentry -, , .

Sentry , , . , , , - SLA .

, , . , , , . .

2019 2020

, , , , . . .

2019- : Moira .

Moira – . , , Telegram Slack. , Grafana. , . Kibana.

, , Kibana, - .

2020- ? Moira . - .

Sentry. , . . , , Sentry.

, , , .

2

– , Sentry – Sentry.

, . .

2019- , Sentry, 26 , post mortem.

, 3 post mortem, - , , .

3 , . . , Moira . , . .

1 . – , Kibana - . , .

, , . ? – , . – , . , , . , . .

, , Sentry. , , , Sentry. 9 19, , Sentry. , .

, . , . . , , . , .

19 . , . , .

5 . - , , . , .

6 . .

5 .

3 , . . Sentry.

. , .

, , . , Sentry , 10-15 % .

, Sentry, , .

, Sentry.

– 2 5. 6, , Sentry. , , Sentry. , , . . . . . , .

. . 2 - . ?

, , , , , . , , . , .

– , , , . . – , , . , , -, - -. .

, . . . , . . soft skills . , , - , , .

post mortems, . . , , . .

, Sentry , , – , Sentry. . . , .

Sentry. , Sentry. Hercules , .

Sentry. . . , .

. ! .

Q/A

: , , . , . ?

: , , . . . . – . .

: 247 , ? ?

: , , . , . . . , . , . , , . Pushover . . , .

: , - ?

: , . , Sentry. Sentry, , . . Sentry . , , . , , . , . , , , , , , 10 . , .

: , , error , ? . . 2 , error?

: . , . , , , . , , . , -, .

: , production , , , -. , , . ? - ?».

: . . - , - , - , - . , events - . , user ID, . , - , , , Slack-, - .

: , - , Sentry . ? - upstream-? , . . , . Sentry ?

: . Sentry . . ? Chrome, Sentry html, , . Sentry. , Chrome, . , , , , . . , - .

: ? ?

: , , . , — Hercules. , . , , -, . span, , . , - , community, . . , , , Elastic. . , . . , .

: , , . «Pushover», ?

: Pushover . , . Pushover. . .

() , PagerDuty android, , , . « », . iOS . , , , . , , - .

: , , ?

: , , , , , . , 80, . , . . , . . , , . , Pushover , .

: , ? , ?

: -. primary-, backup-. . - , : «, . , ». , . , . , . - – , , - .

: ? ?

: . , , . , . , - . . , . , . , .

() , , : 8 , 8 , 8 . – . , . . .

: , . ? . . ? ?

: , . , , . . ? , , - . .

? . . . . , error . - , error. error budget. . . , . , . , , , .

: . . . -, . - retry. , . , -, - tracking error budget. ? Hercules - Sentry ?

: , - – Sentry. , rules . , . . , , . , rule . . .

: . , , error, warning, info debug ?

: , , Sentry. , . , , . , , error, Sentry . , warning , Sentry . . .

: , Sentry, . - stacktrace, ?

: . -, 64 , Sentry. , . . , Sentry .

() stacktrace. . emp, . . , stack. , . , stack overflow error, stack. . .

: , , . , , log -. . , , , . , , , , , connection . . ? ?

: , , , . - , .

: , , . - - , , . . AssertionError – ?

: - , , . , , , – , — . , , , , , , .

() , , . : «, , : « ?»». , - , , , . , , 4 . .

() , , .

: . . , , - , , , ?

: . -, , - , . . . , - , , , . . , . . , - , . . . , .

, , , - . . , , . ? - .

() -, . .

: Sentry, . , , - . , , , CPU. - - ?

: , Moira. . - - , . . Graphite. Moira . Sentry – , . . Sentry . , . - . , - , - , .

. , . , . . , . . , 19- , 20- . . . endpoint . . -. .

, . ? -, stracktraces . , . Stracktaces – - . . stractraces . . , . - -. . . , : «, , , . , ». - Internet Explorer.

- Sentry