SRE: Performance Analysis. Setup method using a simple Go webserver

Performance Analysis and Tuning is a powerful performance compliance validation tool for customers.

Performance analysis can be used to check for bottlenecks in a program, taking a scientific approach to validating tuning experiments. This article defines a general approach to performance analysis and tuning using a Go webserver as an example.

Go is particularly well suited here because it has profiling tools pprofin the standard library.

Strategy

Let's create a pivot list for our structural analysis. We will try to use some of the data to make decisions instead of making changes based on intuition or guesswork. To do this, let's do this:

• We define the boundaries of optimization (requirements);
• We calculate the transaction load for the system;
• We execute the test (create data);
• We are watching;
• We analyze - are all requirements met?
• Setting up in a scientific way, making a hypothesis;
• .

HTTP

HTTP Golang. .

— HTTP-, Postgresql . Prometheus, node_exporter Grafana .

, ( ) :

. ? , ? , , 10 000 .

Google SRE Book . , :

• : 99% 60;
• : , . ;
• : , , , n+1.

, . SRE SLO \ , . - !

. .

Vegeta HTTP, :

$ make load-test LOAD_TEST_RATE=50
echo "POST http://localhost:8080" | vegeta attack -body tests/fixtures/age_no_match.json -rate=50 -duration=0 | tee results.bin | vegeta report

. ( , ) (, CPU, IOPS) , , , .

— , . :

, . Go (pprof) flame graph, . .

, , .

. , , . , . , . : make load-test LOAD_TEST_RATE=X.

50

. , 50 ( ), — . : . HTTP Request Latency SLO 60. , .

:

10000 / 50 = 200 + 1

.

500

, 500 :

, . — , . , , 500 25-40. 99 SLO 60, .

:

10000 / 500 = 20 + 1

.

1000

! , 1000 , SLO. p99 . , p100 , 60. , , .

1000 , pprof , , . HTTP endpoint pprof, curl:

$ curl http://localhost:8080/debug/pprof/profile?seconds=29 > cpu.1000_reqs_sec_no_optimizations.prof

:

$ go tool pprof -http=:12345 cpu.1000_reqs_sec_no_optimizations.prof

, . Brendan Gregg:

X — , ( ), Y , [top]. — . — . — CPU, — . , , , .

—

. . , , , , , , , .

Brendan Gregg . ( ). — , ( ). :

— , . HTTPServe 65% , runtime, runtime.mcall, mstart gc, . : 5% DNS:

, , Postgresql. FindByAge:

, , , : \ , . , DNS, 13% .

: HTTP, .

—

, Postgresql . — . sql go:

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)

if err != nil {
   return nil, err
}

, ,

1000 , p99 SLO 60!

?

10000 / 1000 = 10 + 1

!

2000

, , 2000 , p100 60, p99 SLO.

:

10000 / 2000 = 5 + 1

3000

3000 p99 60. SLO , :

10000 / 3000 = 4 + 1 ( , . )

.

—

3000 :

6% . , , .

: , , , . , .

—

MaxIdleConns ( ):

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)
db.SetMaxIdleConns(8)
if err != nil {
   return nil, err
}

, ,

3000

p99 60 p100!

flame graph , ! pg(*conn).query — .

, . , , — . Go , , .