Clever Geek Handbook

Runaway memory hunting in Go during development

Problems

Escaping memory into a heap has the following potentially fixable problems:

  1. Decreased performance due to memory allocation costs

  2. Decreased performance due to garbage collection costs

  3. An error Out of Memoryappears if the rate of garbage appearance exceeds the rate of its removal

These problems can be solved in several ways:

  1. An increase in the amount of computing resources (memory, processor)

  2. Fine tuning the garbage collector

  3. By minimizing the number of shoots in a heap

In this article, I will only consider the third way.

From scratch

, , , . , : make , new , map ,go . , "", - .

make new . sync.Pool(), habr . , []byte , , - 32 ( go1.14.6 windows/amd64). , ; , , butebufferpool @valyala.

map . map , . , map, , - Discord Rust. .. , , , - . @valyala fastcache, , , , , , slices maps.

go - . , .

, - "" , , fastcache "". , , []byte->[]byte, , . , , - , "rps", "95th percentile latency" .. , , "" , . " " . .. "".

" " : 

git clone https://github.com/valyala/fasthttp

, , , , , :

go test -bench=PServerGet10Req -benchmem -memprofile netmem.out

go test -bench=kServerGet10Req -benchmem  -memprofile fastmem.out

http.Server, - fasthttp.Server. , fasthttp zero-allocation . , netmem.out fastmem.out. -, :

echo top | go tool pprof netmem.out

10 "" :

Showing top 10 nodes out of 53
      flat  flat%   sum%        cum   cum%
  698.15MB 21.85% 21.85%   710.15MB 22.22%  net/textproto.(*Reader).ReadMIMEHeader
  466.13MB 14.59% 36.43%   466.13MB 14.59%  net/http.Header.Clone
  423.07MB 13.24% 49.67%  1738.32MB 54.39%  net/http.(*conn).readRequest
  384.12MB 12.02% 61.69%   384.12MB 12.02%  net/textproto.MIMEHeader.Set
  299.07MB  9.36% 71.05%  1186.24MB 37.12%  net/http.readRequest
  137.02MB  4.29% 75.33%   137.02MB  4.29%  bufio.NewReaderSize
  134.02MB  4.19% 79.53%   134.02MB  4.19%  net/url.parse
  122.45MB  3.83% 83.36%   122.45MB  3.83%  bufio.NewWriterSize (inline)
   99.51MB  3.11% 86.47%   133.01MB  4.16%  context.WithCancel
   87.11MB  2.73% 89.20%    87.11MB  2.73%  github.com/andybalholm/brotli.(*h5).Initialize

: 

go tool pprof -svg netmem.out > netmem.svg

netmem.svg ():

:

go tool pprof -http=:8088 netmem.out

, , , - : Could not execute dot; may need to install graphviz. , Unix- , , Windows chocolatey , , cinst graphviz. - . VIEW/Source:

, make, []byte string. , " ", []byte. "", , , .. return &localVar . , - .

fasthttp , . - fasthttp HTTP/2.0, websockets , fasthttp ( , ), , , . c5.4xlarge 250.000 RPS fasthttp.Server 190.000 RPS http.Server . , , 190.000 RPS? , , , , .

, , . " " , json/yaml , , . - , , , .

, flatbuffers, dynobuffers, - , . , , , .

" ":

Avro         23394 ns/op    11257 B/op
Dyno_Untyped  6437 ns/op      808 B/op
Dyno_Typed    3776 ns/op        0 B/op
Flat          1132 ns/op        0 B/op
Json         87331 ns/op    14145 B/op

" ":

Avro         19311 ns/op    11257 B/op
Dyno_Typed    62.2 ns/op        0 B/op
Flat          19.8 ns/op        0 B/op
Json         83824 ns/op    11073 B/op

, , , linkedin/goavro - .

- . , () , avro "" , flatbuffer.

The problems that lead to poor performance in Go can be dealt with, but keep in mind that this struggle comes at a cost. Before starting it, it is better to experimentally compare the capabilities of the selected data processing method with the requirements for the system - maybe everything will work right out of the box with minimal tweaks.

Links



More articles:


All Articles