The report Andrei Borodin tell how they have taken into account PgBouncer scaling experience in the design of compounds puller Odyssey as it rolled out into production. In addition, we will discuss what functions of the puller we would like to see in new versions: it is important for us not only to cover our needs, but to develop the Odyssey user community .
Video:
Hello! My name is Andrey.
At Yandex, I am developing open source databases. And today we have a topic about connection pooler connections.
If you know how to call connection pooler in Russian, then tell me. I really want to find a good technical term that should be established in the technical literature.
, connection pooler . - , , , Postgres . ( HighLoad++ 2019) Postgres. , , , , , , . , .
Postgres. . . , Postgres.
– primary , ( ), , .
– Postgres, Postgres connection pooler. Connection pooler fencing .
connection pooler?
Postgres . , – , Postgres. , .
, Postgres , procArray. . procArray , . , . , .
3 :
- .
- .
- , . . .
, pooler . PostgreSQL Professional . , . . application-side pool proxy pool.
Application-side pool – . : .
, , .
, , -. client side pooling . – 10 000 . , .
proxy poolers, poolers, . poolers. poolers + . Pgpool Crunchy-Proxy.
, , . , poolers pooling, . . , .
, PgBouncer, transaction pooling, . . .
– . .
, , . loopback - .
, application_name_add_host. Bouncer IP- application_name. application_name .
, – , – , . – application_name, , .
, Bouncer pool, . . , .
? , C++ - , , . 20 000 , . .
, , Bouncer, , . . pool.
Postgres, . . .
, . PgBouncer , . : , , , , , - . . , , .
, .
, Bouncer . . , , pooler.
, PgBouncer’.
https://lwn.net/Articles/542629/
Bouncer.
, TCP- Bouncers. TCP- round-robin’ .
, . . , Bouncer, idle- Bouncer’.
, 3 Bouncers 100 %. Bouncers. ?
TLS. . Postgres TLS TLS, , , TLS handshake .
, . primary , – . . . - primary , -. TLS.
TLS handshake Bouncer, . - . retry exponential backoff, .
16 PgBouncer, 16 100 %.
PgBouncer. , Bouncer. Bouncers TCP handshake, Bouncers pooling, , .
. 18 Bouncers . . , DevOps , , .
, open source, Bouncer . , , PgBouncers . pull request .
https://www.postgresql.org/docs/current/libpq-cancel.html
https://github.com/pgbouncer/pgbouncer/pull/79
. Postgres , - . TCP-reset, . . . Bouncer? . . , , Bouncer , , .
upstream Bouncer’.
, connection pooler, , , , , .
. TLS-.
, Machinarium, . libpq, , : « . IO, , IO ». . . " N, N ", " SYNC, ". , . , . , , (green thread-). , , .
, TCP accept round-robin' workers TPC-.
. cache-friendly.
, , TCP-stack.
, , Odyssey CANCEL ROLLBACK , . . , Odyssey , , .
. application_name_add_host. , , .
.. managed PostgreSQL connection pooler, , . . , , . Bouncer .
.
, . , , , . , connection pooler Odyssey, .
Odyssey PgBouncer. , . - , pull request, issue GitHub, . PgBouncer .
, , error forwarding. , . , , , .
, 100%- PgBouncer. , Bouncer, .
Odyssey.
https://github.com/yandex/odyssey/pull/66
, «Pause /Resume». . Postgres, connection pooler, pg_upgrade, resume. , . . , . ( )
https://github.com/yandex/odyssey/pull/73 —
, PgBouncer, SCRAM Authentication, , .. – .
Odyssey, .
Odyssey, . Kiwi – Postgres’ . . . proto 3 Postgres – , , . Kiwi.
Machinarium – . Machinarium . , , 15 .
Odyssey. , coroutines. accept TCP- workers.
worker . crone- , pool.
Odyssey Postgres. install-check Bouncer Odyssey, div. , Bouncer Odyssey.
, , . Odyssey.
, - : Postgres + Odyssey, PgBouncer + Odyssey, Odyssey + Odyssey , , , Odyssey - , - , .
Odyssey production. , , . , . . , . , production , PostgreSQL Professional , memory leak. , . .
, connection pooler TLS- TLS-. .
Bouncer Odyssey pcache, pcache , Odyssey . , . , 20 000 .
Pluggable Authentication Method – lunux’ . PgBouncer , PAM PgBouncer, PAM.
. . ?
, , PAM- PAM-, PAM- PAM-. , . , .
, , accept . worker pool, TLS handshake.
, 20 000 , . libpq . 3 .
, , retry.
, PgBouncer , throttling TCP-, accept.
, , , , . , handshake , . - , .
Roadmap
Odyssey? ?
2019 .
roadmap Odyssey :
- SCRAM PAM authentication.
- forward standby.
- online-restart.
- .
roadmap , . . , .
forward read-only queries to standby? , . failover switchover. - - . , -, .
, Postgres, 10- session_attrs. , : . , , session_attrs.
, . - , . , , Odyssey , .
Odyssey primary. , , , , . .
, open source. , , 2,5 PgBouncer. Odyssey.
prepared statement. prepared statement . -, SQL-, «prepared». SQL- SQL Bouncer. overkill, , . . . SQL-.
prepared statement proto3. , , prepared statement, . , - prepared statements. , - .
, - , , prepared statements , , . . prepared statement.
, , prepared statement, . , , , , , , , .
, . , PgBouncer. . – : - , - – . .
, , , , .
– 1.0 ( 1.1). , Odyssey 1.0rc, . . release candidate. , , , memory leak.
1.0? Odyssey . , 1 000 000 , , , 1.0.
, 1.0 SCRAM. , production , SCRAM, . , , .
pull request. , Bouncer. . , - , .
, . !
application_name, , transaction pooling Odyssey?
Odyssey Bouncer?
Odyssey. Bouncer .
.
, ?
, . , application_name. , . . , .
, , ! ! , Odyssey . . multi data-source , Odyssey , . . slave, failover .
, , , . storages. . , , failover, pg_recovery. , . - ? . . , . . , C, .
, adoption . , . . , .
. , . , , , . , , , . , , , , , . . . , , .
, . pcache , , real cache , , , .
- , , , , . . .
. , . , , , 10 . , .
, .
. . , , . , .
Nginx has such an option slowly startin the cluster for the server. And he gradually increases the load.
Yes, great idea, we'll give it a try when we get to that.