Clever Geek Handbook

What and why do we do in Open Source databases. Andrey Borodin (Yandex.Cloud)



The contribution of Yandex to the following databases will be considered.



  • ClickHouse
  • Odyssey
  • Point-in-time recovery (WAL-G)
  • PostgreSQL (including logerrors, Amcheck, heapcheck)
  • Greenplum


Video:





Hello world! My name is Andrey Borodin. And at Yandex.Cloud I am engaged in the development of open relational databases in the interests of Yandex.Cloud and Yandex.Cloud clients.





, . ? - , , , . , .



. . , . , , - .



! ? , . , . .





?



  • – . , , , open source, .
  • . . . . . , up getting style - .
  • – . , Google Summer of Code, Google , , .
  • , , . Google, , , , . Google : « , . , ».
  • open source – . , . , .




– ClickHouse. , , ..



open source , ( ) . , .





. ClickHouse Yandex Object Storage, . . .





? , , . , , SSD-, - . , . , . – .





? .



  • ClickHouse over MDS. MDS – . , S3, . . . , , .



  • S3 – , .





, ClickHouse , ., , ClickHouse. ClickHouse over S3, ClickHouse over MDS. .





:



https://github.com/ClickHouse/ClickHouse/pull/7946 "Filesystem abstraction layer”

https://github.com/ClickHouse/ClickHouse/pull/8011 "AWS SDK S3 integration”

https://github.com/ClickHouse/ClickHouse/pull/8649 "Base implementation of IDisk interafce for S3”

https://github.com/ClickHouse/ClickHouse/pull/8356 "Integration of log storage engines with IDisk interface"

https://github.com/ClickHouse/ClickHouse/pull/8862 "Log engine support for S3 and SeekableReadBuffer"

https://github.com/ClickHouse/ClickHouse/pull/9128 "Storage Stripe Log S3 support"

https://github.com/ClickHouse/ClickHouse/pull/9415 "Storage MergeTree initial support for S3"

https://github.com/ClickHouse/ClickHouse/pull/9646 "MergeTree full support for S3"

https://github.com/ClickHouse/ClickHouse/pull/10126 "Support ReplicatedMergeTree over S3"

https://github.com/ClickHouse/ClickHouse/pull/11134 "Add default credentials and custom headers for s3 storage"

https://github.com/ClickHouse/ClickHouse/pull/10576 "S3 with dynamic proxy configuration"

https://github.com/ClickHouse/ClickHouse/pull/10744 "S3 with proxy resolver"



pull request ClickHouse. pull requests.





:



https://github.com/ClickHouse/ClickHouse/pull/9760 "DiskS3 hardlinks optimal implementation"

https://github.com/ClickHouse/ClickHouse/pull/11522 "S3 HTTP client — Avoid copying response stream into memory"

https://github.com/ClickHouse/ClickHouse/pull/11561 "Avoid copying whole response stream into memory in S3 HTTP

client"

https://github.com/ClickHouse/ClickHouse/pull/13076 "Ability to cache mark and index files for S3 disk"

https://github.com/ClickHouse/ClickHouse/pull/13459 "Move parts from DiskLocal to DiskS3 in parallel"



. , , - , .





:



https://github.com/ClickHouse/ClickHouse/pull/12638 "Add SelectedRows and SelectedBytes events"

https://github.com/ClickHouse/ClickHouse/pull/12464 "Add profiling events from S3 request to system.events"

https://github.com/ClickHouse/ClickHouse/pull/13028 "Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQueryTimeMicroseconds"



, .



, , ClickHouse .





, OLTP , .





. .





, , , , Postgres.



Postgres – . , , .



, , . .





, – , , .



, . multiplexer . , connection pooler, , .





https://pgconf.ru/2017/92899



, postgres' . PgBouncer. PgBouncer. , PgBouncer, , , .





https://pgconf.ru/media/2017/04/03/20170316H1_V.Borodin.pdf



. , , , Bouncer, pull request upstream. .



Bouncers. Bouncers, Bouncers. , .





, , Odyssey. .





https://www.pgcon.org/2019/schedule/events/1312.en.html



2019- PgCon . 2 000 GitHub, . . , .



Postgres ., Odyssey, .





? – , , , , , , . .



PgBouncer .



. , pgagroal, Red Hat. , , , , pgagroal.





postgres’ – . , .





. Postgres .



, , . ? , - , . , , , . .





https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-faster-restores-for-postgres/



, CitusData WAL-G. , . CitusData – Microsoft. , , WAL-G, . .





https://github.com/wal-g/wal-g/graphs/contributors



, -10 WAL-G 6 . . , , , , production, , , , WAL-G.





, , , . , Postgres .



? : , .



? , , . , , , .



, , , - , , , .



. , , .





. . . Postgres. MySQL, MongoDB. FoundationDB. .



, , . , Postgres .





? , , , , . pull requests. , . .





, Postgres. Postgres . , Postgres .





, . . - .. , Postgres, , Postgres.



. , . - , .



, Postgres. , , . , . . . PgSQL hacker 40 . - , , .



Postgres – - . - . . .



. , . .





, , .



– - , , .



. , - , , . – - .



, , , . , . – , .



, , , . .





https://commitfest.postgresql.org/23/2171/



, , . , , . Postgres. , , , .



, , - , .





, , . , .



! – .



, Logerrors. , . , , , -.



, , CentOS. , . , open source.





https://www.postgresql.org/message-id/flat/DA9B33AC-53CB-4643-96D4-7A0BBC037FA1@yandex-team.ru



. Amcheck — , , .



, , . . .





https://www.postgresql.org/message-id/flat/59D0DA6B-1652-4D44-B0EF-A582D5824F83@yandex-team.ru



, GiST & GIT . . , .





https://commitfest.postgresql.org/29/2667/



, , , , follower . . . , .



, can… . Crunchy Data . B- Postgres , . . , , . . . , , Postgres, Linux, . , .





https://www.postgresql.org/message-id/flat/38AF687F-8F6B-48B4-AB9E-A60CFD6CC261%40enterprisedb.com#0e86a12c01d967bac04a9bf83cd337cb



, heap, . . , . , .



, Heapcheck. . EnterpriseDB , Heapcheck. PgHeapcheck, Heapcheck. , , . - . open source.



, , , . – , , .





https://www.postgresql.org/message-id/flat/fe9b3722df94f7bdb08768f50ee8fe59%40postgrespro.ru



, . , 1. Postgres , , pg_dump .



. . , . , . , . Postgres.



. , , , , .





https://www.postgresql.org/message-id/flat/fe9b3722df94f7bdb08768f50ee8fe59%40postgrespro.ru



: «, , ».



. , , , , – . , . - , , . . , - . .





? : , . , , . . , , , .



- , , , pull request . , . .





– Greenplum. , Postgres, .





https://greenplum.org/greenplum-database-tables-compression/



Greenplum – append optimized . , . , .



, . . , , , , - .



: «, Postgres. . 20 . ». , , Postgres, 20 – .





https://github.com/greenplum-db/gpdb/commit/179feb77a034c2547021d675082aae0911be40f7



, 20 , . PgConf.Russia Heikki Linakangas Pivotal : « - ? append optimized ?». : « . , . ». : «, , ». : «, , ». , .



pull request, . pull request Pivotal . , .





https://github.com/greenplum-db/gpdb/issues/10150



, pull request Greenplum . , heap- . , . , . – , .





https://github.com/greenplum-db/gpdb/pull/10290



. pull request . .





https://github.com/greenplum-db/gpdb-postgres-merge/pull/53



, Greenplum PostgreSQL 12. . . 20 . , . .





https://github.com/greenplum-db/gpdb/pull/10565



. , .



. , Pivotal. . . , , .



, , . , ? : « , 10 ». ? , scale , , .





? , open source – , . - , - , - , - , - . Greenplum, Greenplum.



! – . . . , , . . – ! . ., , . , , , , .



. .







! . . , . open source. , . , , ., . : « , ?».



WAL-G Go. , . 3 . – . , C. Postgres 30 . C89. Postgres . , ClickHouse C++ . C C++.



, : « open source?».



. . ? , , - . . , 100 % , . , . production. . . ,



: « , ., , ?».



, , . , . , , , , . . , , , . - - , - , . . . .



: « , , , ?». , , ., .



. -, , - . . , . , , , . issues GitHub , DBA, , , , 30 % - , . , . , .



. , . ? Postgres Pro?



, . . , . . , postgres’ . , - .



, ClickHouse ?



, . ClickHouse – . , : « ?». .



: « Greenplum , junior’?». , , , , , , .



, , -. , . . , – . Greenplum .



junior’, . Postgres. , ?



: « ?». - . Postgres, , to do list. , , . . - , - , , . , , . Google Summer of code postgres’ , . , , . WAL-G , . Greenplum , Postgres, Greenplum pull request’ . Postgres – - , Greenplum , . , Greenplum . Greenplum , . , , .



More articles:


All Articles