Introduction

Some time ago, I was tasked with developing a failover cluster for PostgreSQL , operating in several data centers connected by fiber optics within the same city, and capable of withstanding a failure (for example, a power outage) of one data center. I chose Pacemaker as the software responsible for fault tolerance, because this is the official solution from RedHat for creating failover clusters. The good news is that RedHat provides support for it, and the fact that this solution is universal (modular). With its help, it will be possible to provide fault tolerance not only for PostgreSQL, but also for other services, either using standard modules, or creating them for specific needs.

A reasonable question arose to this decision: how fault-tolerant will a failover cluster be? To investigate this, I developed a test bench that simulates various failures on the cluster nodes, waits for health to be restored, rebuilds the failed node, and continues testing in a loop. Initially, this project was called hapgsql, but over time I got bored with the name, which has only one vowel. Therefore, I began to call the fault-tolerant databases (and float IPs pointing to them) krogan (a character from a computer game, in which all important organs are duplicated), and nodes, clusters and the project itself - tuchanka (the planet where krogan live).

Now the management has allowed to open the project for the open source community under the MIT license . The README will soon be translated into English (because Pacemaker and PostgreSQL developers are expected to be the main consumers), and I decided to arrange the old Russian version of the README (partially) in the form of this article.

VirtualBox. 12 ( 36GiB), 4 ( ). PostgreSQL, -, witness c quorum device ( -), 50%/50%, . -: , , quorum device. PostgreSQL, -: , , witness c quorum device. -. , , .

ntpd , ntpd (orphan mode). witness NTP-, , . witness , ( ). HTTP proxy witness, Yum-. , , , witness .

v0. CentOS 7 PostgreSQL 11 VirtualBox 6.1.

-, -. split-brain Pacemaker, STONITH (Shoot The Other Node In The Head) fencing. : , - , , «» , , IPMI UPS. , IPMI UPS . , ( ) -. stonith- (IPMI, UPS ..) .

. , , . «+1» . , , , .. split-brain. , , , watchdog, , IPMI.

( -), 50%/50% (-), . quorum device — , -. ( ), 50%/50%. , quorum device, witness ( repmgr, ).

, , , . , ( ?), IP (float IP). IP, Pacemaker ( ). () , , ( ).

Tuchanka1 ( )

, , slave- hot standby read only- ( ).

- . PostgreSQL ( PostgreSQL , ( ), Pacemaker). , ( float IP). -, . ̒ ( ) (), ( shared_buffers ..), , ( ) -. ( read only-) , .

, .

witness

witness (quorum device) Tuchanka1, . witness , , . 2 3, . .

Tuchanka1

- Tuchanka1. witness -. , float IP.

Tuchanka2 ()

. , . ( read only), float IP: krogan2 — , krogan2s1 — . , .

, .

Tuchanka2

- witness . - , float IP: . , , ( connection ..) float IP. .

Tuchanka4 ( )

. , read-only ( ). Tuchanka4 — , , . . ( ) - , PostgreSQL.

, . , , , -, -, . float IP. , - sql proxy, , . sql proxy, , . , (connection pool), .. ( SQL proxy , ).

Tuchanka4

- (.. ) witness . - : , float IP ( read-write ); , float IP ( read only-).

, : float IP , . , sql proxy float IP; sql proxy , float IP URL . libpq IP, . , , , JDBC, sql proxy. , float IP , , .

: - . , - , , , ( ).

Tuchanka3 (3 -)

, -, . quorum device . - , — . , ANY (slave1, slave2), , , . float IP . Tuchanka4 float IP . read-only SQL- sql proxy ( ), float IP, — .

Tuchanka3

- . float IP , — float IP ( , float IP). . ( ) - ( ).

. , README. .

. test/failure. , . , :

test/failure 2 3

. , . , tmux. Tmux tmux , - default tmux, tmux. . setup.

, ( ) . Tuchanka2. :

1. . :
   
   
   • failure — ( ), .
   • reaction — , . , , , . , , ( (Tuchanka3 Tuchanka4)), , , .
   • deviation — () reaction « ».
   • count — .
2. , . (), ̒ . (> 5 ) - .
3. heart () — . float IP . .
4. beat () — « », heart , float IP. . Tuchanka1 float IP ( , ), (), beat, heart .
5. pcs mon. , .
6. . — . CPU Load ( ), , System Load ( Load Average, 5, 10 15 ), .
7. , . — — .

. , , . , . ( ) - (> 5 , ) , - .

:

1. , .
2. Ready? — ( ).
3. (reaction).
4. Fix — «». .

, :

• ForkBomb: "Out of memory" -.
• OutOfSpace: . , , , , , PostgreSQL .
• Postgres-KILL: PostgreSQL killall -KILL postgres.
• Postgres-STOP: PostgreSQL killall -STOP postgres.
• PowerOff: «» VBoxManage controlvm "" poweroff.
• Reset: VBoxManage controlvm "" reset.
• SBD-STOP: SBD killall -STOP sbd.
• ShutDown: SSH systemctl poweroff, .
• UnLink: , VBoxManage controlvm "" setlinkstate1 off.

tmux "kill-window" Ctrl-b &, "detach-client" Ctrl-b d: , tmux , .

• watchdog sbd , . , , , Corosync Pacemaker, sbd. Corosync PR#83 ( GitHub sbd), master. ( PR#83), Pacemaker - , , RedHat 8 . «» , , , killall -STOP corosync, .

  
  
  

• Pacemaker CentOS 7 sync_timeout quorum device, , . sync_timeout quorum device ( setup/setup1). Pacemaker, ( ), .

  
  
  

• , LC_MESSAGES ( ) , , ru_RU.UTF-8, postgres , locale UTF-8, , ( pacemaker+pgsqlms(paf) postgres), UTF-8 . PostgreSQL , . , LC_MESSAGES=en_US.UTF-8 () .

  
  
  

• wal_receiver_timeout ( 60s), PostgreSQL-STOP tuchanka3 tuchanka4 . , , . wal_receiver_timeout=0 PostgreSQL.

  
  
  

• PostgreSQL ForkBomb ( ). ForkBomb . tuchanka3 tuchanka4, - , , . , - ( ). , . , , .

  
  
  

Deviant Art c :