Clever Geek Handbook

HashiCorp Vault (HA) High Availability Mode

Hashicorp Vault - open-source tool for managing secrets (passwords, API keys, etc.),







Vault can operate in high availability (HA) mode to protect against disruptions by running multiple Vault servers. Vault is typically constrained by the Vault backend I / O limits, not by computational requirements. Some server storage units, such as Consul, provide additional coordination features that enable Vault to operate in a high availability configuration, while others provide a more reliable backup and restore process.







When operating in high availability mode, Vault servers have two additional states: standby and active . In a Vault cluster, only one instance will be active, which will process all requests (read and write), and all standby nodes will forward requests to the active node.













. 0.11, . Performance Standby Nodes Vault Enterprise Premium, Vault Enterprise Pro . . .







Vault Highly Available (HA). , , , .













25













Vault , Vault Consul.













, — Vault HA, :







· 2 Vault: 1 1







· 3- Consul













:













:







  • 1. Consul







  • 2. Consul







  • 3. Consul Vault







  • 4. Vault







  • 5. Vault









Vault Consul; Enterprise.







1. Consul







Consul IP-, :







  • consul_s1: 10.1.42.101



  • consul_s2: 10.1.42.102



  • consul_s3: 10.1.42.103





Consul /usr/local/bin/consul



, , .







, Consul:







{
  "server": true,
  "node_name": "$NODE_NAME",
  "datacenter": "dc1",
  "data_dir": "$CONSUL_DATA_PATH",
  "bind_addr": "0.0.0.0",
  "client_addr": "0.0.0.0",
  "advertise_addr": "$ADVERTISE_ADDR",
  "bootstrap_expect": 3,
  "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"],
  "ui": true,
  "log_level": "DEBUG",
  "enable_syslog": true,
  "acl_enforce_version_8": false
}


, , . Consul :







  • $NODE_NAME — ; consul_s1



    , consul_s2



    consul_s3



    .
  • $CONSUL_DATA_PATH: Consul; , Consul.
  • $ADVERTISE_ADDR: , Consul . 0.0.0.0



    ; IP- Consul 10.1.42.101



    ,10.1.42.102



    10.1.42.103



    .
  • $JOIN1, $JOIN2, $JOIN3: retry_join



    ; , 10.1.42.101



    , 10.1.42.102



    10.1.42.103



    .


, - ("ui": true



), Consul DEBUG ("log_level": "DEBUG"



). acl_enforce_version_8



false



, ACL . , ACL Consul ACL.







Vault /usr/local/etc/consul/client_agent.json



.







consul_s1.json 









{
  "server": true,
  "node_name": "consul_s1",
  "datacenter": "dc1",
  "data_dir": "/var/consul/data",
  "bind_addr": "0.0.0.0",
  "client_addr": "0.0.0.0",
  "advertise_addr": "10.1.42.101",
  "bootstrap_expect": 3,
  "retry_join": ["10.1.42.101", "10.1.42.102", "10.1.42.103"],
  "ui": true,
  "log_level": "DEBUG",
  "enable_syslog": true,
  "acl_enforce_version_8": false
}


consul_s2.json 









{
  "server": true,
  "node_name": "consul_s2",
  "datacenter": "dc1",
  "data_dir": "/var/consul/data",
  "bind_addr": "0.0.0.0",
  "client_addr": "0.0.0.0",
  "advertise_addr": "10.1.42.102",
  "bootstrap_expect": 3,
  "retry_join": ["10.1.42.101", "10.1.42.102", "10.1.42.103"],
  "ui": true,
  "log_level": "DEBUG",
  "enable_syslog": true,
  "acl_enforce_version_8": false
}


consul_s3.json 









{
  "server": true,
  "node_name": "consul_s3",
  "datacenter": "dc1",
  "data_dir": "/var/consul/data",
  "bind_addr": "0.0.0.0",
  "client_addr": "0.0.0.0",
  "advertise_addr": "10.1.42.103",
  "bootstrap_expect": 3,
  "retry_join": ["10.1.42.101", "10.1.42.102", "10.1.42.103"],
  "ui": true,
  "log_level": "DEBUG",
  "enable_syslog": true,
  "acl_enforce_version_8": false
}


systemd









Consul , Consul ; systemd



Linux, , , systemd unit:







### BEGIN INIT INFO
# Provides:          consul
# Required-Start:    $local_fs $remote_fs
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Consul agent
# Description:       Consul service discovery framework
### END INIT INFO

[Unit]
Description=Consul server agent
Requires=network-online.target
After=network-online.target

[Service]
User=consul
Group=consul
PIDFile=/var/run/consul/consul.pid
PermissionsStartOnly=true
ExecStartPre=-/bin/mkdir -p /var/run/consul
ExecStartPre=/bin/chown -R consul:consul /var/run/consul
ExecStart=/usr/local/bin/consul agent \
    -config-file=/usr/local/etc/consul/client_agent.json \
    -pid-file=/var/run/consul/consul.pid
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target


, , . . –







  • config-file
  • pid-file


(, /etc/systemd/system/consul.service



), systemctl daemon-reload



, Consul .







2. Consul







, , data_dir



, Consul :







$ sudo systemctl start consul
$ sudo systemctl status consul
● consul.service - Consul server agent
   Loaded: loaded (/etc/systemd/system/consul.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-03-19 17:33:14 UTC; 24h ago
 Main PID: 2068 (consul)
    Tasks: 13
   Memory: 13.6M
      CPU: 0m 52.784s
   CGroup: /system.slice/consul.service
           └─2068 /usr/local/bin/consul agent -config-file=/usr/local/etc/consul/client_agent.json -pid-file=/var/run/consul/consul.pid


Consul, Consul:







$consul members
Node       Address           Status  Type    Build  Protocol  DC    Segment
consul_s1  10.1.42.101:8301  alive   server  1.0.6  2         dc1   <all>
consul_s2  10.1.42.102:8301  alive   server  1.0.6  2         dc1   <all>
consul_s3  10.1.42.103:8301  alive   server  1.0.6  2         dc1   <all>


, 3 ; , , :







$consul operator raft list-peers
Node                   ID                                    Address           State     Voter  RaftProtocol
consul_s2              536b721f-645d-544a-c10d-85c2ca24e4e4  10.1.42.102:8300  follower  true   3
consul_s1              e10ba554-a4f9-6a8c-f662-81c8bb2a04f5  10.1.42.101:8300  follower  true   3
consul_s3              56370ec8-da25-e7dc-dfc6-bf5f27978a7a  10.1.42.103:8300  leader    true   3


, consul_s3



. Vault.







3. Consul Vault







Vault Consul Vault . Consul , Vault .













Consul







Consul , Consul Vault, Consul , HA ( ).







, Vault Consul.







Consul , Vault, Consul, client_address



, Vault .







Consul:







{
  "server": false,
  "datacenter": "dc1",
  "node_name": "$NODE_NAME",
  "data_dir": "$CONSUL_DATA_PATH",
  "bind_addr": "$BIND_ADDR",
  "client_addr": "127.0.0.1",
  "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"],
  "log_level": "DEBUG",
  "enable_syslog": true,
  "acl_enforce_version_8": false
}


, 1, Consul :







  • $NODE_NAME — ; consul_c1



    consul_c2



    .
  • $CONSUL_DATA_PATH: Consul; , Consul.
  • $BIND_ADDR: , , Consul , 0.0.0.0



    ; IP- Vault 10.1.42.201



    10.1.42.202



    .
  • $JOIN1, $JOIN2, $JOIN3: retry_join



    ; , 10.1.42.101



    , 10.1.42.102



    10.1.42.103



    .


Vault /usr/local/etc/consul/client_agent.json



.







consul_c1.json







{
  "server": false,
  "datacenter": "dc1",
  "node_name": "consul_c1",
  "data_dir": "/var/consul/data",
  "bind_addr": "10.1.42.201",
  "client_addr": "127.0.0.1",
  "retry_join": ["10.1.42.101", "10.1.42.102", "10.1.42.103"],
  "log_level": "DEBUG",
  "enable_syslog": true,
  "acl_enforce_version_8": false
}


consul_c2.json







{
  "server": false,
  "datacenter": "dc1",
  "node_name": "consul_c2",
  "data_dir": "/var/consul/data",
  "bind_addr": "10.1.42.202",
  "client_addr": "127.0.0.1",
  "retry_join": ["10.1.42.101", "10.1.42.102", "10.1.42.103"],
  "log_level": "DEBUG",
  "enable_syslog": true,
  "acl_enforce_version_8": false
}


systemd Consul



Consul , Consul Vault. systemd



:







### BEGIN INIT INFO
# Provides:          consul
# Required-Start:    $local_fs $remote_fs
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Consul agent
# Description:       Consul service discovery framework
### END INIT INFO

[Unit]
Description=Consul client agent
Requires=network-online.target
After=network-online.target

[Service]
User=consul
Group=consul
PIDFile=/var/run/consul/consul.pid
PermissionsStartOnly=true
ExecStartPre=-/bin/mkdir -p /var/run/consul
ExecStartPre=/bin/chown -R consul:consul /var/run/consul
ExecStart=/usr/local/bin/consul agent \
    -config-file=/usr/local/etc/consul/client_agent.json \
    -pid-file=/var/run/consul/consul.pid
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target


:







  • -config-file
  • -pid-file


(, /etc/systemd/system/consul.service



), systemctl daemon-reload



, Consul Vault.







Consul , , , data_dir



, Consul :







$ sudo systemctl start consul
$ sudo systemctl status consul
● consul.service - Consul client agent
   Loaded: loaded (/etc/systemd/system/consul.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-03-20 19:36:49 UTC; 6s ago
 Main PID: 23758 (consul)
    Tasks: 11
   Memory: 9.8M
      CPU: 571ms
   CGroup: /system.slice/consul.service
           └─23758 /usr/local/bin/consul agent -config-file=/usr/local/etc/consul/client_agent.json -pid-file=/var/run/consul/consul.pid


Consul Consul:







$consul members
Node        Address           Status  Type    Build  Protocol  DC    Segment
consul_s1   10.1.42.101:8301  alive   server  1.0.6  2         dc1   <all>
consul_s2   10.1.42.102:8301  alive   server  1.0.6  2         dc1   <all>
consul_s3   10.1.42.103:8301  alive   server  1.0.6  2         dc1   <all>
consul_c1   10.1.42.201:8301  alive   client  1.0.6  2         arus  <default>
consul_c2   10.1.42.202:8301  alive   client  1.0.6  2         arus  <default>


3 Consul 2 Consul . Vault.







4. Vault







, Consul, 3- 2- Vault, Vault , Vault HA.







Vault IP-, :







  • vault_s1: 10.1.42.201
  • vault_s2: 10.1.42.202


:









, Vault /usr/local/bin/vault



.







Vault







listener "tcp" {
  address          = "0.0.0.0:8200"
  cluster_address  = "0.0.0.0:8201"
  tls_disable      = "true"
}

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

api_addr =  "$API_ADDR"
cluster_addr = "$CLUSTER_ADDR"


tcp



-:









(, , Vault ).







Vault (api_addr



cluster_addr



). Consul Vault, Consul Vault. (, Vault ).







, Vault ( ). Client Redirection, .







, , . Vault :







  • $API_ADDR: ( URL) Vault . VAULT_API_ADDR



    . , URL-, . http://10.1.42.201:8200 http://10.1.42.202:8200 .
  • $CLUSTER_ADDR: Vault . VAULT_CLUSTER_ADDR



    . URL, api_addr



    . https://10.1.42.201:8201 https://10.1.42.202:8201 .


, (https) ; TLS / .







vault_s1.hcl 







listener "tcp" {
  address          = "0.0.0.0:8200"
  cluster_address  = "10.1.42.201:8201"
  tls_disable      = "true"
}

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

api_addr = "http://10.1.42.201:8200"
cluster_addr = "https://10.1.42.201:8201"


vault_s2.hcl 







listener "tcp" {
  address          = "0.0.0.0:8200"
  cluster_address  = "10.1.42.202:8201"
  tls_disable      = "true"
}

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

api_addr = "http://10.1.42.202:8200"
cluster_addr = "https://10.1.42.202:8201"


systemd Vault







Vault . Vault . systemd



:







### BEGIN INIT INFO
# Provides:          vault
# Required-Start:    $local_fs $remote_fs
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Vault server
# Description:       Vault secret management tool
### END INIT INFO

[Unit]
Description=Vault secret management tool
Requires=network-online.target
After=network-online.target

[Service]
User=vault
Group=vault
PIDFile=/var/run/vault/vault.pid
ExecStart=/usr/local/bin/vault server -config=/etc/vault/vault_server.hcl -log-level=debug
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=42s
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target


, , . .







  • -config
  • -log-level


, , /etc/systemd/system/vault.service



, systemctl daemon-reload



, Vault .







5. Vault







Vault :







$ sudo systemctl start vault
$ sudo systemctl status vault
● vault.service - Vault secret management tool
   Loaded: loaded (/etc/systemd/system/vault.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-03-20 20:42:10 UTC; 42s ago
 Main PID: 2080 (vault)
    Tasks: 12
   Memory: 71.7M
      CPU: 50s
   CGroup: /system.slice/vault.service
           └─2080 /usr/local/bin/vault server -config=/home/ubuntu/vault_nano/config/vault_server.hcl -log-level=debu


Vault.







, Vault .







Vault:







$ vault status
Key             Value
---             -----
Seal Type       shamir
Sealed          false
Total Shares    5
Threshold       3
Version         0.9.5
Cluster Name    vault
Cluster ID      0ee91bd1-55ec-c84f-3c1d-dcc7f4f644a8
HA Enabled      true
HA Cluster      https://10.1.42.201:8201
HA Mode         active


Vault:







vault status
Key                     Value
---                     -----
Seal Type               shamir
Sealed                  false
Total Shares            5
Threshold               3
Version                 0.9.5
Cluster Name            vaultron
Cluster ID              0ee91bd1-55ec-c84f-3c1d-dcc7f4f644a8
HA Enabled              true
HA Cluster              https://10.1.42.201:8201
HA Mode                 standby
Active Node Address:    http://10.1.42.201:8200


Vault (HA), Vault . , (sudo systemctl stop vault



), , .













Read " Hardening Security " to learn about best practices for deploying Vault to harden security in a production environment.







More articles:


All Articles