A Practical Guide to HashiCorp Consul - Part 1

This is part 1 of a 2-part How-to Guide for HashiCorp Consul. This part is primarily focused on understanding the problems that Consul solves and how it solves them. The second part is more focused on practical application of Consul in a real-life example and will be published next week. Let's start.

How about setting up a discoverable, customizable and secure service mesh with one tool?

What if we told you that this tool is platform independent and ready to run in the cloud?

And it comes as a single binary file.

This is all true. The tool we're talking about is the HashiCorp Consul .

Consul provides service discovery , health checking , load balancing , service graph , identity enforcement with TLS, and distributed service configuration management .

Consul , .

:

. , , .

("") — A, B, C D

("") — A, B, C D

HashiCorp Consul – , , , , , , TLS . Consul control plane service mesh.

HashiCorp Consul ,

HashiCorp Consul 2014 , .

, Consul HashiCorp .

, . , . , .

- (SOA)

, . , .

, .

, HashiCorp Consul, . -, (, A), ( B), ( ) ( D).

, — , — .

. .

: , , A, , .

production.

, , D.

. .

, .

, :

, SOA-

:

A SOA-

, . , .

. . . , . .

, , .

, , . .

, Consul

, . — . , .

, . , , .

A , B. , A B, B?

A

A , B. , . , , , .

, , . .

,

IP-, .

, . IP- , .

, . , . .

. , , . , .

. A B, A B, B. B .

.

. , . , . .

Consul

Consul — .

Consul , . . .

Consul A

A B, B, B. , .

Consul (health-checks) . (health-check), . , , .

, , Consul . , .

Consul .

. Consul , , .

.

, Consul

, , , - YAML, XML JSON. , .

,

, . , .

, . .

, . , .

• . .

  
  
  
  
  
  
  

• , , , .

  
  
  
  
  
  
  

Consul — .

Consul’s KV

Consul . , .

. . .

, Consul

, .

. , .

— . .

— , . .

, . , .

.

. , .

• , - . , .

  
  
  
  
  
  
  

• , , .

  
  
  
  
  
  
  

• , .

  
  
  
  
  
  
  

• - . , , , .

  
  
  
  
  
  
  

SOA

• . , , , , .

Consul TLS.

Consul

Consul , , . Consul Consul Connect.

Consul Connect , , . , , A B, B C.

, IP-. . . , , IP- . .

Consul Connect , TLS. TLS, .

. TLS . .

Consul TLS , . Sidecar. - - .

. , .

Consul

Consul — .

Consul . , .

, Consul, Consul. Consul.

Consul , . /.

Consul — , .

Consul , HashiCorp 3-5 , . Consul. , .

. HashiCorp .

Consul . Consul .

Consul ,

, Consul Consul. Consul Consul.

Consul

Consul , .

, Consul , - HashiCorps Consul GitHub.

Consul .

Consul — Consul.

Consul

Consul .

$ wget https://releases.hashicorp.com/consul/1.4.3/consul_1.4.3_linux_amd64.zip -O consul.zip

--2019-03-10 00:14:07--  https://releases.hashicorp.com/consul/1.4.3/consul_1.4.3_linux_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.37.183, 2a04:4e42:9::439
Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.37.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34777003 (33M) [application/zip]
Saving to: ‘consul.zip’

consul.zip             100%[============================>]  33.17M  4.46MB/s    in 9.2s    

2019-03-10 00:14:17 (3.60 MB/s) - ‘consul.zip’ saved [34777003/34777003]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

zip-.

$ unzip consul.zip

Archive:  consul.zip
  inflating: consul
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

PATH.

$ export PATH="$PATH:/path/to/consul"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

Consul

, .

$ consul agent -dev

==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.4.2'
           Node ID: 'ef46ebb7-3496-346f-f67a-30117cfec0ad'
         Node name: 'devcube'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2019/03/04 00:38:01 [DEBUG] agent: Using random ID "ef46ebb7-3496-346f-f67a-30117cfec0ad" as node ID
    2019/03/04 00:38:01 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:ef46ebb7-3496-346f-f67a-30117cfec0ad Address:127.0.0.1:8300}]
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2019/03/04 00:38:01 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.1
    2019/03/04 00:38:01 [INFO] serf: EventMemberJoin: devcube 127.0.0.1
    2019/03/04 00:38:01 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2019/03/04 00:38:01 [INFO] consul: Handled member-join event for server "devcube.dc1" in area "wan"
    2019/03/04 00:38:01 [DEBUG] agent/proxy: managed Connect proxy manager started
    2019/03/04 00:38:01 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
    2019/03/04 00:38:01 [DEBUG] raft: Votes needed: 1
    2019/03/04 00:38:01 [DEBUG] raft: Vote granted from ef46ebb7-3496-346f-f67a-30117cfec0ad in term 2. Tally: 1
    2019/03/04 00:38:01 [INFO] raft: Election won. Tally: 1
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
    2019/03/04 00:38:01 [INFO] consul: cluster leadership acquired
    2019/03/04 00:38:01 [INFO] consul: New leader elected: devcube
    2019/03/04 00:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2019/03/04 00:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2019/03/04 00:38:01 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2019/03/04 00:38:01 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)
    2019/03/04 00:38:01 [INFO] agent: started state syncer
    2019/03/04 00:38:01 [INFO] connect: initialized primary datacenter CA with provider "consul"
    2019/03/04 00:38:01 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small
    2019/03/04 00:38:01 [INFO] consul: member 'devcube' joined, marking health alive
    2019/03/04 00:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:38:01 [INFO] agent: Synced node info
    2019/03/04 00:38:01 [DEBUG] agent: Node info in sync
    2019/03/04 00:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:38:01 [DEBUG] agent: Node info in sync
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

.

Consul

, Consul.

$ consul members

Node     Address         Status  Type    Build  Protocol  DC   Segment
devcube  127.0.0.1:8301  alive   server  1.4.0  2         dc1  <all>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

, , . , Consul.

gossip .

Consul HTTP API

Consul HTTP API, .

$ curl localhost:8500/v1/catalog/nodes

[
    {
        "ID": "ef46ebb7-3496-346f-f67a-30117cfec0ad",
        "Node": "devcube",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "wan": "127.0.0.1"
        },
        "Meta": {
            "consul-network-segment": ""
        },
        "CreateIndex": 9,
        "ModifyIndex": 10
    }
]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

DNS Consul

Consul DNS- . DNS 8600. .

$ dig @127.0.0.1 -p 8600 devcube.node.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 devcube.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42215
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;devcube.node.consul.       IN  A

;; ANSWER SECTION:
devcube.node.consul.    0   IN  A   127.0.0.1

;; ADDITIONAL SECTION:
devcube.node.consul.    0   IN  TXT "consul-network-segment="

;; Query time: 19 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:45:44 IST 2019
;; MSG SIZE  rcvd: 100
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

Consul , HTTP API.

() Consul

() — . .

, consul.d — ‘.d’ , , consul.

$ mkdir ./consul.d
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

- Django, 80 .

$ echo '{"service": {"name": "web", "tags": ["django"], "port": 80}}' \
    > ./consul.d/web.json
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

- , .

$ consul agent -dev -config-dir=./consul.d

==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.4.2'
           Node ID: '810f4804-dbce-03b1-056a-a81269ca90c1'
         Node name: 'devcube'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2019/03/04 00:55:28 [DEBUG] agent: Using random ID "810f4804-dbce-03b1-056a-a81269ca90c1" as node ID
    2019/03/04 00:55:28 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:810f4804-dbce-03b1-056a-a81269ca90c1 Address:127.0.0.1:8300}]
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2019/03/04 00:55:28 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.1
    2019/03/04 00:55:28 [INFO] serf: EventMemberJoin: devcube 127.0.0.1
    2019/03/04 00:55:28 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2019/03/04 00:55:28 [DEBUG] agent/proxy: managed Connect proxy manager started
    2019/03/04 00:55:28 [INFO] consul: Handled member-join event for server "devcube.dc1" in area "wan"
    2019/03/04 00:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2019/03/04 00:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2019/03/04 00:55:28 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2019/03/04 00:55:28 [INFO] agent: started state syncer
    2019/03/04 00:55:28 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)
    2019/03/04 00:55:28 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
    2019/03/04 00:55:28 [DEBUG] raft: Votes needed: 1
    2019/03/04 00:55:28 [DEBUG] raft: Vote granted from 810f4804-dbce-03b1-056a-a81269ca90c1 in term 2. Tally: 1
    2019/03/04 00:55:28 [INFO] raft: Election won. Tally: 1
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
    2019/03/04 00:55:28 [INFO] consul: cluster leadership acquired
    2019/03/04 00:55:28 [INFO] consul: New leader elected: devcube
    2019/03/04 00:55:28 [INFO] connect: initialized primary datacenter CA with provider "consul"
    2019/03/04 00:55:28 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small
    2019/03/04 00:55:28 [INFO] consul: member 'devcube' joined, marking health alive
    2019/03/04 00:55:28 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:55:28 [INFO] agent: Synced service "web"
    2019/03/04 00:55:28 [DEBUG] agent: Node info in sync
    2019/03/04 00:55:29 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:55:29 [DEBUG] agent: Service "web" in sync
    2019/03/04 00:55:29 [DEBUG] agent: Node info in sync
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

— , web



. Consul . .

DNS Consul

DNS, . :

$ dig @127.0.0.1 -p 8600 web.service.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51488
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;web.service.consul.        IN  A

;; ANSWER SECTION:
web.service.consul. 0   IN  A   127.0.0.1

;; ADDITIONAL SECTION:
web.service.consul. 0   IN  TXT "consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:59:32 IST 2019
;; MSG SIZE  rcvd: 99
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

DNS , , .

$ dig @127.0.0.1 -p 8600 web.service.consul SRV

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 712
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;web.service.consul.        IN  SRV

;; ANSWER SECTION:
web.service.consul. 0   IN  SRV 1 1 80 devcube.node.dc1.consul.

;; ADDITIONAL SECTION:
devcube.node.dc1.consul. 0  IN  A   127.0.0.1
devcube.node.dc1.consul. 0  IN  TXT "consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:59:43 IST 2019
;; MSG SIZE  rcvd: 142
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

TAG, , :

$ dig @127.0.0.1 -p 8600 django.web.service.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 django.web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12278
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;django.web.service.consul. IN  A

;; ANSWER SECTION:
django.web.service.consul. 0    IN  A   127.0.0.1

;; ADDITIONAL SECTION:
django.web.service.consul. 0    IN  TXT "consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 01:01:17 IST 2019
;; MSG SIZE  rcvd: 106
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

Consul HTTP API

HTTP API:

$ curl http://localhost:8500/v1/catalog/service/web

[
    {
        "ID": "810f4804-dbce-03b1-056a-a81269ca90c1",
        "Node": "devcube",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "wan": "127.0.0.1"
        },
        "NodeMeta": {
            "consul-network-segment": ""
        },
        "ServiceKind": "",
        "ServiceID": "web",
        "ServiceName": "web",
        "ServiceTags": [
            "django"
        ],
        "ServiceAddress": "",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {},
        "ServicePort": 80,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 10,
        "ModifyIndex": 10
    }
]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

HTTP API:

$ curl http://localhost:8500/v1/catalog/service/web?passing

[
    {
        "ID": "810f4804-dbce-03b1-056a-a81269ca90c1",
        "Node": "devcube",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "wan": "127.0.0.1"
        },
        "NodeMeta": {
            "consul-network-segment": ""
        },
        "ServiceKind": "",
        "ServiceID": "web",
        "ServiceName": "web",
        "ServiceTags": [
            "django"
        ],
        "ServiceAddress": "",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {},
        "ServicePort": 80,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 10,
        "ModifyIndex": 10
    }
]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

Consul

Consul, .

. SIGHUP , Consul, SIGHUP , HTTP API, , .

$ ps aux | grep [c]onsul

pranav   21289  2.4  0.3 177012 54924 pts/2    Sl+  00:55   0:22 consul agent -dev -config-dir=./consul.d
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

SIGHUP 21289

$ kill -SIGHUP 21289
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

Consul

$ consul reload
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

.

Consul.

...
    2019/03/04 01:10:46 [INFO] agent: Caught signal:  hangup
    2019/03/04 01:10:46 [INFO] agent: Reloading configuration...
    2019/03/04 01:10:46 [DEBUG] agent: removed service "web"
    2019/03/04 01:10:46 [INFO] agent: Synced service "web"
    2019/03/04 01:10:46 [DEBUG] agent: Node info in sync
...
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

- Consul.

Consul - . 8500.

http://localhost:8500. .

Consul , Consul -.

(Exploring) - Consul

, .

(Exploring) - Consul

, .

, - Consul.

, Consul Web UI , Consul.

Consul Zookeeper, doozerd etcd?

Consul , (health-check), -, -.

Zookeeper, doozerd etcd . - , , .

, Consul, , .

, /.

. Consul , .

. , . heartbeats TTL. health checks -. TTL.

Zookeeper, Consul -, , gossip. ( localhost), , .

, Consul HTTP DNS , . , .

- Consul Consul .

HashiCorp Consul

HashiCorp Consul.

Consul HashiCorp:

Consul Template — Consul. , HashiCorp Consul Vault. Consul consul-template.

Envconsul — Consul. Envconsul , HashiCorp Consul Vault.

Consul Replicate — Consul cross-DC KV . Consul consul-replicate.

Consul Migrate — Consul Consul 0.5.1+.

Consul , .

Confd — etcd consul.

Fabio — Fabio — , HTTP(S) TCP , Consul. consul, , Fabio . .

Registrator — Docker . Docker, .

Hashi-UI — HashiCorp Consul & Nomad.

Git2consul — git Consul KVs. git2consul git Consul KVs. , git , , Consul- .

Spring-cloud-consul — Consul Spring Boot Spring Spring. Consul.

Crypt — etcd consul.

Mesos-Consul — Mesos to Consul bridge . Mesos-Consul / , Mesos.

Consul-cli — Consul HTTP API.

. — . HashiCorp Consul , .

, Consul, , .

, , HashiCorp Consul .

, Consul. , . , , .

, - , .

HashiCorp Consul GitHub.

HashiCorp Consul Guides and Code

, .

HashiCorp Consul

PS I will add that you can also install the consul from the package repository: https://www.hashicorp.com/blog/announcing-the-hashicorp-linux-repository