Here will be a translation of 2 posts about grok exporter.

First translation: How to use grok exporter to generate prometheus metrics from unstructured logs

Talk about grok exporter. In this article, I will explain how you can use grok exporter to create prometheus metrics from unstructured journals.

Grok ELK (ElasticSearch, Logstash, Kibana) Fabian Stäber grok exporter.

grok exporter => https://github.com/fstab/grok_exporter

1: Grok exporter

zip grok exporter https://github.com/fstab/grok_exporter/releases.

1. ​ (releases) ( v0.2.7).
2. ​ zip-, . — 64- Linux. .

wget https://github.com/fstab/grok_exporter/releases/download/v0.2.7/grok_exporter-0.2.7.linux-amd64.zip

1. ​ .
2. ​ , grok exporter.

[root@localhost grok_exporter-0.2.7.linux-amd64]# ./grok_exporter -config ./config.yml
Starting server on http://localhost.localdomain:9144/metrics

http://localhost.localdomain:9144/metrics.

2:

Grok exporter. , .

30.07.2016 04:33:03 10.3.4.1 user=Nijil message="logged in"
30.07.2016 06:47:03 10.3.4.2 user=Alex message="logged failed"
30.07.2016 06:55:03 10.3.4.2 user=Alex message="logged in"
30.07.2016 07:03:03 10.3.4.3 user=Alan message="logged in"
30.07.2016 07:37:03 10.3.4.1 user=Nijil message="logged out"
30.07.2016 08:47:03 10.3.4.2 user=Alex message="logged out"
30.07.2016 14:34:03 10.3.4.3 user=Alan message="logged out"

, . Prometheus .

1 , , config.xml, grok exporter. .

global:
    config_version: 2
input:
    type: file
    path: ./example/nijil.log  # Specify the location of the your log
    readall: true              # This should be True if you want to read whole log and False if you want to read only new lines.
grok:
    patterns_dir: ./patterns    
metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'

server:
    port: 9144

.

global:
    # Config version
input:
    # How to read log lines (file or stdin).
grok:
    # Available Grok patterns.
metrics:
    # How to map Grok fields to Prometheus metrics.
server:
    # How to expose the metrics via HTTP(S).

3: Grok exporter

, , Prometheus.

metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'

grok – %{SYNTAX:SEMANTIC}, SYNTAX — , , SEMANTIC – . %{HOSTNAME:instance}, HOSTNAME – grok, IP- , IP- ( ), . , SYNTAX , , IP- . , DATE, TIME, HOSTNAME, USER GREEDYDATA , , " " .

, , . , . , (SEMANTIC of the SYNTAX) . . . . (Counter), grok exporter , .

grok exporter ./grok_exporter -config ./config.yml . , user_activity, .

# TYPE user_activity counter
user_activity{user="Alan"} 2
user_activity{user="Alex"} 3
user_activity{user="Nijil"} 2

Prometheus . , prometheus, Prometheus .

: Apache grok exporter

https://www.robustperception.io/getting-metrics-from-apache-logs-using-the-grok-exporter

, .

, , , , . grok. , Apache, access.log:

x.x.x.x - - [20/Jan/2020:06:25:24 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16061 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16064 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET /blog/rss HTTP/1.1" 301 3478 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET / HTTP/1.1" 200 16065 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET /blog/feed HTTP/1.1" 200 3413 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET /feed HTTP/1.1" 200 6496 "-" "Emacs Elfeed 3.2.0"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"

:

wget https://github.com/fstab/grok_exporter/releases/download/v1.0.0.RC2/grok_exporter-1.0.0.RC2.linux-amd64.zip

unzip grok_exporter-*.zip
cd grok_exporter*amd64

:

cat << 'EOF' > config.yml
global:
    config_version: 2
input:
    type: file
    path: access.log
    readall: true
grok:
    patterns_dir: ./patterns
metrics:
    - type: counter
      name: apache_http_response_codes_total
      help: HTTP requests to Apache
      match: '%{COMBINEDAPACHELOG}'
      labels:
          method: '{{.verb}}'
          path: '{{.request}}'
          code: '{{.response}}'
server:
    port: 9144
EOF

grok exporter:

./grok_exporter -config config.yml

http://localhost:9144/metrics :

# HELP apache_http_response_codes_total HTTP requests to Apache
# TYPE apache_http_response_codes_total counter
apache_http_response_codes_total{code="200",method="GET",path="/"} 5
apache_http_response_codes_total{code="200",method="GET",path="/blog/feed"} 1
apache_http_response_codes_total{code="200",method="GET",path="/feed"} 1
apache_http_response_codes_total{code="301",method="GET",path="/blog/rss"} 1

readall, , .

Grok – , Logstash (Logstash — L ELK). , , Apache. Grok , , . COMMMONAPACHELOG , ,

COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

. , . Go ( Prometheus alerting notification) .

, Grok , :

    - type: summary
      name: apache_http_response_bytes
      help: Size of HTTP responses
      match: '%{COMMONAPACHELOG}'
      value: '{{.bytes}}'

, , :

    - type: gauge 
      name: apache_http_last_request_seconds
      help: Timestamp of the last HTTP request
      match: '%{COMMONAPACHELOG}'
      value: '{{timestamp "02/Jan/2006:15:04:05 -0700" .timestamp}}'

(timestamp) grok exporter, time.Parse Golang. (divide), .

, Grok. , , , .