Clever Geek Handbook

Query language for TSDB. Improving PromQL (Alexander Valyalkin, VictoriaMetrics) (2019)



Good afternoon! Today I will tell you a little about PromQL. It is a query language for time-series databases. Then I'll tell you how we improved it in the VictoriaMetrics database we are developing time-series.







PromQL is a query language used in the Prometheus monitoring system.





PromQL time-series β€” , SQL, InfluxQL Flux, , Graphite. time-series . , PromQL – time-series , , , time-series .





SQL , time-series .



? node_network_receive_bytes_total. , timestamps . , . , JOIN.





, PromQL. PromQL .



s



PromQL Grafana. rate, total. counter, , . , . , rate , .





PromQL SQL.





, . . SQL , PromQL. SQL β€” . :



  • SQL SELECT , , PromQL, β€” , , .
  • PromQL JOIN.
  • GROUP BY. , .
  • . .
  • HAVING.
  • , SQL.




, time-series . SQL β€” , time-series .





https://github.com/grafana/grafana/issues/11451



https://github.com/prometheus/prometheus/issues/3746



https://github.com/prometheus/prometheus/issues/3806



PromQL – data model Prometheus PromQL. Brian Brazil.



- , , , , Brain , .



  • - – , PromQL , , scrape_interval, .. , . , .
  • - – , increase(), , , . , 10, increase() 9,5. - , data model Brian Brazil. . .
  • PromQL , min/max rate. Rate – , . . . rate , , .
  • rate [d]. , , , rate. , PromQL, . , , , , , Prometheus .




PromQL , Grafana, Grafana , .



, Grafana. .





?





- – VictoriaMetrics. time-series , .





  • time-series , long-term remote storage Prometheus, . . Prometheus.
  • Prometheus, Influx, OpenTSDB, Graphite, CVS, JSON lines, Prometheus text exposition format.
  • VictoriaMetrics PromQL, PromQL.
  • VictoriaMetrics PromQL MetricsQL. , . .




:





– rate.



PromQL VictoriaMetrics. VictoriaMetrics PromQL, .



? $__interval. Grafana , . , . VictoriaMetrics, . rate β€” rate(m), rate(m[$__interval]).





. . , , 5. , 5 . PromQL : rate(m[5i]).





. , . . , . , PromQL deriv(), .



, MetricsQL VictoriaMetrics. . , 5 , , 5 . step() . .





VictoriaMetrics -, , , , . , Prometheus , VictoriaMetrics , .



Prometheus rate() , . , , Prometheus . VictoriaMetrics , , , , , scrape_interval time-series , , . . 0,1 . , 1 , , scrape_interval, . . , Prometheus, , -, 10 .





VictoriaMetrics rollup , . , . .



VictoriaMetrics rollup , rollup. , .



rollup_rate. rate. "" (Counters). , β€” . MetricsQL.





rollup_rate . , , . , , .



rate , . , . . , , , rate, , rate .





VictoriaMetrics range_*, . , .



, , , quantile, sum, , .





? , . - rate . rate range_avg(). , , .





default, , default.



, sinus, , 0,3. sinus. rand, . . . , , , . rand 0, .





label’. PromQL label’. : label_replace label_join, β€” , , .



MetricsQLL , label’:



  • label_set – labels time-series. label bar query.
  • label_del – labels query.
  • label_keep – labels, , .
  • label_copy label_move – label.




https://victoriametrics.com/promql/expand-with-exprs



PromQL , PromQL-, , . WITH templates. common table expressions – CTE. .





– . PromQL-. , {instance=~”$node:$port”, job=~”$job”} .



WITH templates . with (cf = {instance=~”$node:$port”, job=~”$job”}). cf – , . , . , cf.





– WITH templates . ru, : free, limit, filters. .



Ru – resource usage. Free – , . Limit – . Filters – .



, resource usage resource utilization, . . .



WITH , . node exporter labels. . , , , . resource usage.





. WITH templates templates. templates. ru, . .



ru MetricsQL VictoriaMetrics, . . WITH . ru.



– , . – . .





MetricsQL, VictoriaMetrics. , - , - . , , .



.



  • if, ifnot.
  • PromQL: rate(q[1.5m]) offset 0.5d.
  • start() end() , .
  • smooth_exponential(), .
  • , , .


MetricsQL , template meta-variables Grafana.





, Grafana template variables. template variables . Grafana , variables. , , .



, PromQL Grafana , template variables. , , - - , , .



Template meta_variables . , WITH templates variables Grafana, , template meta-variables, Grafana.



issue pull request. issue, PR. Grafana PR. , , : ?.





template meta-variables. variables, commonFilters. , variables: job, instance.



, , . WITH templates, Prometheus.





  • .
  • Template meta-variables meta-variables.
  • . , .
  • Template meta-vars Prometheus, datasource Grafana.




VictoriaMetrics , .





single-node , . , , . . time-series , . open source . Update: β€” . single-node





WITH templates Grafana?



. . – template meta-variables, WITH templates .



, , ? InfluxQL, Graphite, Prom’?



, . PromQL MetricsQL, . , , - , MetricsQL.



. . Graphite, ? - ?



time-series , VictoriaMetrics , , . . matric name , value.



Graphite .



Graphite .



Graphite .



VictoriaMetrics . . . .



. . - ?



. Graphite metric name VictoriaMetrics, . . . , Influx line protocol, . (tags & fields). metric names. .



!



alerting. Thanos, PromQL , alerting Prometheus . PromQL alerting, ? alerting?



VictoriaMetrics PromQL, Prometheus. Prometheus’, . Grafana, Prometheus, datasource url VictoriaMetrics. , . .



alerting β€” alerting' VictoriaMetrics. . . , alerting Prometheus, VictoriaMetrics. alerting , Prometheus, VictoriaMetrics, alerting rules Prometheus'. datasource. VictoriaMetrics, alerts, alert manager. Update: β€” vmalert.



VictoriaMetrics ?



VictoriaMetrics , . , . . , . . , . . . , time-series .



. , PromQL.



.



, .



, .



, JOIN. JOIN ? ?



VictoriaMetrics , Prometheus, Graphite, . . . VictoriaMetrics , Prometheus, , VictoriaMetrics.



. . , ?



, . VictoriaMetrics, Thanos, Prometheus', . VictoriaMetrics . . VictoriaMetrics. VictoriaMetrics , . , - , Thanos. Thanos Prometheus', . Prometheus. VictoriaMetrics .



, !



! , VictoriaMetrics single-node, . open source - - ?



Apache2 . , , , . . . , , , VictoriaMetrics.



! InfluxDB, Timescale. ? - benchmarks?



. InfluxDB . . ClickHouse . Altinity, , ClickHouse, . ClickHouse time-series Influx, -. Timescale . . https://altinity.com/blog/tag/benchmark/ .



, . , , . . , .



VictoriaMetrics , ClickHouse, , ClickHouse, . . : - , - .



?



benchmarks , 100 %. ClickHouse, , 100 % CPU .



, !



, ?



. , VictoriaMetrics, , , , . , . storage, google cloud . , - , . . - . . , , . . . , production, , , . Update: VictoriaMetrics β€” . https://victoriametrics.github.io/Cluster-VictoriaMetrics.html#replication-and-data-safety .



. - , , , ?



, , PromQL. Offset . offset 1d, . β€” , , , .



- ? . ., , , , ?



PromQL β€” quantile_over_time. , Prometheus, , , . , quantile_over_time(0.99, m[24h]) 99- 24 .



!



How open will the development model be? You will open up now. After that, two options are possible: either we do merge once every six months, and they are there 5 GB each, or you are developing openly



We will try to conduct development openly. Maybe the question is differently paraphrased? Will we merge third-party pull requests?



This is the second part of the question.



We will, but not all. If we can see that they are useful, we will. But we will try to do it differently from Brain Brazil, that is, we will not say that we have a data model and that's it. If there is a good pull request, then we will merge.



More articles:


All Articles