Today we'll look at collecting and visualizing performance metrics for Red Hat Enterprise Linux 8 using Performance Co-Pilot (PCP), Grafana, and Bpftrace tools, as well as the new performance monitoring capabilities introduced in the RHEL 8.4 web console.
Basics of work
In our examples, we will use the following configuration:
Two RHEL 8.3 servers (named server-1 and server-2).
Grafana version grafana-6.7.4-3.el8.x86_64.
PCP version pcp-5.1.1-3.el8.x86_64.
By default, Performance Co-Pilot is not installed with RHEL 8, so you need to install it yourself. Install it on server-1 and server-2 by running the following command on them:
yum install pcp-zeroconf -y
Once Performance Co-Pilot is installed and running, you can see what built-in performance metrics it offers:
pminfo | wc -l
3101
pminfo | grep kernel | head -n 15
kernel.all.load
kernel.all.intr
kernel.all.pswitch
kernel.all.sysfork
kernel.all.running
kernel.all.blocked
kernel.all.boottime
kernel.all.hz
kernel.all.uptime
kernel.all.idletime
kernel.all.nusers
kernel.all.nroots kernel.all.nsessions
kernel.all.lastpid
kernel.all.runnable
You can display the values of the required metric with the pmrep command:
pmrep kernel.all.load -s 5
k.a.load k.a.load k.a.load
1 minute 5 minute 15 minut
0.020 0.020 0.080
0.020 0.020 0.080
0.020 0.020 0.080
0.020 0.020 0.080
0.020 0.020 0.080
The -s 5 option in the example above says that we only want to display the first 5 metric values.
In order for the metrics to be used by other applications, you need to enable pmproxy and open the corresponding port on the firewall:
systemctl enable pmproxy
systemctl start pmproxy
firewall-cmd --add-service=pmproxy --permanent
firewall-cmd --reload
, Performance Co-Pilot , Grafana, .
grafana server-1, :
yum install grafana grafana-pcp -y
systemctl enable grafana-server
systemctl start grafana-server
firewall-cmd –add-service=grafana –permanent
firewall-cmd –reload
http://server-1:3000 admin admin. .
, , (Configuration), Plugins. «Performance Co-Pilot App», Enable.
Configuration, Data Sources, Add Data Source, PCP Vector Select. , HTTP «http://localhost:44322» URL, Save & Test . , «Data source is working».
(Dashboard), Manage PCP Vector Host Overview.
server-1, :
CPU Percentage
Load Average
Memory Utilization
Disk Utilization
Per-CPU busy (user/system)
CPU user%,system%,interrupt%,wait%
Scheduler context switches per second and runnable processes.
Memory Used/Cached/Free/Page Fault Rate/Hard Fault Rate
Network throughput in/out
Network drops in/out
Network packets in/out
TCP connections/timeouts/close-waits/time wait/established/listen errors/retransmits
Disk latency/iops/throughput/utilization
, .
, pmseries Redis, .
, . server-1 Redis PCP ( ). server-1 Redis. PCP 100 200 . , Redis, pmseries, server-1, /etc/pcp/pmproxy/pmproxy.conf, [pmproxy] :
# support Redis protocol proxying
redis.enabled = true
, , [pmseries] :
# allow REST API queries of fast, scalable time series
enabled = true
, , Redis server-1, :
yum install redis -y
systemctl start redis
systemctl enable redis
:
systemctl restart pmcd pmlogger pmproxy
, server-1 , pmlogger. pmlogger server-1, server-2.
server-2 /etc/sysconfig/pmcd PMCD_LOCAL 1 0, pmcd:
# Behaviour regarding listening on external-facing interfaces;
# unset PMCD_LOCAL to allow connections from remote hosts.
# A value of 0 permits remote connections, 1 permits local only.
PMCD_LOCAL=0
:
firewall-cmd --add-service=pmcd --permanent
firewall-cmd --reload
pcp unreserved:
setsebool -P pcp_bind_all_unreserved_ports on
, pcp:
systemctl restart pmcd pmlogger
server-2 , pmlogger .
server-1 pmlogger, server-2. /etc/pcp/pmlogger/control.d/remote :
server-2 n n PCP_LOG_DIR/pmlogger/server-2 -r -T24h10m -c config.remote
pmlogger:
systemctl restart pmlogger
:
cd /var/log/pcp/pmlogger/server-2
ls *.0
, , «.0»:
20200923.16.41.0
, , server-1 pcp- server-2.
Grafana (http://server-1:3000) , , redis.
Configuration, Data Sources, Add Data Source. PCP Redis Select. , HTTP «http://localhost:44322» URL, Save & Test . , «Data source is working».
, . Dashboard, Manage PCP Redis Host Overview.
PCP Redis Host Overview , . :
load average
memory utilization
per-cpu busy (user/sys)
context switches per second
runnable processes
memory (used/cached/free)
network throughput in/out
network drops in/out
network packets in/out
tcp timeouts/listen errors/retransmits
disk latency/iops/throughput/utilization
, , Redis. , , , Host . , , .
, , ansible – rhel-system-roles.metrics , :
Performance Co-Pilot Bpftrace
, Performance Co-Pilot bpftrace , Linux. , bpftrace Linux - eBPF ( «-, eBPF), Performance Co-Pilot.
, dev- bpftrace Performance Co-Pilot. , Performance Co-Pilot Grafana . server-1 pcp-pmda-bpftrace, :
yum install bpftrace pcp-pmda-bpftrace -y
cd /var/lib/pcp/pmdas/bpftrace
./Install
, bpftrace pmda , :
pmrep bpftrace.scripts.runqlat.data_bytes -s 5
, ( ):
b.s.r.data_bytes
byte/s
N/A
586.165
590.276
588.141
589.113
, bpftrace pmda Performance Co-Pilot, Grafana. dev-, .
, Performance Co-Pilot metrics:
yum install cyrus-sasl-scram cyrus-sasl-lib -y
useradd -r metrics
passwd metrics
saslpasswd2 -a pmcd metrics
chown root:pcp /etc/pcp/passwd.db
chmod 640 /etc/pcp/passwd.db
saslpasswd2 , , metrics sasl- pmcd, pcp.
, /etc/sasl2/pmcd.conf :
mech_list: scram-sha-256
sasldb_path: /etc/pcp/passwd.db
pcp:
systemctl restart pmcd
bpftrace pmda , . : , . /var/lib/pcp/pmdas/bpftrace/bpftrace.conf , [dynamic_scripts] :
enabled=true
allowed_users=root,metrics
bpftrace pmda:
cd /var/lib/pcp/pmdas/bpftrace
./Remove
./Install
Grafana (http://server-1:3000) admin ( ).
Grafana Configuration, Data Sources, Add Data Source, PCP bpftrace, Select.
URL «http://localhost:44322», Auth Basic auth, Basic Auth Details metrics , saslpasswd2. Save & Test, , , «Data source is working».
Dashboards Manage. PCP bpftrace System Analysis, . . CPU usage Edit. , bpftrace-, bpf- @cpu . Grafana bpf- .
, bpftrace , pid , fork. pidpersec.bt , , :
tracepoint:sched:sched_process_fork
{
@ = count();
}
, sched_process_fork bpf, Grafana.
, . PCP bpftrace System Analysis Add Panel, , Add Query.
Query , PCP bpftrace, Query A bpftrace:
tracepoint:sched:sched_process_fork
{
@ = count();
}
General – «Processes Per Second». PCP bpftrace System Analysis , . .
, grafana pcp, bpftrace. dev-, . bpftrace-, bpftrace.
bpftrace pmda. bpftrace, /var/lib/pcp/pmdas/bpftrace/autostart, Performance Co-Pilot , bpftrace pmda.
/var/lib/pcp/pmdas/bpftrace/autostart/pidpersec.bt :
tracepoint:sched:sched_process_fork
{
@ = count();
}
, :
cd /var/lib/pcp/pmdas/bpftrace/
./Remove
./Install
pminfo | grep pidpersec
:
bpftrace.scripts.pidpersec.data.root
bpftrace.scripts.pidpersec.data_bytes
bpftrace.scripts.pidpersec.code
bpftrace.scripts.pidpersec.probes
bpftrace.scripts.pidpersec.error
bpftrace.scripts.pidpersec.exit_code
bpftrace.scripts.pidpersec.pid
bpftrace.scripts.pidpersec.status
bpf-, pmda root. , :
pmrep bpftrace.scripts.pidpersec.data.root -s 5
:
b.s.p.d.root /s N/A 3.984 31.049 0.000 0.000
grafana, grafana- - , Performance Co-Pilot.
Grafana, Dashboards, Manage, PCP Vector Host Overview, Add Panel Add Query.
Query PCP Vector A- :
bpftrace.scripts.pidpersec.data.root
General «Processes per Second».
PCP Vector Host Overview bpf-, bpftrace - . .
pcp-pmda-bpftrace Linux- Grafana.
, bpftrace , , . .
- RHEL 8.4
RHEL 8.4 , , , .
-
, , , , , (saturation) , . ?
, . - RHEL 8.4 , , , , .
, System -> Overview - Usage card. cockpit-pcp (. ), Usage card.
. Usage card, , , :
CPU , . , , - . IO- -, . - .
, , pcp ( ), :
, , . , ; .
– , , . , – (saturation). . , , , .
flame graph
, , , - RHEL. RHEL , flame graphs.
, , , . : , , , .
, , perf js-d3-flame-graph.
flame graph perf. , , , , 30 , , Ctrl+C .
[root@bigdemo1 ~]# perf script record flamegraph -F 99 -g
^C
[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 1.194 MB perf.data (3448 samples) ]
flame graph, :
[root@bigdemo1 ~]# perf script report flamegraph
:
dumping data to flamegraph.html
flamegraph.html, :
30 , perf. -, . , . , user space – .
root, . -, : SetroubleshootP, mysqld, apache2, 240998, rpm, ksoftirqd, kworker, grafana-server redis-server.
( ), , «» mysqld.