Red Hat Enterprise Linux 7 includes a program called Performance Co-Pilot, provided by the pcp RPM package. Performance Co-Pilot, or pcp for short, allows administrators to collect and query data from various subsystems. Performance Co-Pilot has also been backported into Red Hat Enterprise Linux 6, and is available in 6.6 and later.
Installing Performance Co-Pilot
Performance Co-Pilot is installed with the pcp package. After installing pcp, the machine will have the pmcd daemon necessary for collecting subsystem data. Additionally, the machine will also have various command-line tools for querying system performance data.
# yum -y install pcp
There are several services that are part of Performance Co-Pilot, but the one that collects system performance data locally is pmcd, the Performance Metrics Collector Daemon. This service must be running in order to query performance data with the Performance Co-Pilot command-line utilities.
# systemctl start pmcd # systemctl status pmcd ● pmcd.service - Performance Metrics Collector Daemon Loaded: loaded (/usr/lib/systemd/system/pmcd.service; enabled; vendor preset: disabled) Active: active (exited) since Fri 2015-12-11 08:26:38 EST; 8s ago ... Output Truncated ... # systemctl enable pmcd Created symlink from /etc/systemd/system/multi-user.target.wants/pmcd.service to /usr/lib/systemd/system/pmcd.service.
Using the pcp command-line utilities
The pcp package provides a variety of command-line utilities to gather and display data on a machine.
The pmstat command provides information similar to vmstat. Like vmstat, pmstat will support options to adjust the interval between collections (-t) or the number of samples (-s).
# pmstat -s 5
pmatop provides a top-like output of machine statistics and data. It includes disk I/O statistics and network I/O statistics, as well as the CPU, memory, and process information provided by other tools. By default, pmatop will update every five seconds.
Performance Co-Pilot also has a text-based query mechanism for interrogating individually tracked metrics. To obtain a list of the metrics stored in the Performance Co-Pilot database, use the pminfo command, then use the pmval command with the metric name to gather data about the desired item.
# pminfo ... Output Truncated ... proc.nprocs proc.psinfo.pid ... Output Truncated ...
# pminfo -dt proc.nprocs proc.nprocs [instantaneous number of processes] Data Type: 32-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff Semantics: instant Units: none
# pmval -s 5 proc.nprocs metric: proc.nprocs host: geeklab semantics: instantaneous value units: none samples: 5 interval: 1.00 sec 180 180 180 180 180
Retrieving historical performance data
Performance Co-Pilot also has the ability to store data in a log. This facility is provided by the pmlogger service. This service must be enabled for system performance data to be archived by Performance Co-Pilot.
# systemctl start pmlogger # systemctl status pmlogger ● pmlogger.service - Performance Metrics Archive Logger Loaded: loaded (/usr/lib/systemd/system/pmlogger.service; disabled; vendor preset: disabled) Active: active (exited) since Sun 2019-01-06 04:30:55 UTC; 18s ago Docs: man:pmlogger(1) Process: 6481 ExecStart=/usr/share/pcp/lib/pmlogger start (code=exited, status=0/SUCCESS) Main PID: 6481 (code=exited, status=0/SUCCESS) CGroup: /system.slice/pmlogger.service └─11895 /usr/libexec/pcp/bin/pmlogger -P -r -T24h10m -c config.default -v 100mb -m pmlogger_check 20190106.04.30 # systemctl enable pmlogger Created symlink from /etc/systemd/system/multi-user.target.wants/pmlogger.service to /usr/lib/systemd/system pmlogger.service.
By default, pmlogger collects data every second and stores logged data in the /var/log/pcp/pmlogger/HOSTNAME directory. After the data has been collected into a pmlogger archive, tools, like pmval, may be used to query data from it.
The log file name will begin with the ISO-formatted date. In addition to the log file, several other files are created to store metadata and index information.
Once the log has been created, command-line pcp tools use the -a option to indicate that they should be run against a data archive instead of live data. The pmval command also has options to specify the start and end times that should be used if an administrator desired to narrow the data to a specific time window.
# pmval -a /var/log/pcp/pmlogger/thegeekdiary.mylabserver.com/20190106.04.30.0 kernel.all.load metric: kernel.all.load archive: /var/log/pcp/pmlogger/thegeekdiary.mylabserver.com/20190106.04.30.0 host: thegeekdiary.mylabserver.com start: Sun Jan 6 04:31:00 2019 end: Sun Jan 6 04:34:00 2019 semantics: instantaneous value units: none samples: 181 interval: 1.00 sec 04:31:00.572 No values available 1 minute 5 minute 15 minute 04:31:01.572 0.1600 5.000E-02 6.000E-02 04:31:02.572 0.1600 5.000E-02 6.000E-02 04:31:03.572 0.1600 5.000E-02 6.000E-02 04:31:04.572 0.1600 5.000E-02 6.000E-02 04:31:05.572 0.1600 5.000E-02 6.000E-02 04:31:06.572 0.1600 5.000E-02 6.000E-02 04:31:07.572 0.1600 5.000E-02 6.000E-02 04:31:08.572 0.1600 5.000E-02 6.000E-02 04:31:09.572 0.1600 5.000E-02 6.000E-02 ...
If an administrator is interested in the load averages from 03:03:00 until 03:04:00 on December 24, 2018:
~]# pmval -a /var/log/pcp/pmlogger/serverX.example.com/20150224.00.10.0 kernel.all.load -S '@ Tue Dec 24 03:03:00 2018' -T '@ Tue Dec 24 03:04:00 2018'
Alternate time specifications can be found in the PCPIntro(1) man page.