Using the system journal
By default, newer systemd based linux systems now uses two logging services for the system logs:
- systemd-journald, which is configured to only keep logs in memory
rsyslogd, which gets messages sent to it by systemd-journald (and others) and stores them on disk.
To view messages in the system journal, a tool called journalctl can be used. If used without any parameters it will show the full contents of the system journal, presented in a pager (by default less is used). The output of journalctl can be modified by using both options and filters. Options can be used to change the number of lines displayed, to turn on follow mode, change the displayed field, specify a time range, etc. Filters can be used to modify for what services and units information is displayed, which executables to display information for, etc.
journalctl examples
journalctl -ef
Jump to the end of the journal (-e, and enable follow mode (-f). This will keep the journal open on screen, displaying new messages as they come in.
# journalctl -ef
journalctl _SYSTEMD_UNIT=sshd.service
This will display all messages generated by the sshd.service systemd unit.
# journalctl _SYSTEMD_UNIT=sshd.service
journalctl -u sshd.service
This will display all messages generated by, and about, the sshd.service systemd unit.
# journalctl -u sshd.service
journalctl -p emerg..err
Display all messages in the journal with a priority in the range emerg up to and including err.
# journalctl -p emerg..err
If a single priority is specified, for example, -p err, all messages up to and including that level are displayed.
# journalctl -p err
journalctl -b -1
Only show messages from the last system boot. This is useful for searching for information about a system crash. This requires a persistent journal to be configured.
# journalctl -b -1
journalctl –since “2015-02-02 20:30:00” –until “2015-03-31 12:00:00”
Displays all messages between February 2, half past eight in the evening, and noon on March 31st. This requires a persistent journal to be configured.
# journalctl --since "2015-02-02 20:30:00" --until "2015-03-31 12:00:00"
journalctl -o verbose
Use verbose output mode (-o verbose). This will show all fields stored in the journal with their field name and contents. All field names can be used as filters on the journalctl command line.
# journalctl -o verbose
For a complete list of options and filters, refer to the journalctl(1) man page.
# man journalctl
Persisting the journal
By default, CentOS/RHEL 7 stores the system journal in /run/log/journal, which is stored on a tmpfs. This implies that on a reboot all stored information will be lost. If the directory /var/log/journal is present the journal will be stored there, thus enabling a persistent journal across reboots.
Enabling a persistent journal can be done by using the following steps:
1. Create the directory /var/log/journal.
# mkdir /var/log/journal
2. Set the group ownership of the new directory to systemd-journal, and the permissions to 2755.
# chown root:systemd-journal /var/log/journal # chmod 2755 /var/log/journal
3. Inform systemd-journald that the new location should be used by sending a USR1 signal to it. A reboot will also suffice.
# killall -USR1 systemd-journald
Enabling verbose information
Many tools and services can increase the amount of logging they perform, as well as the amount of information they display when run from the command line, by using various configuration options or command-line flags.
Command-line options typically include -v, which can sometimes be specified multiple times, to increase verbosity, or include a –debug option that can be used. Services will typically have configuration options, either in their main configuration file or in /etc/sysconfig/SERVICENAME, that can be used to increase their logging level and/or verbosity as well. Refer to the documentation for these individual services to increase their verbosity and logging levels.