The post outlines the steps to run the syslog daemon in debug mode.
1. Stop the syslogd daemon:
# /etc/init.d/syslog stop ### Prior to Solaris 10 # svcadm disable svc:/system/system-log:default ### Solaris 10 and above
2. The environment variable SYSLOGD_DEBUG affects the amount of output from syslog in debug mode. The default value is 1, which provides the least amount of output. Setting this variable to 5 provides the most output. For example:
# SYSLOGD_DEBUG=5 # export SYSLOGD_DEBUG
3. Start the daemon in debug mode:
# /usr/sbin/syslogd -d
All output will go to the screen. You may want to redirect the output to a file instead. Also, the daemon will run in the foreground.
Interpreting syslogd debug output
The first portion of syslogd debug output contains some network and system configuration:
# /usr/sbin/syslogd -d main(1): Started at time Tue Sep 23 17:00:39 2014 hnc_init(1): hostname cache configured 2037 entry ttl:1200 getnets(1): found 1 addresses, they are: 0.0.0.0.2.2 amiloghost(1): testing 10.135.88.55.2.2 conf_init(1): I am loghost cfline(1): (*.err;kern.notice;auth.notice /dev/sysmsg) cfline(1): (*.err;kern.debug;daemon.notice;mail.crit /var/adm/messages) cfline(1): (auth.debug /var/adm/auth.log) cfline(1): (*.alert;kern.err;daemon.err operator) cfline(1): (*.alert root) cfline(1): (*.emerg *) cfline(1): (auth.debug /var/adm/authlog) cfline(1): (mail.debug /var/log/syslog) syslogd: version 1.105 Started: Tue Sep 23 17:00:39 2014 Input message count: system 0, network 0 # Outputs: 8
The next portion of syslogd debug output is a summary of the syslog selectors (facility.level) and their corresponding actions (where the messages will be logged):
------------------------ priority = [file, facility] ------------------------ 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 -------------------------------------------------- 5 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 X CONSOLE: /dev/sysmsg 7 3 2 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 X FILE: /var/adm/messages X X X X 7 X X X X X X X X X X X X X X X X X X X X FILE: /var/adm/auth.log 3 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 X USERS: operator 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 X USERS: root 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 X WALL: X X X X 7 X X X X X X X X X X X X X X X X X X X X FILE: /var/adm/authlog X X 7 X X X X X X X X X X X X X X X X X X X X X X FILE: /var/log/syslog
There are 25 columns (0-24). The first 24 columns correspond to syslog facilities (0-23) which are described in /usr/include/sys/syslog.h. For example, in Solaris 10 these are:
| Facility | Description |
|---|---|
| 0 | kern |
| 1 | user |
| 2 | |
| 3 | daemon |
| 4 | auth |
| 5 | syslog |
| 6 | lpr |
| 7 | news |
| 8 | uucp |
| 9 | reserved |
| 10 | reserved |
| 11 | reserved |
| 12 | reserved |
| 13 | audit |
| 14 | reserved |
| 15 | cron |
| 16 | local0 |
| 17 | local1 |
| 18 | local2 |
| 19 | local3 |
| 20 | local4 |
| 21 | local5 |
| 22 | local6 |
| 23 | local7 |
The number in each column is the syslog level numbered 0-7 (also described in /usr/include/sys/syslog.h) that is being reported for that facility:
| Syslog Level | Description |
|---|---|
| 0 | emerg |
| 1 | alert |
| 2 | crit |
| 3 | error |
| 4 | warning |
| 5 | notice |
| 6 | info |
| 7 | debug |
Therefore the following line can be decoded as follows:
5 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 X CONSOLE: /dev/sysmsg
– Column 0 is set to 5, which specifies that kern.notice will be logged.
– Column 4 is set to 5, which specifies that auth.notice will be logged.
– All other columns contain 3, which corresponds to *.err.
– Any messages generated by these selectors will be logged to CONSOLE (/dev/sysmsg).
The above agrees with the following entry in /etc/syslog.conf:
*.err;kern.notice;auth.notice /dev/console
Here is another line of syslogd debug output from above:
X X X X 7 X X X X X X X X X X X X X X X X X X X X FILE: /var/adm/auth.log
– Column 4 is set to 7, which specifies that auth.debug will be logged.
– All other columns contain X, which means that these facilities will NOT be logged.
– Any messages generated by these selectors will be logged to /var/adm/auth.log.
And here is the corresponding entry in /etc/syslog.conf:
auth.debug /var/adm/auth.log
Recognizing problems in syslogd debug output
There can be only one level per facility per row. A facility, such as kern, should only be declared once per line in /etc/syslog.conf. For example, the following is an incorrect entry in /etc/syslog.conf:
kern.debug;kern.err;kern.notice /dev/console
This would produce the following syslogd debug output:
7 X X X X X X X X X X X X X X X X X X X X X X X X CONSOLE: /dev/console
The correct entry in /etc/syslog.conf would be:
kern.debug /dev/console
Missing output files
If an output file is specified in /etc/syslog.conf, the file must exist before syslogd is started. Otherwise, the following errors might be seen in syslogd debug output:
cfline(1): (auth.debug /var/adm/auth.log) logerror(1): syslogd: /var/adm/auth.log: No such file or directory logerror_to_console(1): syslogd: /var/adm/auth.log: No such file or directory
or
writemsg(3): Logging msg 'syslogd: /var/adm/auth.log: No such file or directory' to FILE /var/adm/messages
To fix the problem, create the missing file:
# touch /var/adm/auth.log
and restart syslogd.
Testing a specific selector (facility.level) with logger while running syslogd in debug mode
For example, in a separate window:
# logger -p auth.notice "testing auth.notice"
In the window where syslogd was started in debug mode, the following output is seen:
writemsg(2): Logging msg 'Sep 24 09:55:08 dwryder: [ID 702911 auth.notice] testing auth.notice' to FILE /var/adm/auth.log
Stopping syslogd in debug mode and re-starting in normal mode
1. Kill the syslogd debug process or use ^C to stop it in the window where it was started.
2. Re-start syslog in normal mode:
Prior to Solaris 10:
# /etc/init.d/syslog start
Check that the process is running:
# ps -ef|grep syslog|egrep -v grep root 984 1 0 11:08:17 ? 0:00 /usr/sbin/syslogd
In Solaris 10 and above:
# svcadm enable svc:/system/system-log:default
Check that the system-log service is online:
# svcs svc:/system/system-log:default STATE STIME FMRI online 16:12:21 svc:/system/system-log:default