In Solaris, there are so many tools for analyzing and debugging every single part of the OS. the most common is the tools for Process Analysis. Since there are so many tools for process analysis, it can be helpful to group them into general. The general syntax is as follows:
$ ptool pid $ ptool pid/lwpid
In summary:
- prstat – For viewing overall process status<.li>
- ps – To print process status and information
- ptree – To print a process ancestry tree
- pgrep; pkill – To match a process name; to send a signal
- pstop; prun – To freeze a process; to continue a process
- pwait – To wait for a process to finish
- preap – To reap zombies
- pstack – For inspecting stack backtraces
- pmap – For viewing memory segment details
- pfiles – For listing file descriptor details
- ptime – For timing a command
- psig – To list signal handlers
- pldd – To list dynamic libraries
- pflags; pcred – To list tracing flags; to list process credentials
- pargs; pwdx – To list arguments, env; to list working directory
- plockstat – For observing lock activity
The following will explain each tool in brief:
Process Tree: ptree
The process parent-child relationship can be displayed with the ptree command. By default, all processes within the same process group ID are displayed.
$ ptree 22961 301 /usr/lib/ssh/sshd 21571 /usr/lib/ssh/sshd 21578 /usr/lib/ssh/sshd 21580 -ksh 22961 /opt/filebench/bin/filebench 22962 shadow -a shadow -i 1 -s ffffffff10000000 -m /var/tmp/fbench9Ca 22963 shadow -a shadow -i 2 -s ffffffff10000000 -m /var/tmp/fbench9Ca 22964 shadow -a shadow -i 3 -s ffffffff10000000 -m /var/tmp/fbench9Ca 22965 shadow -a shadow -i 4 -s ffffffff10000000 -m /var/tmp/fbench9Ca
Grepping for Processes: pgrep
The pgrep command provides a convenient way to produce a process ID list matching certain criteria.
$ pgrep filebench 22968 22961 22966 22979 ...
Killing Processes: pkill
The pkill command provides a convenient way to send signals to a list or processes matching certain criteria.
$ pkill -HUP in.named
If the signal is not specified, the default is to send a SIGTERM.
Temporarily Stop a Process: pstop
A process can be temporarily suspended with the pstop command.
$ pstop 22961
Making a Process Runnable: prun
A process can be made runnable with the prun command.
$ prun 22961
Wait for Process Completion: pwait
The pwait command blocks and waits for termination of a process.
$ pwait 22961 (sleep...)
Reap a Zombie Process: preap
A zombie process can be reaped with the preap command, which was added in Solaris 9.
$ preap 22961 (sleep...)
Process Stack: pstack
The stacks of all or specific threads within a process can be displayed with the pstack command.
$ pstack 23154 23154: shadow -a shadow -i 193 -s ffffffff10000000 -m /var/tmp/fbench9Cai2S ----------------- lwp# 1 / thread# 1 -------------------- ffffffff7e7ce0f4 lwp_wait (2, ffffffff7fffe9cc) ffffffff7e7c9528 _thrp_join (2, 0, 0, 1, 100000000, ffffffff7fffe9cc) + 38 0000000100018300 threadflow_init (ffffffff3722f1b0, ffffffff10000000, 10006a658, 0, 0, 1000888b0) + 184 00000001000172f8 procflow_exec (6a000, 10006a000, 0, 6a000, 5, ffffffff3722f1b0) + 15c 0000000100026558 main (a3400, ffffffff7ffff948, ffffffff7fffeff8, a4000, 0, 1) + 414 000000010001585c _start (0, 0, 0, 0, 0, 0) + 17c ----------------- lwp# 2 / thread# 2 -------------------- 000000010001ae90 flowoplib_hog (30d40, ffffffff651f3650, 30d40, ffffffff373aa3b8, 1, 2e906) + 68 00000001000194a4 flowop_start (ffffffff373aa3b8, 0, 1, 0, 1, 1000888b0) + 408 ffffffff7e7ccea0 _lwp_start (0, 0, 0, 0, 0, 0)
The pstack command can be very useful for diagnosing process hangs or the status of core dumps. By default it shows a stack backtrace for all the threads within a process. It can also be used as a crude performance analysis technique; by taking a few samples of the process stack, you can often determine where the process is spending most of its time.
Process Memory Map: pmap -x
The pmap command inspects a process, displaying every mapping within the process address space. The amount of resident, nonshared anonymous, and locked memory is shown for each mapping. This allows you to estimate shared and private memory usage.
$ pmap -x 102908 102908: sh Address Kbytes Resident Anon Locked Mode Mapped File 00010000 88 88 - - r-x-- sh 00036000 8 8 8 - rwx-- sh 00038000 16 16 16 - rwx-- [ heap ] FF260000 16 16 - - r-x-- en_.so.2 FF272000 16 16 - - rwx-- en_US.so.2 FF280000 664 624 - - r-x-- libc.so.1 FF336000 32 32 8 - rwx-- libc.so.1 FF360000 16 16 - - r-x-- libc_psr.so.1 FF380000 24 24 - - r-x-- libgen.so.1 FF396000 8 8 - - rwx-- libgen.so.1 FF3A0000 8 8 - - r-x-- libdl.so.1 FF3B0000 8 8 8 - rwx-- [ anon ] FF3C0000 152 152 - - r-x-- ld.so.1 FF3F6000 8 8 8 - rwx-- ld.so.1 FFBFE000 8 8 8 - rw--- [ stack ] -------- ----- ----- ----- ------ total Kb 1072 1032 56 -
Process File Table: pfiles
A list of files open within a process can be obtained with the pfiles command.
# pfiles 21571 21571: /usr/lib/ssh/sshd Current rlimit: 256 file descriptors 0: S_IFCHR mode:0666 dev:286,0 ino:6815752 uid:0 gid:3 rdev:13,2 O_RDWR|O_LARGEFILE /devices/pseudo/mm@0:null 1: S_IFCHR mode:0666 dev:286,0 ino:6815752 uid:0 gid:3 rdev:13,2 O_RDWR|O_LARGEFILE /devices/pseudo/mm@0:null 2: S_IFCHR mode:0666 dev:286,0 ino:6815752 uid:0 gid:3 rdev:13,2 O_RDWR|O_LARGEFILE /devices/pseudo/mm@0:null 3: S_IFCHR mode:0000 dev:286,0 ino:38639 uid:0 gid:0 rdev:215,2 O_RDWR FD_CLOEXEC /devices/pseudo/crypto@0:crypto 4: S_IFIFO mode:0000 dev:294,0 ino:13099 uid:0 gid:0 size:0 O_RDWR|O_NONBLOCK FD_CLOEXEC 5: S_IFDOOR mode:0444 dev:295,0 ino:62 uid:0 gid:0 size:0 O_RDONLY|O_LARGEFILE FD_CLOEXEC door to nscd[89] /var/run/name_service_door
Execution Time Statistics for a Process: ptime
A process can be timed with the ptime command for accurate microstate accounting instrumentation.
$ ptime sleep 1 real 1.203 user 0.022 sys 0.140
Process Signal Disposition: psig
A list of the signals and their current disposition can be displayed with psig.
$ psig $$ 15481: -zsh HUP caught 0 INT blocked,caught 0 QUIT blocked,ignored ILL blocked,default TRAP blocked,default ABRT blocked,default EMT blocked,default FPE blocked,default KILL default BUS blocked,default SEGV blocked,default SYS blocked,default PIPE blocked,default ALRM blocked,caught 0 TERM blocked,ignored USR1 blocked,default USR2 blocked,default CLD caught 0 PWR blocked,default WINCH blocked,caught 0 URG blocked,default POLL blocked,default STOP default
Process Libraries: pldd
A list of the libraries currently mapped into a process can be displayed with pldd. This is useful for verifying which version or path of a library is being dynamically linked into a process.
$ pldd $$ 482764: -ksh /usr/lib/libsocket.so.1 /usr/lib/libnsl.so.1 /usr/lib/libc.so.1 /usr/lib/libdl.so.1 /usr/lib/libmp.so.2
Process Flags: pflags
The pflags command shows a variety of status information for a process. Information includes the mode 32-bit or 64-bit in which the process is running and the current state for each thread within the process. In addition, the top-level function on each threads stack is displayed.
$ pflags $$ 482764: -ksh data model = _ILP32 flags = PR_ORPHAN /1: flags = PR_PCINVAL|PR_ASLEEP [ waitid(0x7,0x0,0xffbff938,0x7) ]
Process Credentials: pcred
The credentials for a process can be displayed with pcred.
$ pcred $$ 482764: e/r/suid=36413 e/r/sgid=10 groups: 10 10512 570
Process Arguments: pargs
The full process arguments and optionally a list of the current environment settings can be displayed for a process with the pargs command.
$ pargs -ae 22961 22961: /opt/filebench/bin/filebench argv[0]: /opt/filebench/bin/filebench envp[0]: _=/opt/filebench/bin/filebench envp[1]: MANPATH=/usr/man:/usr/dt/man:/usr/local/man:/opt/SUNWspro/man:/ws/on998- tools/teamware/man:/home/rmc/local/man envp[2]: VISUAL=/bin/vi ...
Process Working Directory: pwdx
The current working directory of a process can be displayed with the pwdx command.
$ pwdx 22961 22961: /tmp/filebench
Examining User-Level Locks in a Process
With the process lock statistics command, plockstat(1M) , you can observe hot lock behavior in user applications that use user-level locks. The plockstat command uses DTrace to instrument and measure lock statistics.
# plockstat -p 27088 ^C Mutex block Count nsec Lock Caller ------------------------------------------------------------------------------- 102 39461866 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 4 21605652 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 11 19908101 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 12 16107603 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 10 9000198 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 14 5833887 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 10 5366750 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 120 964911 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 48 713877 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 52 575273 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 89 534127 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 14 427750 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28 1 348476 libaio.so.1`__aio_mutex libaio.so.1`_aio_req_add+0x228 Mutex spin
Conclusion
using the above tools will help on figuring out how the process being handled by the OS and therefore, is a good start of troubleshooting the most common issues related to processes.