The Oracle OSWatcher Black Box (OSWbb) product is a collection of shell scripts intended to collect and archive operating system and network metrics to aid in diagnosing performance issues. OSWbb operates as a set of background processes on the server and gathers data on a regular basis, invoking such UNIX utilities as vmstat, netstat, iostat, top, and others.
Beginning with release 4.0.0, OSWbb includes a built-in analyzer called OSWbba, which analyzes the data that OSWbb collects. It provides information on system slowdowns, hangs, and other performance problems. It also provides the ability to graph vmstat and iostat data.
OSWbb is particularly useful for Oracle Real Application Clusters (RAC) and Oracle Grid Infrastructure configurations. OSWbb is included in the RAC-DDT (Diagnostic Data Tool) script file, but is not installed by RAC-DDT.
OSWatcher is certified to run on the following platforms:
- AIX
- Solaris
- HP-UX
- Linux
Installing OSWbb
You must install OSWbb on each node where data is to be collected. For RAC or shared disk systems, each node requires an OSWbb installation into a unique directory (for example, /oswbb_node1 and /oswbb_node2). OSWatcher can be downloade through MOS Doc ID 301137.1 as a tar file. Although the installation and configuration procedure of oswbb for all the Operating systems is almost the same. The post uses RHEL 7 system for the purpose of demonstration.
After downloading the TAR file, copy the file to the directory where OSWbb is to be installed and run the following command:
# tar xvf oswbb.tar
A directory named oswbb is created which houses all the files associated with oswbb. OSWatcher is now installed. If you list the files/directories inside the oswbb directory :
# ls analysis call_uptime.sh Exampleprivate.net ltop.sh OSWatcherFM.sh oswib.sh oswsub.sh src tarupfiles.sh xtop.sh call_du.sh docs ifconfigsub.sh mpsub.sh OSWatcher.sh oswnet.sh profile startOSWbb.sh tar_up_partial_archive.sh call_sar.sh Example_extras.txt iosub.sh nfssub.sh oswbba.jar oswrds.sh psmemsub.sh stopOSWbb.sh vmsub.sh
Starting OSWbb
To start the OSWbb utility, execute the startOSWbb.sh shell script. The startOSWbb.sh script accepts two optional arguments that control the frequency (in seconds) that data is collected and the number of hours worth of data to archive. If you do not enter any arguments, the script runs with default values of 30 and 48, meaning collect data every 30 seconds and store the last 48 hours of data in archive files.
The following example starts the tool and collects data at 60-second intervals and logs the last 10 hours of data to archive files. Some of the output produced when starting the tool is shown:
# ./startOSWbb.sh 60 10 Setting the archive log directory to/data/oswbb/archive Testing for discovery of OS Utilities... VMSTAT found on your system. IOSTAT found on your system. MPSTAT found on your system. IFCONFIG found on your system. NETSTAT found on your system. TOP found on your system. ...... Starting OSWatcher v7.3.3 on Thu Nov 10 17:48:51 IST 2016 With SnapshotInterval = 60 With ArchiveInterval = 10 ..... Data is stored in directory: /data/oswbb/archive Starting Data Collection... oswbb heartbeat:Thu Nov 10 17:48:56 IST 2016 oswbb heartbeat:Thu Nov 10 17:49:56 IST 2016 ....
OSWbb started successfully, the discovery process completed, and data collection begins. You can exit out of the session, once the data collection starts.
Stopping OSWbb
To stop OSWbb, execute the stopOSWbb.sh shell script. This terminates all processes associated with OSWbb and is the normal, graceful mechanism for stopping the tool.
# ./stopOSWbb.sh
Viewing the data collected
OSWatcher invokes these distinct operating system utilities, each as a distinct background process, as data collectors. These utilities will be supported, or their equivalents, as available for each supported target platform.
- ps
- top
- ifconfig
- mpstat
- iostat
- netstat
- traceroute
- vmstat
- meminfo (Linux Only)
- slabinfo (Linux Only)
Use the cd command to change to the archive directory, and then use ls to view the contents of the directory. The archive directory is created when OSWbb is started for the first time. The directory contains 10 subdirectories, one for each data collector.
# cd archive/ # ls -l total 40 drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswifconfig drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswiostat drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswmeminfo drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswmpstat drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswnetstat drwxr-xr-x. 2 root root 4096 Nov 10 17:48 oswprvtnet drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswps drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswslabinfo drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswtop drwxr-xr-x. 2 root root 4096 Nov 10 18:00 oswvmstat
To check a sample, we will cd into a directory oswiostat and view the content of one of the files :
# cd oswiostat/ # ls -l total 20 -rw-r--r--. 1 root root 12067 Nov 10 17:59 localhost.localdomain_iostat_16.11.10.1700.dat -rw-r--r--. 1 root root 7047 Nov 10 18:06 localhost.localdomain_iostat_16.11.10.1800.dat
# cat localhost.localdomain_iostat_16.11.10.1700.dat
Linux OSWbb v7.3.3
zzz ***Thu Nov 10 17:48:56 IST 2016
avg-cpu: %user %nice %system %iowait %steal %idle
2.04 0.00 2.04 2.04 0.00 93.88
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 1.02 0.00 16.33 0.00 32.00 0.03 28.00 28.00 0.00 28.00 2.86
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 1.02 0.00 16.33 0.00 32.00 0.03 28.00 28.00 0.00 28.00 2.86
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zzz ***Thu Nov 10 17:49:56 IST 2016
avg-cpu: %user %nice %system %iowait %steal %idle
2.00 0.00 2.00 0.00 0.00 96.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
As you can see iostat command output is collected at predefined interval. The historical archive data comes handy when you want to gauge the load on the server for a period of time.
Uninstalling oswbb
To de-install OSWatcher issue the following command on the oswbb directory.
# rm -rf oswbb