The Geek Diary

How to Troubleshoot Performance Issues on Linux – Beginners Guide

by

Following is the list of OS commands apart from basic commands output that are useful in diagnosing the problems at OS end causing the slow performance to the Database or application side.

1. iotop
The iotop command is top like utility for disk I/O. It watches I/O usage information output by the Linux kernel (requires v2.6.20 or later) and displays a table of current I/O usage by processes or threads on the system. This requires root user to run the command.

2. To collect the filesystem cache information using “free” command output to get the filesystem cache usage:
“free” command to check for memory usage. “Buffers” represent how much portion of RAM is dedicated to cache disk block. “Cached” is similar like “Buffers”, only this time it caches pages from file reading.

A buffer is something that has yet to be “written” to disk. A cache is something that has been “read” from the disk and stored for later use.

3. To check the list of OS patches applied in Linux using below command. This will help to identify the changes CT did in past:

# rpm -qa --last > last-rpms.txt
Note: This will list with applied patches with date & time.

4. To collect the info about Memory/CPU/IO/NUMA etc for any changes from dmesg output. Can be collected every 30 seconds:

– To display hardware information related to Ethernet port eth0:

$ dmesg | grep -i eth0

– To display total memory available and shared memory details:

$ dmesg | grep -i Memory

– To display tty information:

$ dmesg | grep -i tty

– To display NUMA info:

$ dmesg | grep -i numa

– To display hard disks info:

$ dmesg | grep -i sda

– To display CPU related info:

$ dmesg | grep -i CPU

5. To collect global system messages log:

/var/log/messages
/var/log/dmesg

The dmesg command shows the current content of the kernel syslog ring buffer messages while the /var/log/dmesg file contains what was in that ring buffer when the boot process last completed.

/var/log/dmesg stays unchanged until next reboot. Old mesgs are replaced by new messages in the ring buffer. dmesg is the subset of /var/log/messages and is maintained in ring buffer. /var/log/messages includes all the system messages including from starting of the system along with the messages in dmesg. In a nutshell logs from dmesg are dumped in /var/log/messages.

Common Linux log files names and usage:

  • /var/log/messages: General message and system related stuff.
  • /var/log/auth.log: Authenication logs.
  • /var/log/kern.log: Kernel logs.
  • /var/log/cron.log: Crond logs (cron job).
  • /var/log/maillog: Mail server logs.
  • /var/log/qmail/: Qmail log directory (more files inside this directory).
  • /var/log/httpd/: Apache access and error logs directory.
  • /var/log/lighttpd/: Lighttpd access and error logs directory.
  • /var/log/boot.log: System boot log.
  • /var/log/mysqld.log: MySQL database server log file.
  • /var/log/secure or /var/log/auth.log: Authentication log.
  • /var/log/utmp or /var/log/wtmp: Login records file.
  • /var/log/yum.log: Yum command log file.

6. To check whether hugepages are used. To execute only at start and end of OSW collection:

# grep -i Hugepages /proc/meminfo
# cat /proc/meminfo

7. To check the filesystem used at OS. To be executed only once:

# cat /etc/fstab

8. To collect the info about system activity for the last week:

/var/log/sa has files for sa and sar which has the CPU usage history information.

# ls -al /var/log/sa | grep "Mar 12"
-rw-r--r--. 1 root root 721996 Mar 12 23:50 sa12
-rw-r--r--. 1 root root 655607 Mar 12 23:53 sar12
# sar -W -f /var/log/sa/sa12     ### shows paging/swapping historical info
# less /var/log/sa/sa12

09. To get the CPU Arch information:

# lscpu
# cat /proc/cpuinfo

11. To get the kernel shared memory/semaphores info:

# ipcs -lm
# ipcs -ls
# cat /proc/sys/kernel/shmmax
# cat /proc/sys/kernel/shmmni
# getconf PAGE_SIZE
# cat /proc/sys/kernel/shmall

12. To get info about File handles:

# cat /proc/sys/fs/file-max
# cat /proc/sys/fs/file-nr

13. To get info about ulimit for Open file descriptors/ maxproc:

# su - oracle
# ulimit -n
# ulimit -u

14. To check the maximum IO size per IO request:

# cat /sys/block//queue/max_sectors_kb

To determine the maximum I/O size in bytes when Asynch IO is used:

# cat /proc/sys/fs/aio-max-size

or

# cat /proc/sys/fs/aio-max-nr

15. To show the disk device model, standards, configuration, capacity etc:

Example:

# hdparm -I /dev/sda
# sudo hdparm -i /dev/sda

16. To check the block size for particular device:

Example:

# blockdev --getbsz /dev/xvda1
# tune2fs -l /dev/vgxx/lvolx
NOTE: All the commands mentioned above are tested in Linux environment only.

You May Also Like