The thread dump is a well-known feature of the Java Virtual Machine. It is a snapshot of the JVM that helps the WebLogic administrator analyze which Java method the Java threads were executing in at the exact moment the snapshot was taken. As in the rest of the book, the JVM used in this recipe is Oracle JRockit JVM.
Thread dump provides a snapshot of the current active live threads. It provides the stack trace of all the java threads in the JVM. It is used when the server is hung and we want to see the threads executing and take their dump.
There are different ways to take thread dump.
Command line
use the below command when you know the port number. just run the same command 3 times in 5 sec intervel of gaps.
$ kill -3 PID
WebLogic.Admin utility
javaWebLogic.Admin adminurl -username WebLogic -password WebLogic1 THRED_DUMP
Admin console
Step 1: login to the admin console
Step 2: Click on server
Step 3: Navigate to servers
Step 4: Click monitor tab
Step 5: Click on tread
Step 6: Click on the dumpthread stack.
Location of the Thread Dump:
/DOMAIN_NAME/servers/logs/KK_stdout.log
What is Thread dump?
Thread Dump is a textual dump of all active threads and monitors of Java apps running in a Virtual Machine. When we will take Thread dump? (Scenarios)
- Scenario 1: when server is hang Position, i.e. that time server will not respond to coming requests.
- Scenario 2: While sever is taking more time to restart
- Scenario 3: When we are Getting exception like “java.lang.OutOfMemoryException”
- Scenario 4: Process running out of File descriptors. Server cannot accept further requests because sockets cannot be created
- Scenario 5: Infinite Looping in the code
How many ways take Thread Dumps?
Many types we have to take a Thread dumps. As per your flexibility you can choose one Procedure. For analyzing take dumps some Intervals (like every 10mins, 10mins etc.).
Generating Dump Talking Procedures
1. Take Thread dump from Console by Using of below command:
$ kill -3 PID
For Getting PID, Use this Command ps –ef | grep “java”
Here The Output of the Thread Dump will be generated in the Server STDOUT. Note: If a process is not responding to kill -3 then it’s a JVM bug.
2. Generation Thread Dump via Admin Console.
a. login to Admin Console(with Admin Username/Password)
b. Click on Server, after choose your server
c. Goto Monitoring TAB
d. Goto Threads TAB, after click on “Dump Thread Stack” Button
e. Now you can view the all the Threads in Same page
f. Copy and paste in a txt file.
3. We can Collect Thread Dump Using “WebLogic.Admin” which is deprecated, but still available or may be available in near future as well As I think because it is one of the best debugging utility for Admins.
java WebLogic.Admin -url t3://hostname: port -username Weblogic -password Weblogic THREAD_DUMP
This Thread Dumps will be generated in Servers STDOUT file.
4. Getting Thread Dumps by using Jstack Utility.
a. jstack –m (to connect to a live java process)
b. jstack –m [server_id@] ### (to connect to a remote debug server) (-m Means print both java and native frames (mixed mode))
5. By Using WLST Script, can contain extension of (.py)
connect(‘weblogic’,'weblogic’,'t3://hostname:port′) cd (”Servers’) ls() cd (‘AdminServer’) ls() threadDump()
Execute this Script in console.
What can I Analysis with Thread Dumps?
We need to analyze the thread dumps for analyzing running threads and their states to identifying.
How can I analysis thread dumps?
At this time, Oracle does not have its own Thread Dump analysis tool that would be offered to customers with licensing and support. However, there are several tools recommended by Oracle support which are useful in assessing performance issues with customer environments for Fusion Middleware Products.
Some of these tools include the following:
Samurai
Description: Samurai is a light weight open source tool that extract thread dumps and GC stats from log files, and presents them in different table views with color code for easy visual analysis.
Thread Dump Analyzer
Description: Thread Dump Analyzer is a swing GUI for analyzing thread dumps and heap information.
Thread Logic
Description: Thread Logic is built upon Thread Dump Analyzer by adding logic for common patterns found in application servers.
Actions taken for Issue resolving
1. Classic Dead Locks
Look for the threads waiting for monitor entry
For Example:
ExecuteThread: '95' for queue: 'default'" daemon prio=5 tid=0x411cf8 nid=0x6c waiting for monitor entry [0xd0f80000..0xd0f819d8]
at weblogic.common.internal.ResourceAllocator.release(ResourceAllocator.java:766)
at weblogic.jdbc.common.internal.ConnectionEnv.destroy(ConnectionEnv.java:590)
Reason: The above thread is waiting to acquire lock on Resource Allocator object. The next step is to identify the thread that is holding the Resource Allocator object.
ExecuteThread: '0' for queue: '__weblogic_admin_rmi_queue'" daemon prio=5 tid=0x41b978 nid=0x77 waiting for monitor entry [0xd0480000..0xd04819d8]
at weblogic.jdbc.common.internal.ConnectionEnv.getPrepStmtCacheHits(ConnectionEnv.java:174)
at weblogic.common.internal.ResourceAllocator.getPrepStmtCacheHitCount (ResourceAllocator.java:1525)
Reason: This thread is holding lock on source Allocator object, but is waiting for Connection Env object. This is a classic deadlock.
2. Threads in wait() state:
A sample dump:
ExecuteThread: '10' for queue: 'SERV_EJB_QUEUE'" daemon prio=5 tid=0x005607f0 nid=0x30 in Object.wait() [83300000..83301998] at java.lang.Object.wait(Native Method) - waiting on <0xc357bf18> (a weblogic.ejb20.pool.StatelessSessionPool) at weblogic.ejb20.pool.StatelessSessionPool.waitForBean(StatelessSessionPool.java:222) Reason: The above thread would come out of wait() under two conditions (Depending on application logic)
1. One of the thread available in the execute queue pool would call notify() on this object when an instance is available. (If the wait() is indefinite). This can cause the thread to hang for ever if server never does a notify() to this object.
2. If the timeout exceeds, the thread would throw an exception and back to execute queue thread pool.