How to take Heap Dump in WebLogic and analyze Heap Dump

Heap Dump is very important in terms of application performance analysis running in WebLogic. Here is a short and nice tutorial describing how heap dump is been taken in run time and analyze the same with specific tool set.

 

Heap Dump is very important in terms of application performance analysis running in WebLogic. Here is a short and nice tutorial describing how heap dump is been taken in run time and analyze the same with specific tool set.

• Taking Heap Dump:
  
  You can take heap dump by several java inbuilt tool few from command line and few from GUI. Few tools like Jmap, Jhat or Jrcmd command line tool which come free with java packages (JDK or JRockit). You can use your GUI based local system heap monitoring tools like JConsole, Jvisualvm, etc. and by connecting the WebLogic sever remotely.
  
  
• Command Line Tool: Personally I like to use Jrcmd for taking heap dump. Although rest tools are also very good, but here I will explain the procedure taking reference with Jrcmd (JRockit tool).

1. Login into the server with application user (su – application user) where WebLogic is running.
2. Go to the jrmc location of WebLogic: /opt/weblogic-10.3.3/jrmc-4.0.1-1.6.0/bin
3. If you run jrmcd script, it will show all the running PIDs of WebLogic server and JVM:[weblogic@LinuxProd01 bin]# ./jrcmd540 weblogic.Server5187 jrockit.tools.jrcmd.JrCmd
4. To take a heap dump, put the following command with proper application PID (Take heap dump in .hprof/.bin format):[weblogic@LinuxProd01 bin]$ ./jrcmd 540 hprofdump filename=/tmp/heapdump.bin

It will create the heap dump in /tmp location.

Note: Taking heap dump in running server can slow its performance.

GUI Tool:

Go to your Java directory in C drive and open Jvisualvm. Connect to the remote server by JMX remote port where you will get a scope to acquire heap dump.

• Analyzing Heap Dump: Analyzing heap dump is a very important thing and luckily there are several open platform tools to analyze the same including Eclipse Memory Analyzer, IBM HeapAnalyzer, VisualVM, Jhat etc. Here I will elaborate the process of heap dump analysis via Eclipse Memory Analyzer.

1. Start Eclipse Memory Analyzer and load the heapdump.bin file.

1. 

1. Once loaded, it will start analyzing the leak suspected objects in the dump
   
   

2. Once the primary analysis is done, the tool will come up with biggest objects from within heap dump mapped in a pie chart.

3. Now click on the Leak Suspect link at the bottom and wait for sometime. The tool will come up with probable list of Leak Suspected object list and the related loads of classes. Basically these are the objects, which are called a lot of times consuming a huge proportion of the  heap memory. You have to be careful about the analysis part. All the leak suspects are not criminals.

4. Sometimes you can not solely rely on the analysis part based on leak suspect report. You need to see with thorough depth that how many types of classes and threads are occupying most of the heap memory. Class loading is entirely dependent on thread utilization. So if there are piles of threads get accumulated for a longer period of time, may be that is where the connections are not getting properly closed resulting into leakage of memory which can only be re-utilized after a graceful restart of the server, which will not help for production environment if you are concerned about the downtime though.
So you need to check the top consumers of the heap memory and it looks something like this

5. There is also an option of checking biggest top level dominator class from the heap dump. It will give you a notion which classes are mostly loaded during its life cycle which sometimes does not match with top consumers. Top dominator classes can give a notion that they might be quickly loaded and unloaded in a regular fashion leaving a small leak which can ultimately block the small amount of memory with each cycle.

6. Finally based on the analysis, it would be quite easier for you to dig down at the root cause. Most of the time, application code with little error can cause a major business impact. It is the responsibility of a Heap Dump Analyst to identify the error keeping in mind the application usage pattern with relationship to threads and JVM utilization. A typical study of heap dump will reveal across such scenario:

From the above analysis, you can see jdbc Connection Factory is opened too many times accumulating a larger chuck of heap space. But few classes were not closing the connections after task completion resulting in the leak. Later on, it was fixed from application end and the issue (WebLogic server not responding) got resolved.

Hope this tutorial helps further. But remember, this analysis is very crucial. You should       have a very good knowledge in JVM and thread utilization. But this comes with hands on work in practical. Try this to explore more on this and have fun.

Accessing the Data in Core Dumps

 

Accessing the data from core dumps is fairly simple, and it's well worth the effort.

If you're like most UNIX administrators, you probably have a crontab or some other housekeeping program that regularly searches your systems for core dumps, backs them up (maybe), and then deletes them.

This leaves you to try to puzzle out their cause from information contained in the AIX error log. Of course, about half the time, core dumps aren't flagged, so other methods must be used to discern their cause, such as examining them with a utility like strings or some other reader.

Chances are, that's as far as you'll get. You'll dutifully inform your application developers or DBAs about the location of the core dump and when it occurred, and then move on to other duties. Meanwhile, the core dump will sit in its archive, forgotten about. And that's unfortunate.

Core dumps contain a snapshot of the memory address space for an application -- or a database, utility or middleware -- at the time of the event that caused the core dump to happen. That's a wealth of data that, if diagnosed, could help you avoid future core dumps. So how do you access this data? It can actually be done quickly and fairly easily.

Seriously, this information can be gathered in about 10 minutes. By entering a few commands, you'll not only know which program caused the core dump, but also the file or function that lies at fault and the working thread that was interrupted.

First, you need the proper tool to examine your core dump. The dbx utility is a standard debugger that ships with AIX; it's included on the operating system distribution DVDs. Part of the bos.adt.debug fileset, dbx is usually installed by default. It has many uses, from providing a controlled environment for running programs to diagnosing misbehaving processes live and giving you the ability to step through a program one line at a time. And yes, dbx can be used to examine core dumps. Here's how: To start your examination, change your working directory to its location and call up the debugger on the core file:

 dbx -C ./core

Your output will be similar to this:

[lpar:/home/raym] dbx -C ./core
Type 'help' for help.
warning: Object file is not specified, and the object file "perl5.10.1" mentioned in
 the core file doesn't exist in the current directory or doesn't match with
 the core file (ignored). Some info may not be available.
[using memory image in ./core]
reading symbolic information ...warning: cannot open perl5.10.1
warning: no source compiled with -g
Segmentation fault in extend_brk at 0xd011119c ($t1)
0xd011119c (extend_brk+0x2dc) 90040004            stw   r0,0x4(r4)
(dbx)

In this example, I'm using a core dump that was generated on a system I was working on that was using a 32-bit version of Perl and had recently been updated to AIX 7.1. This initial screen above is basically a summary of the conditions in memory at the time of the core dump. Note the object file name in line 3; it will likely point to the executable that caused the core dump. If you're only looking to identify the program that caused the dump, most of the time, this screen will give you your answer. However, this information can be a bit nebulous, so it's often worth digging deeper.

The last line of the output (dbx) is the dbx utility's command prompt. This is where you'll enter all of your commands. To determine which program caused the dump, enter the corefile command at the (dbx) prompt:

(dbx) corefile
 Process Name:  perl5.10.1
 Version:       430
 Flags:         FULL_CORE | CORE_VERSION_1 | UBLOCK_VALID | USTACK_VALID | LE_VALID
 Signal:        SEGV
 Process Mode:  32 bit

At this point you'll be presented with several lines of information, including the name of the process that core dumped, the signal that caused the core dump -- in this case a segmentation violation (SEGV pr); most core dumps are generated by this signal -- and whether the process was executing in 32- or 64-bit mode. The identified process will, of course, point back to the executable having problems. But where in the programming code of that executable did the fault happen? To determine this, issue the dump command:

(dbx) dump
extend_brk(internal error: assertion failed at line 3915 in file frame.c
??, internal error: assertion failed at line 3915 in file frame.c
??, internal error: assertion failed at line 3915 in file frame.c
??) at 0xd011119c

In this case, we have a failed assert called from the frame.c file. An assert is simply a reaction to a Boolean expression that should always be true; a "false" reaction will generate a fault, causing the core dump and the execuatble to fail. Sometimes dbx points to a location in a specific file; sometimes it points to an offending function.

So with just two commands, you've pinpointed the program that faulted as well as the area of programming code that caused the fault. Now you're ready to fix the code. (In this case, as noted, we were running a 32-bit version of Perl on a system that had recently been upgrded to AIX v7.1. When we upgraded to a 64-bit version of Perl, our problems went away.)

Most of the time, running corefile and dump is sufficient for starting remediation of the problem that caused the core dump. However, if you work with application developers who require more information to write a patch or an upgrade, you should be aware of several other dbx commands that tell you about the status of the working thread that encountered the error:

(dbx) th -
 thread kernel                   user                         state  flag  held
        tid      pri sched state tid      pri nice sched state
        -----------------------------------------------------------------------
        mode   scope    cancellation    join- boosted cursig wchan      function
                        pending state   able

>$t1    0x2ea02af 112 other run   0x000001 1   58   other running   0x0002 0x000 no
        kernel system   no      ed      yes   0       0

Entering th - at a dbx prompt tells you a lot about the thread's state at the time of the fault, and proc gives you a full dump of the process's memory usage. Enter coremap and you'll see all of the libraries the process had loaded as well as whether they were text or data segment. Likewise, the word map by itself gives you verbose information on all of the object files in use by your process.

Analyzing core dumps with the dbx utility provides a comprehensive picture of their causes as well as clear options for dealing with them. Details about individual core dumps can be packaged in an easily understood form and presented to your vendor, the appropriate support personal and even to non-technical personnel in your management chain.

Perhaps best of all, by using the dbx utility, in most instances you'll be able to diagnose and remediate the conditions that create core dumps with any outside help. This should lead to more reliable and stable AIX systems.

Apache Configuration Redirects

Apache Configuration Redirects version 2.1.1.11

1. How to configure apache to redirect complete site to https?

Configuration:- 

Below are 3 different way to configure apache

------------------------------

RewriteEngine On
RewriteCond %{SERVER_PORT} !=443
RewriteCond %{HTTP_HOST} ^(staging2\.)?virgin-atlantic\.com$ [NC]
RewriteRule ^$ https://staging2.virgin-atlantic.com%{REQUEST_URI} [R=301,NC]

RewriteEngine On
RewriteCond %{SERVER_PORT} !=443
RewriteCond %{HTTP_HOST} ^(staging2\.)?virgin-atlantic\.com$ [NC]
RewriteRule ^$ https://staging2.virgin-atlantic.com%{REQUEST_URI} [R=301,NC]

RewriteEngine on
RewriteCond %{SERVER_PORT} !^443
RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [NE,R]

2. How to redirect to catch IE7 & 8 out of date browser?

Configuration:-

RewriteCond %{HTTP_USER_AGENT} "Trident/4.0" [NC]
RewriteCond %{HTTP_USER_AGENT} "MSIE [7-8]" [NC]
RewriteCond %{REQUEST_URI} !^/.*browser-support-ie-outdated.html
RewriteCond %{REQUEST_URI} !^/.*\.(png|gif|jpg|jpeg)
RewriteCond %{HTTP_REFERER} !.*browser-support-ie-outdated.html
RewriteCond %{env:VAA_HTTPS}_%{HTTP_HOST} (.*)_(.+) [NV]
RewriteRule ^/([^/]*)/en.html http%1://%2/$1/en/browser-support-ie-outdated.html [R=301,L]

3. How to allow external webservice through apache to the application server?

Configuration:-

# Always set these headers.
Header always set Access-Control-Allow-Origin "https://secure-dev3.ecnp.bankofamerica.com/applynow/confirmation.go"
Header always set Access-Control-Allow-Methods "POST, GET, OPTIONS, DELETE, PUT"
Header always set Access-Control-Allow-Credentials "true"
Header always set Access-Control-Max-Age "1000"
Header always set Access-Control-Allow-Headers "DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type"
# Added a rewrite to respond with a 200 SUCCESS on every OPTIONS request.

4. How to prevent Poodle attack?

Configuration:-

SSLEnable
SSLProtocolDisable SSLv2 SSLv3
SSLCipherSpec ALL -SSL_RSA_WITH_RC4_128_MD5 -SSL_RSA_WITH_RC4_128_SHA

5. How to implement http to https for any URL?

Configuration:-

Say we need to redirect the below URL from http to https

http://www.travel.com/gb/en/travel-information/customer-service/eu-claims/passenger-eu-claims.html

# START to make 1 URLs HTTPS: CHANGE-1 OUT OF 2
RewriteCond %{REQUEST_URI} !^(.*/travel-information/customer-service/eu-claims/passenger-eu-claims/.*\.html)
RewriteCond %{REQUEST_URI} !^(.*/travel-information/customer-service/eu-claims/passenger-eu-claims.html)
# FINISH to make 1 URLs HTTPS: CHANGE-1 OUT OF 2

# START to make 1 URLs HTTPS: CHANGE-2 OUT OF 2
RewriteCond %{HTTP:WL-Proxy-SSL} "!True"
RewriteCond %{env:VAA_HTTPS}_%{HTTP_HOST} (.*)_(.+) [NV]
RewriteRule ^(.*/travel-information/customer-service/eu-claims/passenger-eu-claims.*) https://%{HTTP_HOST}$1 [NE,L]
# FINISH to make 1 URLs HTTPS: CHANGE-2 OUT OF 2

6. How to set rule to block the request for any string?

Configuration:-

RewriteCond %{QUERY_STRING} ^.*(bxss.me).* [NC]
RewriteRule ^(.*) - [F,L]

7. How to respond 404 page for a Non-exsistent pages?

Configuration:-

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
#RewriteCond %{REQUEST_FILENAME} !-d
#RewriteCond %{REQUEST_FILENAME} !-l
RewriteRule . http://acc.virgin-atlantic.com/gb/en/404.html [L,R=301]

URLs that do not exist on our site currently result in a redirection chain to a custom 404 page based on the current cookie. This is causing SEO issues. To fix this we have applied redirect rule for any non-existent URL to return a 404 response at source. It should not result in redirection.

RewriteCond %{REQUEST_FILENAME} !-f: The rewriting to "page Not found for the request url"page will take place if the requested filename is not a regular file or if the file doesn't exist.

8. How to configure apache for IE to keep using highest version of IE browser and don’t take user to lower IE version?

Configuration:-

In addition to release deployment, we need to get below changes done in Apache httpd.conf file. Once changes are done, apache need to be restarted as well. These changes are required for IE to keep using highest version of IE browser and don’t take user to lower IE version.

1. Check if below line is uncommented in file

LoadModule headers_module modules/mod_headers.so

2. Add below section in file

IfModule headers_module  --- Put with < >
   Header set X-UA-Compatible: IE=edge
/IfModule --- Put with < >

9. How to enable server status URL in Apache?

Configuration:-

# Server-status enabled
Location /server-status --- put with < >
SetHandler server-status
Order deny,allow
Allow from all
/Location --- put with < >

How to create a self-signed SSL Certificate using OpenSSL and install in Apache server

How to create a self-signed SSL Certificate using OpenSSL and install in Apache server

Overview

The following is an extremely simplified view of how SSL is implemented and what part the certificate plays in the entire process.

Step 1: Generate a Private Key

The openssl toolkit is used to generate an RSA Private Key and CSR (Certificate Signing Request). It can also be used to generate self-signed certificates which can be used for testing purposes or internal usage.

The first step is to create your RSA Private Key. This key is a 1024 bit RSA key which is encrypted using Triple-DES and stored in a PEM format so that it is readable as ASCII text.

$ openssl genrsa -des3 -out server.key 1024

Generating RSA private key, 1024 bit long modulus
.........................................................++++++
........++++++
e is 65537 (0x10001)
Enter PEM pass phrase:
Verifying password - Enter PEM pass phrase:

Step 2: Generate a CSR (Certificate Signing Request)

Once the private key is generated a Certificate Signing Request can be generated. The CSR is then used in one of two ways. Ideally, the CSR will be sent to a Certificate Authority, such as Thawte or Verisign who will verify the identity of the requestor and issue a signed certificate. The second option is to self-sign the CSR, which will be demonstrated in the next section.

During the generation of the CSR, you will be prompted for several pieces of information. These are the X.509 attributes of the certificate. One of the prompts will be for "Common Name (e.g., YOUR name)". It is important that this field be filled in with the fully qualified domain name of the server to be protected by SSL. If the website to be protected will be https://public.pictc.com, then enter public.pictc.com at this prompt. The command to generate the CSR is as follows:

 $ openssl req -new -key server.key -out server.csr 

Country Name (2 letter code) [GB]:IN
State or Province Name (full name) [Berkshire]:Chandigarh
Locality Name (eg, city) [Newbury]:sector1
Organization Name (eg, company) [My Company Ltd]:PICTC
Organizational Unit Name (eg, section) []:Information Technology
Common Name (eg, your name or your server's hostname) []:public.pictc.com
Email Address []:ankur dot agarwal at pictc dot com
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Step 3: Remove Passphrase from Key

One unfortunate side-effect of the pass-phrased private key is that Apache will ask for the pass-phrase each time the web server is started. Obviously this is not necessarily convenient as someone will not always be around to type in the pass-phrase, such as after a reboot or crash. mod_ssl includes the ability to use an external program in place of the built-in pass-phrase dialog, however, this is not necessarily the most secure option either. It is possible to remove the Triple-DES encryption from the key, thereby no longer needing to type in a pass-phrase. If the private key is no longer encrypted, it is critical that this file only be readable by the root user! If your system is ever compromised and a third party obtains your unencrypted private key, the corresponding certificate will need to be revoked. With that being said, use the following command to remove the pass-phrase from the key:

cp server.key server.key.org
openssl rsa -in server.key.org -out server.key

The newly created server.key file has no more passphrase in it.

-rw-r--r-- 1 root root 745 Jun 29 12:19 server.csr
-rw-r--r-- 1 root root 891 Jun 29 13:22 server.key
-rw-r--r-- 1 root root 963 Jun 29 13:22 server.key.org

Step 4: Generating a Self-Signed Certificate

At this point you will need to generate a self-signed certificate because you either don't plan on having your certificate signed by a CA, or you wish to test your new SSL implementation while the CA is signing your certificate. This temporary certificate will generate an error in the client browser to the effect that the signing certificate authority is unknown and not trusted.

To generate a temporary certificate which is good for 365 days, issue the following command:

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=IN/ST=Chandigarh/L=sector1/O=PICTC AG/OU=Information
Technology/CN=public.pictc.com/Email=ankur dot agarwal at pictc dot com
Getting Private key

Step 5: Installing the Private Key and Certificate

When Apache with mod_ssl is installed, it creates several directories in the Apache config directory. The location of this directory will differ depending on how Apache was compiled.

cp server.crt /usr/local/apache/conf/ssl.crt
cp server.key /usr/local/apache/conf/ssl.key

Step 6: Configuring SSL Enabled Virtual Hosts

SSLEngine on
SSLCertificateFile /usr/local/apache/conf/ssl.crt/server.crt
SSLCertificateKeyFile /usr/local/apache/conf/ssl.key/server.key
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
CustomLog logs/ssl_request_log \
   "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

Step 7: Restart Apache and Test

/etc/init.d/httpd stop
/etc/init.d/httpd start

Steps to create a self-signed certificate and configure Custom Identity and Custom Trust with Weblogic Server using Keytool...

Steps to create a self-signed certificate and configure Custom Identity and Custom Trust with Weblogic Server using Keytool...

Below are the steps to create a self signed certificate :

Command 1 :

keytool -genkey -alias mykey -keyalg RSA -keysize 1024 -validity 365 -keypass privatepassword -keystore identity.jks -storepass password
Note :

List of keytool commands which are changed in java 1.6 :

-export, renamed to -exportcert

-genkey, renamed to -genkeypair

-import, renamed to -importcert

All previous commands are still supported in this release ( keytool in java 1.6 ) and will continue to be supported in future releases.

To create a 2048 bit SHA2/SHA256 certificate use the following command :

Command :

keytool -genkey -alias mykey -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -validity 365 -keypass privatepassword -keystore identity.jks -storepass password

Command 2 :

keytool -export -alias mykey -file root.cer -keystore identity.jks -storepass password

Command 3 :

keytool -import -alias mykey -file root.cer -keystore trust.jks -storepass password







To see the contents of the keystore use the following command :

Command :

keytool -list -v -keystore identity.jks -storepass password

To see the contents of an individual certificate ( like root.cer in our case ).

Command :

keytool -printcert -file root.cer



Copy the keystore files in the domain_home location :




Below are the steps to configure Custom Identity and Custom Trust with Weblogic Server :

Step 1 :

Login to Weblogic Admin console --> Environment --> Servers --> Configuration -> General --> SSL Listen Port Enabled ( Check )

Note : The default SSL Listen Port would be 7002, change it if required.



Step 2 :

Click on Keystores tab under " Configuration " tab :

Step 2a :

Click on the drop down menu next to Keystores and select " Custom Identity and Custom Trust "

Step 2b :

Now fill in the following information :

---Identity---

Custom Identity Keystore : < location_of_identity_keystore_that_you_have_created>

NOTE : By default WLS will look for this keystore file in domain_home location.

Custom Identity Keystore Type : jks
Custom Identity Keystore Passphrase: < This_would_be_your_storepass >

---Trust---

Custom Trust Keystore : < location_of_trust_keystore_that_you_have_created>

NOTE : By default WLS will look for this keystore file in domain_home location.

Custom Trust Keystore Type : jks
Custom Trust Keystore Passphrase: < This_would_be_your_storepass >




Step 2c :

Now save the changes and click on " SSL " tab :

Private Key Alias: < This_would_be_your_certificate_alias >

Private Key Passphrase: < This_would_be_your_keypass >


Step 3 :

Save the changes and click on the " >Advanced " field under the " SSL " tab :

Set the " Hostname Verification: " to None ( from the drop down menu ).

Note : We need to select the hostname verification as none if the CN of the certificate is not the same as the hostname of the machine where WLS is installed.



Now access your Weblogic Admin console over https URL :

" https://localhost:7002/console "

Python script to generate thread dump in Weblogic Server.

Python script to generate thread dump as below:

Step 1: Create a script vi ThreadDump.py in Unix terminal and copy paste the below snippet

$ vi ThreadDump.py

connect('weblogic','weblogic','t3://localhost:7001')
cd('Servers')
cd('AdminServer')
threadDump(writeToFile='true',fileName='ThreadDump.txt')
disconnect()
exit()

Edit the username, password, Adminserver URL as per your environment

Step 2: Now run setWLSEnv.sh script to setup the classpath & path

/u01/app/oracle/product/fmw/wlserver_10.3/server/bin

$ ../setWLSEnv.sh

Step 3 : Finally run the below command to generate the thread dump using WLST

$ java weblogic.WLST ThreadDump.py

Analyzing Java Thread Dumps in Weblogic Instances

Analyzing Java Thread Dumps

Thread Dumps is an excellent mechanism to trouble shoot typical production issues. Analyzing Java Thread dumps will provide us the clear understanding of production issues. In this blog entry we will discuss about what is a thread dump, how to take the thread dump,when to take thread dump and how to analyze the thread dumps.

What is a thread?

A thread is a Light-Weight process. Java Threads are Light-Weight Processes of the JVM process. A WebLogic JVM process contains multiple threads

What is a thread dump?

A thread dump is a snapshot of all the Java threads that are currently running in a Java Virtual Machine (JVM) at a given point of time. A thread dump might contain single thread or multiple threads. Thread Dump will provide the stack traces of all the JVM threads and much more information about a particular thread what it is doing at that point of time.

When will you take thread dumps?

• Java application is running much slower than expected (poor response time) 
• High CPU Utilization 
• When Server/Application is hanging (server become unresponsive or not responding). 

If you want to understand what is actively executed in the system during the load, you can take the thread dump and go through it.

How will you take the thread dumps?

There are several ways to take thread dumps

(1) Using jrcmd (jrockit 1.6)

jrcmd print_threads (or)
jrcmd print_threads nativestack=true jvmmonitors=true

Note: The THREAD_DUMPS will be generated in the Servers STDOUT file.

The above two commands will print out the thread dump. It is possible to redirect the above thread dump to a file as below:

jrcmd print_threads nativestack=true jvmmonitors=true > /tmp/thread_dump.txt

Here is the java process id

nativestack=true will print C-level stacktraces as well as Java traces.

jvmmonitors=true will also print the JRockit JVM's internal native locks

Example:

/u01/app/oracle/product/fmw/jrockit_160_29_D1.2.0-10/bin/jrcmd 2789 print_threads nativestack=true jvmmonitors=true > /tmp/thread_dump_2789.txt

(2) Using jstack (Sun JDK)

jstack (get the process id using jps -l)

Note: The THREAD_DUMPS will be generated in the Servers STDOUT file.

You can redirect the output to a file as below

jstack > threaddump.log

When Java process is hung or not responding, you can take the thread dump as below:

jstack -F -l (or)

jstack -F -l > threaddump.log

Options:

-F to force a thread dump. Use when jstack does not respond (process is hung)

-l long listing. Prints additional information about locks

(3) Using WebLogic Admin Console

Login to Admin Console

Click on Servers under domain tree > Click on the Server (for which you want to take thread dump) > Monitoring > Threads >Click on "Dump Thread Stack" Button. This will generate Thread dump in the Admin Console

(4) Using kill command in Unix/Linux

kill -3 (get the process id using ps -ef | grep "java")

Note: The THREAD_DUMPS will be generated in the Servers STDOUT file.

You can redirect the output to a file as below

kill -3 > /tmp/threaddump.log

(5) Using weblogic.Admin utility (deprecated, still you can use)

You can take the thread dump for WebLogic Admin java process using below comamnd:

java weblogic.Admin -url t3://: -username --password THREAD_DUMP

For WebLogic managed servers you can take thread dump as below:

java weblogic.Admin -url t3://: -username --password THREAD_DUMP

Note: The THREAD_DUMPS will be generated in the Servers STDOUT file.

Eg:

To take thread dump of AdminServer running on AdminHost with the port 8001

java weblogic.Admin -url t3://AdminHost:8001 -username weblogic -password weblogic THREAD_DUMP

To take thread dump of managed server running MngHost on localhost with the port 8003

java weblogic.Admin -url t3://MngHost:8001 -username weblogic -password weblogic THREAD_DUMP

(6) Using WLST

In WLST threadDump command displays a thread dump for the specified server.

Syntax:

threadDump([writeToFile], [fileName], [serverName])

writeToFile: This argument defaults to true, indicating that output is saved to a file.

fileName: Name of the file to which the output is written. It is pptional.

serverName:This argument defaults to the server to which WLST is connected. It is optional. If you don't specify the serverName it displays the thread dump for connected server

Example:

You can follow the below steps to take the thread dump.

$ pwd

/u01/app/oracle/product/fmw/wlserver_10.3/common/bin

$ ./wlst.sh

wls:/offline> connect('weblogic','weblogic','t3://localhost:7001')

Connecting to t3://localhost:7001 with userid weblogic ...

Successfully connected to Admin Server 'AdminServer' that belongs to domain 'test_domain'.

Warning: An insecure protocol was used to connect to the

server. To ensure on-the-wire security, the SSL port or

Admin port should be used instead.

wls:/test_domain/serverConfig> threadDump()

Thread dump for the running server: AdminServer

===== FULL THREAD DUMP ===============

Sat May 31 11:23:50 2014

Oracle JRockit(R) R28.2.4-14-151097-1.6.0_33-20120618-1634-linux-x86_64

"Main Thread" id=1 idx=0x4 tid=2487 prio=5 alive, waiting, native_blocked

-- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xf18a00b8[fat lock]

at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

at java/lang/Object.wait(J)V(Native Method)

at java/lang/Object.wait(Object.java:485)

at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:981)

^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xf18a00b8[fat lock]

at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:490)

at weblogic/Server.main(Server.java:71)

at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

-- end of trace

........

........

........

===== END OF THREAD DUMP ===============

The Thread Dump for server AdminServer

has been successfully written to Thread_Dump_AdminServer.txt

wls:/test_domain/serverConfig>

$ [server1@OEL bin]$ ls Thread_Dump_AdminServer.txt

Thread_Dump_AdminServer.txt

You can also write a python script to generate thread dump as below:

$ cat ThreadDump.py

connect('weblogic','weblogic','t3://localhost:7001')

cd('Servers')

cd('AdminServer')

threadDump(writeToFile='true',fileName='ThreadDump.txt')

disconnect()

exit()

Now run setWLSEnv.sh script to setup the classpath & path

/u01/app/oracle/product/fmw/wlserver_10.3/server/bin

$ . ./setWLSEnv.sh

Finally run the below command to generate the thread dump using WLST

$ java weblogic.WLST ThreadDump.py

What are different thread states?

For analyzing thread dumps, you need to know the status of threads. The status of threads are

NEW: The thread is created but not available to process the requests.

RUNNABLE (ACTIVE): The thread is either currently processing the request or ready to run when it gets its CPU. It may be in WAITING status due to the OS's resource distribution. JRockit thread dumps refer to this state as ACTIVE.

BLOCKED or Locked (MW): Waiting for Monitor Entry - The thread is waiting to get the lock( a different thread may be holding the lock)

WAITING or Waiting on Monitor or CW: The thread is waiting by using a wait, join or park method. For example, In WebLogc server the idle execute threads are in this condition and they wait till a socket reader thread notify them of some new work to be done.

What are the different types of WebLogic thread states?

Weblogic administrators need to understand different thread states and monitoring WebLogic threads.

• Active Execute Threads
• Idle Execute Threads
• Standby Threads
• Hogging Threads
• Stuck Threads

The thread monitoring section can be accessed for Admin and each managed server Managed server under the Monitoring > Threads tab.

From the above you can see, the thread monitoring tab provides a complete view of each WebLogic thread along with its state. Now we can try to understand WebLogic thread states.

Active Execute Threads: The threads which are currently processing the requests or ready to process the requests(idle threads). When thread demand goes up, WebLogic will start promoting threads from Standby to Active state which will enable them to process future client requests.

Execute Thread Idle Count: This is the number of Active idle threads currently “available” to process a client request.

Standby Thread Count: This is the number of threads waiting to be marked “eligible” to process client requests. These threads are created and visible from the JVM Thread Dump but not available yet to process a client request.

Execute Thread Total Count: This is the total number of threads “created” from the Weblogic self-tuning pool and visible from the JVM Thread Dump.You can calcualte these threads as below.

Execute Thread Total Count = Active Execute Threads + Standby Threads

Hogging Thread Count: This is the number of threads taking much more time to process the request than the average current execution time.

Stuck Threads: The threads which are taking more time to process the request than the configured stuck thread time. When facing slowdown conditions, the WebLogic threads transition from the Hogging state to Stuck stage, depending how long these threads remain stuck executing their current request.

In WebLogic Admin console you can configure Stuck thread related parameters for Admin and each Managed server under the Configuration > Tuning tab.

Stuck Thread Max Time: The number of seconds that a thread must be continually working before this server considers the thread stuck. The default value is 600 seconds(10mins)

Stuck Thread Timer Interval: The number of seconds after which WebLogic Server periodically scans threads to see if they have been continually working for the configured maximum length of time. The default value is 60 seconds

********************************Article Ends***************************************