Saturday, February 8, 2014

Troubleshoot Hung Database

To troubleshoot hung database

Few Tools that can help us to troubleshoot such situation:
1. OSwatcher must be installed and running on the database server.
2. Hung analyzer in 9.0.1.0 onwards is a good tool to analyze where the problem lies.
 A database can be hung due to following:

1. A network or firewall issue where users are unable to login.
2. Listener Hang.
3. RAM shortage hung. like SWAP area full.
4. 100% CPU.
5. Data Hang due to blocks.
6. A bad disk.
7. OS limit exceeded.

Steps to be followed while sitting on this kind of situation:

1. As a Top down approach. Try to figure what are the OS statistics showing. Commands that can help us figure out are:
Ø  dmesg: check error logs on the server.
Ø  tail -1000f /var/adm/messages to check errors logs on the server.
Ø  vmstat and top provides you the clear picture what resources is being consumed heavily.
Ø  Netstat –a this will give you network statistics.

2. Check alertlog for any errors.

3. Try hung analyzer. Hung analyzer will generate trace files which will provide a quick view of hang chains occurring at the time of a hang being experienced.

Sometimes, database may actually just be very slow and not actually hanging. It is therefore recommended,  where possible to get 2 hanganalyze and 2 systemstate dumps in order to determine whether processes are moving at all or whether they are "frozen".

Steps to perform Hung analyzer:
Hanganalyze
sqlplus '/ as sysdba'
oradebug setmypid
oradebug unlimit
oradebug hanganalyze 3
-- Wait one minute before getting the second hanganalyze
oradebug hanganalyze 3
oradebug tracefile_name
exit
Systemstate
sqlplus '/ as sysdba'
oradebug setmypid
oradebug unlimit
oradebug dump systemstate 266
oradebug dump systemstate 266
oradebug tracefile_name
exit


4. Check if there is any block or locks that are waiting. Remove the locks and blocks.
5. Run the AWR, ADDM and ASH report for the time period.

Once you are done to figure out what OS resource is exhausted and where the issue lies. You can try to figure out which pid is consuming the most of the resources. As per application design Kill those hung chain sessions.

If unable to do any of the following before rebouncing the database on server, run hung analyze in prelim session as mentioned below:

root> sqlplus –prelim
  1. SQL> oradebug hanganalyze 3
  2. Wait at least 2 minutes to give time to identify process state changes.
  3. SQL> oradebug hanganalyze 3
  4. Open a separate SQL session and immediately generate a system state dump.
  5. SQL> alter session set events 'immediate trace name SYSTEMSTATE level 10';

SQLT installation

Steps for sqlT installation:
1.       Download the attached sqlt and upload to the server in desired location.
2.       Unzip the Sqlt.zip.
3.       Run @sqlt/install/sqcreate.sql as below:
SQL> conn / as sysdba
Connected.
SQL> @sqcreate.sql
  adding: 140204144155_01_sqcreate.log (deflated 81%)

zip error: Nothing to do! (SQLT_installation_logs_archive.zip)

PL/SQL procedure successfully completed.


Specify optional Connect Identifier (as per Oracle Net)
Include "@" symbol, ie. @PROD
If not applicable, enter nothing and hit the "Enter" key.
You *MUST* provide a connect identifier when installing
SQLT in a Pluggable Database in 12c
This connect identifier is only used while exporting SQLT
repository everytime you execute one of the main methods.

Optional Connect Identifier (ie: @PROD):


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


Define SQLTXPLAIN password (hidden and case sensitive).

Password for user SQLTXPLAIN:
Re-enter password:


PL/SQL procedure successfully completed.

... please wait
Specify PERMANENT tablespace to be used by SQLTXPLAIN.

Tablespace name is case sensitive.

Default tablespace [UNKNOWN]: USERS

PL/SQL procedure successfully completed.

... please wait

Specify TEMPORARY tablespace to be used by SQLTXPLAIN.

Tablespace name is case sensitive.

Temporary tablespace [UNKNOWN]: TEMP

PL/SQL procedure successfully completed.


The main application user of SQLT is the schema
owner that issued the SQL to be analyzed.
For example, on an EBS application you would
enter APPS.
You will not be asked to enter its password.
To add more SQLT users after this installation
is completed simply grant them the SQLT_USER_ROLE
role.

Main application user of SQLT: system

PL/SQL procedure successfully completed.


SQLT can make extensive use of licensed features
provided by the Oracle Diagnostic and the Oracle
Tuning Packs, including SQL Tuning Advisor (STA),
SQL Monitoring and Automatic Workload Repository
(AWR).
To enable or disable access to these features
from the SQLT tool enter one of the following
values when asked:

"T" if you have license for Diagnostic and Tuning
"D" if you have license only for Oracle Diagnostic
"N" if you do not have these two licenses

Oracle Pack license [T]: T

PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.

TADOBJ completed.

PL/SQL procedure successfully completed.


SQDOLD completed. Ignore errors from this script
  adding: 140204144333_01_sqcreate.log (deflated 89%)






SQCUSR completed. Some errors are expected.

Procedure created.

No errors.
  adding: 140204144409_02_sqcusr.log (deflated 85%)

TAUTLTEST completed.
  adding: 140204144410_09_tautltest.log (deflated 60%)






SQUTLTEST completed.
  adding: 140204144410_10_squtltest.log (deflated 61%)






no rows selected

TACOBJ completed.
  adding: 140204144410_03_tacobj.log (deflated 86%)

SQL> PRO Dropping Libraries for TRCA
Dropping Libraries for TRCA
SQL> SET TERM OFF;
tool_repository_schema: "SQLTXPLAIN"
tool_administer_schema: "SQLTXADMIN"
role_name: "SQLT_USER_ROLE"
Creating Procedures
Creating Package Specs TRCA$G
No errors.
Creating Package Specs TRCA$P
No errors.
Creating Package Specs TRCA$T
No errors.
Creating Package Specs TRCA$I
No errors.
Creating Package Specs TRCA$E
No errors.
Creating Package Specs TRCA$R
No errors.
Creating Package Specs TRCA$X
No errors.
Creating Views
Creating Package Body TRCA$G
No errors.
Creating Package Body TRCA$P
No errors.
Creating Package Body TRCA$T
No errors.
Creating Package Body TRCA$I
No errors.
Creating Package Body TRCA$E
No errors.
Creating Package Body TRCA$R
No errors.
Creating Package Body TRCA$X
No errors.
Creating Grants on Libraries

Tool Version
----------------
12.1.06

Install Date
----------------
20140204

Directories
--------------------------------------------------------------------------------------------------------------------------------
TRCA$INPUT1(VALID)      /opt/app/p3isc1d6/oracle/diag/rdbms/p3isc1d6/p3isc1d6/trace
TRCA$INPUT2(VALID)      /opt/app/p3isc1d6/oracle/diag/rdbms/p3isc1d6/p3isc1d6/trace
TRCA$STAGE(VALID)       /opt/app/p3isc1d6/oracle/diag/rdbms/p3isc1d6/p3isc1d6/trace
user_dump_dest          /opt/app/p3isc1d6/oracle/diag/rdbms/p3isc1d6/p3isc1d6/trace
background_dump_dest    /opt/app/p3isc1d6/oracle/diag/rdbms/p3isc1d6/p3isc1d6/trace

Libraries
--------------------------------------------------------------------------------------------------------------------------------
VALID PACKAGE TRCA$I /* $Header: 224270.1 tacpkgi.pks 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE TRCA$E /* $Header: 224270.1 tacpkge.pks 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE TRCA$G /* $Header: 224270.1 tacpkgg.pks 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE TRCA$P /* $Header: 224270.1 tacpkgp.pks 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE TRCA$R /* $Header: 224270.1 tacpkgr.pks 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE TRCA$T /* $Header: 224270.1 tacpkgt.pks 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE TRCA$X /* $Header: 224270.1 tacpkgx.pks 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE BODY TRCA$I /* $Header: 224270.1 tacpkgi.pkb 11.4.5.1 2012/11/27 carlos.sierra $ */
VALID PACKAGE BODY TRCA$E /* $Header: 224270.1 tacpkge.pkb 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE BODY TRCA$G /* $Header: 224270.1 tacpkgg.pkb 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE BODY TRCA$P /* $Header: 224270.1 tacpkgp.pkb 11.4.5.8 2013/05/10 carlos.sierra $ */
VALID PACKAGE BODY TRCA$R /* $Header: 224270.1 tacpkgr.pkb 11.4.5.0 2012/11/21 carlos.sierra $ */
VALID PACKAGE BODY TRCA$T /* $Header: 224270.1 tacpkgt.pkb 11.4.5.7 2013/04/05 carlos.sierra $ */
VALID PACKAGE BODY TRCA$X /* $Header: 224270.1 tacpkgx.pkb 11.4.5.0 2012/11/21 carlos.sierra $ */
TACPKG completed.

PL/SQL procedure successfully completed.

  adding: 140204144416_04_tacpkg.log (deflated 80%)






PL/SQL procedure successfully completed.


SQCOBJ completed. Some errors are expected.
  adding: 140204144531_05_sqcobj.log (deflated 93%)


PL/SQL procedure successfully completed.


SQLT can make extensive use of licensed features
provided by the Oracle Diagnostic and the Oracle
Tuning Packs, including SQL Tuning Advisor (STA),
SQL Monitoring, Automatic Workload Repository
(AWR) and SQL Tuning Sets (STS).
To enable or disable access to these features
from the SQLT tool enter one of the following
values when asked:

"T" if you have license for Diagnostic and Tuning
"D" if you have license only for Oracle Diagnostic
"N" if you do not have these two licenses

pack_license: "T"
enable_tuning_pack_access

PL/SQL procedure successfully completed.


Specify optional Connect Identifier (as per Oracle Net)
Include "@" symbol, ie. @PROD
If not applicable, enter nothing and hit the "Enter" key

connect_identifier: ""

PL/SQL procedure successfully completed.


Table truncated.

PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


Procedure created.

No errors.

Table truncated.


PL/SQL procedure successfully completed.


Procedure dropped.


Commit complete.


SQSEED completed.
  adding: 140204144601_07_sqseed.log (deflated 82%)


PL/SQL procedure successfully completed.

... dropping packages for SQLT
... creating package specs for SQLT$A
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package specs for SQLT$C
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package specs for SQLT$D
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package specs for SQLT$E
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package specs for SQLT$H
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package specs for SQLT$I
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package specs for SQLT$M
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package specs for SQLT$R
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package specs for SQLT$S
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package specs for SQLT$T
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating views

PL/SQL procedure successfully completed.

... creating package body for SQLT$A
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package body for SQLT$C
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package body for SQLT$D
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package body for SQLT$E
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package body for SQLT$H
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package body for SQLT$I
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package body for SQLT$M
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package body for SQLT$R
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package body for SQLT$S
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;
... creating package body for SQLT$T
SQL> SHOW ERRORS;
No errors.
SQL> --
SQL> SET TERM ON ECHO OFF;

Creating Grants on Packages ...


LIBRARIES
----------------------------------------------------------------
VALID   PACKAGE      12.1.06  SQLT$A
VALID   PACKAGE      11.4.5.0 SQLT$C
VALID   PACKAGE      12.1.06  SQLT$D
VALID   PACKAGE      12.1.03  SQLT$E
VALID   PACKAGE      11.4.5.0 SQLT$H
VALID   PACKAGE      12.1.04  SQLT$I
VALID   PACKAGE      11.4.5.0 SQLT$M
VALID   PACKAGE      12.1.03  SQLT$R
VALID   PACKAGE      11.4.5.0 SQLT$S
VALID   PACKAGE      11.4.5.6 SQLT$T
VALID   PACKAGE      11.4.5.0 TRCA$E
VALID   PACKAGE      11.4.5.0 TRCA$G
VALID   PACKAGE      11.4.5.0 TRCA$I
VALID   PACKAGE      11.4.5.0 TRCA$P
VALID   PACKAGE      11.4.5.0 TRCA$R
VALID   PACKAGE      11.4.5.0 TRCA$T
VALID   PACKAGE      11.4.5.0 TRCA$X
VALID   PACKAGE BODY 12.1.06  SQLT$A
VALID   PACKAGE BODY 12.1.03  SQLT$C
VALID   PACKAGE BODY 12.1.06  SQLT$D
VALID   PACKAGE BODY 12.1.03  SQLT$E

LIBRARIES
----------------------------------------------------------------
VALID   PACKAGE BODY 12.1.06  SQLT$H
VALID   PACKAGE BODY 12.1.05  SQLT$I
VALID   PACKAGE BODY 12.1.06  SQLT$M
VALID   PACKAGE BODY 12.1.06  SQLT$R
VALID   PACKAGE BODY 11.4.5.0 SQLT$S
VALID   PACKAGE BODY 12.1.06  SQLT$T
VALID   PACKAGE BODY 11.4.5.0 TRCA$E
VALID   PACKAGE BODY 11.4.5.0 TRCA$G
VALID   PACKAGE BODY 11.4.5.1 TRCA$I
VALID   PACKAGE BODY 11.4.5.8 TRCA$P
VALID   PACKAGE BODY 11.4.5.0 TRCA$R
VALID   PACKAGE BODY 11.4.5.7 TRCA$T
VALID   PACKAGE BODY 11.4.5.0 TRCA$X

Deleting CBO statistics for SQLTXPLAIN objects ...

14:47:54    0 sqlt$a: -> delete_sqltxplain_stats
14:47:57    3 sqlt$a: <- delete_sqltxplain_stats

PL/SQL procedure successfully completed.


SQCPKG completed.
  adding: 140204144615_08_sqcpkg.log (deflated 80%)






TAUTLTEST completed.
  adding: 140204144757_09_tautltest.log (deflated 59%)






SQUTLTEST completed.
  adding: 140204144757_10_squtltest.log (deflated 59%)


SQLT users must be granted SQLT_USER_ROLE before using this tool.

SQCREATE completed. Installation completed successfully.


To remove SQT

Run @sqdrop.sql

Oracle Enterprise Manager Cloud Control 13c (OMS upgrade from 13.3 or 13.4 to OMS 13.5)

 OMS upgrade from 13.3 or 13.4 to OMS 13.5 PHASE -I Planning 1. OMS 13.5 is the latest Oracle Cloud Control version available. 2. To directl...