Performance issues check list for Oracle EBS applications

Performance issues check list for Oracle EBS applications

1. Verify all the servers that are part of the applications are accessible.
-> login to the servers that confirms servers are up and accessiable.
-> verify the load on the servers to see if any hung processes.
top  - command to verify the processes and resource utilization
-> verify disk spaces.
$df -h
-> verify if disks are accessible.

2. Verify all application services are up and running normal.

3. Verify database health check.
-> check any blocking locks in the database.
-> check any tablespace running out of space.
-> check Gather schema statistics running periodically and verify if any scheduled job failed.
set linesize 200
col table_name for a30
select table_name, last_analyzed from dba_tables order by last_analyzed desc;
-> check alert log for errors.
-> check temporary space, undo retention.
-> verify if any recent changes to database configuration parameters.
-> verify if any recent changes to database Oracle home like patching/upgrade ..
-> If any changes to Oracle home consider applying CPU/PS patches to Oracle home to make sure latest bugs are addressed.
-> verify if any indexes status is invalid
select index_name , status, last_analyzed from dba_indexes where  STATUS != 'VALID';
-> verify if any recent jobs scheduled in the database like Mviews refresh
-> Verify SGA configuration is appropricate.
-> Check if sufficient database sessions, processes, cursors are configured. It not it may throw error into alert log.

4. Concurrent Managers performance issues and Oracle notes:
Note 1075684.1 Concurrent Managers are consuming high CPU and memory
Note 1360118.1 Concurrent Manager Slow Performance Leaving Requests in Pending Status.
Note 1060736.1 Deadlock Error During Concurrent Request Termination
Note 866298.1 Concurrent Processing - ORA-00060: Deadlock Detected - UPDATE FND_CONCURRENT_QUEUES
Note 822368.1 Concurrent Processing - How To Run the Purge Concurrent Request FNDCPPUR, Which Tables Are Purged, And Known Issues Like Files Are Not Deleted From File System or Slow Performance
Note 1060707.1 Purge Concurrent Requests/Manager Data, FNDCPPUR, Not Removing Files From Filesystem
Note: Also reference Note 986272.1 "How To Performance Tune The Concurrent Processing Server"
EBPERF FAQ - Collecting Statistics in Oracle EBS 11i and R12 (Doc ID 368252.1)

5. Forms issues
11i: Troubleshooting Tips For Spinning/Hanging F60WEBMX Processes [ID 457381.1]
11i: Troubleshooting Tips For Spinning/Hanging F60WEBMX Processes (Doc ID 457381.1)


6. Turn on debugging for specific issue:
I.
a. cd to your $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config directory.
b. Make a back up your oc4j.properties (e.g. cp -p oc4j.properties oc4j.properties_oss)
c. Add the values to your oc4j.properties file
 a. AFLOG_ENABLED=true
 b. AFLOG_LEVEL=statement
 c. AFLOG_MODULE=%
 d. AFLOG_FILENAME=/tmp/aflog_oss.log

2. Enable http ODL logging
II.
 a. Backup your [ $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf ] file
 Edit the httpd.conf file and add the following to the end of file
 OraLogMode oracle
 OraLogSeverity TRACE:32
 OraLogDir $LOG_HOME/ora/10.1.3/Apache/oracle
 Please use the full path to $LOG_HOME
 (e.g. OraLogDir /oracle/v1204/inst/apps/V1204_bertha/logs/ora/10.1.3/Apache/oracle)
 b. Make the following directory which will be where the Apache ODL log files are created
 mkdir $LOG_HOME/ora/10.1.3/Apache/oracle

III. Increase OPMN Logging
 -------------------------------
 Backup of your [ $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml ] file.
 Edit the opmn.xml file and make the following change.
 ( i.e.
 BEFORE:
 <debug path="/oracle/v1204/inst/apps/V1204_bertha/logs/ora/10.1.3/opmn/opmn.dbg"
 comp=""
 rotation-size="1500000"
 rotation-hour="0" />

 AFTER:
 <debug path="/oracle/v1204/inst/apps/V1204_bertha/logs/ora/10.1.3/opmn/opmn.dbg"
 comp="internal;ons;pm"
 <<-- [ debug statements.. ]
 rotation-size="1500000"
 rotation-hour="0" />

IV. Shutdown middle (web) tier ( $ADMIN_SCRIPTS_HOME/adopmnctl.sh stopall )

V. Clean out the Apache/oc4j log files for middle tier iAS 10g (10.1.3):
 ( From an Unix shell where the Apps environment has been sourced )

 a-1. $ cd $LOG_HOME/ora/10.1.3/Apache/
 a-2. $ pwd (verify that you are in the "Apache logs directory")
 a-3. $ rm -r *
 b-1. $ cd $LOG_HOME/ora/10.1.3/opmn/
 b-2. $ pwd (verify that you are in the "opmn logs directory")
 b-3. $ rm -r *

 [ Note: ] For EACH j2ee log group cleaning below:
 Could have multiple jvms per each type of log groups so
 "default_group_#" refers to generic #. Whereby the steps
 below for cleaning should be for EACH jvm.

 c-1. $ cd $LOG_HOME/ora/10.1.3/j2ee/forms/[ forms_default_group_# ]
 c-2. $ pwd (verify that you are in the "forms_default_group_#" directory)
 c-3. $ rm -r *

 d-1. $ cd $LOG_HOME/ora/10.1.3/j2ee/oacore/[ oacore_default_group_# ]
 d-2. $ pwd (verify that you are in the "oacore_default_group_#" directory)
 d-3. $ rm -r *
 e-1. $ cd $LOG_HOME/ora/10.1.3/j2ee/oafm/
 e-2. $ pwd (verify that you are in the oafm directory)
 e-3. $ rm -r *

 f-1. $ cd $LOG_HOME/ora/10.1.3/j2ee/oafm/[ oacore_default_group_# ]
 f-2. $ pwd (verify that you are in the "oafm_default_group_#" directory)
 f-3. $ rm -r *

VI. Startup middle (web) tier ( $ADMIN_SCRIPTS_HOME/adopmnctl.sh startall )

 a. Reproduce the problem

 b. Run the following (aoljtests) tests as well.
 Note(s):
 Upload screen shots from each of the tests below.
 -or-
 Save the pages from your browser. (File --> Save Page)

 Go to..
 http://[machine:port]/OA_HTML/jsp/fnd/aoljtest.jsp
 Enter in the following Info...
 Environment Type: non-RAC ( unless you have a RAC system,
 which I do not believe to be true )
 Apps User Name:
 Apps Password:
 Oracle SID
 Host Name: <-- this is the database server- (fully qualified domain)
 Port Number: <-- Database listener port

VII. The tests we want you to run are the following -
 *Connection Test
 Locate DBC File
 Verify DBC Settings
 Verify Guest User
 *APPS_WEB_AGENT
 Virtual Directory Settings
 *APPS_SERVLET AGENT
 Virtual Directory Settings
 Servlet Ping
 *APPS_FRAMEWORK_AGENT
 Virtual Directory Settings
 OA Framework System Info
 *Tool Launcher Profile Settings
 ICX_FORMS_LAUNCHER <<-- Actually click on the link that comes up and see if forms launches
 *Application Login
 Login Page <<-- Try and log in, if possible then click on the two tests below
 Show Responsibilities
 Show Current Session Settings

VIII. Shutdown middle (web) tier ( $ADMIN_SCRIPTS_HOME/adopmnctl.sh stopall )

IX. Run the following from a Unix shell with the Apps environment sourced and upload the resulting zip file
 under your [ tmp ] directory.
 zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_iAS1013.zip \
 $ORA_CONFIG_HOME/10.1.3/Apache/* \
 $ORA_CONFIG_HOME/10.1.3/j2ee/* \
 $ORA_CONFIG_HOME/10.1.3/opmn/* \
 $ORA_CONFIG_HOME/10.1.3/javacache/* \
 $LOG_HOME/ora/10.1.3/Apache/* \
 $LOG_HOME/ora/10.1.3/opmn/* \
 $LOG_HOME/ora/10.1.3/j2ee/* \
 $APPLRGF/javacache.log \
 /tmp/aflog_oss.log \
 $CONTEXT_FILE