Node ID does not exist for the current application server id.

Node ID  does not exist for the current application server id.

On entering Oracle Applications on the web, the following error appears:

Node ID  does not exist for the current application server id.

Cause:

There is a mismatch between server_id in the  fnd_nodes table and appl_server_id in the dbc file and
hence we are getting this  "node id does not exist" error.

Solution:

Stop application.
Run adgendbc.sh script to configure apps tier dbc file with the correct id from the database instance.
Start the application and retest.

Location of script:
11i: $COMMON_TOP/admin/install/$CONTEXT_NAME
R12: $INST_TOP/admin/install

[appl@test install]$ sh adgendbc.sh

To disable virbr0, enter:

*To disable virbr0, enter:
*
virsh net-destroy default

virsh net-undefine default

service libvirtd restart

ifconfig

Linux: How to monitor the nproc limit

Linux: How to monitor the nproc limit

You probably know about 'nproc' limits in Linux which are set in /etc/limits.conf and checked with 'ulimit -u'. But do you know how to handle the monitoring and be alerted when you're close the fixed limit?

Nproc and ps
Nproc is defined at OS level to limit the number of processes per user. Oracle 11.2.0.4 documentation recommends the following:

oracle soft nproc 2047
oracle hard nproc 16384
 
But that is often too low, especially when you have the Enterprise Manager agent or other java programs running.

Do you want to check that you are far from the limit? then you can use 'ps'. But beware, 'ps' by default does not show all processes.
 
In Linux, when doing multithreading, each thread is implemented as a light-weight process (LWP). And you must use the '-L' to see all of them.

Let's take an example. I have a system where 'ps -u oracle' returns 243 lines. But including LWPs shows a lot more processes which is near the limit:

$ ps h -Led -o user | sort | uniq -c | sort -n
      1 dbus
      1 ntp
      1 rpc
      1 rpcuser
      2 avahi
      2 haldaemon
      2 postfix
    166 grid
    400 root
   1370 oracle
 
So the 'oracle' user has 1370 processes. That's high. And this is the actual number where the nproc limit applies.

'ps -Lf' can show the detail. And even without '-L' we can display the NLWP which is the number of threads per process:

ps -o nlwp,pid,lwp,args -u oracle | sort -n
NLWP   PID   LWP COMMAND
   1  8444  8444 oracleOPRODP3 (LOCAL=NO)
   1  9397  9397 oracleOPRODP3 (LOCAL=NO)
   1  9542  9542 oracleOPRODP3 (LOCAL=NO)
   1  9803  9803 /u00/app/oracle/product/agent12c/core/12.1.0.3.0/perl/bin/perl /u00/app/oracle/product/agent12c/core/12.1.0.3.0/bin/emwd.pl agent /u00/app/oracle/product/agent12c/agent_inst/sysman/log/emagent.nohup
  19 11966 11966 /u00/app/11.2.0/grid/bin/oraagent.bin
1114  9963  9963 /u00/app/oracle/product/agent12c/core/12.1.0.3.0/jdk/bin/java ... emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain
 
The Oracle 12c EM agent has started 1114 threads and the grid infrastructure 'oraagent.bin' has 19 threads. In addition to that I've a lot of other monothreaded processes. This is how we reach 1370 which is the exact value to compare to the nproc limit.

So what are the good values to set? About the high number of threads for EM agent 12c, there are a few bugs. And I suspect that 1000 threads is too much, especially when checking them with 'jstack' I see that they are "CRSeOns" threads that should not be used in 11.2.0.2 and higher. But that's another problem which I'm currently investigating. When you reach the nproc limit, the user will not be able to create new processes. clone() calls will return EAGAIN and that is reported by Oracle as:

ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
And that is clearly bad when it concerns an +ASM instance or archiver processes.

The goal of the nproc limit is only to prevent 'fork bombs' where a process forks forever and exhausts all resources. So there is no problem to increase this limit. However if you set it high for some users ('oracle' and 'grid' usually), it can be a good idea to monitor the number of processes with the ps h -L above. Because having too many processes is suspect and increasing the limit just hides a process leak and defer the failure.

In 'ps h -L -o' The argument 'h' is there to remove the header line, and '-L' to show all processes including LWP. Then you can count with 'wc -l'.

The proof
In order to be sure that 'ps h -L' gives the exact number, I have tested it. In case you want to check this on your system, here is how to do it. And please report any difference.

First, set your limit to 1024 processes. This is a limit for my user, and the limit is set for my shell and all its child processes:

[oracle@VM211 ocm]$ ulimit -u 1024
Now you can check it:

[oracle@VM211 ocm]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15919
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[root@VM211 ocm]# ps h -Lu oracle | wc -l
1023
 
Recommended values for Oracle
Currently this is what is set on Oracle linux 6 for 11gR2 by the preinstall package (in /etc/security/limits.conf):

oracle   soft   nproc    16384
oracle   hard   nproc    16384

For 12c, these are set in /etc/security/limits.d/oracle-rdbms-server-12cR1-preinstall.conf which overrides /etc/security/limits.conf:

oracle soft nproc 16384
oracle hard nproc 16384
And just for your information, here is what is set in the ODA X4-2:

oracle soft nproc 131072

So what do you want to set? You probably don't want it too low and experience 'resource temporarily unavailable'. But what you don't want either is 100000 processes on your server. So my recommendation is to set it high but monitor it when the number of processes reaches something that is not sensible. Then you prevent having the system down in case of process leak, but you can detect it and ask for a patch.

Special thanks to Mr. Franck Pachot for such easy illustration.
Regards
Shahzad

Master Note For Oracle Database Server Installation (Doc ID 1156586.1)

Master Note For Oracle Database Server Installation (Doc ID 1156586.1)

-------------------------------------------------------------------------------------------

Applies to:
Oracle Server - Enterprise Edition - Version: 8.1.7.0 to 11.2.0.2.0 - Release: 8.1.7 to 11.2
Information in this document applies to any platform.

Purpose
This Master Note is intended to provide an index and references to the most frequently used My Oracle Support Notes with respect to Oracle Database Installation.This Master Note is subdivided into categories to allow for easy access and reference to notes that are applicable to your area of interest, within INSTALLATION.This includes the following categories:

* Certification

* Installation Guides

* OS Requirements

* Scripts

* Silent Install

* Diagnostic Tools

* Known Issues

* General Information

* Using My Oracle Support Effectively

* Generic Link



Scope and Application
This note guides you through the process of Oracle Database Installation.

This note applies to the following versions of these products:

Oracle Server - Enterprise Edition - Version: 8.1.7.1 to 8.1.7.x
Oracle Server - Enterprise Edition - Version: 9.0.1.1 to 9.0.1.x
Oracle Server - Enterprise Edition - Version: 9.2.0.1 to 9.2.0.x
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 10.2.0.x
Oracle Server - Enterprise Edition - Version: 11.1.0.6 to 11.1.0.x
Oracle Server - Enterprise Edition - Version: 11.2.0.1 to 11.2.0.x



Master Note For Oracle Database Server Installation
Certification
For the latest Database Certification always refer to My Oracle Support (MOS) Certify section.

At the same time the documentation listed below help you in understanding the Oracle Database Certification:

Note.415166.1 Oracle Database Client Certification Notes 10g Release 2 (10.2.0.3) for Microsoft Windows Vista
Note.161818.1 Oracle Database (RDBMS) Releases Support Status Summary
Note.207303.1 Client / Server / Interoperability Support Between Different Oracle Versions
Note.77627.1 Oracle Database Server product support Matrix for Windows 2000
Note.266043.1 Support of Linux and Oracle Products on Linux
Note.464754.1 Certified Software on Oracle VM
Note.753601.1 SUPPORT FOR ORACLE DATABASE ON WPARS UNDER AIX 6.1

Installation Guides
Oracle Database documentation, 11g Release 2 (11.2).
Installation manuals and platform-specific books are included in this link.

Oracle Database documentation, 11g Release 1 (11.1).
Installation manuals and platform-specific books are included in this link.

Oracle Database documentation, 10g Release 2 (10.2).
Installation manuals and platform-specific books are included in this link.

Oracle Database documentation, 10g Release 1 (10.1).
Installation manuals and platform-specific books are included in this link.

Oracle9i documentation, Release 2 (9.2).
Oracle9i documentation, Release 1 (9.0.1).
Installation manuals and platform-specific books are included in this search, for 9.2 only.

OS Requirements
Most of the installation and relinking issues appear if you don't satisfy the minimum requirements for the installation. Therefore if you are raising any SR for installation issue it is better to ensure that you have all these requirements mentioned in the below articles before raising the SR.

The documentation listed below help you in understanding the OS Requirements for Oracle Database Installation:
Generic
Note.169706.1 Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
Note.268025.1 Server Installation - Unix Knowledge Browser Product Page
Note.750540.1 What are the requirements for an Oracle Home Name in the Inventory?
Note.389678.1 Are Compilers Necessary for Oracle RDBMS Installation?

Linux
Note.376183.1 Defining a "default RPMs" installation of the RHEL OS
Note.851598.1 Master Note of Linux OS Requirements for Database Server
Note.761261.1 Recommended metalink articles for Oracle Database 10gR2 installation on zLinux (s390x)
Note.415182.1 DB Install Requirements Quick Reference - zSeries based Linux
Note.341507.1 Oracle Database Server on Linux on IBM POWER

AIX
Note.889464.1 Is Oracle 10g And 11g Database Compatible With AIX Version 6.1
Note.859856.1 Oracle 10g R2 database server software for AIX 6.1
Note.417451.1 How To Determine Whether an APAR has been Fixed in AIX Version/Maintenance Level
Note.201019.1 AIX Quick Start Guide - 9.2.0 RDBMS Installation
Note.182037.1 AIX Quick Start Guide - 9.0.1 RDBMS Installation
Note.109843.1 AIX Quick Start Guide - 8.1.X RDBMS Installation
Note.109842.1 AIX Quick Start Guide - 8.0.X RDBMS Installation

Solaris
Note.201021.1 SOLARIS Quick Start Guide - 9.2.0 RDBMS Installation
Note.148673.1 SOLARIS Quick Start Guide - 9.0.x RDBMS Installation
Note.108413.1 SOLARIS Quick Start Guide - 8.0.X RDBMS Installation
Note.108306.1 SOLARIS Quick Start Guide - 8.1.X RDBMS Installation
Note.743042.1 Requirements for Installing Oracle 11gR1 RDBMS on Solaris 10 SPARC 64-bit
Note 971464.1 FAQ - 11gR2 requires Solaris 10 update 6 or greater

HP-UX
Note.201023.1 HP-UX Quick Start Guide - 9.2.0 RDBMS Installation
Note.154360.1 HP-UX Quick Start Guide - 9.0.x RDBMS Installation
Note.108387.1 HP-UX Quick Start Guide - 8.1.X RDBMS Installation
Note.108381.1 HP-UX Quick Start Guide - 8.0.X RDBMS Installation

Scripts
Note.296665.1 Pre-Install checks for 10gR1 RDBMS (10.1.x) - Linux AMD64/EM64T Platforms
Note.283749.1 Pre-Install checks for 10gR1 RDBMS (10.1.x) - Linux Itanium 64 (IA64) Platforms
Note.283748.1 Pre-Install checks for 10gR1 RDBMS (10.1.x) - Linux x86 Platforms
Note.189256.1 UNIX Script to Verify Installation Requirements for Oracle 8.0.5 to 9.2 versions of RDBMS

Silent Install
Note.136660.1 Working Response files for a silent install of Oracle Server on MS-Windows- Part 1 of 3
Note.136661.1 Database Configuration Response file for performing silent install - Part 2 of 3
Note 136662.1 Network Configuration Reponse file for performing silent install - Part 3 of 3
Note 388451.1 How to create a Response File for Silent Installation?
Note 885643.1 How to install 11gR2 Database/Client Software in Silent mode without using response file
Note 808275.1 How to install Oracle Database Patchset software in Silent mode without using response file
Note 782919.1 How to install Oracle Database Server software silently with customized listener configuration
Note 782918.1 How to install/deinstall Oracle Database Server software silently without response file
Note 435680.1 Response File Installation of Database Server 10.2 (Suppressed or Silent mode)
Note 434803.1 Silent Installation for Oracle Server 10.1
Note 434315.1 How Can You Install Oracle Without Using The Oracle Universal Installer( runInstaller) GUI and Xserver especially When Doing A Remote Installation ?
Note 405124.1 How to Prevent a Silent Install from Prompting for User Input to Run the 'root.sh' Script
Note 228916.1 How to Troubleshoot a Silent Installation

Diagnostic Tools
Note.181459.1 How To Troubleshoot an Oracle Database Software Installation on UNIX
Note.780551.1 How To Diagnose Linking Errors During Base Install of Oracle RDBMS
Note.757964.1 Oracle Universal Installer ( OUI ) Error Messages and Solution Reference List

Known Issues
Note.833732.1 How To quick Deal with Relink, make, Failed on Linux ? Errors bin/sh /usr/bin/cc cannot execute binary file. or undefined reference to `dl_iterate_phdr@GLIBC_x.x.y'.
Note.848700.1 11gR1 on Solaris failing at relink phase with errors like "ld fatal relocation error" and then "ld fatal library -lclntsh not found"

Note.820135.1 What Does -ignoreSysPrereqs Ignore When Invoking runInstaller?
Note.867562.1 11gR1 OUI Error "fatal libCrun.so.1 open failed No such file or directory"
Note.852241.1 Installation of 11G fails on relink with INFO ld fatal library -lumem not found
Note.849336.1 Unable to launch OUI
Note.832074.1 AIX 6.1 "Dependent module /usr/lib/libXpm.a(shr_64.o) could not be loaded" Error Attempting To Execute 11.1.0 runInstaller

Note.1083353.1 11gR2 Pre-requirement APAR's check fail for AIX 6.1
Note.879102.1 Stand Alone 11.1.0.6 Silent Installation on AIX 5.3 Fails On Prerequisite Check For rsct.basic.rte(0.0) and rsct.compat.clients.rte(0.0)
NOTE:966929.1 - 11gR2 INSTALL ERRORS on Solaris 10
NOTE:989727.1 - 11GR2 INSTALL ON SOLARIS FAILS MANY PREREQUISITE CHECKS

General Information
Note.168607.1 Oracle Server Installation FAQ
Note.787705.1 How to identify correct Oracle Database software to install on Linux platform
Note.317257.1 Running Oracle Database in Solaris 10 Containers - Best Practices
Note.131321.1 How to Relink Oracle Database Software on UNIX
Note.738771.1 How to Log Good Service Request for Relink issues?
Note.458893.1 Oracle Universal Installer (OUI) FAQ
Note.736819.1 How to Log Good Service Request for Oracle Universal Installer (OUI) issues?
Note.429074.1 Can You Run Different Releases Of The Oracle Database On The Same Computer At The Same Time?

Using My Oracle Support Effectively
Note 374370.1 New Customers Start Here
Note 747242.5 My Oracle Support Configuration Management FAQ
Note 868955.1 My Oracle Support Health Checks Catalog
Note 166650.1 Working Effectively With Global Customer Support
Note 199389.1 Escalating Service Requests with Oracle Support Services

Generic Links
Note 854428.1 Patch Set Updates for Oracle Products
Note 1061295.1 Patch Set Updates - One-off Patch Conflict Resolution
Note 268895.1 Oracle Database Server Patchset Information, Versions: 8.1. 7 to 11.2.0
Note 161549.1 Oracle Database Server and Networking Patches for Microsoft Platforms
Note 161818.1 Oracle Database (RDBMS) Releases Support Status Summary
Information on Critical Patch Updates (CPU's)
About Oracle
Oracle Corporation, an enterprise software company, engages in the development, manufacture, distribution, servicing, and marketing of database, middleware, and application software worldwide.
The company was founded in 1977 and is headquartered in Redwood City, California.

ORAchk to check databases (single and RAC)

ORAchk to check databases (single and RAC)
Oracle ORAchk utility will replace RACchk utility and allow you to check also single instance databases. You can also send notification emails throught it. And schedule automatic check via cron jobs.

Here is quick guide to install and use ORAchk utility:

1. Download from My Oracle Support (you'll need MOS account for this):
Doc ID 1268927.2

2.  Install as oracle user in host you want to check:

      - unzip orachk.zip

3. Set notifications Email and automatic cron checks:
      - ./orachk -set "AUTORUN_SCHEDULE=3 1 * *; NOTIFICATION_EMAIL=<your_email_address>"

NOTE! Outlook (+ virus scanner) may not work proberly with these notification emails. So you might need some testing to get this work with it.
cron parameters in this set are (left to right):  hour (0 - 23), day of month (1 - 31), month (1 - 12), day of week (0 - 6) (0 to 6 are Sunday to Saturday)

4. Check settings:
./orachk -get all

5. Start daemon:
./orachk -d start

NOTE! with RAC this ask's root pwd for audit checks

6.  Stop daemon:
./orachk -d stop


7. To see all parameters:
./orachk -help

NOTE! you can also set auto restart. So script will be started automatically on node restarts.

How to create Oracle AWR report for a multiple instance (RAC) Oracle database

How to create Oracle AWR report for a multiple instance (RAC) Oracle database

To create the AWR report for a multiple instance Oracle RAC database (use awrrpti.sql) as:
 
1. Find the script
2. Login as sysdba
3. Execute the script. While executing supply the following:
a. format of the report (html/text)
b. enter db id
c. enter instance id
d. for number of days (don’t enter anything if you want to generate for specific snapshots). press enter
e. enter starting snapshot id from the displayed list
f. enter ending snapshot id from the displayed list
g. give a name for the report including file extension (txt/html/htm)
4. Once report is generated you exit sql*plus and view the report in the server or
5. transfer the report to your local machine and view it.
Here are the steps:

[oracle@DEVSERV ~]$ ls -l /u01/app/oracle/product/11.1.0/db_1/rdbms/admin/awrr*
-rw-r–r– 1 oracle oinstall 7575 Apr 18  2005 /u01/app/oracle/product/11.1.0/db_1/rdbms/admin/awrrpti.sql
-rw-r–r– 1 oracle oinstall 1999 Oct 24  2003 /u01/app/oracle/product/11.1.0/db_1/rdbms/admin/awrrpt.sql
[oracle@DEVSERV ~]$
[oracle@DEVSERV ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 – Production on Wed Apr 20 15:39:08 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 – 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> set pages 999 line 300
SQL>
SQL> @/u01/app/oracle/product/11.1.0/db_1/rdbms/admin/awrrpti.sql
Specify the Report Type
~~~~~~~~~~~~~~~~~~~~~~~
Would you like an HTML report, or a plain text report?
Enter ‘html’ for an HTML report, or ‘text’ for plain text
Defaults to ‘html’
Enter value for report_type: text
Type Specified:  text
Instances in this Workload Repository schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DB Id     Inst Num DB Name      Instance     Host
———— ——– ———— ———— ————
* 2330100236        1 DEVDB        DEVDB        DEVSERV
Enter value for dbid: 2330100236
Using 2330100236 for database Id
Enter value for inst_num: 1
Using 1 for instance number
Specify the number of days of snapshots to choose from
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering the number of days (n) will result in the most recent
(n) days of snapshots being listed.  Pressing <return> without
specifying a number lists all completed snapshots.
Enter value for num_days:
Listing all Completed Snapshots
Snap
Instance     DB Name        Snap Id    Snap Started    Level
———— ———— ——— —————— —–
DEVDB         DEVDB           24128 07 Apr 2011 00:00      1
24129 07 Apr 2011 01:00      1
24130 07 Apr 2011 02:00      1
24131 07 Apr 2011 03:00      1
24132 07 Apr 2011 04:00      1
24133 07 Apr 2011 05:00      1
24134 07 Apr 2011 06:00      1
24135 07 Apr 2011 07:00      1
24136 07 Apr 2011 08:00      1
24137 07 Apr 2011 09:00      1
24138 07 Apr 2011 10:00      1
24139 07 Apr 2011 11:00      1
24140 07 Apr 2011 12:00      1
24141 07 Apr 2011 13:00      1
24142 07 Apr 2011 14:00      1
24143 07 Apr 2011 15:00      1
24144 07 Apr 2011 16:00      1
24145 08 Apr 2011 18:00      1
24146 08 Nov 2011 18:00      1
24147 08 Feb 2012 13:56      1
24148 12 Oct 2013 18:00      1
Specify the Begin and End Snapshot Ids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap: 24147
Begin Snapshot Id specified: 24147
Enter value for end_snap: 24148
End   Snapshot Id specified: 24148
Specify the Report Name
~~~~~~~~~~~~~~~~~~~~~~~
The default report file name is awrrpt_1_24147_24148.txt.  To use this name,
press <return> to continue, otherwise enter an alternative.
Enter value for report_name: awr_report_24147_24148.txt
Using the report name awr_report_24147_24148.txt
WORKLOAD REPOSITORY report for
DB Name         DB Id    Instance     Inst Num Startup Time    Release     RAC
———— ———– ———— ——– ————— ———– —
DEVDB         2330100236 DEVDB               1 11-Mar-11 12:42 11.1.0.7.0  NO
Host Name        Platform                         CPUs Cores Sockets Memory(GB)
—————- ——————————– —- —– ——- ———-
DEVSERV          Linux x86 64-bit                    8     8       4      31.29
Snap Id      Snap Time      Sessions Curs/Sess
——— ——————- ——– ———
Begin Snap:     24147 08-Feb-12 13:56:12       287       3.1
End Snap:     24148 12-Oct-13 18:00:50       123        .9
Elapsed:          881,524.64 (mins)
DB Time:        5,752,234.99 (mins)
Cache Sizes                       Begin        End
~~~~~~~~~~~                  ———- ———-
Buffer Cache:     1,536M     1,536M  Std Block Size:         8K
Shared Pool Size:     2,560M     2,560M      Log Buffer:   157,036K
Load Profile              Per Second    Per Transaction   Per Exec   Per Call
~~~~~~~~~~~~         —————    ————— ———- ———-
DB Time(s):                6.5            2,543.7     123.07     345.85
DB CPU(s):                0.0                0.1       0.00       0.01
Redo size:               34.4           13,407.1
Logical reads:                4.8            1,877.2
Block changes:                0.2               63.0
Physical reads:                0.0                0.2
Physical writes:                0.0                1.9
User calls:                0.0                7.4
Parses:                0.0               15.3
Hard parses:                0.0                0.1
W/A MB processed:           15,009.8        5,851,091.6
Logons:                0.0                0.4
Executes:                0.1               20.7
Rollbacks:                0.0                0.0
Transactions:                0.0
————————————————————-
————————————————————-
————————————————————-
————————————————————-
End of Report
Report written to awr_report_24147_24148.txt
SQL>

FRM-92095: Oracle JInitiator version too low. Please install version 1.1.8.2 or higher

ERROR :

FRM-92095: Oracle JInitiator version too low. Please install version 1.1.8.2 or higher

Solution:-

Create System Environment variable in that PC with following name and value :

Name : JAVA_TOOL_OPTIONS
Value : -Djava.vendor="Sun Microsystems Inc."

Troubleshooting ORA-27300 ORA-27301 ORA-27302 Errors

Troubleshooting ORA-27300 ORA-27301 ORA-27302 Errors

NOTE:848387.1 - With NUMA Enabled, Database Fails To Open With ORA-600[ksbmoveme4], ORA-27300, ORA-27301, ORA-27302 Errors Reported
NOTE:300956.1 - Ora-27302: sskgxpsnd1 - Starting Instance
NOTE:314179.1 - Instance Startup Fails With Error ORA-27154,ORA-27300,ORA-27301,ORA-27302
NOTE:3411021.8 - Bug 3411021 - Oracle process may die due to ORA-27300/ORA-27301/ORA-27302 at skgxppost1
NOTE:746888.1 - ORA-27302: Failure Occurred at: skgxpvfymmtu Signalled in the Alert.log
BUG:7620133 - HIGH NUMBER OF DBW* PROCESSES SPAWNED IN 11.1.0.7
NOTE:458442.1 - 10.2.0.3 PMON Crashes on Startup on AIX 5L 5.3 ML05 -- Works on ML06
NOTE:277399.1 - DBMS_SCHEDULER Fails To Execute Jobs With Program_type EXECUTABLE On HP-UX
NOTE:392006.1 - Ora-27300 OS system dependent operation:fork failed with status: 11
NOTE:295832.1 - Unable To Create Database Ora-00603, ORA-27300
NOTE:1252265.1 - ORA-27300 ORA-27301 ORA-27302: failure occurred at: skgpalive1
NOTE:438205.1 - ORA-27300 ORA-27301 ORA-27302 ORA-27157 Database Crash
NOTE:453959.1 - Cannot Connect As "/ as sysdba" ORA-27140 ORA-27300 ORA-27301 ORA-27302 And ORA-27303 In Trace File
NOTE:356640.1 - ORA-27300, ORA-27301, ORA-27302: Failure Occurred At: Skgpalive1 During 'Shutdown'
NOTE:465002.1 - Database Crash With Error ORA-00490
NOTE:466370.1 - ORA-7445 [ACCESS_VIOLATION] [unable_to_trans_pc] [UNABLE_TO_WRITE] ORA-27301: OS failure message: Not enough storage ORA-27300 ORA-27302
NOTE:225349.1 - Implementing Address Windowing Extensions (AWE) or VLM on 32-bit Windows Platforms
NOTE:371074.1 - ORA-27300 ORA-27301 ORA-27302 in alert log. Cannot connect to database.
BUG:7232946 - ORA-600[KSKRECONFIGNUMA2] CAUSES INSTANCE CRASH
NOTE:580552.1 - Database Crashes With ORA-04030 ORA-07445 ORA-27300 ORA-27301 ORA-27302
NOTE:557153.1 - ORA-27370 ORA-27301 (Permission denied) When Running Job Of Type EXECUTABLE
BUG:3411021 - LMD PROCESS CRASH DUE TO ORA-27300/ORA-27301/ORA-27302 AT SKGXPPOST1
NOTE:560309.1 - Database Cannot Start Due to Lack of Memory
BUG:6991131 - ORA-27300, ORA-27301 AND ORA-27302 HAPPENED DURING SHUTDOWN USING SRVCTL
NOTE:364353.1 - ORA-00603 ORA-27504 ORA-27300 ORA-27504 in the Alert Log
NOTE:1274030.1 - Startup Instance Failed with ORA-27140 ORA-27300 ORA-27301 ORA-27302 and ORA-27303 on skgpwinit6
NOTE:949468.1 - Database Startup Fails with ORA-27300: OS system dependent operation:semget failed with status: 28
NOTE:6629265.8 - Bug 6629265 - Intermittent ORA-27504 / ORA-27300 ioctl error 11
BUG:9059053 - ORA-00603 ORA-27504 ORA-27300 ORA-27301 ORA-27302 IN ALERT.LOG
NOTE:1038058.6 - LIST OF UNIX ERRORS AND EXPLANATIONS
NOTE:297498.1 - Resolving Instance Evictions on Windows Platforms
BUG:10010310 - ORA-27300: INVALID_PROCESS_ID FAILED WITH STATUS: 0; ORA-27302: SKGPALIVE1
NOTE:6441119.8 - Bug 6441119 - Instance crash due to ORA-27300 / ORA-27152

How to Apply patch on Oracle RAC

How to Apply patch on Oracle RAC
 
Prerequisite:-
 
>> Check Oracle service status
[oracle@rac-node1 ~]$crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE dbsr...ode1
ora....E1.lsnr application ONLINE ONLINE dbsr...ode1
ora....de1.gsd application ONLINE ONLINE dbsr...ode1
ora....de1.ons application ONLINE ONLINE dbsr...ode1
ora....de1.vip application ONLINE ONLINE dbsr...ode1
ora....SM2.asm application ONLINE ONLINE dbsr...ode2
ora....E2.lsnr application ONLINE ONLINE dbsr...ode2
ora....de2.gsd application ONLINE ONLINE dbsr...ode2
ora....de2.ons application ONLINE ONLINE dbsr...ode2
ora....de2.vip application ONLINE ONLINE dbsr...ode2
ora.prod.db application ONLINE ONLINE dbsr...ode1
ora....b1.inst application ONLINE ONLINE dbsr...ode1
ora....b2.inst application ONLINE ONLINE dbsr...ode2
[oracle@rac-node1 ~]$

>> Stop database instance one by one
[oracle@rac-node1 ~]$ srvctl stop instance -i prod1 -d prod
[oracle@rac-node1 ~]$ srvctl stop instance -i prod2 -d prod
 
>>  Stop ASM instance on each node one by one
[oracle@rac-node1 ~]$ srvctl stop asm -n rac-node1
[oracle@rac-node1 ~]$ srvctl stop asm -n rac-node2
 
>> Stop node applications one by one on each node
[oracle@rac-node1 ~]$ srvctl stop nodeapps -n rac-node1
[oracle@rac-node1 ~]$ srvctl stop nodeapps -n rac-node2
 
>> Check status again
[oracle@rac-node1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application OFFLINE OFFLINE
ora....E1.lsnr application OFFLINE OFFLINE
ora....de1.gsd application OFFLINE OFFLINE
ora....de1.ons application OFFLINE OFFLINE
ora....de1.vip application OFFLINE OFFLINE
ora....SM2.asm application OFFLINE OFFLINE
ora....E2.lsnr application OFFLINE OFFLINE
ora....de2.gsd application OFFLINE OFFLINE
ora....de2.ons application OFFLINE OFFLINE
ora....de2.vip application OFFLINE OFFLINE
ora.prod.db application OFFLINE OFFLINE
ora....b1.inst application OFFLINE OFFLINE
ora....b2.inst application OFFLINE OFFLINE
[oracle@rac-node1 ~]
 
>> Download Patch from Metalink.

>> Unzip patch folder after that read readme file.

>> Go to Opatch top.
[oracle@rac-node1 rman]$ cd 9949948
OPatch should be in your path as follows
Set Oracle_Home.
[oracle@rac-node1 9949948]$ export PATH=$PATH:/u01/app/oracle/product/10.2.0 /db_1/OPatch
 
>> Start apply Opatch
[oracle@rac-node1 9949948]$ opatch apply
Invoking OPatch 10.2.0.4.9
Oracle Interim Patch Installer version 10.2.0.4.9
Copyright (c) 2009, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/10.2.0/db_1
Central Inventory : /u01/app/oracle/OraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.9
OUI version : 10.2.0.5.0
OUI location : /u01/app/oracle/product/10.2.0/db_1/oui
Log file location : /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2012-09-20_23-38-29PM.log
Patch history file: /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt
ApplySession applying interim patch '9949948' to OH '/u01/app/oracle/product/10.2.0/db_1'
Running prerequisite checks...
OPatch detected the node list and the local node from the inventory. OPatch will patch the local system then propagate This node is part of an Oracle Real Application Cluster.
Remote nodes: 'rac-node2'
Local node: 'rac-node1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/10.2.0/db_1')
Is the local system ready for patching? [y|n]
Y ==> I put yes
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '9949948' for restore. This might take a while...
Backing up files affected by the patch '9949948' for rollback. This might take a while...
Patching component oracle.rdbms, 10.2.0.5.0...
Updating archive file "/u01/app/oracle/product/10.2.0/db_1/lib/libserver10.a" with "lib/libserver10.a/ksfd.o"
Running make for target ioracle
ApplySession adding interim patch '9949948' to inventory
Verifying the update...
Inventory check OK: Patch ID 9949948 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9949948 are present in Oracle Home.
The local system has been patched. You can restart Oracle instances on it.
Patching in rolling mode.
The node 'rac-node2' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'rac-node2'.
(Oracle Home = '/u01/app/oracle/product/10.2.0/db_1')
Is the node ready for patching? [y|n]
y ==> I put yes for node2
User Responded with: Y
Updating nodes 'rac-node2'
Apply-related files are:
FP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_files.txt"
DP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_dirs.txt"
MP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/make_cmds.txt"
RC = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/remote_cmds.txt"
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_Propagating files to remote nodes...
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_Propagating directories to remote nodes...
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/make_Running command on remote node 'rac-node2':
cd /u01/app/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/The node 'rac-node2' has been patched. You can restart Oracle instances on it.
There were relinks on remote nodes. Remember to check the binary size and timestamp on the nodes 'rac-node2' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/'
OPatch succeeded.
Start Oracle services on both nodes
[oracle@rac-node1 9949948]$ srvctl start nodeapps -n rac-node1
[oracle@rac-node1 9949948]$ srvctl start nodeapps -n rac-node2
[oracle@rac-node1 9949948]$ srvctl start asm -n rac-node1
[oracle@rac-node1 9949948]$ srvctl start asm -n rac-node2
[oracle@rac-node1 9949948]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE dbsr...ode1
ora....E1.lsnr application ONLINE ONLINE dbsr...ode1
ora....de1.gsd application ONLINE ONLINE dbsr...ode1
ora....de1.ons application ONLINE ONLINE dbsr...ode1
ora....de1.vip application ONLINE ONLINE dbsr...ode1
ora....SM2.asm application ONLINE ONLINE dbsr...ode2
ora....E2.lsnr application ONLINE ONLINE dbsr...ode2
ora....de2.gsd application ONLINE ONLINE dbsr...ode2
ora....de2.ons application ONLINE ONLINE dbsr...ode2
ora....de2.vip application ONLINE ONLINE dbsr...ode2
ora.prod.db application OFFLINE OFFLINE
ora....b1.inst application OFFLINE OFFLINE
ora....b2.inst application OFFLINE OFFLINE
 
>> Start database instances on both nodes:-
[oracle@rac-node1 9949948]$ srvctl start instance -i prod1 -d prod
[oracle@rac-node1 9949948]$ srvctl start instance -i prod1 -d prod
[oracle@rac-node1 9949948]$ crs-stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE dbsr...ode1
ora....E1.lsnr application ONLINE ONLINE dbsr...ode1
ora....de1.gsd application ONLINE ONLINE dbsr...ode1
ora....de1.ons application ONLINE ONLINE dbsr...ode1
ora....de1.vip application ONLINE ONLINE dbsr...ode1
ora....SM2.asm application ONLINE ONLINE dbsr...ode2
ora....E2.lsnr application ONLINE ONLINE dbsr...ode2
ora....de2.gsd application ONLINE ONLINE dbsr...ode2
ora....de2.ons application ONLINE ONLINE dbsr...ode2
ora....de2.vip application ONLINE ONLINE dbsr...ode2
ora.prod.db application ONLINE ONLINE dbsr...ode2
ora....b1.inst application ONLINE ONLINE dbsr...ode1
ora....b2.inst application ONLINE ONLINE dbsr...ode2
 
[oracle@rac-node1 script]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.5.0 – Production on Thu Sep 20 23:44:01 2012
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.
 
SQL>conn /as sysdba
Connected.
SQL>