Easy to Learn Oracle Database and Solve Your Problem. : May 2016

Saturday, May 21, 2016

How to Restore spfile from RMAN Backup and Create pfile.

C:\Windows\system32>RMAN TARGET /

Recovery Manager: Release 11.2.0.1.0 - Production on Wed May 18 23:13:55 2016

Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

connected to target database (not started)

RMAN> STARTUP FORCE NOMOUNT

startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file 'D:\APP\RAJIB.PRADHAN\PRODUCT\11.2.0\DBHOME_1\DATABASE\INITASMDB.ORA'

starting Oracle instance without parameter file for retrieval of spfile
Oracle instance started

Total System Global Area 158662656 bytes

Fixed Size 2173840 bytes
Variable Size 92275824 bytes
Database Buffers 58720256 bytes
Redo Buffers 5492736 bytes

RMAN> restore spfile from 'D:\app\rman_backup\spfile_ASMDB_16_20160519.bak';

Starting restore at 18-MAY-16
using channel ORA_DISK_1

channel ORA_DISK_1: restoring spfile from AUTOBACKUP D:\app\rman_backup\spfile_ASMDB_16_20160519.bak
channel ORA_DISK_1: SPFILE restore from AUTOBACKUP complete
Finished restore at 18-MAY-16

RMAN> exit

Recovery Manager complete.

C:\Windows\system32>sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.1.0 Production on Wed May 18 23:20:51 2016

Copyright (c) 1982, 2010, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> create pfile='D:\APP\RAJIB.PRADHAN\PRODUCT\11.2.0\DBHOME_1\DATABASE\INITASMDB.ORA' from spfile;

File created.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

ORA-01624: log file needed for crash recovery of instance

Cause: The logfile group 2 contain committed data which are not written into data file because of checkpoint not completed.

Solution: Perform the manual checkpoint using ALTER SYSTEM CHECKPOINT GLOBAL command and drop the logfile.

SQL> alter database drop logfile group 2;
*
ERROR at line 2:
ORA-01624: log 2 needed for crash recovery of instance
ORA-00312: online log 2 thread 1: '+DATA/CBSDB/ONLINELOG/redo-02-01.log'

So to make ACTIVE to INACTIVE exeute system level checkpoint.

SQL> ALTER SYSTEM CHECKPOINT GLOBAL;

System altered.

SQL> alter database drop logfile group 2;

Database altered.

SQL>

Opatch Error libclntsh.so.11.1 executables are active

Cause: Opatch failed because of process active, find the process id and kill the process. You can find out the process which are currently active and stop the process using ps –ef |grep (process id)

Following executables are active :
/u01/app/oracle/product/11.2.4/db_1/lib/libclntsh.so.11.1
[May 20, 2016 1:27:58 PM] OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
[May 20, 2016 1:27:58 PM] Finishing UtilSession at Fri May 20 13:27:58 EDT 2016
[May 20, 2016 1:27:58 PM] Log file location: /u01/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/22502456_May_20_2016_13_27_39/rollback 2016-05-20_13-27-39PM_1.log
[May 20, 2016 1:27:58 PM] Stack Description: java.lang.RuntimeException: Prerequisite check "CheckActiveFilesAndExecutables" failed.
[May 20, 2016 1:27:58 PM] StackTrace: oracle.opatch.OPatchSessionHelper.runRollbackPrereqs(OPatchSessionHelper.java:4315)
[May 20, 2016 1:27:58 PM] StackTrace: oracle.opatch.opatchutil.NRollback.process(NRollback.java:769)
[May 20, 2016 1:27:58 PM] StackTrace: oracle.opatch.opatchutil.OUSession.nrollback(OUSession.java:1105)
[May 20, 2016 1:27:58 PM] StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[May 20, 2016 1:27:58 PM] StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[May 20, 2016 1:27:58 PM] StackTrace: sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[May 20, 2016 1:27:58 PM] StackTrace: java.lang.reflect.Method.invoke(Method.java:592)
[May 20, 2016 1:27:58 PM] StackTrace: oracle.opatch.UtilSession.process(UtilSession.java:317)
[May 20, 2016 1:27:58 PM] StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:2580)
[May 20, 2016 1:27:58 PM] StackTrace: oracle.opatch.OPatch.main(OPatch.java:634)

Find the process id for /u01/app/oracle/product/11.2.4/db_1/lib/libclntsh.so.11.1

[root@racn1 oracle]# /sbin/fuser /u01/app/oracle/product/11.2.4/db_1/lib/libclntsh.so.11.1
/u01/app/oracle/product/11.2.4/db_1/lib/libclntsh.so.11.1: 22762m

Now kill the process 22762

[root@racn1 oracle]# kill -9 22762
[root@racn1 oracle]# Killed

How to find Session ID/ SQL ID/ SQL Statement Using OS Process ID

1. It will show you the basic Session Information.

SELECT P.SPID OS_PROCESS_ID,
S.SID SESSION_ID,
S.SERIAL#,
S.USERNAME,
S.STATUS,
S.LAST_CALL_ET,
P.PROGRAM,
P.TERMINAL,
LOGON_TIME,
MODULE,
S.OSUSER
FROM GV$PROCESS P, GV$SESSION S
WHERE S.PADDR = P.ADDR
AND S.INST_ID=P.INST_ID
AND P.SPID=&Process_ID
ORDER BY S.INST_ID;

2. It Will Show You the SQL ID which SQL Statements currently Executing.

SELECT S.SQL_ID
FROM GV$PROCESS P, GV$SESSION S
WHERE S.PADDR = P.ADDR
AND S.INST_ID=P.INST_ID
AND P.SPID=&Process_ID
ORDER BY S.INST_ID;

3. It Will Show You the SQL Statements Details which SQL Statements are currently Running.

SELECT *
FROM GV$SQLAREA
WHERE SQL_ID =
(SELECT S.SQL_ID
FROM GV$PROCESS P, GV$SESSION S
WHERE S.PADDR = P.ADDR
AND S.INST_ID = P.INST_ID
AND P.SPID = &Process_ID)
ORDER BY INST_ID;

Monday, May 16, 2016

How to export dump file from Standby Database.

Physical standby database are open in Read Only mode and data pump job perform some write activity in database for completing export/import job, to support this activity Create database link in your UAT or any other database which are opened as read write mode.

1. Create Database Link With Standby Database.

CREATE DATABASE LINK STDBY
CONNECT TO PRODUSER
IDENTIFIED BY pass123
USING '10.11.201.200:1521/dcdbstdby';

2. Now submit data pump job using NETWORK_LINK parameter.

expdp system/pass123 schemas=PRODUSER directory=DUMP_FILE_DIR dumpfile=PRODUSER_DATA_DUMP.dmp logfile=PRODUSER_DATA_DUMP.log NETWORK_LINK=STDBY

This policy can be used to perform import dump file into remote database.

How to debug PL-SQL Program in RAC Database.

To debug PL-SQL code in RAC environment first create TNS in tnsnames.ora file using the only one instance. Connect into database using the new TNS name.

CBS_PROD_DB =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.11.201.200)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = CBSDB)
(INSTANCE_NAME = CBSDB1)
)
)

PSU patching updates (APR 2016) in RAC Database.

Two Node RAC Database with following environments :

1. dcdb01 (Hostname Node1)
2. dcdb02 (Hostname Node2)
3. dcdb (RAC DB Name)
4. dcdb1 (RAC DB Instance 1)
5. dcdb2 (RAC DB Instance 2)
6. /u01/stage/patch/psu_apr_2016/ (Patch staging directory for April 2016)
7. /u01/app/11.2.4/grid (Grid Home)
8. /u01/app/oracle/product/11.2.4/db_1 (RAC DB Home)

The APR2016 PSU patching updates the Grid Infrastucture (GI), Database (RAC), and OJVM components of both the GI and RAC installations. The GI patches are applied first and then the RAC patches are applied. The OJVM patches are applied during both the GI and RAC patching sessions.

The following link for downloading PSU for April 2016.

https://updates.oracle.com/download/22738793.html

Step 1 to 5 perform in all of rac node if Oracle RAC and GI home are not shared.

1. Download Opatch and move to Oracle RAC Database Home and GI Home.

$ unzip <OPATCH-ZIP> -d <ORACLE_HOME>
$ <ORACLE_HOME>/OPatch/opatch version

unzip /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip -d /u01/app/11.2.4/grid/

$opatch version

The version output of the previous command should be 11.2.0.3.6 or later.

2. OCM Configuration perform the following steps from GI Home.

export PATH=$PATH:/u01/app/11.2.0.2/grid/OPatch

export JAVA_HOME=/u01/app/oracle/product/11.2.4/db_1/jdk

$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u01/stage/patch/psu_apr_2016/ocm.rsp

chmod 775 /u01/stage/patch/psu_apr_2016/ocm.rsp

3. Validation of Oracle Inventory for Oracle RAC Database Home and GI Home. This should run from grid and oracle user.

$ <ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

$ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME

4. Download and Unzip the Patch. The directory should be empty and not /tmp

Unzip the patch as grid home owner.

cd <UNZIPPED_PATCH_LOCATION>

unzip /u01/stage/patch/psu_apr_2016/p22738793_112040_Linux-x86-64.zip

5. Stop EM Agent Processes Prior to Patching and Prior to Rolling Back the Patch

As the Oracle RAC database home owner execute:

$ <ORACLE_HOME>/bin/emctl stop dbconsole

emctl stop dbconsole

6. From the Oracle database home, make sure to stop the Oracle RAC databases running on all nodes. As the database home owner execute:

$ <ORACLE_HOME>/bin/srvctl stop database –d <db-unique-name>

srvctl stop database –d dcdb

7. On the 1st node, apply the patch to the GI Home using the opatch auto command. As root user, execute the following command:

# opatch auto <UNZIPPED_PATCH_LOCATION>/22738793 -ocmrf <ocm response file>

opatch auto /u01/stage/patch/psu_apr_2016/22738793 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp

8. On the 1st node, apply the patch to the Database home using the opatch auto command. Since the Database home is shared, this operation will patch the Database home across the cluster. Note that a USM only patch cannot be applied to a database home. As root user, execute the following command:

# opatch auto <UNZIPPED_PATCH_LOCATION>/22738793 -oh <DATABASE_HOME> -ocmrf <ocm response file>

opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp

9. On the 1st node only, restart the Oracle instance, which you have previously stopped in Step 1. As the database home owner execute:

$ <ORACLE_HOME>/bin/srvctl start instance –d <db-unique-name> -n <nodename>

srvctl start instance -d dcdb -n dcdb01

10. On the 2nd node, apply the patch to GI Home using the opatch auto command. As root user, execute the following command:

# opatch auto <UNZIPPED_PATCH_LOCATION>/22738793 -oh <GI_HOME> -ocmrf <ocm response file>

opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/11.2.4/grid -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp

11. On the 2st node, apply the patch to the Database home using the opatch auto command.

# opatch auto <UNZIPPED_PATCH_LOCATION>/22738793 -oh <DATABASE_HOME> -ocmrf <ocm response file>

opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp

12. On the 2nd node only, restart the Oracle instance, which you have previously stopped in Step 1. As the database home owner execute:

$ <ORACLE_HOME>/bin/srvctl start instance –d <db-unique-name> -n <nodename>

srvctl start instance -d dcdb -n dcdb02

##################### Loading Modified SQL Files into the Database #####################

1. load modified SQL files into the database in Oracle RAC environment, perform these steps on only one node.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> SHUTDOWN
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

The catbundle.sql execution is reflected in the dba_registry_history view by a row associated with bundle series PSU.

2. If the OJVM PSU was applied for a previous GI PSU patch, you may see invalid Java classes after execution of the catbundle.sql script in the previous step. If this is the case, run utlrp.sql to re-validate these Java classes.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql

3. Check the following log files in $ORACLE_BASE/cfgtoollogs/catbundle for any errors:

catbundle_PSU_<database SID>_APPLY_<TIMESTAMP>.log
catbundle_PSU_<database SID>_GENERATE_<TIMESTAMP>.log
where TIMESTAMP is of the form YYYYMMMDD_HH_MM_SS.

##################### Upgrade Oracle Recovery Manager Catalog #####################

If you are using the Oracle Recovery Manager, the catalog needs to be upgraded. Enter the following command to upgrade it:

$ rman catalog username/password@alias
RMAN> UPGRADE CATALOG;

##################### Post install steps for OJVM #####################

The following steps load modified SQL files into the database. For an Oracle RAC environment, perform these steps on only one node.

Install the SQL portion of the patch by running the following command for a single instance environment.

cd $ORACLE_HOME/sqlpatch/22674697
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> SHUTDOWN
SQL> STARTUP
SQL> alter system set cluster_database=false scope=spfile;
SQL> SHUTDOWN
SQL> STARTUP UPGRADE
SQL> @postinstall.sql
SQL> alter system set cluster_database=true scope=spfile;
SQL> SHUTDOWN
SQL> STARTUP

After installing the SQL portion of the patch, some packages could become INVALID. This will get recompiled upon access or you can run utlrp.sql to get them back into a VALID state.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql

##################### Log ####################
[grid@dcdb01 ~]$ cd /u01/stage/patch/psu_apr_2016/
[grid@dcdb01 psu_apr_2016]$ ls -ltr
total 774564
-rw-r--r-- 1 oracle oinstall 52733900 May 12 17:05 p6880880_112000_Linux-x86-64.zip
-rw-r--r-- 1 oracle oinstall 740284621 May 13 01:06 p22738793_112040_Linux-x86-64.zip
[grid@dcdb01 ~]$ unzip /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip -d /u01/app/11.2.4/grid/
Archive: /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip
inflating: /u01/app/11.2.4/grid/OPatch/opatchdiag
inflating: /u01/app/11.2.4/grid/OPatch/opatch.pl
.....
[grid@dcdb02 ~]$ cd /u01/stage/patch/psu_apr_2016/
[grid@dcdb02 psu_apr_2016]$ ls -ltr
total 774564
-rw-r--r-- 1 oracle oinstall 52733900 May 12 17:05 p6880880_112000_Linux-x86-64.zip
-rw-r--r-- 1 oracle oinstall 740284621 May 13 01:06 p22738793_112040_Linux-x86-64.zip
[grid@dcdb02 ~]$ unzip /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip -d /u01/app/11.2.4/grid/
Archive: /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip
inflating: /u01/app/11.2.4/grid/OPatch/opatchdiag
inflating: /u01/app/11.2.4/grid/OPatch/opatch.pl
.....
[grid@dcdb01 ~]$ export JAVA_HOME=/u01/app/oracle/product/11.2.4/db_1/jdk
[grid@dcdb01 ~]$ $ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u01/stage/patch/psu_apr_2016/ocm.rsp
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
The OCM configuration response file (/u01/stage/patch/psu_apr_2016/ocm.rsp) was successfully created.
[grid@dcdb01 ~]$ chmod 775 /u01/stage/patch/psu_apr_2016/ocm.rsp
[grid@dcdb01 ~]$

[grid@dcdb02 ~]$ export JAVA_HOME=/u01/app/oracle/product/11.2.4/db_1/jdk
[grid@dcdb02 ~]$ $ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u01/stage/patch/psu_apr_2016/ocm.rsp
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
The OCM configuration response file (/u01/stage/patch/psu_apr_2016/ocm.rsp) was successfully created.
[grid@dcdb02 ~]$ chmod 775 /u01/stage/patch/psu_apr_2016/ocm.rsp

[grid@dcdb01 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/11.2.4/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/11.2.4/grid/oraInst.loc
OPatch version : 11.2.0.3.12
OUI version : 11.2.0.4.0
Log file location : /u01/app/11.2.4/grid/cfgtoollogs/opatch/opatch2016-05-15_17-34-08PM_1.log

Lsinventory Output file location : /u01/app/11.2.4/grid/cfgtoollogs/opatch/lsinv/lsinventory2016-05-15_17-34-08PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: sbldcdb02.spftl.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.

Installed Products (89):

Agent Required Support Files 10.2.0.4.5
Assistant Common Files 11.2.0.4.0
Automatic Storage Management Assistant 11.2.0.4.0
Bali Share 1.1.18.0.0
Buildtools Common Files 11.2.0.4.0
Character Set Migration Utility 11.2.0.4.0
Cluster Ready Services Files 11.2.0.4.0
Cluster Verification Utility Common Files 11.2.0.4.0
Cluster Verification Utility Files 11.2.0.4.0
Database SQL Scripts 11.2.0.4.0
Deinstallation Tool 11.2.0.4.0
Enterprise Manager Common Core Files 10.2.0.4.5
Enterprise Manager Common Files 10.2.0.4.5
Enterprise Manager plugin Common Files 11.2.0.4.0
Expat libraries 2.0.1.0.1
HAS Common Files 11.2.0.4.0
HAS Files for DB 11.2.0.4.0
Installation Common Files 11.2.0.4.0
Installation Plugin Files 11.2.0.4.0
Installer SDK Component 11.2.0.4.0
Java Development Kit 1.5.0.51.10
LDAP Required Support Files 11.2.0.4.0
OLAP SQL Scripts 11.2.0.4.0
Oracle Advanced Security 11.2.0.4.0
Oracle Clusterware RDBMS Files 11.2.0.4.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle Containers for Java 11.2.0.4.0
Oracle Core Required Support Files 11.2.0.4.0
Oracle Database 11g 11.2.0.4.0
Oracle Database 11g Multimedia Files 11.2.0.4.0
Oracle Database Deconfiguration 11.2.0.4.0
Oracle Database User Interface 2.2.13.0.0
Oracle Database Utilities 11.2.0.4.0
Oracle DBCA Deconfiguration 11.2.0.4.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle Globalization Support 11.2.0.4.0
Oracle Globalization Support 11.2.0.4.0
Oracle Grid Infrastructure 11g 11.2.0.4.0
Oracle Help For Java 4.2.9.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle Internet Directory Client 11.2.0.4.0
Oracle Java Client 11.2.0.4.0
Oracle JDBC/OCI Instant Client 11.2.0.4.0
Oracle JDBC/THIN Interfaces 11.2.0.4.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle JVM 11.2.0.4.0
Oracle LDAP administration 11.2.0.4.0
Oracle Locale Builder 11.2.0.4.0
Oracle Multimedia 11.2.0.4.0
Oracle Multimedia Client Option 11.2.0.4.0
Oracle Multimedia Java Advanced Imaging 11.2.0.4.0
Oracle Multimedia Locator 11.2.0.4.0
Oracle Multimedia Locator RDBMS Files 11.2.0.4.0
Oracle Net 11.2.0.4.0
Oracle Net Listener 11.2.0.4.0
Oracle Net Required Support Files 11.2.0.4.0
Oracle Netca Client 11.2.0.4.0
Oracle Notification Service 11.2.0.3.0
Oracle Notification Service (eONS) 11.2.0.4.0
Oracle One-Off Patch Installer 11.2.0.3.4
Oracle Quality of Service Management (Client) 11.2.0.4.0
Oracle Quality of Service Management (Server) 11.2.0.4.0
Oracle RAC Deconfiguration 11.2.0.4.0
Oracle RAC Required Support Files-HAS 11.2.0.4.0
Oracle Recovery Manager 11.2.0.4.0
Oracle Security Developer Tools 11.2.0.4.0
Oracle Text Required Support Files 11.2.0.4.0
Oracle Universal Installer 11.2.0.4.0
Oracle USM Deconfiguration 11.2.0.4.0
Oracle Wallet Manager 11.2.0.4.0
Parser Generator Required Support Files 11.2.0.4.0
Perl Interpreter 5.10.0.0.2
Perl Modules 5.10.0.0.1
PL/SQL 11.2.0.4.0
PL/SQL Embedded Gateway 11.2.0.4.0
Platform Required Support Files 11.2.0.4.0
Precompiler Required Support Files 11.2.0.4.0
RDBMS Required Support Files 11.2.0.4.0
RDBMS Required Support Files for Instant Client 11.2.0.4.0
RDBMS Required Support Files Runtime 11.2.0.4.0
Required Support Files 11.2.0.4.0
Secure Socket Layer 11.2.0.4.0
SQL*Plus 11.2.0.4.0
SQL*Plus Files for Instant Client 11.2.0.4.0
SQL*Plus Required Support Files 11.2.0.4.0
SSL Required Support Files for InstantClient 11.2.0.4.0
Universal Storage Manager Files 11.2.0.4.0
XDK Required Support Files 11.2.0.4.0
XML Parser for Java 11.2.0.4.0
There are 89 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Rac system comprising of multiple nodes
Local node = dcdb01
Remote node = dcdb02

--------------------------------------------------------------------------------

OPatch succeeded.

[grid@dcdb02 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/11.2.4/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/11.2.4/grid/oraInst.loc
OPatch version : 11.2.0.3.12
OUI version : 11.2.0.4.0
Log file location : /u01/app/11.2.4/grid/cfgtoollogs/opatch/opatch2016-05-15_17-34-08PM_1.log

Lsinventory Output file location : /u01/app/11.2.4/grid/cfgtoollogs/opatch/lsinv/lsinventory2016-05-15_17-34-08PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: sbldcdb02.spftl.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.

Installed Products (89):

Agent Required Support Files 10.2.0.4.5
Assistant Common Files 11.2.0.4.0
Automatic Storage Management Assistant 11.2.0.4.0
Bali Share 1.1.18.0.0
Buildtools Common Files 11.2.0.4.0
Character Set Migration Utility 11.2.0.4.0
Cluster Ready Services Files 11.2.0.4.0
Cluster Verification Utility Common Files 11.2.0.4.0
Cluster Verification Utility Files 11.2.0.4.0
Database SQL Scripts 11.2.0.4.0
Deinstallation Tool 11.2.0.4.0
Enterprise Manager Common Core Files 10.2.0.4.5
Enterprise Manager Common Files 10.2.0.4.5
Enterprise Manager plugin Common Files 11.2.0.4.0
Expat libraries 2.0.1.0.1
HAS Common Files 11.2.0.4.0
HAS Files for DB 11.2.0.4.0
Installation Common Files 11.2.0.4.0
Installation Plugin Files 11.2.0.4.0
Installer SDK Component 11.2.0.4.0
Java Development Kit 1.5.0.51.10
LDAP Required Support Files 11.2.0.4.0
OLAP SQL Scripts 11.2.0.4.0
Oracle Advanced Security 11.2.0.4.0
Oracle Clusterware RDBMS Files 11.2.0.4.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle Containers for Java 11.2.0.4.0
Oracle Core Required Support Files 11.2.0.4.0
Oracle Database 11g 11.2.0.4.0
Oracle Database 11g Multimedia Files 11.2.0.4.0
Oracle Database Deconfiguration 11.2.0.4.0
Oracle Database User Interface 2.2.13.0.0
Oracle Database Utilities 11.2.0.4.0
Oracle DBCA Deconfiguration 11.2.0.4.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle Globalization Support 11.2.0.4.0
Oracle Globalization Support 11.2.0.4.0
Oracle Grid Infrastructure 11g 11.2.0.4.0
Oracle Help For Java 4.2.9.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle Internet Directory Client 11.2.0.4.0
Oracle Java Client 11.2.0.4.0
Oracle JDBC/OCI Instant Client 11.2.0.4.0
Oracle JDBC/THIN Interfaces 11.2.0.4.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle JVM 11.2.0.4.0
Oracle LDAP administration 11.2.0.4.0
Oracle Locale Builder 11.2.0.4.0
Oracle Multimedia 11.2.0.4.0
Oracle Multimedia Client Option 11.2.0.4.0
Oracle Multimedia Java Advanced Imaging 11.2.0.4.0
Oracle Multimedia Locator 11.2.0.4.0
Oracle Multimedia Locator RDBMS Files 11.2.0.4.0
Oracle Net 11.2.0.4.0
Oracle Net Listener 11.2.0.4.0
Oracle Net Required Support Files 11.2.0.4.0
Oracle Netca Client 11.2.0.4.0
Oracle Notification Service 11.2.0.3.0
Oracle Notification Service (eONS) 11.2.0.4.0
Oracle One-Off Patch Installer 11.2.0.3.4
Oracle Quality of Service Management (Client) 11.2.0.4.0
Oracle Quality of Service Management (Server) 11.2.0.4.0
Oracle RAC Deconfiguration 11.2.0.4.0
Oracle RAC Required Support Files-HAS 11.2.0.4.0
Oracle Recovery Manager 11.2.0.4.0
Oracle Security Developer Tools 11.2.0.4.0
Oracle Text Required Support Files 11.2.0.4.0
Oracle Universal Installer 11.2.0.4.0
Oracle USM Deconfiguration 11.2.0.4.0
Oracle Wallet Manager 11.2.0.4.0
Parser Generator Required Support Files 11.2.0.4.0
Perl Interpreter 5.10.0.0.2
Perl Modules 5.10.0.0.1
PL/SQL 11.2.0.4.0
PL/SQL Embedded Gateway 11.2.0.4.0
Platform Required Support Files 11.2.0.4.0
Precompiler Required Support Files 11.2.0.4.0
RDBMS Required Support Files 11.2.0.4.0
RDBMS Required Support Files for Instant Client 11.2.0.4.0
RDBMS Required Support Files Runtime 11.2.0.4.0
Required Support Files 11.2.0.4.0
Secure Socket Layer 11.2.0.4.0
SQL*Plus 11.2.0.4.0
SQL*Plus Files for Instant Client 11.2.0.4.0
SQL*Plus Required Support Files 11.2.0.4.0
SSL Required Support Files for InstantClient 11.2.0.4.0
Universal Storage Manager Files 11.2.0.4.0
XDK Required Support Files 11.2.0.4.0
XML Parser for Java 11.2.0.4.0
There are 89 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Rac system comprising of multiple nodes
Local node = dcdb02
Remote node = dcdb01

--------------------------------------------------------------------------------

OPatch succeeded.

[grid@dcdb01 ~]$
[grid@dcdb01 ~]$ cd /u01/stage/patch/psu_apr_2016/
[grid@dcdb01 psu_apr_2016]$ unzip p22738793_112040_Linux-x86-64.zip
Archive: p22738793_112040_Linux-x86-64.zip
creating: 22738793/
creating: 22738793/22674697/
creating: 22738793/22674697/etc/
creating: 22738793/22674697/etc/config/
inflating: 22738793/22674697/etc/config/deploy.xml
inflating: 22738793/22674697/etc/config/inventory.xml
inflating: 22738793/22674697/etc/config/actions.xml
creating: 22738793/22674697/etc/xml/
.....
[grid@dcdb02 ~]$ cd /u01/stage/patch/psu_apr_2016/
[grid@dcdb02 psu_apr_2016]$ unzip p22738793_112040_Linux-x86-64.zip
Archive: p22738793_112040_Linux-x86-64.zip
creating: 22738793/
creating: 22738793/22674697/
creating: 22738793/22674697/etc/
creating: 22738793/22674697/etc/config/
inflating: 22738793/22674697/etc/config/deploy.xml
inflating: 22738793/22674697/etc/config/inventory.xml
inflating: 22738793/22674697/etc/config/actions.xml
creating: 22738793/22674697/etc/xml/
.....

[oracle@dcdb01 ~]$ emctl stop dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.4.0
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
https://dcdb01.dc.com:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 11g Database Control ...
... Stopped.
[oracle@dcdb01 ~]$

[oracle@dcdb02 ~]$ emctl stop dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.4.0
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
https://dcdb01.dc.com:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 11g Database Control ...
... Stopped.
[oracle@dcdb02 ~]$
[oracle@dcdb01 oracle]# srvctl stop database -d dcdb
[oracle@dcdb01 oracle]# srvctl status database -d dcdb
Instance dcdb1 is not running on node dcdb01
Instance dcdb1 is not running on node dcdb02
[oracle@dcdb01 ~]$ su grid
Password:
[grid@dcdb01 oracle]$ cd
[grid@dcdb01 ~]$
[grid@dcdb01 ~]$ . ./.bash_profile
[grid@dcdb01 ~]$ su root
Password:
[root@dcdb01 grid]#
[root@dcdb01 grid]# echo $ORACLE_HOME
/u01/app/11.2.4/grid
[root@dcdb01 grid]#
[root@dcdb01 ~]# opatch auto /u01/stage/patch/psu_apr_2016/22738793 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp
Executing /u01/app/11.2.4/grid/perl/bin/perl /u01/app/11.2.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/stage/patch/psu_apr_2016 -patchn 22738793 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp -paramfile /u01/app/11.2.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_13-33-15.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_13-33-15.report.log

2016-05-15 13:33:15: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.4/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.4/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.4/db_1 successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456 apply successful for home /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22502549/custom/server/22502549 apply successful for home /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22674697 apply successful for home /u01/app/oracle/product/11.2.4/db_1

Stopping CRS...
Stopped CRS successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456 apply successful for home /u01/app/11.2.4/grid
patch /u01/stage/patch/psu_apr_2016/22738793/22502549 apply successful for home /u01/app/11.2.4/grid
patch /u01/stage/patch/psu_apr_2016/22738793/22502505 apply successful for home /u01/app/11.2.4/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.4/db_1 ...
Started RAC /u01/app/oracle/product/11.2.4/db_1 successfully

opatch auto succeeded.
[root@dcdb01 ~]#
[root@dcdb01 ~]# cd /home/oracle/
[root@dcdb01 oracle]# . ./.bash_profile
[root@dcdb01 oracle]# echo $ORACLE_HOME
/u01/app/oracle/product/11.2.4/db_1
[root@dcdb01 oracle]#
[root@dcdb01 oracle]# opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp
Executing /u01/app/11.2.4/grid/perl/bin/perl /u01/app/11.2.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/stage/patch/psu_apr_2016 -patchn 22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp -paramfile /u01/app/11.2.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_13-51-45.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_13-51-45.report.log

2016-05-15 13:51:45: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.4/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.4/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.4/db_1 successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456 apply successful for home /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22502549/custom/server/22502549 apply successful for home /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22674697 apply successful for home /u01/app/oracle/product/11.2.4/db_1

Starting RAC /u01/app/oracle/product/11.2.4/db_1 ...
Started RAC /u01/app/oracle/product/11.2.4/db_1 successfully

opatch auto succeeded.
[root@dcdb01 oracle]#
[root@dcdb01 oracle]# su oracle
[oracle@dcdb01 ~]$ srvctl start instance -d dcdb -n dcdb01
[oracle@dcdb01 ~]$
[oracle@dcdb01 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/11.2.4/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/11.2.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.12
OUI version : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/opatch2016-05-15_13-57-42PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/lsinv/lsinventory2016-05-15_13-57-42PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: dcdb01.dc.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Database 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.

Interim patches (3) :

Patch 22674697 : applied on Sun May 15 13:37:40 BDT 2016
Unique Patch ID: 20038581
Patch description: "OJVM PATCH SET UPDATE 11.2.0.4.160419"
Created on 17 Mar 2016, 07:51:04 hrs PST8PDT
Bugs fixed:
19058059, 18933818, 19176885, 17201047, 19007266, 19554117, 14774730
17285560, 19153980, 21911849, 18166577, 22670385, 19187988, 18458318
19374518, 19006757, 17056813, 21811517, 19909862, 19223010, 22118835
19895326, 22253904, 20408829, 19852360, 17804361, 21047766, 19231857
17528315, 21566944

Patch 22502549 : applied on Sun May 15 13:37:12 BDT 2016
Unique Patch ID: 19880366
Patch description: "OCW Patch Set Update : 11.2.0.4.160419 (22502549)"
Created on 11 Mar 2016, 01:34:41 hrs PST8PDT
Bugs fixed:
19270660, 18328800, 18508710, 18691572, 20038451, 21251192, 21232394
20365005, 17750548, 17387214, 17617807, 14497275, 20219458, 17733927
18180541, 18962892, 17292250, 17378618, 16759171, 20110156, 17843489
17065496, 13991403, 17273020, 17155238, 20012766, 21245437, 18261183
18053580, 20218012, 17013634, 17886392, 20995001, 17039197, 17947785
16317771, 10052729, 22353346, 20340620, 16237657, 20317221, 15917869
18199185, 18399991, 20186278, 17374271, 18024089, 16849642, 20746251
20246071, 14270845, 20552947, 18882642, 18414137, 17001914, 17927970
14378120, 16346413, 15986647, 18068871, 21222147, 18143836, 16206997
21982225, 19168690, 20235511, 18343490, 21875360, 16613232, 19276791
17722664, 20440643, 12928658, 18952577, 18520351, 16249829, 18226143
16076412, 18265482, 18229842, 17172091, 20676340, 17818075, 20091753
18231837, 14373486, 20136892, 17483479, 20551654, 18120545, 18729166
13843841, 21225209, 17405302, 18709496, 18330979, 18744838, 17087371
20531190, 14525998, 18187697, 20598625, 14385860, 18348155, 19479503
12928592, 17516024, 18370031, 17764053, 19272663, 17551223, 14671408
18272135, 14207615, 21255373, 17500165, 18875012, 14769643, 18464784
19558324, 18848125, 19241857, 14851828, 17955615, 20315294, 14693336
16284825, 17352230, 20014326, 17238586, 17089344, 17405605, 21327402
17531342, 19398098, 17159489, 17640316, 13823394, 16543190, 22024217
17983675, 20795241, 17598201, 17481314, 16281493, 18346135, 15986311
19601468, 17208793, 18700935, 18999857, 14076173, 18428146, 18352845
17435488, 20408163, 17592037, 18352846, 19616601, 17391726, 17387779
14777968, 15851860, 16206882, 20141091, 21113068, 20175174, 17305100
15832129, 19885321, 16901346, 17985714, 18536826, 17780903, 18752378
18946768, 16876500, 16875342, 17769597, 19955755, 16429265, 18336452
17273003, 17209968, 16988311, 20094984, 19319357, 17059927, 17046460
18053631, 16867761, 18774591, 21442094, 20235486, 19359787, 15869775
19642566, 17447588, 16798862, 15920201

Patch 22502456 : applied on Sun May 15 13:36:11 BDT 2016
Unique Patch ID: 19856194
Patch description: "Database Patch Set Update : 11.2.0.4.160419 (22502456)"
Created on 21 Mar 2016, 11:49:22 hrs
Sub-patch 21948347; "Database Patch Set Update : 11.2.0.4.160119 (21948347)"
Sub-patch 21352635; "Database Patch Set Update : 11.2.0.4.8 (21352635)"
Sub-patch 20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
Sub-patch 20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
Sub-patch 19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch 19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch 18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch 17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
Bugs fixed:
17288409, 21051852, 17811429, 18607546, 17205719, 20506699, 17816865
17922254, 17754782, 16934803, 13364795, 17311728, 17441661, 17284817
16992075, 17446237, 14015842, 19972569, 21756677, 21538558, 20925795
17449815, 17375354, 19463897, 13866822, 17982555, 17235750, 17478514
18317531, 14338435, 18235390, 20803583, 13944971, 20142975, 17811789
16929165, 18704244, 20506706, 17546973, 20334344, 14054676, 17088068
17346091, 18264060, 17343514, 21538567, 19680952, 18471685, 19211724
13951456, 21847223, 16315398, 18744139, 16850630, 19049453, 18673304
17883081, 19915271, 18641419, 18262334, 17006183, 16065166, 18277454
16833527, 10136473, 18051556, 17865671, 17852463, 18554871, 17853498
18334586, 17551709, 17588480, 19827973, 17344412, 17842825, 18828868
17025461, 11883252, 13609098, 17239687, 17602269, 19197175, 22195457
18316692, 17313525, 12611721, 19544839, 18964939, 17600719, 18191164
19393542, 17571306, 18482502, 20777150, 19466309, 17040527, 17165204
18098207, 16785708, 17465741, 17174582, 16180763, 16777840, 12982566
19463893, 22195465, 16875449, 12816846, 17237521, 19358317, 17811438
17811447, 21983325, 17945983, 18762750, 16912439, 17184721, 18061914
17282229, 18331850, 18202441, 17082359, 18723434, 21972320, 19554106
14034426, 18339044, 19458377, 17752995, 20448824, 17891943, 17258090
17767676, 16668584, 18384391, 17040764, 17381384, 15913355, 18356166
14084247, 20596234, 20506715, 21756661, 13853126, 18203837, 14245531
21756699, 16043574, 22195441, 17848897, 17877323, 21453153, 17468141
20861693, 17786518, 17912217, 17037130, 18155762, 16956380, 17478145
17394950, 18641461, 18189036, 18619917, 17027426, 21352646, 16268425
22195492, 19584068, 18436307, 17265217, 17634921, 13498382, 21526048
19258504, 20004087, 17443671, 22195485, 18000422, 20004021, 22321756
17571039, 21067387, 16344544, 18009564, 14354737, 21286665, 18135678
18614015, 20441797, 18362222, 17835048, 16472716, 17936109, 17050888
17325413, 14010183, 18747196, 17761775, 16721594, 17082983, 20067212
21179898, 17302277, 18084625, 15990359, 18203835, 17297939, 17811456
16731148, 21168487, 13829543, 17215560, 14133975, 17694209, 17385178
18091059, 8322815, 17586955, 17201159, 17655634, 18331812, 19730508
18868646, 17648596, 16220077, 16069901, 17348614, 17393915, 17274537
17957017, 18096714, 17308789, 18436647, 14285317, 19289642, 14764829
18328509, 17622427, 22195477, 16943711, 22502493, 14368995, 17346671
18996843, 17783588, 21343838, 16618694, 17672719, 18856999, 18783224
17851160, 17546761, 17798953, 18273830, 22092979, 16596890, 19972566
16384983, 17726838, 17360606, 22321741, 13645875, 18199537, 16542886
21787056, 17889549, 14565184, 17071721, 17610798, 20299015, 21343897
22893153, 20657441, 17397545, 18230522, 16360112, 19769489, 12905058
18641451, 12747740, 18430495, 17016369, 17042658, 14602788, 17551063
19972568, 21517440, 18508861, 19788842, 14657740, 17332800, 13837378
19972564, 17186905, 18315328, 19699191, 17437634, 22353199, 18093615
19006849, 19013183, 17296856, 18674024, 17232014, 16855292, 17762296
14692762, 21051840, 17705023, 19121551, 21330264, 19854503, 21868720
19309466, 18681862, 18554763, 20558005, 17390160, 18456514, 16306373
13955826, 18139690, 17501491, 17752121, 21668627, 17299889, 17889583
18673325, 19721304, 18293054, 17242746, 17951233, 17649265, 18094246
19615136, 17011832, 16870214, 17477958, 18522509, 20631274, 16091637
17323222, 16595641, 16524926, 18228645, 18282562, 17596908, 17156148
18031668, 16494615, 22683225, 17545847, 17655240, 17614134, 13558557
17341326, 17891946, 17716305, 16392068, 19271443, 21351877, 18092127
18440047, 17614227, 14106803, 16903536, 18973907, 18673342, 19032867
17389192, 17612828, 16194160, 17006570, 17721717, 17390431, 17570240
16863422, 18325460, 19727057, 16422541, 19972570, 17267114, 18244962
21538485, 18765602, 18203838, 16198143, 17246576, 14829250, 17835627
18247991, 14458214, 21051862, 16692232, 17786278, 17227277, 16042673
16314254, 16228604, 16837842, 17393683, 17787259, 20331945, 20074391
15861775, 16399083, 18018515, 22683212, 18260550, 21051858, 17036973
16613964, 17080436, 16579084, 18384537, 18280813, 20296213, 16901385
15979965, 18441944, 16450169, 9756271, 17892268, 11733603, 16285691
17587063, 21343775, 16538760, 18180390, 18193833, 21387964, 21051833
17238511, 17824637, 16571443, 18306996, 14852021, 18674047, 17853456
12364061, 22195448

Rac system comprising of multiple nodes
Local node = dcdb01
Remote node = dcdb02

--------------------------------------------------------------------------------

OPatch succeeded.
[oracle@dcdb01 ~]$

[root@dcdb01 oracle]# ssh grid@dcdb02
grid@dcdb02's password:
Last login: Mon May 1 15:40:54 2016 from X.X.X.X
[grid@dcdb02 ~]$
[root@dcdb02 grid]# opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/11.2.4/grid -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp
Executing /u01/app/11.2.4/grid/perl/bin/perl /u01/app/11.2.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/stage/patch/psu_apr_2016 -patchn 22738793 -oh /u01/app/11.2.4/grid -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp -paramfile /u01/app/11.2.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_17-39-47.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_17-39-47.report.log

2016-05-15 17:39:47: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.4/grid/crs/install/crsconfig_params

Stopping CRS...
Stopped CRS successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456 apply successful for home /u01/app/11.2.4/grid
patch /u01/stage/patch/psu_apr_2016/22738793/22502549 apply successful for home /u01/app/11.2.4/grid
patch /u01/stage/patch/psu_apr_2016/22738793/22502505 apply successful for home /u01/app/11.2.4/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

opatch auto succeeded.
[root@dcdb02 grid]#
[root@dcdb02 ~]# cd /home/oracle/
[root@dcdb02 oracle]# . ./.bash_profile
[root@dcdb02 oracle]# echo $ORACLE_HOME
/u01/app/oracle/product/11.2.4/db_1
[root@dcdb02 oracle]#
[root@dcdb02 oracle]# opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp
Executing /u01/app/11.2.4/grid/perl/bin/perl /u01/app/11.2.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/stage/patch/psu_apr_2016 -patchn 22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp -paramfile /u01/app/11.2.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_17-52-14.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_17-52-14.report.log

2016-05-15 17:52:14: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.4/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.4/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.4/db_1 successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456 apply successful for home /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22502549/custom/server/22502549 apply successful for home /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22674697 apply successful for home /u01/app/oracle/product/11.2.4/db_1

Starting RAC /u01/app/oracle/product/11.2.4/db_1 ...
Started RAC /u01/app/oracle/product/11.2.4/db_1 successfully

opatch auto succeeded.
[root@dcdb02 oracle]# exit
[oracle@dcdb02 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/11.2.4/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/11.2.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.12
OUI version : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/opatch2016-05-16_12-25-55PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/lsinv/lsinventory2016-05-16_12-25-55PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: dcdb02.dc.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Database 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.

Interim patches (3) :

Patch 22674697 : applied on Sun May 15 17:56:06 BDT 2016
Unique Patch ID: 20038581
Patch description: "OJVM PATCH SET UPDATE 11.2.0.4.160419"
Created on 17 Mar 2016, 07:51:04 hrs PST8PDT
Bugs fixed:
19058059, 18933818, 19176885, 17201047, 19007266, 19554117, 14774730
17285560, 19153980, 21911849, 18166577, 22670385, 19187988, 18458318
19374518, 19006757, 17056813, 21811517, 19909862, 19223010, 22118835
19895326, 22253904, 20408829, 19852360, 17804361, 21047766, 19231857
17528315, 21566944

Patch 22502549 : applied on Sun May 15 17:55:38 BDT 2016
Unique Patch ID: 19880366
Patch description: "OCW Patch Set Update : 11.2.0.4.160419 (22502549)"
Created on 11 Mar 2016, 01:34:41 hrs PST8PDT
Bugs fixed:
19270660, 18328800, 18508710, 18691572, 20038451, 21251192, 21232394
20365005, 17750548, 17387214, 17617807, 14497275, 20219458, 17733927
18180541, 18962892, 17292250, 17378618, 16759171, 20110156, 17843489
17065496, 13991403, 17273020, 17155238, 20012766, 21245437, 18261183
18053580, 20218012, 17013634, 17886392, 20995001, 17039197, 17947785
16317771, 10052729, 22353346, 20340620, 16237657, 20317221, 15917869
18199185, 18399991, 20186278, 17374271, 18024089, 16849642, 20746251
20246071, 14270845, 20552947, 18882642, 18414137, 17001914, 17927970
14378120, 16346413, 15986647, 18068871, 21222147, 18143836, 16206997
21982225, 19168690, 20235511, 18343490, 21875360, 16613232, 19276791
17722664, 20440643, 12928658, 18952577, 18520351, 16249829, 18226143
16076412, 18265482, 18229842, 17172091, 20676340, 17818075, 20091753
18231837, 14373486, 20136892, 17483479, 20551654, 18120545, 18729166
13843841, 21225209, 17405302, 18709496, 18330979, 18744838, 17087371
20531190, 14525998, 18187697, 20598625, 14385860, 18348155, 19479503
12928592, 17516024, 18370031, 17764053, 19272663, 17551223, 14671408
18272135, 14207615, 21255373, 17500165, 18875012, 14769643, 18464784
19558324, 18848125, 19241857, 14851828, 17955615, 20315294, 14693336
16284825, 17352230, 20014326, 17238586, 17089344, 17405605, 21327402
17531342, 19398098, 17159489, 17640316, 13823394, 16543190, 22024217
17983675, 20795241, 17598201, 17481314, 16281493, 18346135, 15986311
19601468, 17208793, 18700935, 18999857, 14076173, 18428146, 18352845
17435488, 20408163, 17592037, 18352846, 19616601, 17391726, 17387779
14777968, 15851860, 16206882, 20141091, 21113068, 20175174, 17305100
15832129, 19885321, 16901346, 17985714, 18536826, 17780903, 18752378
18946768, 16876500, 16875342, 17769597, 19955755, 16429265, 18336452
17273003, 17209968, 16988311, 20094984, 19319357, 17059927, 17046460
18053631, 16867761, 18774591, 21442094, 20235486, 19359787, 15869775
19642566, 17447588, 16798862, 15920201

Patch 22502456 : applied on Sun May 15 17:54:42 BDT 2016
Unique Patch ID: 19856194
Patch description: "Database Patch Set Update : 11.2.0.4.160419 (22502456)"
Created on 21 Mar 2016, 11:49:22 hrs
Sub-patch 21948347; "Database Patch Set Update : 11.2.0.4.160119 (21948347)"
Sub-patch 21352635; "Database Patch Set Update : 11.2.0.4.8 (21352635)"
Sub-patch 20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
Sub-patch 20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
Sub-patch 19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch 19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch 18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch 17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
Bugs fixed:
17288409, 21051852, 17811429, 18607546, 17205719, 20506699, 17816865
17922254, 17754782, 16934803, 13364795, 17311728, 17441661, 17284817
16992075, 17446237, 14015842, 19972569, 21756677, 21538558, 20925795
17449815, 17375354, 19463897, 13866822, 17982555, 17235750, 17478514
18317531, 14338435, 18235390, 20803583, 13944971, 20142975, 17811789
16929165, 18704244, 20506706, 17546973, 20334344, 14054676, 17088068
17346091, 18264060, 17343514, 21538567, 19680952, 18471685, 19211724
13951456, 21847223, 16315398, 18744139, 16850630, 19049453, 18673304
17883081, 19915271, 18641419, 18262334, 17006183, 16065166, 18277454
16833527, 10136473, 18051556, 17865671, 17852463, 18554871, 17853498
18334586, 17551709, 17588480, 19827973, 17344412, 17842825, 18828868
17025461, 11883252, 13609098, 17239687, 17602269, 19197175, 22195457
18316692, 17313525, 12611721, 19544839, 18964939, 17600719, 18191164
19393542, 17571306, 18482502, 20777150, 19466309, 17040527, 17165204
18098207, 16785708, 17465741, 17174582, 16180763, 16777840, 12982566
19463893, 22195465, 16875449, 12816846, 17237521, 19358317, 17811438
17811447, 21983325, 17945983, 18762750, 16912439, 17184721, 18061914
17282229, 18331850, 18202441, 17082359, 18723434, 21972320, 19554106
14034426, 18339044, 19458377, 17752995, 20448824, 17891943, 17258090
17767676, 16668584, 18384391, 17040764, 17381384, 15913355, 18356166
14084247, 20596234, 20506715, 21756661, 13853126, 18203837, 14245531
21756699, 16043574, 22195441, 17848897, 17877323, 21453153, 17468141
20861693, 17786518, 17912217, 17037130, 18155762, 16956380, 17478145
17394950, 18641461, 18189036, 18619917, 17027426, 21352646, 16268425
22195492, 19584068, 18436307, 17265217, 17634921, 13498382, 21526048
19258504, 20004087, 17443671, 22195485, 18000422, 20004021, 22321756
17571039, 21067387, 16344544, 18009564, 14354737, 21286665, 18135678
18614015, 20441797, 18362222, 17835048, 16472716, 17936109, 17050888
17325413, 14010183, 18747196, 17761775, 16721594, 17082983, 20067212
21179898, 17302277, 18084625, 15990359, 18203835, 17297939, 17811456
16731148, 21168487, 13829543, 17215560, 14133975, 17694209, 17385178
18091059, 8322815, 17586955, 17201159, 17655634, 18331812, 19730508
18868646, 17648596, 16220077, 16069901, 17348614, 17393915, 17274537
17957017, 18096714, 17308789, 18436647, 14285317, 19289642, 14764829
18328509, 17622427, 22195477, 16943711, 22502493, 14368995, 17346671
18996843, 17783588, 21343838, 16618694, 17672719, 18856999, 18783224
17851160, 17546761, 17798953, 18273830, 22092979, 16596890, 19972566
16384983, 17726838, 17360606, 22321741, 13645875, 18199537, 16542886
21787056, 17889549, 14565184, 17071721, 17610798, 20299015, 21343897
22893153, 20657441, 17397545, 18230522, 16360112, 19769489, 12905058
18641451, 12747740, 18430495, 17016369, 17042658, 14602788, 17551063
19972568, 21517440, 18508861, 19788842, 14657740, 17332800, 13837378
19972564, 17186905, 18315328, 19699191, 17437634, 22353199, 18093615
19006849, 19013183, 17296856, 18674024, 17232014, 16855292, 17762296
14692762, 21051840, 17705023, 19121551, 21330264, 19854503, 21868720
19309466, 18681862, 18554763, 20558005, 17390160, 18456514, 16306373
13955826, 18139690, 17501491, 17752121, 21668627, 17299889, 17889583
18673325, 19721304, 18293054, 17242746, 17951233, 17649265, 18094246
19615136, 17011832, 16870214, 17477958, 18522509, 20631274, 16091637
17323222, 16595641, 16524926, 18228645, 18282562, 17596908, 17156148
18031668, 16494615, 22683225, 17545847, 17655240, 17614134, 13558557
17341326, 17891946, 17716305, 16392068, 19271443, 21351877, 18092127
18440047, 17614227, 14106803, 16903536, 18973907, 18673342, 19032867
17389192, 17612828, 16194160, 17006570, 17721717, 17390431, 17570240
16863422, 18325460, 19727057, 16422541, 19972570, 17267114, 18244962
21538485, 18765602, 18203838, 16198143, 17246576, 14829250, 17835627
18247991, 14458214, 21051862, 16692232, 17786278, 17227277, 16042673
16314254, 16228604, 16837842, 17393683, 17787259, 20331945, 20074391
15861775, 16399083, 18018515, 22683212, 18260550, 21051858, 17036973
16613964, 17080436, 16579084, 18384537, 18280813, 20296213, 16901385
15979965, 18441944, 16450169, 9756271, 17892268, 11733603, 16285691
17587063, 21343775, 16538760, 18180390, 18193833, 21387964, 21051833
17238511, 17824637, 16571443, 18306996, 14852021, 18674047, 17853456
12364061, 22195448

Rac system comprising of multiple nodes
Local node = dcdb02
Remote node = dcdb01

--------------------------------------------------------------------------------

OPatch succeeded.
[oracle@dcdb02 ~]$
[oracle@dcdb02 ~]$ srvctl start instance -d dcdb -n dcdb02

[oracle@dcdb01 ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@dcdb01 admin]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Sun May 15 18:03:45 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> SHUTDOWN
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> STARTUP;
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size 2265224 bytes
Variable Size 3472887672 bytes
Database Buffers 2919235584 bytes
Redo Buffers 19292160 bytes
Database mounted.
Database opened.
SQL> @catbundle.sql psu apply

PL/SQL procedure successfully completed.

Function created.

Function created.

....

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@dcdb01 admin]$
[oracle@dcdb01 admin]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Sun May 15 18:03:45 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @utlrp.sql

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN 2016-05-15 18:18:38

DOC> The following PL/SQL block invokes UTL_RECOMP to recompile invalid
DOC> objects in the database. Recompilation time is proportional to the
DOC> number of invalid objects in the database, so this command may take
DOC> a long time to execute on a database with a large number of invalid
DOC> objects.

......
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@dcdb01 admin]$
[grid@dcdb01 ~]$ srvctl stop database -d dcdb
[oracle@dcdb01 catbundle]$ cd $ORACLE_HOME/sqlpatch/22674697
[oracle@dcdb01 22674697]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Sun May 15 18:58:42 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size 2265224 bytes
Variable Size 3472887672 bytes
Database Buffers 2919235584 bytes
Redo Buffers 19292160 bytes
SQL> alter systemd set cluster_database=false scope=spfile;

System altered.

SQL> shut
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup upgrade
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size 2265224 bytes
Variable Size 3472887672 bytes
Database Buffers 2919235584 bytes
Redo Buffers 19292160 bytes
Database mounted.
Database opened.
SQL> @postinstall.sql

PL/SQL procedure successfully completed.

Function created.

PL/SQL procedure successfully completed.

...

SQL> alter system set cluster_database=true scope=spfile;

System altered.

SQL> SHUTDOWN
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
[oracle@dcdb01 22674697]$
[oracle@dcdb01 22674697]$ srvctl start database -d dcdb

Reference:

Patch Set Updates for Oracle Products (Doc ID 854428.1)

My Oracle Support Document 966023.1

How to Create an OCM Response file to Apply a Patch in Silent Mode - opatch silent (Doc ID 966023.1)

Sunday, May 15, 2016

ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE

SQL> alter system set cluster_database=false scope=spfile;

[oracle@dcdb01 22674697]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Sun May 15 18:56:08 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup upgrade
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size 2265224 bytes
Variable Size 3472887672 bytes
Database Buffers 2919235584 bytes
Redo Buffers 19292160 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE
Process ID: 27450
Session ID: 194 Serial number: 3

SQL> startup nomount;
ORA-24324: service handle not initialized
ORA-01041: internal error. hostdef extension doesn't exist
SQL>
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@dcdb01 22674697]$
[oracle@dcdb01 22674697]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Sun May 15 18:58:42 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size 2265224 bytes
Variable Size 3472887672 bytes
Database Buffers 2919235584 bytes
Redo Buffers 19292160 bytes
SQL> alter system set cluster_database=false scope=spfile;

System altered.

SQL> shut
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup upgrade
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size 2265224 bytes
Variable Size 3472887672 bytes
Database Buffers 2919235584 bytes
Redo Buffers 19292160 bytes
Database mounted.
Database opened.

SQL>

Easy to Learn Oracle Database and Solve Your Problem.

Query

Search Blog