Showing posts with label RAC. Show all posts
Showing posts with label RAC. Show all posts

Thursday, December 22, 2016

CRS-2730: Resource 'ora.erpdb.db' depends on resource 'ora.RMAN_BKP.dg'

Cause : If you want to delete resource from clusterware which are using by another service then you have to remove dependence.

Solution : Remove the dependence before remove service.


[grid@erpdb01 ~]$ crsctl delete res ora.RMAN_BKP.dg
CRS-2730: Resource 'ora.erpdb.db' depends on resource 'ora.RMAN_BKP.dg'
CRS-4000: Command Delete failed, or completed with errors.
[grid@erpdb01 ~]$ srvctl config database -d erpdb
Database unique name: erpdb
Database name: erpdb
Oracle home: /u01/app/oracle/product/11.2.4/db_1
Oracle user: oracle
Spfile: +DATA/erpdb/spfileerpdb.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: erpdb
Database instances: erpdb1,erpdb2
Disk Groups: DATA,FRA,RMAN_BKP
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[grid@erpdb01 ~]$ su oracle
Password:
[oracle@erpdb01 grid]$ cd
[oracle@erpdb01 ~]$ . ./.bash_profile
[oracle@erpdb01 ~]$ srvctl modify database -d erpdb -a "DATA,FRA"
[oracle@erpdb01 ~]$ srvctl config database -d erpdb
Database unique name: erpdb
Database name: erpdb
Oracle home: /u01/app/oracle/product/11.2.4/db_1
Oracle user: oracle
Spfile: +DATA/erpdb/spfileerpdb.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: erpdb
Database instances: erpdb1,erpdb2
Disk Groups: DATA,FRA
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@erpdb01 ~]$
[grid@erpdb01 ~]$ crsctl delete res ora.RMAN_BKP.dg
[grid@erpdb01 ~]$

Tuesday, November 22, 2016

How to perform flashback database in RAC Database.

bash-4.3$ srvctl stop database -d dcdbrac

bash-4.3$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Tue Nov 22 07:15:20 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> STARTUP MOUNT EXCLUSIVE;
ORACLE instance started.

Total System Global Area 6.3069E+10 bytes
Fixed Size                  2393728 bytes
Variable Size            9059699072 bytes
Database Buffers         5.3989E+10 bytes
Redo Buffers               18116608 bytes
Database mounted.
SQL> FLASHBACK DATABASE TO TIMESTAMP SYSDATE-(1/8);


Flashback complete.

SQL> SQL>
SQL>
SQL> ALTER DATABASE OPEN RESETLOGS;

Database altered.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,
OLAP, Data Mining, Oracle Database Vault and Real Application Testing options
bash-4.3$ srvctl status database -d dcdbrac
Instance dcdbrac1 is running on node dcdbrac01
Instance dcdbrac2 is not running on node dcdbrac02
bash-4.3$ srvctl stop instance -d dcdbrac -i dcdbrac1
bash-4.3$ srvctl start database -d dcdbrac
bash-4.3$

Monday, November 21, 2016

Enable Flashback in oracle RAC database


bash-4.3$ srvctl stop database -d dcdbrac
bash-4.3$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Tue Nov 22 00:52:44 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 6.3069E+10 bytes
Fixed Size                  2393728 bytes
Variable Size            9059699072 bytes
Database Buffers         5.3989E+10 bytes
Redo Buffers               18116608 bytes
Database mounted.
SQL> alter database flashback on;

Database altered.

SQL> show parameter DB_FLASHBACK_RETENTION_TARGET

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_flashback_retention_target        integer     1440
SQL> alter database open;

Database altered.

SQL> select flashback_on  from v$database;

FLASHBACK_ON
------------------
YES

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,
OLAP, Data Mining, Oracle Database Vault and Real Application Testing options
bash-4.3$
bash-4.3$ srvctl stop instance -d dcdbrac -i dcdbrac1
bash-4.3$ srvctl start database -d dcdbrac
bash-4.3$

Sunday, November 20, 2016

Enable/Disable the automatic startup of the Oracle Clusterware (CRS|HAS).

[root@dcdbrac01 ~]# crsctl config crs
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@dcdbrac01 ~]#
[root@dcdbrac01 ~]# crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@dcdbrac01 ~]#
[root@dcdbrac01 ~]# cat /etc/oracle/scls_scr/dcdbrac01/root/ohasdstr
disable
[root@dcdbrac01 ~]# crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@dcdbrac01 ~]#
[root@dcdbrac01 ~]# crsctl enable crs
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@dcdbrac01 ~]#
[root@dcdbrac01 ~]# crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.
[root@dcdbrac01 ~]# crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.
[root@dcdbrac01 ~]# crsctl config crs
CRS-4621: Oracle High Availability Services autostart is disabled.
[root@dcdbrac01 ~]# crsctl config has
CRS-4621: Oracle High Availability Services autostart is disabled.

How to Restore OCR in RAC Database.

My Environment :

Host 1 : dcdbrac01
Host 2 : dcdbrac02
Cluser : dcdbrac-cluster

1. Find out the latest backup of OCR ( check in each server location of $CRS_HOME/cdata/cluster_name/)

[root@dcdbrac01 ~]$ echo $CRS_HOME
/u01/app/11.2.4/grid
[root@dcdbrac01 ~]$ cd $CRS_HOME/cdata/dcdbrac-cluster/
[root@dcdbrac01 dcdbrac-cluster]# ls -ltr
total 51408
-rw------- 1 root root 7520256 Oct  3 04:03 week.ocr
-rw------- 1 root root 7520256 Oct 10 04:04 week_.ocr
-rw------- 1 root root 7520256 Nov 16 11:41 day.ocr
-rw------- 1 root root 7520256 Nov 17 03:41 backup02.ocr
-rw------- 1 root root 7520256 Nov 17 07:41 backup01.ocr
-rw------- 1 root root 7520256 Nov 17 11:41 backup00.ocr
-rw------- 1 root root 7520256 Nov 17 11:41 day_.ocr
[root@dcdbrac01 dcdbrac-cluster]#

[root@dcdbrac02 dcdbrac-cluster]# ls -ltr
total 51408
-rw------- 1 root root 7520256 Nov  7 08:53 week.ocr
-rw------- 1 root root 7520256 Nov 14 08:53 week_.ocr
-rw------- 1 root root 7520256 Nov 19 15:23 day.ocr
-rw------- 1 root root 7520256 Nov 20 07:23 backup02.ocr
-rw------- 1 root root 7520256 Nov 20 11:23 backup01.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 backup00.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 day_.ocr
[root@dcdbrac02 dcdbrac-cluster]#

2. Force stop Grid Infrastructure in each node.

[root@dcdbrac01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'dcdbrac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.RMAN_BKP.dg' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.sblcbs.db' on 'dcdbrac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.dcdbrac01.vip' on 'dcdbrac01'
CRS-2677: Stop of 'ora.RMAN_BKP.dg' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'dcdbrac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.sblcbs.db' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dcdbrac01'
CRS-2677: Stop of 'ora.FRA.dg' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.dcdbrac01.vip' on 'dcdbrac01' succeeded
CRS-2672: Attempting to start 'ora.dcdbrac01.vip' on 'dcdbrac02'
CRS-2676: Start of 'ora.scan1.vip' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02'
CRS-2676: Start of 'ora.dcdbrac01.vip' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac01'
CRS-2677: Stop of 'ora.asm' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dcdbrac01'
CRS-2677: Stop of 'ora.ons' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dcdbrac01'
CRS-2677: Stop of 'ora.net1.network' on 'dcdbrac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dcdbrac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.ctssd' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dcdbrac01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.cssd' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dcdbrac01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.crf' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.gipcd' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.gpnpd' on 'dcdbrac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dcdbrac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@dcdbrac01 ~]#

[root@dcdbrac02 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'dcdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.dcdbrac01.vip' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.RMAN_BKP.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.sblcbs.db' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.dcdbrac01.vip' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.dcdbrac02.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.registry.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.RMAN_BKP.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.dcdbrac02.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.sblcbs.db' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dcdbrac02'
CRS-2677: Stop of 'ora.DATA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dcdbrac02'
CRS-2677: Stop of 'ora.ons' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dcdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'dcdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dcdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.evmd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cssd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dcdbrac02'
CRS-2677: Stop of 'ora.crf' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gipcd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'dcdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@dcdbrac02 ~]#

3. Start the CRS stack in exclusive mode in node which contain latest OCR backup without CRS demon (In my case node 2 contain latest OCR backup).

[root@dcdbrac02 dcdbrac-cluster]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'dcdbrac02'
CRS-2676: Start of 'ora.mdnsd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dcdbrac02'
CRS-2676: Start of 'ora.gpnpd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dcdbrac02'
CRS-2672: Attempting to start 'ora.gipcd' on 'dcdbrac02'
CRS-2676: Start of 'ora.cssdmonitor' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.gipcd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dcdbrac02'
CRS-2672: Attempting to start 'ora.diskmon' on 'dcdbrac02'
CRS-2676: Start of 'ora.diskmon' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.cssd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'dcdbrac02'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2672: Attempting to start 'ora.ctssd' on 'dcdbrac02'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2676: Start of 'ora.drivers.acfs' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.ctssd' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dcdbrac02'
CRS-2676: Start of 'ora.asm' on 'dcdbrac02' succeeded
[root@dcdbrac02 dcdbrac-cluster]#

4. Restore latest OCR 

[root@dcdbrac02 dcdbrac-cluster]# cd $CRS_HOME/cdata/dcdbrac-cluster/
[root@dcdbrac02 dcdbrac-cluster]# ls -ltr
total 51408
-rw------- 1 root root 7520256 Nov  7 08:53 week.ocr
-rw------- 1 root root 7520256 Nov 14 08:53 week_.ocr
-rw------- 1 root root 7520256 Nov 19 15:23 day.ocr
-rw------- 1 root root 7520256 Nov 20 07:23 backup02.ocr
-rw------- 1 root root 7520256 Nov 20 11:23 backup01.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 backup00.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 day_.ocr
[root@dcdbrac02 dcdbrac-cluster]#
[root@dcdbrac02 dcdbrac-cluster]# ocrconfig -restore backup00.ocr
[root@dcdbrac02 dcdbrac-cluster]#

5. Stop and Start the CRS

[root@dcdbrac02 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'dcdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.dcdbrac01.vip' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cvu' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.dcdbrac01.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.RMAN_BKP.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.dcdbrac02.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.FRA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.RMAN_BKP.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.dcdbrac02.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dcdbrac02'
CRS-2677: Stop of 'ora.ons' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dcdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'dcdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dcdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2677: Stop of 'ora.mdnsd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.cssd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dcdbrac02'
CRS-2677: Stop of 'ora.crf' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gipcd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'dcdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@dcdbrac02 dcdbrac-cluster]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@dcdbrac02 dcdbrac-cluster]#

6. Now chech the status of Cluser 

[root@dcdbrac02 dcdbrac-cluster]# crsctl check cluster -all
**************************************************************
dcdbrac02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@dcdbrac02 dcdbrac-cluster]#

Wednesday, October 19, 2016

Enable/Disable Automatic Startup Oracle HAS

[grid@OEL564ASMN1 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@OEL564ASMN1 ~]$
[grid@OEL564ASMN1 ~]$
[grid@OEL564ASMN1 ~]$ crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.
[grid@OEL564ASMN1 ~]$
[grid@OEL564ASMN1 ~]$ crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.
[grid@OEL564ASMN1 ~]$ crsctl config has
CRS-4621: Oracle High Availability Services autostart is disabled.
[grid@OEL564ASMN1 ~]$ crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.
[grid@OEL564ASMN1 ~]$ crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.
[grid@OEL564ASMN1 ~]$

Tuesday, August 23, 2016

ORA-39070: Unable to open the log file. (in RAC/ASM)

Cause : In ASM directory impdp/expdp are not able to create log file.

Solution: Create directory in file system and use this for log file directories.


[oracle@sbldcdb01 ~]$ impdp system/pass directory=DUMPDIR dumpfile=CBSDB_BODAY080816-%U.dmp logfile=impCBSDB_BODAY080816.log parallel=4 remap_schema=CBSDB_BODAY:CBSDB_RAC transform=oid:n

Import: Release 11.2.0.4.0 - Production on Wed Aug 24 12:43:34 2016

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORA-39002: invalid operation
ORA-39070: Unable to open the log file.
ORA-29283: invalid file operation
ORA-06512: at "SYS.UTL_FILE", line 536
ORA-29283: invalid file operation


[oracle@sbldcdb01 ~]$ PWD
-bash: PWD: command not found
[oracle@sbldcdb01 ~]$
[oracle@sbldcdb01 ~]$ pwd
/home/oracle
[oracle@sbldcdb01 ~]$
[oracle@sbldcdb01 ~]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Wed Aug 24 12:46:19 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> CREATE OR REPLACE DIRECTORY
LOGPDIR AS
'/home/oracle/';
  2    3
Directory created.

SQL> GRANT READ, WRITE ON DIRECTORY LOGPDIR TO PUBLIC;

Grant succeeded.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@sbldcdb01 ~]$ impdp system/pass directory=DUMPDIR dumpfile=CBSDB_BODAY080816-%U.dmp logfile=LOGPDIR:impCBSDB_BODAY080816.log parallel=4 remap_schema=CBSDB_BODAY:CBSDB_RAC transform=oid:n

Import: Release 11.2.0.4.0 - Production on Wed Aug 24 12:47:31 2016

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
Master table "SYSTEM"."SYS_IMPORT_FULL_03" successfully loaded/unloaded
Starting "SYSTEM"."SYS_IMPORT_FULL_03":  system/******** directory=DUMPDIR dumpfile=CBSDB_BODAY080816-%U.dmp logfile=LOGPDIR:impCBSDB_BODAY080816.log parallel=4 remap_schema=CBSDB_BODAY:CBSDB_RAC transform=oid:n
Processing object type SCHEMA_EXPORT/USER
Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/TABLESPACE_QUOTA
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/SYNONYM/SYNONYM
Processing object type SCHEMA_EXPORT/TYPE/TYPE_SPEC
Processing object type SCHEMA_EXPORT/DB_LINK
Processing object type SCHEMA_EXPORT/SEQUENCE/SEQUENCE
Processing object type SCHEMA_EXPORT/TABLE/TABLE

How to change VIP in 11gR2 RAC

1. Find the existing configuration 

srvctl config nodeapps -a

[grid@sbldcdb01 ~]$ srvctl config nodeapps -a
Network exists: 1/172.31.36.0/255.255.255.128/eth0, type static
VIP exists: /sbldcdb01-vip/172.31.36.26/172.31.36.0/255.255.255.128/eth0, hosting node sbldcdb01
VIP exists: /sbldcdb02-vip/172.31.36.27/172.31.36.0/255.255.255.128/eth0, hosting node sbldcdb02
[grid@sbldcdb01 ~]$

2. Stop VIP Service 

srvctl stop vip -n sbldcdb01 -f

srvctl stop instance -d sblcbs -n sbldcdb01 (Optional)

[grid@sbldcdb01 ~]$ srvctl stop vip -n sbldcdb01
PRCR-1014 : Failed to stop resource ora.sbldcdb01.vip
PRCR-1065 : Failed to stop resource ora.sbldcdb01.vip
CRS-2529: Unable to act on 'ora.sbldcdb01.vip' because that would require stopping or relocating 'ora.LISTENER.lsnr', but the force option was not specified
[grid@sbldcdb01 ~]$

[grid@sbldcdb01 ~]$ srvctl stop vip -n sbldcdb01 -f
[grid@sbldcdb01 ~]$ srvctl stop instance -d sblcbs -n sbldcdb01

[IF problem found in above statement srvctl stop vip -n sbldcdb01 -f]

3. Modify the VIP

srvctl modify nodeapps -n sbldcdb01 -A 172.31.36.8/255.255.255.0/eth0

srvctl modify network -k 1 -S 110.11.70.0/255.255.255.0/eth2 [If want to change the network]

4. Changes into /etc/hosts


5. Verify the new VIP

/sbin/ifconfig -a | egrep '(eth0|Mask)'

[grid@sbldcdb01 ~]$ /sbin/ifconfig -a | egrep '(eth0|Mask)'
eth0      Link encap:Ethernet  HWaddr 00:50:56:9E:B5:3E
          inet addr:172.31.36.18  Bcast:172.31.36.127  Mask:255.255.255.128
eth0:1    Link encap:Ethernet  HWaddr 00:50:56:9E:B5:3E
          inet addr:172.31.36.24  Bcast:172.31.36.127  Mask:255.255.255.128
eth0:3    Link encap:Ethernet  HWaddr 00:50:56:9E:B5:3E
          inet addr:172.31.36.8  Bcast:172.31.36.127  Mask:255.255.255.128
          inet addr:172.31.36.162  Bcast:172.31.36.191  Mask:255.255.255.192
          inet addr:169.254.233.149  Bcast:169.254.255.255  Mask:255.255.0.0
          inet addr:127.0.0.1  Mask:255.0.0.0
[grid@sbldcdb01 ~]$

6. Start VIP

srvctl stop vip -n node1

Tuesday, August 9, 2016

Oracle 11.2.0.4 Grid Installation Failed in Red Hat Linux 7.1 (ohasd failed to start)

I got this problem at the time of installation Oracle 11.2.0.4 grid in Red Hat Linux 7.1. I tryed several way to solve this problem but failed finally this problem has beed solved by the applying patch 18370031. This is bug for Oracle Clusterware 11g Release 2 (11.2.0.4.0). The following steps you have to perform to resolve this issue.

1. Deinstall previous GRID configuration.
2. Download and Apply the patch 18370031 using OPatch after orainstRoot.sh and  before executing root.sh scripts.
3. Execute root.sh scripts.

[root@spftldb02 app]# /home/grid/app/grid/product/11.2.4/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /home/grid/app/grid/product/11.2.4/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /home/grid/app/grid/product/11.2.4/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node spftldb02 successfully pinned.
Adding Clusterware entries to inittab
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2016-08-04 13:04:22.340:
[client(10083)]CRS-2101:The OLR was formatted using version 3.
2016-08-04 13:04:22.768:
[client(10110)]CRS-1001:The OCR was formatted using version 3.

ohasd failed to start at /home/grid/app/grid/product/11.2.4/grid/crs/install/roothas.pl line 377, <ALERTLOG> line 4.
/home/grid/app/grid/product/11.2.4/grid/perl/bin/perl -I/home/grid/app/grid/product/11.2.4/grid/perl/lib -I/home/grid/app/grid/product/11.2.4/grid/crs/install /home/grid/app/grid/product/11.2.4/grid/crs/install/roothas.pl execution failed

[grid@spftldb02 ~]$ cd /home/grid/app/grid/product/11.2.4/grid/deinstall
[grid@spftldb02 deinstall]$ ./deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2016-08-04_02-21-58PM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################### CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /home/grid/app/grid/product/11.2.4/grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Standalone Server
Oracle Base selected for deinstall is: /home/grid/app/grid
Checking for existence of central inventory location /home/grid/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /home/grid/app/grid/product/11.2.4/grid
Checking for sufficient temp space availability on node(s) : 'spftldb02.spftl.com'

## [END] Install check configuration ##

Traces log file: /tmp/deinstall2016-08-04_02-21-58PM/logs//crsdc.log

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2016-08-04_02-21-58PM/logs/netdc_check2016-08-04_02-22-10-PM.log

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2016-08-04_02-21-58PM/logs/asmcadc_check2016-08-04_02-22-10-PM.log

ASM configuration was not detected in this Oracle home. Was ASM configured in this Oracle home (y|n) [n]: n
ASM was not detected in the Oracle Home

######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /home/grid/app/grid/product/11.2.4/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:null
Oracle Home selected for deinstall is: /home/grid/app/grid/product/11.2.4/grid
Inventory Location where the Oracle home registered is: /home/grid/app/oraInventory
ASM was not detected in the Oracle Home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2016-08-04_02-21-58PM/logs/deinstall_deconfig2016-08-04_02-22-08-PM.out'
Any error messages from this session will be written to: '/tmp/deinstall2016-08-04_02-21-58PM/logs/deinstall_deconfig2016-08-04_02-22-08-PM.err'

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /tmp/deinstall2016-08-04_02-21-58PM/logs/asmcadc_clean2016-08-04_02-23-25-PM.log
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2016-08-04_02-21-58PM/logs/netdc_clean2016-08-04_02-23-25-PM.log

De-configuring backup files...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END


---------------------------------------->

Run the following command as the root user or the administrator on node "spftldb02".

/tmp/deinstall2016-08-04_02-21-58PM/perl/bin/perl -I/tmp/deinstall2016-08-04_02-21-58PM/perl/lib -I/tmp/deinstall2016-08-04_02-21-58PM/crs/install /tmp/deinstall2016-08-04_02-21-58PM/crs/install/roothas.pl -force  -deconfig -paramfile "/tmp/deinstall2016-08-04_02-21-58PM/response/deinstall_Ora11g_gridinfrahome1.rsp"

Press Enter after you finish running the above commands

<----------------------------------------

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/home/grid/app/grid/product/11.2.4/grid' from the central inventory on the local node : Done

Delete directory '/home/grid/app/grid/product/11.2.4/grid' on the local node : Done

Delete directory '/home/grid/app/oraInventory' on the local node : Done

Delete directory '/home/grid/app/grid' on the local node : Done

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2016-08-04_02-21-58PM' on node 'spftldb02'

## [END] Oracle install clean ##


######################### CLEAN OPERATION END #########################


####################### CLEAN OPERATION SUMMARY #######################
Oracle Restart was already stopped and de-configured on node "spftldb02"
Oracle Restart is stopped and de-configured successfully.
Successfully detached Oracle home '/home/grid/app/grid/product/11.2.4/grid' from the central inventory on the local node.
Successfully deleted directory '/home/grid/app/grid/product/11.2.4/grid' on the local node.
Successfully deleted directory '/home/grid/app/oraInventory' on the local node.
Successfully deleted directory '/home/grid/app/grid' on the local node.
Oracle Universal Installer cleanup was successful.


Run 'rm -rf /etc/oraInst.loc' as root on node(s) 'spftldb02' at the end of the session.

Run 'rm -rf /opt/ORCLfmap' as root on node(s) 'spftldb02' at the end of the session.
Run 'rm -rf /etc/oratab' as root on node(s) 'spftldb02' at the end of the session.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL & DECONFIG TOOL END #############

[grid@spftldb02 deinstall]$

[root@spftldb02 ~]# rm -rf /etc/oraInst.loc
[root@spftldb02 ~]# rm -rf /opt/ORCLfmap
[root@spftldb02 ~]# rm -rf /etc/oratab


[grid@spftldb02 OPatch]$ ./opatch napply -local /home/grid/app/grid/product/11.2.4/grid/18370031
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


Oracle Home       : /home/grid/app/grid/product/11.2.4/grid
Central Inventory : /home/grid/app/oraInventory
   from           : /home/grid/app/grid/product/11.2.4/grid/oraInst.loc
OPatch version    : 11.2.0.3.12
OUI version       : 11.2.0.4.0
Log file location : /home/grid/app/grid/product/11.2.4/grid/cfgtoollogs/opatch/opatch2016-08-04_14-41-51PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   18370031

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y



Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/home/grid/app/grid/product/11.2.4/grid')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '18370031' to OH '/home/grid/app/grid/product/11.2.4/grid'

Patching component oracle.crs, 11.2.0.4.0...
Patch 18370031 successfully applied.
Log file location: /home/grid/app/grid/product/11.2.4/grid/cfgtoollogs/opatch/opatch2016-08-04_14-41-51PM_1.log

OPatch succeeded.
[grid@spftldb02 OPatch]$
[grid@spftldb02 OPatch]$


[root@spftldb02 ~]# /home/grid/app/grid/product/11.2.4/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /home/grid/app/grid/product/11.2.4/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /home/grid/app/grid/product/11.2.4/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node spftldb02 successfully pinned.
Adding Clusterware entries to oracle-ohasd.service

spftldb02     2016/08/04 14:43:37     /home/grid/app/grid/product/11.2.4/grid/cdata/spftldb02/backup_20160804_144337.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
[root@spftldb02 ~]#

Monday, May 16, 2016

PSU patching updates (APR 2016) in RAC Database.

Two Node RAC Database with following environments :

1. dcdb01 (Hostname Node1)
2. dcdb02 (Hostname Node2)
3. dcdb (RAC DB Name)
4. dcdb1 (RAC DB Instance 1)
5. dcdb2 (RAC DB Instance 2)
6. /u01/stage/patch/psu_apr_2016/ (Patch staging directory for April 2016)
7. /u01/app/11.2.4/grid (Grid Home)
8. /u01/app/oracle/product/11.2.4/db_1 (RAC DB Home)

The APR2016 PSU patching updates the Grid Infrastucture (GI), Database (RAC), and OJVM components of both the GI and RAC installations. The GI patches are applied first and then the RAC patches are applied. The OJVM patches are applied during both the GI and RAC patching sessions.

The following link for downloading PSU for April 2016.

https://updates.oracle.com/download/22738793.html 

Step 1 to 5 perform in all of rac node if Oracle RAC and GI home are not shared.

1. Download Opatch and move to Oracle RAC Database Home and GI Home.

$ unzip <OPATCH-ZIP> -d <ORACLE_HOME>
$ <ORACLE_HOME>/OPatch/opatch version

unzip /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip -d /u01/app/11.2.4/grid/

$opatch version

The version output of the previous command should be 11.2.0.3.6 or later.

2. OCM Configuration perform the following steps from GI Home.

export PATH=$PATH:/u01/app/11.2.0.2/grid/OPatch

export JAVA_HOME=/u01/app/oracle/product/11.2.4/db_1/jdk

$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u01/stage/patch/psu_apr_2016/ocm.rsp

chmod 775 /u01/stage/patch/psu_apr_2016/ocm.rsp

3. Validation of Oracle Inventory for Oracle RAC Database Home and GI Home. This should run from grid and oracle user.

$ <ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

$ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME

4. Download and Unzip the Patch. The  directory should be empty and not /tmp

Unzip the patch as grid home owner.

cd <UNZIPPED_PATCH_LOCATION>

unzip /u01/stage/patch/psu_apr_2016/p22738793_112040_Linux-x86-64.zip

5. Stop EM Agent Processes Prior to Patching and Prior to Rolling Back the Patch

As the Oracle RAC database home owner execute:

$ <ORACLE_HOME>/bin/emctl stop dbconsole

emctl stop dbconsole

6. From the Oracle database home, make sure to stop the Oracle RAC databases running on all nodes. As the database home owner execute:

$ <ORACLE_HOME>/bin/srvctl stop database –d <db-unique-name>

srvctl stop database –d dcdb

7. On the 1st node, apply the patch to the GI Home using the opatch auto command. As root user, execute the following command:

# opatch auto <UNZIPPED_PATCH_LOCATION>/22738793 -ocmrf <ocm response file>

opatch auto /u01/stage/patch/psu_apr_2016/22738793 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp

8. On the 1st node, apply the patch to the Database home using the opatch auto command. Since the Database home is shared, this operation will patch the Database home across the cluster. Note that a USM only patch cannot be applied to a database home. As root user, execute the following command:

# opatch auto <UNZIPPED_PATCH_LOCATION>/22738793 -oh <DATABASE_HOME> -ocmrf <ocm response file>

 opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp

9. On the 1st node only, restart the Oracle instance, which you have previously stopped in Step 1. As the database home owner execute:

$ <ORACLE_HOME>/bin/srvctl start instance –d <db-unique-name> -n <nodename>

srvctl start instance -d dcdb -n dcdb01

10. On the 2nd node, apply the patch to GI Home using the opatch auto command. As root user, execute the following command:

# opatch auto <UNZIPPED_PATCH_LOCATION>/22738793 -oh <GI_HOME> -ocmrf <ocm response file>

opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/11.2.4/grid -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp

11. On the 2st node, apply the patch to the Database home using the opatch auto command.

# opatch auto <UNZIPPED_PATCH_LOCATION>/22738793 -oh <DATABASE_HOME> -ocmrf <ocm response file>

opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp

12. On the 2nd node only, restart the Oracle instance, which you have previously stopped in Step 1. As the database home owner execute:

$ <ORACLE_HOME>/bin/srvctl start instance –d <db-unique-name> -n <nodename>

srvctl start instance -d dcdb -n dcdb02

#####################   Loading Modified SQL Files into the Database   #####################

1. load modified SQL files into the database in Oracle RAC environment, perform these steps on only one node.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> SHUTDOWN
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

The catbundle.sql execution is reflected in the dba_registry_history view by a row associated with bundle series PSU.

2. If the OJVM PSU was applied for a previous GI PSU patch, you may see invalid Java classes after execution of the catbundle.sql script in the previous step. If this is the case, run utlrp.sql to re-validate these Java classes.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql

3. Check the following log files in $ORACLE_BASE/cfgtoollogs/catbundle for any errors:

catbundle_PSU_<database SID>_APPLY_<TIMESTAMP>.log
catbundle_PSU_<database SID>_GENERATE_<TIMESTAMP>.log
where TIMESTAMP is of the form YYYYMMMDD_HH_MM_SS.

#####################   Upgrade Oracle Recovery Manager Catalog   #####################

If you are using the Oracle Recovery Manager, the catalog needs to be upgraded. Enter the following command to upgrade it:

$ rman catalog username/password@alias
RMAN> UPGRADE CATALOG;

#####################   Post install steps for OJVM   #####################

The following steps load modified SQL files into the database. For an Oracle RAC environment, perform these steps on only one node.

Install the SQL portion of the patch by running the following command for a single instance environment.

cd $ORACLE_HOME/sqlpatch/22674697
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> SHUTDOWN
SQL> STARTUP
SQL> alter system set cluster_database=false scope=spfile;
SQL> SHUTDOWN
SQL> STARTUP UPGRADE
SQL> @postinstall.sql
SQL> alter system set cluster_database=true scope=spfile;
SQL> SHUTDOWN
SQL> STARTUP

After installing the SQL portion of the patch, some packages could become INVALID. This will get recompiled upon access or you can run utlrp.sql to get them back into a VALID state.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql



##################### Log ####################
[grid@dcdb01 ~]$ cd /u01/stage/patch/psu_apr_2016/
[grid@dcdb01 psu_apr_2016]$ ls -ltr
total 774564
-rw-r--r-- 1 oracle oinstall  52733900 May 12 17:05 p6880880_112000_Linux-x86-64.zip
-rw-r--r-- 1 oracle oinstall 740284621 May 13 01:06 p22738793_112040_Linux-x86-64.zip
[grid@dcdb01 ~]$ unzip /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip -d /u01/app/11.2.4/grid/
Archive:  /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip
  inflating: /u01/app/11.2.4/grid/OPatch/opatchdiag
  inflating: /u01/app/11.2.4/grid/OPatch/opatch.pl
  .....
[grid@dcdb02 ~]$ cd /u01/stage/patch/psu_apr_2016/
[grid@dcdb02 psu_apr_2016]$ ls -ltr
total 774564
-rw-r--r-- 1 oracle oinstall  52733900 May 12 17:05 p6880880_112000_Linux-x86-64.zip
-rw-r--r-- 1 oracle oinstall 740284621 May 13 01:06 p22738793_112040_Linux-x86-64.zip
[grid@dcdb02 ~]$ unzip /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip -d /u01/app/11.2.4/grid/
Archive:  /u01/stage/patch/psu_apr_2016/p6880880_112000_Linux-x86-64.zip
  inflating: /u01/app/11.2.4/grid/OPatch/opatchdiag
  inflating: /u01/app/11.2.4/grid/OPatch/opatch.pl
  .....
[grid@dcdb01 ~]$ export JAVA_HOME=/u01/app/oracle/product/11.2.4/db_1/jdk
[grid@dcdb01 ~]$ $ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u01/stage/patch/psu_apr_2016/ocm.rsp
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (/u01/stage/patch/psu_apr_2016/ocm.rsp) was successfully created.
[grid@dcdb01 ~]$ chmod 775 /u01/stage/patch/psu_apr_2016/ocm.rsp
[grid@dcdb01 ~]$

[grid@dcdb02 ~]$ export JAVA_HOME=/u01/app/oracle/product/11.2.4/db_1/jdk
[grid@dcdb02 ~]$ $ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /u01/stage/patch/psu_apr_2016/ocm.rsp
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (/u01/stage/patch/psu_apr_2016/ocm.rsp) was successfully created.
[grid@dcdb02 ~]$ chmod 775 /u01/stage/patch/psu_apr_2016/ocm.rsp

[grid@dcdb01 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/11.2.4/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.4/grid/oraInst.loc
OPatch version    : 11.2.0.3.12
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.4/grid/cfgtoollogs/opatch/opatch2016-05-15_17-34-08PM_1.log

Lsinventory Output file location : /u01/app/11.2.4/grid/cfgtoollogs/opatch/lsinv/lsinventory2016-05-15_17-34-08PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: sbldcdb02.spftl.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 products installed in this Oracle Home.


Installed Products (89):

Agent Required Support Files                                         10.2.0.4.5
Assistant Common Files                                               11.2.0.4.0
Automatic Storage Management Assistant                               11.2.0.4.0
Bali Share                                                           1.1.18.0.0
Buildtools Common Files                                              11.2.0.4.0
Character Set Migration Utility                                      11.2.0.4.0
Cluster Ready Services Files                                         11.2.0.4.0
Cluster Verification Utility Common Files                            11.2.0.4.0
Cluster Verification Utility Files                                   11.2.0.4.0
Database SQL Scripts                                                 11.2.0.4.0
Deinstallation Tool                                                  11.2.0.4.0
Enterprise Manager Common Core Files                                 10.2.0.4.5
Enterprise Manager Common Files                                      10.2.0.4.5
Enterprise Manager plugin Common Files                               11.2.0.4.0
Expat libraries                                                       2.0.1.0.1
HAS Common Files                                                     11.2.0.4.0
HAS Files for DB                                                     11.2.0.4.0
Installation Common Files                                            11.2.0.4.0
Installation Plugin Files                                            11.2.0.4.0
Installer SDK Component                                              11.2.0.4.0
Java Development Kit                                                1.5.0.51.10
LDAP Required Support Files                                          11.2.0.4.0
OLAP SQL Scripts                                                     11.2.0.4.0
Oracle Advanced Security                                             11.2.0.4.0
Oracle Clusterware RDBMS Files                                       11.2.0.4.0
Oracle Configuration Manager Deconfiguration                         10.3.1.0.0
Oracle Containers for Java                                           11.2.0.4.0
Oracle Core Required Support Files                                   11.2.0.4.0
Oracle Database 11g                                                  11.2.0.4.0
Oracle Database 11g Multimedia Files                                 11.2.0.4.0
Oracle Database Deconfiguration                                      11.2.0.4.0
Oracle Database User Interface                                       2.2.13.0.0
Oracle Database Utilities                                            11.2.0.4.0
Oracle DBCA Deconfiguration                                          11.2.0.4.0
Oracle Extended Windowing Toolkit                                    3.4.47.0.0
Oracle Globalization Support                                         11.2.0.4.0
Oracle Globalization Support                                         11.2.0.4.0
Oracle Grid Infrastructure 11g                                       11.2.0.4.0
Oracle Help For Java                                                  4.2.9.0.0
Oracle Ice Browser                                                    5.2.3.6.0
Oracle Internet Directory Client                                     11.2.0.4.0
Oracle Java Client                                                   11.2.0.4.0
Oracle JDBC/OCI Instant Client                                       11.2.0.4.0
Oracle JDBC/THIN Interfaces                                          11.2.0.4.0
Oracle JFC Extended Windowing Toolkit                                4.2.36.0.0
Oracle JVM                                                           11.2.0.4.0
Oracle LDAP administration                                           11.2.0.4.0
Oracle Locale Builder                                                11.2.0.4.0
Oracle Multimedia                                                    11.2.0.4.0
Oracle Multimedia Client Option                                      11.2.0.4.0
Oracle Multimedia Java Advanced Imaging                              11.2.0.4.0
Oracle Multimedia Locator                                            11.2.0.4.0
Oracle Multimedia Locator RDBMS Files                                11.2.0.4.0
Oracle Net                                                           11.2.0.4.0
Oracle Net Listener                                                  11.2.0.4.0
Oracle Net Required Support Files                                    11.2.0.4.0
Oracle Netca Client                                                  11.2.0.4.0
Oracle Notification Service                                          11.2.0.3.0
Oracle Notification Service (eONS)                                   11.2.0.4.0
Oracle One-Off Patch Installer                                       11.2.0.3.4
Oracle Quality of Service Management (Client)                        11.2.0.4.0
Oracle Quality of Service Management (Server)                        11.2.0.4.0
Oracle RAC Deconfiguration                                           11.2.0.4.0
Oracle RAC Required Support Files-HAS                                11.2.0.4.0
Oracle Recovery Manager                                              11.2.0.4.0
Oracle Security Developer Tools                                      11.2.0.4.0
Oracle Text Required Support Files                                   11.2.0.4.0
Oracle Universal Installer                                           11.2.0.4.0
Oracle USM Deconfiguration                                           11.2.0.4.0
Oracle Wallet Manager                                                11.2.0.4.0
Parser Generator Required Support Files                              11.2.0.4.0
Perl Interpreter                                                     5.10.0.0.2
Perl Modules                                                         5.10.0.0.1
PL/SQL                                                               11.2.0.4.0
PL/SQL Embedded Gateway                                              11.2.0.4.0
Platform Required Support Files                                      11.2.0.4.0
Precompiler Required Support Files                                   11.2.0.4.0
RDBMS Required Support Files                                         11.2.0.4.0
RDBMS Required Support Files for Instant Client                      11.2.0.4.0
RDBMS Required Support Files Runtime                                 11.2.0.4.0
Required Support Files                                               11.2.0.4.0
Secure Socket Layer                                                  11.2.0.4.0
SQL*Plus                                                             11.2.0.4.0
SQL*Plus Files for Instant Client                                    11.2.0.4.0
SQL*Plus Required Support Files                                      11.2.0.4.0
SSL Required Support Files for InstantClient                         11.2.0.4.0
Universal Storage Manager Files                                      11.2.0.4.0
XDK Required Support Files                                           11.2.0.4.0
XML Parser for Java                                                  11.2.0.4.0
There are 89 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


Rac system comprising of multiple nodes
  Local node = dcdb01
  Remote node = dcdb02

--------------------------------------------------------------------------------

OPatch succeeded.

[grid@dcdb02 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/11.2.4/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.4/grid/oraInst.loc
OPatch version    : 11.2.0.3.12
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.4/grid/cfgtoollogs/opatch/opatch2016-05-15_17-34-08PM_1.log

Lsinventory Output file location : /u01/app/11.2.4/grid/cfgtoollogs/opatch/lsinv/lsinventory2016-05-15_17-34-08PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: sbldcdb02.spftl.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 products installed in this Oracle Home.


Installed Products (89):

Agent Required Support Files                                         10.2.0.4.5
Assistant Common Files                                               11.2.0.4.0
Automatic Storage Management Assistant                               11.2.0.4.0
Bali Share                                                           1.1.18.0.0
Buildtools Common Files                                              11.2.0.4.0
Character Set Migration Utility                                      11.2.0.4.0
Cluster Ready Services Files                                         11.2.0.4.0
Cluster Verification Utility Common Files                            11.2.0.4.0
Cluster Verification Utility Files                                   11.2.0.4.0
Database SQL Scripts                                                 11.2.0.4.0
Deinstallation Tool                                                  11.2.0.4.0
Enterprise Manager Common Core Files                                 10.2.0.4.5
Enterprise Manager Common Files                                      10.2.0.4.5
Enterprise Manager plugin Common Files                               11.2.0.4.0
Expat libraries                                                       2.0.1.0.1
HAS Common Files                                                     11.2.0.4.0
HAS Files for DB                                                     11.2.0.4.0
Installation Common Files                                            11.2.0.4.0
Installation Plugin Files                                            11.2.0.4.0
Installer SDK Component                                              11.2.0.4.0
Java Development Kit                                                1.5.0.51.10
LDAP Required Support Files                                          11.2.0.4.0
OLAP SQL Scripts                                                     11.2.0.4.0
Oracle Advanced Security                                             11.2.0.4.0
Oracle Clusterware RDBMS Files                                       11.2.0.4.0
Oracle Configuration Manager Deconfiguration                         10.3.1.0.0
Oracle Containers for Java                                           11.2.0.4.0
Oracle Core Required Support Files                                   11.2.0.4.0
Oracle Database 11g                                                  11.2.0.4.0
Oracle Database 11g Multimedia Files                                 11.2.0.4.0
Oracle Database Deconfiguration                                      11.2.0.4.0
Oracle Database User Interface                                       2.2.13.0.0
Oracle Database Utilities                                            11.2.0.4.0
Oracle DBCA Deconfiguration                                          11.2.0.4.0
Oracle Extended Windowing Toolkit                                    3.4.47.0.0
Oracle Globalization Support                                         11.2.0.4.0
Oracle Globalization Support                                         11.2.0.4.0
Oracle Grid Infrastructure 11g                                       11.2.0.4.0
Oracle Help For Java                                                  4.2.9.0.0
Oracle Ice Browser                                                    5.2.3.6.0
Oracle Internet Directory Client                                     11.2.0.4.0
Oracle Java Client                                                   11.2.0.4.0
Oracle JDBC/OCI Instant Client                                       11.2.0.4.0
Oracle JDBC/THIN Interfaces                                          11.2.0.4.0
Oracle JFC Extended Windowing Toolkit                                4.2.36.0.0
Oracle JVM                                                           11.2.0.4.0
Oracle LDAP administration                                           11.2.0.4.0
Oracle Locale Builder                                                11.2.0.4.0
Oracle Multimedia                                                    11.2.0.4.0
Oracle Multimedia Client Option                                      11.2.0.4.0
Oracle Multimedia Java Advanced Imaging                              11.2.0.4.0
Oracle Multimedia Locator                                            11.2.0.4.0
Oracle Multimedia Locator RDBMS Files                                11.2.0.4.0
Oracle Net                                                           11.2.0.4.0
Oracle Net Listener                                                  11.2.0.4.0
Oracle Net Required Support Files                                    11.2.0.4.0
Oracle Netca Client                                                  11.2.0.4.0
Oracle Notification Service                                          11.2.0.3.0
Oracle Notification Service (eONS)                                   11.2.0.4.0
Oracle One-Off Patch Installer                                       11.2.0.3.4
Oracle Quality of Service Management (Client)                        11.2.0.4.0
Oracle Quality of Service Management (Server)                        11.2.0.4.0
Oracle RAC Deconfiguration                                           11.2.0.4.0
Oracle RAC Required Support Files-HAS                                11.2.0.4.0
Oracle Recovery Manager                                              11.2.0.4.0
Oracle Security Developer Tools                                      11.2.0.4.0
Oracle Text Required Support Files                                   11.2.0.4.0
Oracle Universal Installer                                           11.2.0.4.0
Oracle USM Deconfiguration                                           11.2.0.4.0
Oracle Wallet Manager                                                11.2.0.4.0
Parser Generator Required Support Files                              11.2.0.4.0
Perl Interpreter                                                     5.10.0.0.2
Perl Modules                                                         5.10.0.0.1
PL/SQL                                                               11.2.0.4.0
PL/SQL Embedded Gateway                                              11.2.0.4.0
Platform Required Support Files                                      11.2.0.4.0
Precompiler Required Support Files                                   11.2.0.4.0
RDBMS Required Support Files                                         11.2.0.4.0
RDBMS Required Support Files for Instant Client                      11.2.0.4.0
RDBMS Required Support Files Runtime                                 11.2.0.4.0
Required Support Files                                               11.2.0.4.0
Secure Socket Layer                                                  11.2.0.4.0
SQL*Plus                                                             11.2.0.4.0
SQL*Plus Files for Instant Client                                    11.2.0.4.0
SQL*Plus Required Support Files                                      11.2.0.4.0
SSL Required Support Files for InstantClient                         11.2.0.4.0
Universal Storage Manager Files                                      11.2.0.4.0
XDK Required Support Files                                           11.2.0.4.0
XML Parser for Java                                                  11.2.0.4.0
There are 89 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


Rac system comprising of multiple nodes
  Local node = dcdb02
  Remote node = dcdb01

--------------------------------------------------------------------------------

OPatch succeeded.

[grid@dcdb01 ~]$
[grid@dcdb01 ~]$ cd /u01/stage/patch/psu_apr_2016/
[grid@dcdb01 psu_apr_2016]$ unzip p22738793_112040_Linux-x86-64.zip
Archive:  p22738793_112040_Linux-x86-64.zip
   creating: 22738793/
   creating: 22738793/22674697/
   creating: 22738793/22674697/etc/
   creating: 22738793/22674697/etc/config/
  inflating: 22738793/22674697/etc/config/deploy.xml
  inflating: 22738793/22674697/etc/config/inventory.xml
  inflating: 22738793/22674697/etc/config/actions.xml
   creating: 22738793/22674697/etc/xml/
.....
[grid@dcdb02 ~]$ cd /u01/stage/patch/psu_apr_2016/
[grid@dcdb02 psu_apr_2016]$ unzip p22738793_112040_Linux-x86-64.zip
Archive:  p22738793_112040_Linux-x86-64.zip
   creating: 22738793/
   creating: 22738793/22674697/
   creating: 22738793/22674697/etc/
   creating: 22738793/22674697/etc/config/
  inflating: 22738793/22674697/etc/config/deploy.xml
  inflating: 22738793/22674697/etc/config/inventory.xml
  inflating: 22738793/22674697/etc/config/actions.xml
   creating: 22738793/22674697/etc/xml/
.....

[oracle@dcdb01 ~]$ emctl stop dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.4.0
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
https://dcdb01.dc.com:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 11g Database Control ...
 ...  Stopped.
[oracle@dcdb01 ~]$

[oracle@dcdb02 ~]$ emctl stop dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.4.0
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
https://dcdb01.dc.com:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 11g Database Control ...
 ...  Stopped.
[oracle@dcdb02 ~]$
[oracle@dcdb01 oracle]# srvctl stop database -d dcdb
[oracle@dcdb01 oracle]# srvctl status database -d dcdb
Instance dcdb1 is not running on node dcdb01
Instance dcdb1 is not running on node dcdb02
[oracle@dcdb01 ~]$ su grid
Password:
[grid@dcdb01 oracle]$ cd
[grid@dcdb01 ~]$
[grid@dcdb01 ~]$ . ./.bash_profile
[grid@dcdb01 ~]$ su root
Password:
[root@dcdb01 grid]#
[root@dcdb01 grid]# echo $ORACLE_HOME
/u01/app/11.2.4/grid
[root@dcdb01 grid]#
[root@dcdb01 ~]# opatch auto /u01/stage/patch/psu_apr_2016/22738793 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp
Executing /u01/app/11.2.4/grid/perl/bin/perl /u01/app/11.2.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/stage/patch/psu_apr_2016 -patchn 22738793 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp -paramfile /u01/app/11.2.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_13-33-15.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_13-33-15.report.log

2016-05-15 13:33:15: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.4/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.4/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.4/db_1 successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456  apply successful for home  /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22502549/custom/server/22502549  apply successful for home  /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22674697  apply successful for home  /u01/app/oracle/product/11.2.4/db_1

Stopping CRS...
Stopped CRS successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456  apply successful for home  /u01/app/11.2.4/grid
patch /u01/stage/patch/psu_apr_2016/22738793/22502549  apply successful for home  /u01/app/11.2.4/grid
patch /u01/stage/patch/psu_apr_2016/22738793/22502505  apply successful for home  /u01/app/11.2.4/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.4/db_1 ...
Started RAC /u01/app/oracle/product/11.2.4/db_1 successfully

opatch auto succeeded.
[root@dcdb01 ~]#
[root@dcdb01 ~]# cd /home/oracle/
[root@dcdb01 oracle]# . ./.bash_profile
[root@dcdb01 oracle]#  echo $ORACLE_HOME
/u01/app/oracle/product/11.2.4/db_1
[root@dcdb01 oracle]#
[root@dcdb01 oracle]# opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp
Executing /u01/app/11.2.4/grid/perl/bin/perl /u01/app/11.2.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/stage/patch/psu_apr_2016 -patchn 22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp -paramfile /u01/app/11.2.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_13-51-45.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_13-51-45.report.log

2016-05-15 13:51:45: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.4/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.4/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.4/db_1 successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456  apply successful for home  /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22502549/custom/server/22502549  apply successful for home  /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22674697  apply successful for home  /u01/app/oracle/product/11.2.4/db_1

Starting RAC /u01/app/oracle/product/11.2.4/db_1 ...
Started RAC /u01/app/oracle/product/11.2.4/db_1 successfully

opatch auto succeeded.
[root@dcdb01 oracle]#
[root@dcdb01 oracle]# su oracle
[oracle@dcdb01 ~]$ srvctl start instance -d dcdb -n dcdb01
[oracle@dcdb01 ~]$
[oracle@dcdb01 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.4/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.4/db_1/oraInst.loc
OPatch version    : 11.2.0.3.12
OUI version       : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/opatch2016-05-15_13-57-42PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/lsinv/lsinventory2016-05-15_13-57-42PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: dcdb01.dc.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.4.0
There are 1 products installed in this Oracle Home.


Interim patches (3) :

Patch  22674697     : applied on Sun May 15 13:37:40 BDT 2016
Unique Patch ID:  20038581
Patch description:  "OJVM PATCH SET UPDATE 11.2.0.4.160419"
   Created on 17 Mar 2016, 07:51:04 hrs PST8PDT
   Bugs fixed:
     19058059, 18933818, 19176885, 17201047, 19007266, 19554117, 14774730
     17285560, 19153980, 21911849, 18166577, 22670385, 19187988, 18458318
     19374518, 19006757, 17056813, 21811517, 19909862, 19223010, 22118835
     19895326, 22253904, 20408829, 19852360, 17804361, 21047766, 19231857
     17528315, 21566944

Patch  22502549     : applied on Sun May 15 13:37:12 BDT 2016
Unique Patch ID:  19880366
Patch description:  "OCW Patch Set Update : 11.2.0.4.160419 (22502549)"
   Created on 11 Mar 2016, 01:34:41 hrs PST8PDT
   Bugs fixed:
     19270660, 18328800, 18508710, 18691572, 20038451, 21251192, 21232394
     20365005, 17750548, 17387214, 17617807, 14497275, 20219458, 17733927
     18180541, 18962892, 17292250, 17378618, 16759171, 20110156, 17843489
     17065496, 13991403, 17273020, 17155238, 20012766, 21245437, 18261183
     18053580, 20218012, 17013634, 17886392, 20995001, 17039197, 17947785
     16317771, 10052729, 22353346, 20340620, 16237657, 20317221, 15917869
     18199185, 18399991, 20186278, 17374271, 18024089, 16849642, 20746251
     20246071, 14270845, 20552947, 18882642, 18414137, 17001914, 17927970
     14378120, 16346413, 15986647, 18068871, 21222147, 18143836, 16206997
     21982225, 19168690, 20235511, 18343490, 21875360, 16613232, 19276791
     17722664, 20440643, 12928658, 18952577, 18520351, 16249829, 18226143
     16076412, 18265482, 18229842, 17172091, 20676340, 17818075, 20091753
     18231837, 14373486, 20136892, 17483479, 20551654, 18120545, 18729166
     13843841, 21225209, 17405302, 18709496, 18330979, 18744838, 17087371
     20531190, 14525998, 18187697, 20598625, 14385860, 18348155, 19479503
     12928592, 17516024, 18370031, 17764053, 19272663, 17551223, 14671408
     18272135, 14207615, 21255373, 17500165, 18875012, 14769643, 18464784
     19558324, 18848125, 19241857, 14851828, 17955615, 20315294, 14693336
     16284825, 17352230, 20014326, 17238586, 17089344, 17405605, 21327402
     17531342, 19398098, 17159489, 17640316, 13823394, 16543190, 22024217
     17983675, 20795241, 17598201, 17481314, 16281493, 18346135, 15986311
     19601468, 17208793, 18700935, 18999857, 14076173, 18428146, 18352845
     17435488, 20408163, 17592037, 18352846, 19616601, 17391726, 17387779
     14777968, 15851860, 16206882, 20141091, 21113068, 20175174, 17305100
     15832129, 19885321, 16901346, 17985714, 18536826, 17780903, 18752378
     18946768, 16876500, 16875342, 17769597, 19955755, 16429265, 18336452
     17273003, 17209968, 16988311, 20094984, 19319357, 17059927, 17046460
     18053631, 16867761, 18774591, 21442094, 20235486, 19359787, 15869775
     19642566, 17447588, 16798862, 15920201

Patch  22502456     : applied on Sun May 15 13:36:11 BDT 2016
Unique Patch ID:  19856194
Patch description:  "Database Patch Set Update : 11.2.0.4.160419 (22502456)"
   Created on 21 Mar 2016, 11:49:22 hrs
Sub-patch  21948347; "Database Patch Set Update : 11.2.0.4.160119 (21948347)"
Sub-patch  21352635; "Database Patch Set Update : 11.2.0.4.8 (21352635)"
Sub-patch  20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
Sub-patch  20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
Sub-patch  19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch  19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch  18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch  18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
   Bugs fixed:
     17288409, 21051852, 17811429, 18607546, 17205719, 20506699, 17816865
     17922254, 17754782, 16934803, 13364795, 17311728, 17441661, 17284817
     16992075, 17446237, 14015842, 19972569, 21756677, 21538558, 20925795
     17449815, 17375354, 19463897, 13866822, 17982555, 17235750, 17478514
     18317531, 14338435, 18235390, 20803583, 13944971, 20142975, 17811789
     16929165, 18704244, 20506706, 17546973, 20334344, 14054676, 17088068
     17346091, 18264060, 17343514, 21538567, 19680952, 18471685, 19211724
     13951456, 21847223, 16315398, 18744139, 16850630, 19049453, 18673304
     17883081, 19915271, 18641419, 18262334, 17006183, 16065166, 18277454
     16833527, 10136473, 18051556, 17865671, 17852463, 18554871, 17853498
     18334586, 17551709, 17588480, 19827973, 17344412, 17842825, 18828868
     17025461, 11883252, 13609098, 17239687, 17602269, 19197175, 22195457
     18316692, 17313525, 12611721, 19544839, 18964939, 17600719, 18191164
     19393542, 17571306, 18482502, 20777150, 19466309, 17040527, 17165204
     18098207, 16785708, 17465741, 17174582, 16180763, 16777840, 12982566
     19463893, 22195465, 16875449, 12816846, 17237521, 19358317, 17811438
     17811447, 21983325, 17945983, 18762750, 16912439, 17184721, 18061914
     17282229, 18331850, 18202441, 17082359, 18723434, 21972320, 19554106
     14034426, 18339044, 19458377, 17752995, 20448824, 17891943, 17258090
     17767676, 16668584, 18384391, 17040764, 17381384, 15913355, 18356166
     14084247, 20596234, 20506715, 21756661, 13853126, 18203837, 14245531
     21756699, 16043574, 22195441, 17848897, 17877323, 21453153, 17468141
     20861693, 17786518, 17912217, 17037130, 18155762, 16956380, 17478145
     17394950, 18641461, 18189036, 18619917, 17027426, 21352646, 16268425
     22195492, 19584068, 18436307, 17265217, 17634921, 13498382, 21526048
     19258504, 20004087, 17443671, 22195485, 18000422, 20004021, 22321756
     17571039, 21067387, 16344544, 18009564, 14354737, 21286665, 18135678
     18614015, 20441797, 18362222, 17835048, 16472716, 17936109, 17050888
     17325413, 14010183, 18747196, 17761775, 16721594, 17082983, 20067212
     21179898, 17302277, 18084625, 15990359, 18203835, 17297939, 17811456
     16731148, 21168487, 13829543, 17215560, 14133975, 17694209, 17385178
     18091059, 8322815, 17586955, 17201159, 17655634, 18331812, 19730508
     18868646, 17648596, 16220077, 16069901, 17348614, 17393915, 17274537
     17957017, 18096714, 17308789, 18436647, 14285317, 19289642, 14764829
     18328509, 17622427, 22195477, 16943711, 22502493, 14368995, 17346671
     18996843, 17783588, 21343838, 16618694, 17672719, 18856999, 18783224
     17851160, 17546761, 17798953, 18273830, 22092979, 16596890, 19972566
     16384983, 17726838, 17360606, 22321741, 13645875, 18199537, 16542886
     21787056, 17889549, 14565184, 17071721, 17610798, 20299015, 21343897
     22893153, 20657441, 17397545, 18230522, 16360112, 19769489, 12905058
     18641451, 12747740, 18430495, 17016369, 17042658, 14602788, 17551063
     19972568, 21517440, 18508861, 19788842, 14657740, 17332800, 13837378
     19972564, 17186905, 18315328, 19699191, 17437634, 22353199, 18093615
     19006849, 19013183, 17296856, 18674024, 17232014, 16855292, 17762296
     14692762, 21051840, 17705023, 19121551, 21330264, 19854503, 21868720
     19309466, 18681862, 18554763, 20558005, 17390160, 18456514, 16306373
     13955826, 18139690, 17501491, 17752121, 21668627, 17299889, 17889583
     18673325, 19721304, 18293054, 17242746, 17951233, 17649265, 18094246
     19615136, 17011832, 16870214, 17477958, 18522509, 20631274, 16091637
     17323222, 16595641, 16524926, 18228645, 18282562, 17596908, 17156148
     18031668, 16494615, 22683225, 17545847, 17655240, 17614134, 13558557
     17341326, 17891946, 17716305, 16392068, 19271443, 21351877, 18092127
     18440047, 17614227, 14106803, 16903536, 18973907, 18673342, 19032867
     17389192, 17612828, 16194160, 17006570, 17721717, 17390431, 17570240
     16863422, 18325460, 19727057, 16422541, 19972570, 17267114, 18244962
     21538485, 18765602, 18203838, 16198143, 17246576, 14829250, 17835627
     18247991, 14458214, 21051862, 16692232, 17786278, 17227277, 16042673
     16314254, 16228604, 16837842, 17393683, 17787259, 20331945, 20074391
     15861775, 16399083, 18018515, 22683212, 18260550, 21051858, 17036973
     16613964, 17080436, 16579084, 18384537, 18280813, 20296213, 16901385
     15979965, 18441944, 16450169, 9756271, 17892268, 11733603, 16285691
     17587063, 21343775, 16538760, 18180390, 18193833, 21387964, 21051833
     17238511, 17824637, 16571443, 18306996, 14852021, 18674047, 17853456
     12364061, 22195448



Rac system comprising of multiple nodes
  Local node = dcdb01
  Remote node = dcdb02

--------------------------------------------------------------------------------

OPatch succeeded.
[oracle@dcdb01 ~]$

[root@dcdb01 oracle]# ssh grid@dcdb02
grid@dcdb02's password:
Last login: Mon May  1 15:40:54 2016 from X.X.X.X
[grid@dcdb02 ~]$
[root@dcdb02 grid]# opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/11.2.4/grid -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp
Executing /u01/app/11.2.4/grid/perl/bin/perl /u01/app/11.2.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/stage/patch/psu_apr_2016 -patchn 22738793 -oh /u01/app/11.2.4/grid -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp -paramfile /u01/app/11.2.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_17-39-47.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_17-39-47.report.log

2016-05-15 17:39:47: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.4/grid/crs/install/crsconfig_params

Stopping CRS...
Stopped CRS successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456  apply successful for home  /u01/app/11.2.4/grid
patch /u01/stage/patch/psu_apr_2016/22738793/22502549  apply successful for home  /u01/app/11.2.4/grid
patch /u01/stage/patch/psu_apr_2016/22738793/22502505  apply successful for home  /u01/app/11.2.4/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

opatch auto succeeded.
[root@dcdb02 grid]#
[root@dcdb02 ~]# cd /home/oracle/
[root@dcdb02 oracle]# . ./.bash_profile
[root@dcdb02 oracle]#  echo $ORACLE_HOME
/u01/app/oracle/product/11.2.4/db_1
[root@dcdb02 oracle]#
[root@dcdb02 oracle]# opatch auto /u01/stage/patch/psu_apr_2016/22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp
Executing /u01/app/11.2.4/grid/perl/bin/perl /u01/app/11.2.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/stage/patch/psu_apr_2016 -patchn 22738793 -oh /u01/app/oracle/product/11.2.4/db_1 -ocmrf /u01/stage/patch/psu_apr_2016/ocm.rsp -paramfile /u01/app/11.2.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_17-52-14.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.4/grid/cfgtoollogs/opatchauto2016-05-15_17-52-14.report.log

2016-05-15 17:52:14: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.4/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.4/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.4/db_1 successfully

patch /u01/stage/patch/psu_apr_2016/22738793/22502456  apply successful for home  /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22502549/custom/server/22502549  apply successful for home  /u01/app/oracle/product/11.2.4/db_1
patch /u01/stage/patch/psu_apr_2016/22738793/22674697  apply successful for home  /u01/app/oracle/product/11.2.4/db_1

Starting RAC /u01/app/oracle/product/11.2.4/db_1 ...
Started RAC /u01/app/oracle/product/11.2.4/db_1 successfully

opatch auto succeeded.
[root@dcdb02 oracle]# exit
[oracle@dcdb02 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.4/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.4/db_1/oraInst.loc
OPatch version    : 11.2.0.3.12
OUI version       : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/opatch2016-05-16_12-25-55PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/lsinv/lsinventory2016-05-16_12-25-55PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: dcdb02.dc.com
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.4.0
There are 1 products installed in this Oracle Home.


Interim patches (3) :

Patch  22674697     : applied on Sun May 15 17:56:06 BDT 2016
Unique Patch ID:  20038581
Patch description:  "OJVM PATCH SET UPDATE 11.2.0.4.160419"
   Created on 17 Mar 2016, 07:51:04 hrs PST8PDT
   Bugs fixed:
     19058059, 18933818, 19176885, 17201047, 19007266, 19554117, 14774730
     17285560, 19153980, 21911849, 18166577, 22670385, 19187988, 18458318
     19374518, 19006757, 17056813, 21811517, 19909862, 19223010, 22118835
     19895326, 22253904, 20408829, 19852360, 17804361, 21047766, 19231857
     17528315, 21566944

Patch  22502549     : applied on Sun May 15 17:55:38 BDT 2016
Unique Patch ID:  19880366
Patch description:  "OCW Patch Set Update : 11.2.0.4.160419 (22502549)"
   Created on 11 Mar 2016, 01:34:41 hrs PST8PDT
   Bugs fixed:
     19270660, 18328800, 18508710, 18691572, 20038451, 21251192, 21232394
     20365005, 17750548, 17387214, 17617807, 14497275, 20219458, 17733927
     18180541, 18962892, 17292250, 17378618, 16759171, 20110156, 17843489
     17065496, 13991403, 17273020, 17155238, 20012766, 21245437, 18261183
     18053580, 20218012, 17013634, 17886392, 20995001, 17039197, 17947785
     16317771, 10052729, 22353346, 20340620, 16237657, 20317221, 15917869
     18199185, 18399991, 20186278, 17374271, 18024089, 16849642, 20746251
     20246071, 14270845, 20552947, 18882642, 18414137, 17001914, 17927970
     14378120, 16346413, 15986647, 18068871, 21222147, 18143836, 16206997
     21982225, 19168690, 20235511, 18343490, 21875360, 16613232, 19276791
     17722664, 20440643, 12928658, 18952577, 18520351, 16249829, 18226143
     16076412, 18265482, 18229842, 17172091, 20676340, 17818075, 20091753
     18231837, 14373486, 20136892, 17483479, 20551654, 18120545, 18729166
     13843841, 21225209, 17405302, 18709496, 18330979, 18744838, 17087371
     20531190, 14525998, 18187697, 20598625, 14385860, 18348155, 19479503
     12928592, 17516024, 18370031, 17764053, 19272663, 17551223, 14671408
     18272135, 14207615, 21255373, 17500165, 18875012, 14769643, 18464784
     19558324, 18848125, 19241857, 14851828, 17955615, 20315294, 14693336
     16284825, 17352230, 20014326, 17238586, 17089344, 17405605, 21327402
     17531342, 19398098, 17159489, 17640316, 13823394, 16543190, 22024217
     17983675, 20795241, 17598201, 17481314, 16281493, 18346135, 15986311
     19601468, 17208793, 18700935, 18999857, 14076173, 18428146, 18352845
     17435488, 20408163, 17592037, 18352846, 19616601, 17391726, 17387779
     14777968, 15851860, 16206882, 20141091, 21113068, 20175174, 17305100
     15832129, 19885321, 16901346, 17985714, 18536826, 17780903, 18752378
     18946768, 16876500, 16875342, 17769597, 19955755, 16429265, 18336452
     17273003, 17209968, 16988311, 20094984, 19319357, 17059927, 17046460
     18053631, 16867761, 18774591, 21442094, 20235486, 19359787, 15869775
     19642566, 17447588, 16798862, 15920201

Patch  22502456     : applied on Sun May 15 17:54:42 BDT 2016
Unique Patch ID:  19856194
Patch description:  "Database Patch Set Update : 11.2.0.4.160419 (22502456)"
   Created on 21 Mar 2016, 11:49:22 hrs
Sub-patch  21948347; "Database Patch Set Update : 11.2.0.4.160119 (21948347)"
Sub-patch  21352635; "Database Patch Set Update : 11.2.0.4.8 (21352635)"
Sub-patch  20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
Sub-patch  20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
Sub-patch  19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch  19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch  18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch  18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
   Bugs fixed:
     17288409, 21051852, 17811429, 18607546, 17205719, 20506699, 17816865
     17922254, 17754782, 16934803, 13364795, 17311728, 17441661, 17284817
     16992075, 17446237, 14015842, 19972569, 21756677, 21538558, 20925795
     17449815, 17375354, 19463897, 13866822, 17982555, 17235750, 17478514
     18317531, 14338435, 18235390, 20803583, 13944971, 20142975, 17811789
     16929165, 18704244, 20506706, 17546973, 20334344, 14054676, 17088068
     17346091, 18264060, 17343514, 21538567, 19680952, 18471685, 19211724
     13951456, 21847223, 16315398, 18744139, 16850630, 19049453, 18673304
     17883081, 19915271, 18641419, 18262334, 17006183, 16065166, 18277454
     16833527, 10136473, 18051556, 17865671, 17852463, 18554871, 17853498
     18334586, 17551709, 17588480, 19827973, 17344412, 17842825, 18828868
     17025461, 11883252, 13609098, 17239687, 17602269, 19197175, 22195457
     18316692, 17313525, 12611721, 19544839, 18964939, 17600719, 18191164
     19393542, 17571306, 18482502, 20777150, 19466309, 17040527, 17165204
     18098207, 16785708, 17465741, 17174582, 16180763, 16777840, 12982566
     19463893, 22195465, 16875449, 12816846, 17237521, 19358317, 17811438
     17811447, 21983325, 17945983, 18762750, 16912439, 17184721, 18061914
     17282229, 18331850, 18202441, 17082359, 18723434, 21972320, 19554106
     14034426, 18339044, 19458377, 17752995, 20448824, 17891943, 17258090
     17767676, 16668584, 18384391, 17040764, 17381384, 15913355, 18356166
     14084247, 20596234, 20506715, 21756661, 13853126, 18203837, 14245531
     21756699, 16043574, 22195441, 17848897, 17877323, 21453153, 17468141
     20861693, 17786518, 17912217, 17037130, 18155762, 16956380, 17478145
     17394950, 18641461, 18189036, 18619917, 17027426, 21352646, 16268425
     22195492, 19584068, 18436307, 17265217, 17634921, 13498382, 21526048
     19258504, 20004087, 17443671, 22195485, 18000422, 20004021, 22321756
     17571039, 21067387, 16344544, 18009564, 14354737, 21286665, 18135678
     18614015, 20441797, 18362222, 17835048, 16472716, 17936109, 17050888
     17325413, 14010183, 18747196, 17761775, 16721594, 17082983, 20067212
     21179898, 17302277, 18084625, 15990359, 18203835, 17297939, 17811456
     16731148, 21168487, 13829543, 17215560, 14133975, 17694209, 17385178
     18091059, 8322815, 17586955, 17201159, 17655634, 18331812, 19730508
     18868646, 17648596, 16220077, 16069901, 17348614, 17393915, 17274537
     17957017, 18096714, 17308789, 18436647, 14285317, 19289642, 14764829
     18328509, 17622427, 22195477, 16943711, 22502493, 14368995, 17346671
     18996843, 17783588, 21343838, 16618694, 17672719, 18856999, 18783224
     17851160, 17546761, 17798953, 18273830, 22092979, 16596890, 19972566
     16384983, 17726838, 17360606, 22321741, 13645875, 18199537, 16542886
     21787056, 17889549, 14565184, 17071721, 17610798, 20299015, 21343897
     22893153, 20657441, 17397545, 18230522, 16360112, 19769489, 12905058
     18641451, 12747740, 18430495, 17016369, 17042658, 14602788, 17551063
     19972568, 21517440, 18508861, 19788842, 14657740, 17332800, 13837378
     19972564, 17186905, 18315328, 19699191, 17437634, 22353199, 18093615
     19006849, 19013183, 17296856, 18674024, 17232014, 16855292, 17762296
     14692762, 21051840, 17705023, 19121551, 21330264, 19854503, 21868720
     19309466, 18681862, 18554763, 20558005, 17390160, 18456514, 16306373
     13955826, 18139690, 17501491, 17752121, 21668627, 17299889, 17889583
     18673325, 19721304, 18293054, 17242746, 17951233, 17649265, 18094246
     19615136, 17011832, 16870214, 17477958, 18522509, 20631274, 16091637
     17323222, 16595641, 16524926, 18228645, 18282562, 17596908, 17156148
     18031668, 16494615, 22683225, 17545847, 17655240, 17614134, 13558557
     17341326, 17891946, 17716305, 16392068, 19271443, 21351877, 18092127
     18440047, 17614227, 14106803, 16903536, 18973907, 18673342, 19032867
     17389192, 17612828, 16194160, 17006570, 17721717, 17390431, 17570240
     16863422, 18325460, 19727057, 16422541, 19972570, 17267114, 18244962
     21538485, 18765602, 18203838, 16198143, 17246576, 14829250, 17835627
     18247991, 14458214, 21051862, 16692232, 17786278, 17227277, 16042673
     16314254, 16228604, 16837842, 17393683, 17787259, 20331945, 20074391
     15861775, 16399083, 18018515, 22683212, 18260550, 21051858, 17036973
     16613964, 17080436, 16579084, 18384537, 18280813, 20296213, 16901385
     15979965, 18441944, 16450169, 9756271, 17892268, 11733603, 16285691
     17587063, 21343775, 16538760, 18180390, 18193833, 21387964, 21051833
     17238511, 17824637, 16571443, 18306996, 14852021, 18674047, 17853456
     12364061, 22195448



Rac system comprising of multiple nodes
  Local node = dcdb02
  Remote node = dcdb01

--------------------------------------------------------------------------------

OPatch succeeded.
[oracle@dcdb02 ~]$
[oracle@dcdb02 ~]$ srvctl start instance -d dcdb -n dcdb02

[oracle@dcdb01 ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@dcdb01 admin]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Sun May 15 18:03:45 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> SHUTDOWN
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> STARTUP;
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size    2265224 bytes
Variable Size 3472887672 bytes
Database Buffers 2919235584 bytes
Redo Buffers   19292160 bytes
Database mounted.
Database opened.
SQL> @catbundle.sql psu apply

PL/SQL procedure successfully completed.


Function created.


Function created.

....

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@dcdb01 admin]$
[oracle@dcdb01 admin]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Sun May 15 18:03:45 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @utlrp.sql

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN  2016-05-15 18:18:38

DOC>   The following PL/SQL block invokes UTL_RECOMP to recompile invalid
DOC>   objects in the database. Recompilation time is proportional to the
DOC>   number of invalid objects in the database, so this command may take
DOC>   a long time to execute on a database with a large number of invalid
DOC>   objects.

......
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@dcdb01 admin]$
[grid@dcdb01 ~]$ srvctl stop database -d dcdb
[oracle@dcdb01 catbundle]$ cd $ORACLE_HOME/sqlpatch/22674697
[oracle@dcdb01 22674697]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Sun May 15 18:58:42 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size    2265224 bytes
Variable Size 3472887672 bytes
Database Buffers 2919235584 bytes
Redo Buffers   19292160 bytes
SQL> alter systemd  set cluster_database=false scope=spfile;

System altered.

SQL> shut
ORA-01507: database not mounted


ORACLE instance shut down.
SQL> startup upgrade
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size    2265224 bytes
Variable Size 3472887672 bytes
Database Buffers 2919235584 bytes
Redo Buffers   19292160 bytes
Database mounted.
Database opened.
SQL> @postinstall.sql

PL/SQL procedure successfully completed.


Function created.


PL/SQL procedure successfully completed.

...

SQL> alter system set cluster_database=true scope=spfile;

System altered.

SQL> SHUTDOWN
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
[oracle@dcdb01 22674697]$
[oracle@dcdb01 22674697]$ srvctl start database -d dcdb

Reference:

Patch Set Updates for Oracle Products (Doc ID 854428.1)

My Oracle Support Document 966023.1

How to Create an OCM Response file to Apply a Patch in Silent Mode - opatch silent (Doc ID 966023.1)