Sunday, November 20, 2016

How to Restore OCR in RAC Database.

My Environment :

Host 1 : dcdbrac01
Host 2 : dcdbrac02
Cluser : dcdbrac-cluster

1. Find out the latest backup of OCR ( check in each server location of $CRS_HOME/cdata/cluster_name/)

[root@dcdbrac01 ~]$ echo $CRS_HOME
/u01/app/11.2.4/grid
[root@dcdbrac01 ~]$ cd $CRS_HOME/cdata/dcdbrac-cluster/
[root@dcdbrac01 dcdbrac-cluster]# ls -ltr
total 51408
-rw------- 1 root root 7520256 Oct  3 04:03 week.ocr
-rw------- 1 root root 7520256 Oct 10 04:04 week_.ocr
-rw------- 1 root root 7520256 Nov 16 11:41 day.ocr
-rw------- 1 root root 7520256 Nov 17 03:41 backup02.ocr
-rw------- 1 root root 7520256 Nov 17 07:41 backup01.ocr
-rw------- 1 root root 7520256 Nov 17 11:41 backup00.ocr
-rw------- 1 root root 7520256 Nov 17 11:41 day_.ocr
[root@dcdbrac01 dcdbrac-cluster]#

[root@dcdbrac02 dcdbrac-cluster]# ls -ltr
total 51408
-rw------- 1 root root 7520256 Nov  7 08:53 week.ocr
-rw------- 1 root root 7520256 Nov 14 08:53 week_.ocr
-rw------- 1 root root 7520256 Nov 19 15:23 day.ocr
-rw------- 1 root root 7520256 Nov 20 07:23 backup02.ocr
-rw------- 1 root root 7520256 Nov 20 11:23 backup01.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 backup00.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 day_.ocr
[root@dcdbrac02 dcdbrac-cluster]#

2. Force stop Grid Infrastructure in each node.

[root@dcdbrac01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'dcdbrac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.RMAN_BKP.dg' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.sblcbs.db' on 'dcdbrac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.dcdbrac01.vip' on 'dcdbrac01'
CRS-2677: Stop of 'ora.RMAN_BKP.dg' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'dcdbrac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.sblcbs.db' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dcdbrac01'
CRS-2677: Stop of 'ora.FRA.dg' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.dcdbrac01.vip' on 'dcdbrac01' succeeded
CRS-2672: Attempting to start 'ora.dcdbrac01.vip' on 'dcdbrac02'
CRS-2676: Start of 'ora.scan1.vip' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02'
CRS-2676: Start of 'ora.dcdbrac01.vip' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac01'
CRS-2677: Stop of 'ora.asm' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dcdbrac01'
CRS-2677: Stop of 'ora.ons' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dcdbrac01'
CRS-2677: Stop of 'ora.net1.network' on 'dcdbrac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dcdbrac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.ctssd' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dcdbrac01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.cssd' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dcdbrac01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.crf' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.gipcd' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.gpnpd' on 'dcdbrac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dcdbrac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@dcdbrac01 ~]#

[root@dcdbrac02 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'dcdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.dcdbrac01.vip' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.RMAN_BKP.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.sblcbs.db' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.dcdbrac01.vip' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.dcdbrac02.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.registry.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.RMAN_BKP.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.dcdbrac02.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.sblcbs.db' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dcdbrac02'
CRS-2677: Stop of 'ora.DATA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dcdbrac02'
CRS-2677: Stop of 'ora.ons' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dcdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'dcdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dcdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.evmd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cssd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dcdbrac02'
CRS-2677: Stop of 'ora.crf' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gipcd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'dcdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@dcdbrac02 ~]#

3. Start the CRS stack in exclusive mode in node which contain latest OCR backup without CRS demon (In my case node 2 contain latest OCR backup).

[root@dcdbrac02 dcdbrac-cluster]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'dcdbrac02'
CRS-2676: Start of 'ora.mdnsd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dcdbrac02'
CRS-2676: Start of 'ora.gpnpd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dcdbrac02'
CRS-2672: Attempting to start 'ora.gipcd' on 'dcdbrac02'
CRS-2676: Start of 'ora.cssdmonitor' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.gipcd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dcdbrac02'
CRS-2672: Attempting to start 'ora.diskmon' on 'dcdbrac02'
CRS-2676: Start of 'ora.diskmon' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.cssd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'dcdbrac02'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2672: Attempting to start 'ora.ctssd' on 'dcdbrac02'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2676: Start of 'ora.drivers.acfs' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.ctssd' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dcdbrac02'
CRS-2676: Start of 'ora.asm' on 'dcdbrac02' succeeded
[root@dcdbrac02 dcdbrac-cluster]#

4. Restore latest OCR 

[root@dcdbrac02 dcdbrac-cluster]# cd $CRS_HOME/cdata/dcdbrac-cluster/
[root@dcdbrac02 dcdbrac-cluster]# ls -ltr
total 51408
-rw------- 1 root root 7520256 Nov  7 08:53 week.ocr
-rw------- 1 root root 7520256 Nov 14 08:53 week_.ocr
-rw------- 1 root root 7520256 Nov 19 15:23 day.ocr
-rw------- 1 root root 7520256 Nov 20 07:23 backup02.ocr
-rw------- 1 root root 7520256 Nov 20 11:23 backup01.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 backup00.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 day_.ocr
[root@dcdbrac02 dcdbrac-cluster]#
[root@dcdbrac02 dcdbrac-cluster]# ocrconfig -restore backup00.ocr
[root@dcdbrac02 dcdbrac-cluster]#

5. Stop and Start the CRS

[root@dcdbrac02 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'dcdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.dcdbrac01.vip' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cvu' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.dcdbrac01.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.RMAN_BKP.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.dcdbrac02.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.FRA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.RMAN_BKP.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.dcdbrac02.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dcdbrac02'
CRS-2677: Stop of 'ora.ons' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dcdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'dcdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dcdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2677: Stop of 'ora.mdnsd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.cssd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dcdbrac02'
CRS-2677: Stop of 'ora.crf' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gipcd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'dcdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@dcdbrac02 dcdbrac-cluster]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@dcdbrac02 dcdbrac-cluster]#

6. Now chech the status of Cluser 

[root@dcdbrac02 dcdbrac-cluster]# crsctl check cluster -all
**************************************************************
dcdbrac02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@dcdbrac02 dcdbrac-cluster]#

No comments:

Post a Comment