Showing posts with label Cluster. Show all posts
Showing posts with label Cluster. Show all posts

Friday, December 2, 2016

How to fix timezone issue in RAC Database.

When mismatch between OS time zone and clusterware time zone SYSDATE and  CURRENT_DATE will return different value, because of at the time of clusterware starting lintener started using clusterware time zone.

Solution : 

1. Shutdown clusterware from root user

crsctl stop crs

2. Change the clusterware time $GRID_HOME/crs/install/s_crsconfig__env.txt file using vi editor from root user. Before change take backup of this file.

bash-4.3# cat s_crsconfig_sbldcdb01_env.txt
### This file can be used to modify the NLS_LANG environment variable, which determines the charset to be used for messages.
### For example, a new charset can be configured by setting NLS_LANG=JAPANESE_JAPAN.UTF8
### Do not modify this file except to change NLS_LANG, or under the direction of Oracle Support Services

#TZ=America/New_York
TZ=Asia/Dhaka
NLS_LANG=AMERICAN_AMERICA.WE8ISO8859P1
RT_GRQ=ON
EXTSHM=OFF
TNS_ADMIN=
ORACLE_BASE=

3. Startup the clusterware from root user

crsctl start crs

Sunday, November 20, 2016

Enable/Disable the automatic startup of the Oracle Clusterware (CRS|HAS).

[root@dcdbrac01 ~]# crsctl config crs
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@dcdbrac01 ~]#
[root@dcdbrac01 ~]# crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@dcdbrac01 ~]#
[root@dcdbrac01 ~]# cat /etc/oracle/scls_scr/dcdbrac01/root/ohasdstr
disable
[root@dcdbrac01 ~]# crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@dcdbrac01 ~]#
[root@dcdbrac01 ~]# crsctl enable crs
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@dcdbrac01 ~]#
[root@dcdbrac01 ~]# crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.
[root@dcdbrac01 ~]# crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.
[root@dcdbrac01 ~]# crsctl config crs
CRS-4621: Oracle High Availability Services autostart is disabled.
[root@dcdbrac01 ~]# crsctl config has
CRS-4621: Oracle High Availability Services autostart is disabled.

How to Restore OCR in RAC Database.

My Environment :

Host 1 : dcdbrac01
Host 2 : dcdbrac02
Cluser : dcdbrac-cluster

1. Find out the latest backup of OCR ( check in each server location of $CRS_HOME/cdata/cluster_name/)

[root@dcdbrac01 ~]$ echo $CRS_HOME
/u01/app/11.2.4/grid
[root@dcdbrac01 ~]$ cd $CRS_HOME/cdata/dcdbrac-cluster/
[root@dcdbrac01 dcdbrac-cluster]# ls -ltr
total 51408
-rw------- 1 root root 7520256 Oct  3 04:03 week.ocr
-rw------- 1 root root 7520256 Oct 10 04:04 week_.ocr
-rw------- 1 root root 7520256 Nov 16 11:41 day.ocr
-rw------- 1 root root 7520256 Nov 17 03:41 backup02.ocr
-rw------- 1 root root 7520256 Nov 17 07:41 backup01.ocr
-rw------- 1 root root 7520256 Nov 17 11:41 backup00.ocr
-rw------- 1 root root 7520256 Nov 17 11:41 day_.ocr
[root@dcdbrac01 dcdbrac-cluster]#

[root@dcdbrac02 dcdbrac-cluster]# ls -ltr
total 51408
-rw------- 1 root root 7520256 Nov  7 08:53 week.ocr
-rw------- 1 root root 7520256 Nov 14 08:53 week_.ocr
-rw------- 1 root root 7520256 Nov 19 15:23 day.ocr
-rw------- 1 root root 7520256 Nov 20 07:23 backup02.ocr
-rw------- 1 root root 7520256 Nov 20 11:23 backup01.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 backup00.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 day_.ocr
[root@dcdbrac02 dcdbrac-cluster]#

2. Force stop Grid Infrastructure in each node.

[root@dcdbrac01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'dcdbrac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.RMAN_BKP.dg' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.sblcbs.db' on 'dcdbrac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.dcdbrac01.vip' on 'dcdbrac01'
CRS-2677: Stop of 'ora.RMAN_BKP.dg' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'dcdbrac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.sblcbs.db' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dcdbrac01'
CRS-2677: Stop of 'ora.FRA.dg' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.dcdbrac01.vip' on 'dcdbrac01' succeeded
CRS-2672: Attempting to start 'ora.dcdbrac01.vip' on 'dcdbrac02'
CRS-2676: Start of 'ora.scan1.vip' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02'
CRS-2676: Start of 'ora.dcdbrac01.vip' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac01'
CRS-2677: Stop of 'ora.asm' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dcdbrac01'
CRS-2677: Stop of 'ora.ons' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dcdbrac01'
CRS-2677: Stop of 'ora.net1.network' on 'dcdbrac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dcdbrac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.ctssd' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dcdbrac01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.cssd' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dcdbrac01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dcdbrac01' succeeded
CRS-2677: Stop of 'ora.crf' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.gipcd' on 'dcdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dcdbrac01'
CRS-2677: Stop of 'ora.gpnpd' on 'dcdbrac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dcdbrac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@dcdbrac01 ~]#

[root@dcdbrac02 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'dcdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.dcdbrac01.vip' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.RMAN_BKP.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.sblcbs.db' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.dcdbrac01.vip' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.dcdbrac02.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.registry.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.RMAN_BKP.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.dcdbrac02.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.sblcbs.db' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dcdbrac02'
CRS-2677: Stop of 'ora.DATA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dcdbrac02'
CRS-2677: Stop of 'ora.ons' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dcdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'dcdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dcdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.evmd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cssd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dcdbrac02'
CRS-2677: Stop of 'ora.crf' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gipcd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'dcdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@dcdbrac02 ~]#

3. Start the CRS stack in exclusive mode in node which contain latest OCR backup without CRS demon (In my case node 2 contain latest OCR backup).

[root@dcdbrac02 dcdbrac-cluster]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'dcdbrac02'
CRS-2676: Start of 'ora.mdnsd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dcdbrac02'
CRS-2676: Start of 'ora.gpnpd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dcdbrac02'
CRS-2672: Attempting to start 'ora.gipcd' on 'dcdbrac02'
CRS-2676: Start of 'ora.cssdmonitor' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.gipcd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dcdbrac02'
CRS-2672: Attempting to start 'ora.diskmon' on 'dcdbrac02'
CRS-2676: Start of 'ora.diskmon' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.cssd' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'dcdbrac02'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2672: Attempting to start 'ora.ctssd' on 'dcdbrac02'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2676: Start of 'ora.drivers.acfs' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.ctssd' on 'dcdbrac02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dcdbrac02'
CRS-2676: Start of 'ora.asm' on 'dcdbrac02' succeeded
[root@dcdbrac02 dcdbrac-cluster]#

4. Restore latest OCR 

[root@dcdbrac02 dcdbrac-cluster]# cd $CRS_HOME/cdata/dcdbrac-cluster/
[root@dcdbrac02 dcdbrac-cluster]# ls -ltr
total 51408
-rw------- 1 root root 7520256 Nov  7 08:53 week.ocr
-rw------- 1 root root 7520256 Nov 14 08:53 week_.ocr
-rw------- 1 root root 7520256 Nov 19 15:23 day.ocr
-rw------- 1 root root 7520256 Nov 20 07:23 backup02.ocr
-rw------- 1 root root 7520256 Nov 20 11:23 backup01.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 backup00.ocr
-rw------- 1 root root 7520256 Nov 20 15:23 day_.ocr
[root@dcdbrac02 dcdbrac-cluster]#
[root@dcdbrac02 dcdbrac-cluster]# ocrconfig -restore backup00.ocr
[root@dcdbrac02 dcdbrac-cluster]#

5. Stop and Start the CRS

[root@dcdbrac02 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'dcdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.dcdbrac01.vip' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cvu' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.dcdbrac01.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.RMAN_BKP.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dcdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.dcdbrac02.vip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.FRA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.RMAN_BKP.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.dcdbrac02.vip' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dcdbrac02'
CRS-2677: Stop of 'ora.ons' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dcdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'dcdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dcdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'dcdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'dcdbrac02'
CRS-2677: Stop of 'ora.mdnsd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dcdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dcdbrac02' succeeded
CRS-2677: Stop of 'ora.cssd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dcdbrac02'
CRS-2677: Stop of 'ora.crf' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gipcd' on 'dcdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dcdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'dcdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dcdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@dcdbrac02 dcdbrac-cluster]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@dcdbrac02 dcdbrac-cluster]#

6. Now chech the status of Cluser 

[root@dcdbrac02 dcdbrac-cluster]# crsctl check cluster -all
**************************************************************
dcdbrac02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@dcdbrac02 dcdbrac-cluster]#