一、先关闭GI，命令不再贴出来了，参见：
http://blog.csdn.net/msdnchina/article/details/49402299

二、模拟OLR损坏

[root@hostb cdata]# ls -lrt
total 2884
drwxr-xr-x 2 grid oinstall      4096 Oct 30 13:21 hostb
drwxr-xr-x 2 grid oinstall      4096 Oct 30 19:06 localhost drwxrwxr-x 2 grid oinstall      4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan  8 13:38 hostb.olr
[root@hostb cdata]# mv hostb.olr  back_hostb.olr ----------->  mv掉OLR
[root@hostb cdata]# ls -lrt
total 2884
drwxr-xr-x 2 grid oinstall      4096 Oct 30 13:21 hostb
drwxr-xr-x 2 grid oinstall      4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall      4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan  8 13:38 back_hostb.olr
[root@hostb cdata]# /u01/app/11.2.0/grid/bin/crsctl start crs   --------->起不来。
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: command Start failed, or completed with errors.
[root@hostb cdata]# pwd
/u01/app/11.2.0/grid/cdata
[root@hostb cdata]#

[oracle@hostb ohasd]$ pwd
/u01/app/11.2.0/grid/log/hostb/ohasd
[oracle@hostb ohasd]$ ls -lrt
total 95720
-rw-r--r-- 1 root root 10540929 Nov  4 03:51 ohasd.l09
-rw-r--r-- 1 root root 10556558 Nov  8 16:28 ohasd.l08
-rw-r--r-- 1 root root 10558423 Nov 13 12:39 ohasd.l07
-rw-r--r-- 1 root root 10558697 Nov 18 05:44 ohasd.l06
-rw-r--r-- 1 root root 10557602 Nov 22 18:15 ohasd.l05
-rw-r--r-- 1 root root 10532670 Nov 27 08:24 ohasd.l04
-rw-r--r-- 1 root root 10556869 Dec  2 01:30 ohasd.l03
-rw-r--r-- 1 root root 10557883 Dec  6 23:49 ohasd.l02
-rw-r--r-- 1 root root 10542530 Dec 21 11:49 ohasd.l01
-rw-r--r-- 1 root root     6700 Jan  8 13:42 ohasdOUT.log
-rw-r--r-- 1 root root  2986628 Jan  8 13:42 ohasd.log
[oracle@hostb ohasd]$ vi ohasd.log

2016-01-08 13:42:59.562: [ default][3149834016] Initializing OLR
2016-01-08 13:42:59.564: [  OCROSD][3149834016]utopen:6m': failed in stat OCR file/disk /u01/app/11.2.0/grid/cdata/hostb.olr, errno=2, os err string=No such file or directory
2016-01-08 13:42:59.564: [  OCROSD][3149834016]utopen:7: failed to open any OCR file/disk, errno=2, os err string=No such file or directory
2016-01-08 13:42:59.564: [  OCRRAW][3149834016]proprinit: Could not open raw device
2016-01-08 13:42:59.564: [  OCRAPI][3149834016]a_init:16!: Backend init unsuccessful : [26]
2016-01-08 13:42:59.564: [  CRSOCR][3149834016] OCR context init failure.  Error: PROCL-26: Error while accessing the physical storage Operating system error [No such file or directory] [2]
2016-01-08 13:42:59.565: [ default][3149834016] Created alert : (:OHAS00106:) :  OLR initialization failed, error: PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
2016-01-08 13:42:59.565: [ default][3149834016][PANIC] OHASD exiting; Could not init OLR
2016-01-08 13:42:59.565: [ default][3149834016] Done.

开始对本地节点（主机名为hostb）进行OLR的recreate,参考文章：
Top 5 Grid Infrastructure Startup Issues (文档 ID 1368382.1) ，搜索关键字为"OLR"

[root@hostb bin]# ./crsctl stop crs
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.

第一步：# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force

[root@hostb install]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully deconfigured Oracle clusterware stack on this node
[root@hostb install]#

此时的/u01/app/11.2.0/grid/cdata目录情况：

[grid@hostb cdata]$ ls -lrt
total 13124
drwxr-xr-x 2 grid oinstall      4096 Oct 30 13:21 hostb
drwxr-xr-x 2 grid oinstall      4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall      4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan  8 13:38 back_hostb.olr
-rw-r--r-- 1 root root      10485760 Jan 12 18:08 hostb.olr
[grid@hostb cdata]$

第二步：# /u01/app/11.2.0/grid/root.sh

[root@hostb install]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'hostb'
CRS-2676: Start of 'ora.mdnsd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hostb'
CRS-2676: Start of 'ora.gpnpd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hostb'
CRS-2672: Attempting to start 'ora.gipcd' on 'hostb'
CRS-2676: Start of 'ora.cssdmonitor' on 'hostb' succeeded
CRS-2676: Start of 'ora.gipcd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hostb'
CRS-2672: Attempting to start 'ora.diskmon' on 'hostb'
CRS-2676: Start of 'ora.diskmon' on 'hostb' succeeded
CRS-2676: Start of 'ora.cssd' on 'hostb' succeeded
PRKO-2190 : VIP exists for node hostb, VIP name hostb-vip
PRCS-1037 : Single Client Access Name VIPs already exist
PRCS-1028 : Single Client Access Name listeners already exist
OC4J could not be created as it already exists
PRCR-1086 : resource ora.oc4j is already registered
PRCR-1086 : resource ora.cvu is already registered
PRCC-1014 : scan1 was already running
PRCR-1004 : Resource ora.scan1.vip is already running
PRCR-1079 : Failed to start resource ora.scan1.vip
CRS-5702: Resource 'ora.scan1.vip' is already running on 'hostb'

PRCC-1014 : LISTENER_SCAN1 was already running
PRCR-1004 : Resource ora.LISTENER_SCAN1.lsnr is already running
PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr
CRS-5702: Resource 'ora.LISTENER_SCAN1.lsnr' is already running on 'hostb'

PRCC-1014 : oc4j was already running
PRCR-1004 : Resource ora.oc4j is already running
PRCR-1079 : Failed to start resource ora.oc4j
CRS-5702: Resource 'ora.oc4j' is already running on 'hostb'
CRS-2672: Attempting to start 'ora.registry.acfs' on 'hostb'
CRS-2676: Start of 'ora.registry.acfs' on 'hostb' succeeded
PRCC-1014 : cvu was already running
PRCR-1004 : Resource ora.cvu is already running
PRCR-1079 : Failed to start resource ora.cvu
CRS-5702: Resource 'ora.cvu' is already running on 'hostb'
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@hostb install]#

完成后的/u01/app/11.2.0/grid/cdata目录情况：

[grid@hostb cdata]$ ls -lrt
total 5612
drwxr-xr-x 2 grid oinstall      4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall      4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan  8 13:38 back_hostb.olr
drwxr-xr-x 2 grid oinstall      4096 Jan 12 18:19 hostb
-rw------- 1 root oinstall 272756736 Jan 12 18:19 hostb.olr
[grid@hostb cdata]$

[grid@hostb cdata]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       hostb                                      
ora.DATAFILE.dg
               ONLINE  ONLINE       hostb                                      
ora.FRA.dg
               ONLINE  ONLINE       hostb                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       hostb                                      
ora.asm
               ONLINE  ONLINE       hostb                   Started            
ora.gsd
               OFFLINE OFFLINE      hostb                                      
ora.net1.Network ONLINE  ONLINE       hostb                                      
ora.ons
               ONLINE  ONLINE       hostb                                      
ora.registry.acfs
               ONLINE  ONLINE       hostb                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       hostb                                      
ora.cvu
      1        ONLINE  ONLINE       hostb                                      
ora.orcl.db
      1        ONLINE  OFFLINE                                                  
      2        ONLINE  ONLINE       hostb                   Open               
ora.orcl.orclxa.svc
      1        ONLINE  ONLINE       hostb                                      
ora.orcla.vip
      1        ONLINE  INTERMEDIATE hostb                   FAILED OVER        
ora.hostb.vip
      1        ONLINE  ONLINE       hostb                                      
ora.oc4j
      1        ONLINE  ONLINE       hostb                                      
ora.scan1.vip
      1        ONLINE  ONLINE       hostb                                      
[grid@hostb cdata]$

[grid@hostb cdata]$ lsnrctl status

LSNRCTL for linux: version 11.2.0.3.0 - Production on 12-JAN-2016 18:21:34

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                12-JAN-2016 18:18:21
Uptime                    0 days 0 hr. 3 min. 13 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/hostb/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.188.182)(PORT=1528)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.188.184)(PORT=1528)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "orcl" has 1 instance(s).
  Instance "orcl2", status READY, has 1 handler(s) for this service...
Service "orclXDB" has 1 instance(s).
  Instance "orcl2", status READY, has 1 handler(s) for this service...
Service "orclxa" has 1 instance(s).
  Instance "orcl2", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@hostb cdata]$

[oracle@hostb ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Jan 12 18:29:52 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE