之前有同学在我的Oracle Allstarts群里讨论关于丢失包含ocr和votedisk的ASM diskgroup导致11gR2 RAC cluster无法正常启动的问题，最早我在《在11gR2 RAC中修改ASM DISK Path磁盘路径》一文中介绍了，如何在不启动CRS的情况下启动11.2中的ASM实例并实施操作， 这里也需要用到同样的命令”crsctl start crs -excl -nocrs “;
注意以下的恢复手段，针对ASM中单独的ocr或者单独的votedisk丢失也有效，因为11.2中普遍把ocr和votedisk存放在ASM中，而ASM的启动又依赖于ocr和votedisk，所以在丢失ocr或votedisk仍一一者都会导致cluter无法正常启动；这里我们仅仅讨论如何让CRS正常启动，如果丢失的diskgroup中还存放有数据库的话，数据的恢复不属于本篇文章的讨论范畴。
前提：恢复的前提是你仍有和故障前一样多的ASM LUN DISK，且你有OCR的自动备份，注意默认情况下每4个小时会自动备份一次，只要你没有删除$GI_HOME，一般都会有备份可用；不要求有votedisk备份
恢复场景： 利用dd命令清空ocr和votedisk所在diskgroup header，模拟diskgroup corruption:
1. 检查votedisk和 ocr备份
[root@vrh1 ~]# crsctl query css votedisk
|
1
2
3
4
5
6
7
8
|
## STATE File Universal Id File Name Disk group
— —— ———————— ———— ———
1.
ONLINE a853d6204bbc4feabfd8c73d4c3b3001 (/dev/asm-diskh) [SYSTEMDG]
2.
ONLINE a5b37704c3574f0fbf21d1d9f58c4a6b (/dev/asm-diskg) [SYSTEMDG]
3.
ONLINE 36e5c51ff0294fc3bf2a042266650331 (/dev/asm-diski) [SYSTEMDG]
4.
ONLINE af337d1512824fe4bf6ad45283517aaa (/dev/asm-diskj) [SYSTEMDG]
5.
ONLINE 3c4a349e2e304ff6bf64b2b1c9d9cf5d (/dev/asm-diskk) [SYSTEMDG]
Located
5
voting disk(s).
|
[grid@vrh1 ~]$ ocrconfig -showbackup
|
1
2
3
4
5
6
7
|
PROT
-26
: Oracle Cluster Registry backup locations were retrieved
from
a local copy
vrh1
2012
/
08
/
09
01
:
59
:
56
/g01/
11.2.
0
/maclean/grid/cdata/vrh-cluster/backup00.ocr
vrh1
2012
/
08
/
08
21
:
59
:
56
/g01/
11.2.
0
/maclean/grid/cdata/vrh-cluster/backup01.ocr
vrh1
2012
/
08
/
08
17
:
59
:
55
/g01/
11.2.
0
/maclean/grid/cdata/vrh-cluster/backup02.ocr
vrh1
2012
/
08
/
08
05
:
59
:
54
/g01/
11.2.
0
/grid/cdata/vrh-cluster/day.ocr
vrh1
2012
/
08
/
08
05
:
59
:
54
/g01/
11.2.
0
/grid/cdata/vrh-cluster/week.ocr
PROT
-25
: Manual backups
for
the Oracle Cluster Registry are
not
available
|
2. 彻底关闭所有节点上的clusterware ,OHASD
[root@vrh1 ~]$crsctl stop has -f
3. 请养成良好的习惯，做危险操作前备份asm header
4. 使用dd 命令 破坏ocr和votedisk所在diskgroup
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
[root@vrh1 ~]# dd
if
=/dev/zero of=/dev/asm-diskh bs=1024k count=
1
1
+
0
records
in
1
+
0
records out
1048576
bytes (
1.0
MB) copied,
0.00423853
seconds,
247
MB/s
[root@vrh1 ~]# dd
if
=/dev/zero of=/dev/asm-diskg bs=1024k count=
1
1
+
0
records
in
1
+
0
records out
1048576
bytes (
1.0
MB) copied,
0.0045179
seconds,
232
MB/s
[root@vrh1 ~]# dd
if
=/dev/zero of=/dev/asm-diski bs=1024k count=
1
1
+
0
records
in
1
+
0
records out
1048576
bytes (
1.0
MB) copied,
0.00469976
seconds,
223
MB/s
[root@vrh1 ~]# dd
if
=/dev/zero of=/dev/asm-diskj bs=1024k count=
1
1
+
0
records
in
1
+
0
records out
1048576
bytes (
1.0
MB) copied,
0.00344262
seconds,
305
MB/s
[root@vrh1 ~]# dd
if
=/dev/zero of=/dev/asm-diskk bs=1024k count=
1
1
+
0
records
in
1
+
0
records out
1048576
bytes (
1.0
MB) copied,
0.0053518
seconds,
196
MB/s
|
5. 在一个节点上尝试重新启动HAS
[root@vrh1 ~]# crsctl start has
CRS-4123: Oracle High Availability Services has been started.
但是因为ocr和votedisk所在diskgroup丢失，所以CSS将无法正常启动，如以下日志所示：
查看日志：alertvrh1.log
|
1
2
3
4
5
6
7
8
9
10
|
[cssd(
5162
)]CRS-
1714
:Unable to discover any voting files, retrying discovery in
15
seconds; Details at (:CSSNM00070:) in /g01/
11.2
.
0
/grid/log/vrh1/cssd/ocssd.log
2012
-
08
-
09
03
:
35
:
41.207
[cssd(
5162
)]CRS-
1714
:Unable to discover any voting files, retrying discovery in
15
seconds; Details at (:CSSNM00070:) in /g01/
11.2
.
0
/grid/log/vrh1/cssd/ocssd.log
2012
-
08
-
09
03
:
35
:
56.240
[cssd(
5162
)]CRS-
1714
:Unable to discover any voting files, retrying discovery in
15
seconds; Details at (:CSSNM00070:) in /g01/
11.2
.
0
/grid/log/vrh1/cssd/ocssd.log
2012
-
08
-
09
03
:
36
:
11.284
[cssd(
5162
)]CRS-
1714
:Unable to discover any voting files, retrying discovery in
15
seconds; Details at (:CSSNM00070:) in /g01/
11.2
.
0
/grid/log/vrh1/cssd/ocssd.log
2012
-
08
-
09
03
:
36
:
26.305
[cssd(
5162
)]CRS-
1714
:Unable to discover any voting files, retrying discovery in
15
seconds; Details at (:CSSNM00070:) in /g01/
11.2
.
0
/grid/log/vrh1/cssd/ocssd.log
2012
-
08
-
09
03
:
36
:
41.328
|
ocssd.log：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
2012
-08
-09
03
:
40
:
26.662
: [ CSSD][
1078700352
]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*)
2012
-08
-09
03
:
40
:
26.662
: [ CSSD][
1078700352
]clssnmvDDiscThread: using discovery string /dev/asm*
for
initial discovery
2012
-08
-09
03
:
40
:
26.662
: [ SKGFD][
1078700352
]Discovery
with
str:/dev/asm*:
2012
-08
-09
03
:
40
:
26.662
: [ SKGFD][
1078700352
]UFS discovery
with
:/dev/asm*:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diskf:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diskb:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diskj:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diskh:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diskc:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diskd:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diske:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diskg:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diski:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Fetching UFS disk :/dev/asm-diskk:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]OSS discovery
with
:/dev/asm*:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Handle
0xdf22a0
from
lib :UFS::
for
disk :/dev/asm-diskf:
2012
-08
-09
03
:
40
:
26.665
: [ SKGFD][
1078700352
]Handle
0xf412a0
from
lib :UFS::
for
disk :/dev/asm-diskb:
2012
-08
-09
03
:
40
:
26.666
: [ SKGFD][
1078700352
]Handle
0xf3a680
from
lib :UFS::
for
disk :/dev/asm-diskj:
2012
-08
-09
03
:
40
:
26.666
: [ SKGFD][
1078700352
]Handle
0xf93da0
from
lib :UFS::
for
disk :/dev/asm-diskh:
2012
-08
-09
03
:
40
:
26.667
: [ CSSD][
1078700352
]clssnmvDiskVerify: Successful discovery of
0
disks
2012
-08
-09
03
:
40
:
26.667
: [ CSSD][
1078700352
]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2012
-08
-09
03
:
40
:
26.667
: [ CSSD][
1078700352
]clssnmvFindInitialConfigs: No voting files found
2012
-08
-09
03
:
40
:
26.667
: [ CSSD][
1078700352
](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file
not
found. Retrying discovery
in
15
seconds
|
正式的恢复ocr和votedisk所在diskgroup的步骤如下：
1. 以-excl -nocrs 方式启动cluster，这将可以启动ASM实例 但不启动CRS
[root@vrh1 vrh1]# crsctl start crs -excl -nocrs
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
CRS
-4123
: Oracle High Availability Services has been started.
CRS
-2672
: Attempting to start ‘ora.mdnsd’
on
‘vrh1’
CRS
-2676
: Start of ‘ora.mdnsd’
on
‘vrh1’ succeeded
CRS
-2672
: Attempting to start ‘ora.gpnpd’
on
‘vrh1’
CRS
-2676
: Start of ‘ora.gpnpd’
on
‘vrh1’ succeeded
CRS
-2672
: Attempting to start ‘ora.cssdmonitor’
on
‘vrh1’
CRS
-2672
: Attempting to start ‘ora.gipcd’
on
‘vrh1’
CRS
-2676
: Start of ‘ora.cssdmonitor’
on
‘vrh1’ succeeded
CRS
-2676
: Start of ‘ora.gipcd’
on
‘vrh1’ succeeded
CRS
-2672
: Attempting to start ‘ora.cssd’
on
‘vrh1’
CRS
-2672
: Attempting to start ‘ora.diskmon’
on
‘vrh1’
CRS
-2676
: Start of ‘ora.diskmon’
on
‘vrh1’ succeeded
CRS
-2676
: Start of ‘ora.cssd’
on
‘vrh1’ succeeded
CRS
-2679
: Attempting to clean ‘ora.cluster_interconnect.haip’
on
‘vrh1’
CRS
-2672
: Attempting to start ‘ora.ctssd’
on
‘vrh1’
CRS
-2681
: Clean of ‘ora.cluster_interconnect.haip’
on
‘vrh1’ succeeded
CRS
-2672
: Attempting to start ‘ora.cluster_interconnect.haip’
on
‘vrh1’
CRS
-2676
: Start of ‘ora.ctssd’
on
‘vrh1’ succeeded
CRS
-2676
: Start of ‘ora.cluster_interconnect.haip’
on
‘vrh1’ succeeded
CRS
-2672
: Attempting to start ‘ora.asm’
on
‘vrh1’
CRS
-2676
: Start of ‘ora.asm’
on
‘vrh1’ succeeded
|
2.重建原ocr和votedisk所在diskgroup,注意compatible.asm必须是11.2:
|
1
2
3
4
5
6
7
8
9
10
11
|
[root@vrh1 vrh1]# su - grid
[grid@vrh1 ~]$ sqlplus /
as
sysasm
SQL*Plus: Release
11.2.
0.3.
0
Production
on
Thu Aug
9
04
:
16
:
58
2012
Copyright (c)
1982
,
2011
, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release
11.2.
0.3.
0
- 64bit Production
With the Real Application Clusters
and
Automatic Storage Management options
SQL> create diskgroup systemdg high redundancy
disk ‘/dev/asm-diskh’,’/dev/asm-diskg’,’/dev/asm-diski’,’/dev/asm-diskj’,’/dev/asm-diskk’
ATTRIBUTE ‘compatible.rdbms’ = ’
11.2
’, ‘compatible.asm’ = ’
11.2
’;
|
3.从ocr backup中恢复ocr并做ocrcheck检验:
[root@vrh1 ~]# ocrconfig -restore /g01/11.2.0/grid/cdata/vrh-cluster/backup00.ocr
[root@vrh1 ~]# ocrcheck
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
Status of Oracle Cluster Registry is
as
follows :
Version :
3
Total space (kbytes) :
262120
Used space (kbytes) :
3180
Available space (kbytes) :
258940
ID :
1238458014
Device/File Name : +systemdg
Device/File integrity check succeeded
Device/File
not
configured
Device/File
not
configured
Device/File
not
configured
Device/File
not
configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
|
4. 准备恢复votedisk ，可能会遇到下面的错误:
[grid@vrh1 ~]$ crsctl replace votedisk +SYSTEMDG
|
1
2
3
4
5
6
7
|
CRS
-4602
: Failed
27
to add voting file 2e4e0fe285924f86bf5473d00dcc0388.
CRS
-4602
: Failed
27
to add voting file 4fa54bb0cc5c4fafbf1a9be5479bf389.
CRS
-4602
: Failed
27
to add voting file a109ead9ea4e4f28bfe233188623616a.
CRS
-4602
: Failed
27
to add voting file 042c9fbd71b54f5abfcd3ab3408f3cf3.
CRS
-4602
: Failed
27
to add voting file 7b5a8cd24f954fafbf835ad78615763f.
Failed to
replace
voting disk group
with
+SYSTEMDG.
CRS
-4000
: Command Replace failed,
or
completed
with
errors.
|
需要重新配置一下ASM的参数，并重启ASM:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
|
SQL> alter system set asm_diskstring=‘/dev/asm*’;
System altered.
SQL> create spfile
from
memory;
File created.
SQL> startup force mount;
ORA
-32004
: obsolete
or
deprecated parameter(s) specified
for
ASM instance
ASM instance started
Total System Global Area
283930624
bytes
Fixed Size
2227664
bytes
Variable Size
256537136
bytes
ASM Cache
25165824
bytes
ASM diskgroups mounted
SQL> show parameter spfile
NAME TYPE VALUE
—————————————————— ————— ———————————————
spfile string /g01/
11.2.
0
/grid/dbs/spfi le+ASM1.ora
[grid@vrh1 trace]$ crsctl
replace
votedisk +SYSTEMDG
CRS
-4256
: Updating the profile
Successful addition of voting disk 85edc0e82d274f78bfc58cdc73b8c68a.
Successful addition of voting disk 201ffffc8ba44faabfe2efec2aa75840.
Successful addition of voting disk 6f2a25c589964faabf6980f7c5f621ce.
Successful addition of voting disk 93eb315648454f25bf3717df1a2c73d5.
Successful addition of voting disk 3737240678964f88bfbfbd31d8b3829f.
Successfully replaced voting disk group
with
+SYSTEMDG.
CRS
-4256
: Updating the profile
CRS
-4266
: Voting file(s) successfully replaced
|
5. 重启has服务，检验cluster是否正常：
[root@vrh1 ~]# crsctl check crs
|
1
2
3
4
|
CRS
-4638
: Oracle High Availability Services is online
CRS
-4537
: Cluster Ready Services is online
CRS
-4529
: Cluster Synchronization Services is online
CRS
-4533
: Event Manager is online
|
[root@vrh1 ~]# crsctl query css votedisk
|
1
2
3
4
5
6
7
8
|
## STATE File Universal Id File Name Disk group
——————————————— —— ———————— ———— ————
1
. ONLINE 85edc0e82d274f78bfc58cdc73b8c68a (/dev/asm-diskh) [SYSTEMDG]
2
. ONLINE 201ffffc8ba44faabfe2efec2aa75840 (/dev/asm-diskg) [SYSTEMDG]
3
. ONLINE 6f2a25c589964faabf6980f7c5f621ce (/dev/asm-diski) [SYSTEMDG]
4
. ONLINE 93eb315648454f25bf3717df1a2c73d5 (/dev/asm-diskj) [SYSTEMDG]
5
. ONLINE 3737240678964f88bfbfbd31d8b3829f (/dev/asm-diskk) [SYSTEMDG]
Located
5
voting disk(s).
|
[root@vrh1 ~]# crsctl stat res -t
|
1
2
3
4
5
6
7
8
9
10
|
————————————————————————————————————————
NAME TARGET STATE SERVER STATE_DETAILS Local Resources
————————————————————————————————————————
ora.BACKUPDG.dg ONLINE ONLINE vrh1
ora.DATA.dg ONLINE ONLINE vrh1
ora.LISTENER.lsnr ONLINE ONLINE vrh1
ora.LSN_MACLEAN.lsnr ONLINE ONLINE vrh1
ora.SYSTEMDG.dg ONLINE ONLINE vrh1
ora.asm ONLINE ONLINE vrh1
......
|
本文转自 客居天涯 51CTO博客，原文链接：http://blog.51cto.com/tiany/1566416，如需转载请自行联系原作者
