数据库版本:
p18317531_112040_Linux（建议应用！建议应用该补丁,否则可能所有节点会down.)
11.2.0.4测试环境:
TEST ASMGRP NORMAL diskgoup.
DG_NAME DG_STATE TYPE DSK_NO DSK_NAME PATH MOUNT_S FAILGROUP STATE
--------------- ---------- ------ ---------- ---------- -------------------------------------------------- ------- -------------------- --------
ASMGRP MOUNTED NORMAL 0 DISK1 /dev/oracleasm/disks/DB01DATE01 CACHED FG1 NORMAL
ASMGRP MOUNTED NORMAL 1 DISK2 /dev/oracleasm/disks/DB01DATE02 CACHED FG1 NORMAL
ASMGRP MOUNTED NORMAL 2 DISK3 /dev/oracleasm/disks/DB02DATE01 CACHED FG2 NORMAL
ASMGRP MOUNTED NORMAL 3 DISK4 /dev/oracleasm/disks/DB02DATE02 CACHED FG2 NORMAL
CRS MOUNTED NORMAL 0 CRS_0000 /dev/asm-diskcc CACHED CRS_0000 NORMAL
CRS MOUNTED NORMAL 1 CRS_0001 /dev/asm-diskdd CACHED CRS_0001 NORMAL
CRS MOUNTED NORMAL 2 CRS_0002 /dev/asm-diskee CACHED CRS_0002 NORMAL
7 rows selected.
DISK_NUMBER NAME PATH HEADER_STATUS OS_MB TOTAL_MB FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- -------------------------------------------------- -------------------- ---------- ---------- ---------- ------------ - -------
0 /dev/asm-test102 FORMER 3072 0 0 0 N REGULAR
1 /dev/asm-test101 PROVISIONED 3072 0 0 0 N REGULAR
2 CRS_0002 /dev/asm-diskee MEMBER 5120 5120 22 0 Y REGULAR
1 CRS_0001 /dev/asm-diskdd MEMBER 5120 5120 36 0 Y REGULAR
0 CRS_0000 /dev/asm-diskcc MEMBER 5120 5120 30 0 Y REGULAR
1 DISK2 /dev/oracleasm/disks/DB01DATE02 MEMBER 5119 5119 4446 0 N REGULAR
0 DISK1 /dev/oracleasm/disks/DB01DATE01 MEMBER 5120 5120 4443 0 N REGULAR
3 DISK4 /dev/oracleasm/disks/DB02DATE02 MEMBER 5119 5119 4444 0 N REGULAR
2 DISK3 /dev/oracleasm/disks/DB02DATE01 MEMBER 5120 5120 4445 0 N REGULAR
9 rows selected.
GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
1 CRS 11.2.0.0.0 10.1.0.0.0 Y
2 ASMGRP 11.2.0.0.0 11.2.0.0.0 N
STRONG LONK DOWN:
pdb01 db log
Mon May 27 22:43:37 2019
Starting background process SMCO
Mon May 27 22:43:37 2019
SMCO started with pid=30, OS id=126181
Mon May 27 22:43:39 2019
Thread 1 advanced to log sequence 4 (LGWR switch)
Current log# 2 seq# 4 mem# 0: +ASMGRP/ora/onlinelog/group_2.262.1009406213
Mon May 27 22:46:02 2019
WARNING: Write Failed. group:2 disk:3 AU:675 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/ora/ora1/trace/ora1_ckpt_125688.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
Mon May 27 22:46:02 2019
WARNING: Read Failed. group:2 disk:3 AU:675 offset:16384 size:16384
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group 2 on disk 3 allocation unit 675
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group [2.3431740168] from disk DISK4 allocation unit 675 reason error; if possible, will try another mirror side
NOTE: process _ckpt_ora1 (125688) initiating offline of disk 3.3916219397 (DISK4) with mask 0x7e in group 2
NOTE: successfully read mirror side 2 of virtual extent 0 logical extent 1 of file 260 in group [2.3431740168] from disk DISK1 allocation unit 677
Mon May 27 22:46:02 2019
WARNING: Read Failed. group:2 disk:3 AU:677 offset:49152 size:16384
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 260 in group [2.3431740168] from disk DISK4 allocation unit 677 reason error; if possible, will try another mirror side
WARNING: Read Failed. group:2 disk:2 AU:681 offset:524288 size:16384
WARNING: Read Failed. group:2 disk:3 AU:675 offset:65536 size:16384
WARNING: failed to read mirror side 1 of virtual extent 14 logical extent 0 of file 260 in group [2.3431740168] from disk DISK3 allocation unit 681 reason error; if possible, will try another mirror side
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group [2.3431740168] from disk DISK4 allocation unit 675 reason error; if possible, will try another mirror side
NOTE: successfully read mirror side 2 of virtual extent 14 logical extent 1 of file 260 in group [2.3431740168] from disk DISK1 allocation unit 684
NOTE: successfully read mirror side 2 of virtual extent 0 logical extent 1 of file 260 in group [2.3431740168] from disk DISK1 allocation unit 677
WARNING: Read Failed. group:2 disk:2 AU:0 offset:0 size:4096
ERROR: cannot read disk header of disk DISK3 (2:3916219398)
WARNING: Read Failed. group:2 disk:3 AU:677 offset:49152 size:16384
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 260 in group [2.3431740168] from disk DISK4 allocation unit 677 reason error; if possible, will try another mirror side
WARNING: failed to offline some disks corresponding to the mirrors of virtual extent 0 of file 260 in group 2
NOTE: process _lgwr_ora1 (125686) initiating offline of disk 2.3916219398 (DISK3) with mask 0x7e in group 2
Errors in file /u01/app/oracle/diag/rdbms/ora/ora1/trace/ora1_ckpt_125688.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+ASMGRP/ora/controlfile/current.260.1009406211'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15040: diskgroup is incomplete
Errors in file /u01/app/oracle/diag/rdbms/ora/ora1/trace/ora1_ckpt_125688.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+ASMGRP/ora/controlfile/current.260.1009406211'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15040: diskgroup is incomplete
CKPT (ospid: 125688): terminating the instance due to error 221
Mon May 27 22:46:02 2019
System state dump requested by (instance=1, osid=125688 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/ora/ora1/trace/ora1_diag_125656_20190527224602.trc
Dumping diagnostic data in directory=[cdmp_20190527224602], requested by (instance=1, osid=125688 (CKPT)), summary=[abnormal instance termination].
Instance terminated by CKPT, pid = 125688
db01 asm log:
Mon May 27 22:46:02 2019
WARNING: Write Failed. group:2 disk:2 AU:1 offset:1044480 size:4096
WARNING: Waited 127 secs for write IO to PST disk 0 in group 2.
WARNING: Hbeat write to PST disk 2.3916219398 in group 2 failed. [4]
Mon May 27 22:46:02 2019
NOTE: process _user125738_+asm1 (125738) initiating offline of disk 3.3916219397 (DISK4) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
WARNING: Disk DISK1 in mode 0x7f is now being offlined
GMON checking disk modes for group 2 at 16 for pid 32, osid 125738
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 2 done.
NOTE: initiating PST update: grp = 2, dsk = 3/0xe96cc405, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 17 for pid 32, osid 125738
ERROR: no read quorum in group: required 2, found 0 disks
Mon May 27 22:46:02 2019
NOTE: cache dismounting (not clean) group 2/0xCC8C3308 (ASMGRP)
WARNING: Offline for disk DISK4 in mode 0x7f failed.
ERROR: no read quorum in group: required 2, found 0 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
NOTE: messaging CKPT to quiesce pins Unix process pid: 126328, image: oracle@prod01 (B001)
Mon May 27 22:46:02 2019
NOTE: process _b000_+asm1 (126326) initiating offline of disk 2.3916219398 (DISK3) with mask 0x7e in group 2
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST: grp = 2
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
Mon May 27 22:46:02 2019
NOTE: halting all I/Os to diskgroup 2 (ASMGRP)
Mon May 27 22:46:02 2019
NOTE: LGWR doing non-clean dismount of group 2 (ASMGRP)
NOTE: LGWR sync ABA=2.267 last written ABA 2.267
GMON checking disk modes for group 2 at 18 for pid 28, osid 126326
ERROR: no read quorum in group: required 2, found 0 disks
Mon May 27 22:46:02 2019
kjbdomdet send to inst 2
detach from dom 2, sending detach message to inst 2
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 2 done.
ERROR: Could not read PST for grp 2. Force dismounting the disk group.NOTE: initiating PST update: grp = 2, dsk = 2/0xe96cc406, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 19 for pid 28, osid 126326
ERROR: no read quorum in group: required 2, found 0 disks
ERROR: no read quorum in group: required 2, found 0 disks
WARNING: Offline for disk DISK3 in mode 0x7f failed.
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
Mon May 27 22:46:02 2019
NOTE: process _user125747_+asm1 (125747) initiating offline of disk 2.3916219398 (DISK3) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
GMON checking disk modes for group 2 at 20 for pid 33, osid 125747
WARNING: Read Failed. group:2 disk:2 AU:1 offset:0 size:4096
WARNING: Read Failed. group:2 disk:2 AU:1 offset:4096 size:4096
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 2 done.
NOTE: initiating PST update: grp = 2, dsk = 2/0xe96cc406, mask = 0x6a, op = clear
WARNING: Read Failed. group:2 disk:2 AU:1 offset:0 size:4096
ERROR: no read quorum in group: required 2, found 0 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
Mon May 27 22:46:02 2019
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
GMON updating disk modes for group 2 at 21 for pid 33, osid 125747
ERROR: no read quorum in group: required 2, found 0 disks
ERROR: no read quorum in group: required 2, found 0 disks
WARNING: Offline for disk DISK3 in mode 0x7f failed.
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE
362 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Mon May 27 22:46:02 2019
WARNING: dirty detached from domain 2
NOTE: cache dismounted group 2/0xCC8C3308 (ASMGRP)
SQL> alter diskgroup ASMGRP dismount force /* ASM SERVER:3431740168 */
Mon May 27 22:46:02 2019
NOTE: ASM client ora1:ora disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_125701.trc
Mon May 27 22:46:03 2019
NOTE: cache deleting context for group ASMGRP 2/0xcc8c3308
GMON dismounting group 2 at 22 for pid 30, osid 126328
NOTE: Disk DISK1 in mode 0x7f marked for de-assignment
NOTE: Disk DISK2 in mode 0x7f marked for de-assignment
NOTE: Disk DISK3 in mode 0x7f marked for de-assignment
NOTE: Disk DISK4 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 2
Mon May 27 22:46:33 2019
SUCCESS: diskgroup ASMGRP was dismounted
SUCCESS: alter diskgroup ASMGRP dismount force /* ASM SERVER:3431740168 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group ASMGRP
Mon May 27 22:46:33 2019
NOTE: diskgroup resource ora.ASMGRP.dg is offline
Mon May 27 22:46:34 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_126375.trc:
ORA-17503: ksfdopn:2 Failed to open file +ASMGRP/ora/spfileora.ora
ORA-15001: diskgroup "ASMGRP" does not exist or is not mounted
Mon May 27 22:46:54 2019
SQL> ALTER DISKGROUP ASMGRP MOUNT /* asm agent *//* {0:9:31} */
NOTE: cache registered group ASMGRP number=2 incarn=0xcc8c331d
NOTE: cache began mount (not first) of group ASMGRP number=2 incarn=0xcc8c331d
NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/DB01DATE02)
NOTE: Assigning number (2,0) to disk (/dev/oracleasm/disks/DB01DATE01)
NOTE: cache dismounting (clean) group 2/0xCC8C331D (ASMGRP)
NOTE: messaging CKPT to quiesce pins Unix process pid: 122630, image: oracle@prod01 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 2/0xCC8C331D (ASMGRP)
NOTE: cache ending mount (fail) of group ASMGRP number=2 incarn=0xcc8c331d
NOTE: cache deleting context for group ASMGRP 2/0xcc8c331d
Mon May 27 22:46:54 2019
GMON dismounting group 2 at 24 for pid 29, osid 122630
NOTE: Disk DISK1 in mode 0x9 marked for de-assignment
NOTE: Disk DISK2 in mode 0x9 marked for de-assignment
ERROR: diskgroup ASMGRP was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "ASMGRP" cannot be mounted
ORA-15040: diskgroup is incomplete
ERROR: ALTER DISKGROUP ASMGRP MOUNT /* asm agent *//* {0:9:31} */
db01 crs log
019-05-27 22:46:02.880:
[/u01/app/11.2.0/grid/bin/oraagent.bin(125610)]CRS-5011:Check of resource "ora" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/prod01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2019-05-27 22:46:02.885:
[crsd(122432)]CRS-2765:Resource 'ora.ora.db' has failed on server 'prod01'.
2019-05-27 22:46:54.679:
[crsd(122432)]CRS-2878:Failed to restart resource 'ora.ora.db'
2019-05-27 22:46:54.680:
[crsd(122432)]CRS-2769:Unable to failover resource 'ora.ora.db'.db02 db log:
Mon May 27 22:43:13 2019
Thread 2 advanced to log sequence 2 (LGWR switch)
Current log# 4 seq# 2 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Mon May 27 22:43:36 2019
Thread 2 cannot allocate new log, sequence 3
Checkpoint not complete
Current log# 4 seq# 2 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Thread 2 advanced to log sequence 3 (LGWR switch)
Current log# 3 seq# 3 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
Mon May 27 22:45:56 2019
WARNING: Read Failed. group:2 disk:0 AU:679 offset:0 size:16384
WARNING: failed to read mirror side 1 of virtual extent 5 logical extent 0 of file 260 in group [2.3430969731] from disk DISK1 allocation unit 679 reason error; if possible, will try another mirror side
NOTE: successfully read mirror side 2 of virtual extent 5 logical extent 1 of file 260 in group [2.3430969731] from disk DISK4 allocation unit 678
WARNING: Read Failed. group:2 disk:0 AU:0 offset:0 size:4096
ERROR: cannot read disk header of disk DISK1 (0:3913318812)
NOTE: process _ckpt_ora2 (39745) initiating offline of disk 0.3913318812 (DISK1) with mask 0x7e in group 2
Mon May 27 22:45:56 2019
WARNING: Write Failed. group:2 disk:1 AU:765 offset:4096 size:1024
Errors in file /u01/app/oracle/diag/rdbms/ora/ora2/trace/ora2_lgwr_39743.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1024
WARNING: failed to write mirror side 2 of virtual extent 0 logical extent 1 of file 265 in group 2 on disk 1 allocation unit 765
Mon May 27 22:45:59 2019
NOTE: disk 0 (DISK1) in group 2 (ASMGRP) is offline for reads
Mon May 27 22:45:59 2019
Dumping diagnostic data in directory=[cdmp_20190527224602], requested by (instance=1, osid=125688 (CKPT)), summary=[abnormal instance termination].
Mon May 27 22:46:00 2019
WARNING: Write Failed. group:2 disk:0 AU:677 offset:65536 size:16384
Errors in file /u01/app/oracle/diag/rdbms/ora/ora2/trace/ora2_lmon_39723.trc:
ORA-15081: failed to submit an I/O operation to a disk
WARNING: failed to write mirror side 2 of virtual extent 0 logical extent 1 of file 260 in group 2 on disk 0 allocation unit 677
NOTE: disk 0 (DISK1) in group 2 (ASMGRP) is offline for writes
NOTE: process _lgwr_ora2 (39743) initiating offline of disk 1.3913318811 (DISK2) with mask 0x7e in group 2
NOTE: successfully read mirror side 2 of virtual extent 5 logical extent 1 of file 260 in group [2.3430969731] from disk DISK4 allocation unit 678
NOTE: disk 1 (DISK2) in group 2 (ASMGRP) is offline for reads
NOTE: disk 1 (DISK2) in group 2 (ASMGRP) is offline for writes
Reconfiguration started (old inc 4, new inc 6)
List of instances:
2 (myinst: 2)
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon May 27 22:46:01 2019
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Mon May 27 22:46:01 2019
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Mon May 27 22:46:01 2019
Instance recovery: looking for dead threads
Mon May 27 22:46:01 2019
minact-scn: Inst 2 is now the master inc#:6 mmon proc-id:39755 status:0x7
Submitted all GCS remote-cache requests
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.000e4595 gcalc-scn:0x0000.000e45a4
minact-scn: master found reconf/inst-rec before recscn scan old-inc#:6 new-inc#:6
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Thread 2 advanced to log sequence 4 (LGWR switch)
Current log# 4 seq# 4 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Beginning instance recovery of 1 threads
Thread 2 cannot allocate new log, sequence 5
Checkpoint not complete
Current log# 4 seq# 4 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
parallel recovery started with 7 processes
Started redo scan
Completed redo scan
read 0 KB redo, 0 data blocks need recovery
Started redo application at
Thread 1: logseq 4, block 10, scn 935405
Recovery of Online Redo Log: Thread 1 Group 2 Seq 4 Reading mem 0
Mem# 0: +ASMGRP/ora/onlinelog/group_2.262.1009406213
Completed redo application of 0.00MB
Completed instance recovery at
Thread 1: logseq 4, block 10, scn 955406
0 data blocks read, 0 data blocks written, 0 redo k-bytes read
Thread 1 advanced to log sequence 5 (thread recovery)
Thread 2 advanced to log sequence 5 (LGWR switch)
Current log# 3 seq# 5 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
minact-scn: master continuing after IR
Mon May 27 22:46:23 2019
Thread 2 cannot allocate new log, sequence 6
Checkpoint not complete
Current log# 3 seq# 5 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
Thread 2 advanced to log sequence 6 (LGWR switch)
Current log# 4 seq# 6 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Thread 2 advanced to log sequence 7 (LGWR switch)
Current log# 3 seq# 7 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
Thread 2 advanced to log sequence 8 (LGWR switch)
Current log# 4 seq# 8 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Mon May 27 22:46:38 2019
Thread 2 advanced to log sequence 9 (LGWR switch)
Current log# 3 seq# 9 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
Thread 2 advanced to log sequence 10 (LGWR switch)
Current log# 4 seq# 10 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Thread 2 advanced to log sequence 11 (LGWR switch)
Current log# 3 seq# 11 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
Mon May 27 22:47:01 2019
Decreasing number of real time LMS from 2 to 0
db02 asm log:
Mon May 27 22:45:56 2019
NOTE: process _user39832_+asm2 (39832) initiating offline of disk 0.3913318812 (DISK1) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
Mon May 27 22:45:56 2019
GMON checking disk modes for group 2 at 7 for pid 33, osid 39832
NOTE: checking PST for grp 2 done.
NOTE: sending set offline flag message 1221974703 to 1 disk(s) in group 2
WARNING: Disk DISK1 in mode 0x7f is now being offlined
NOTE: initiating PST update: grp = 2, dsk = 0/0xe940819c, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 8 for pid 33, osid 39832
WARNING: Write Failed. group:2 disk:1 AU:1 offset:1044480 size:4096
WARNING: found another non-responsive disk 1.3913318811 (DISK2) that will be offlined
NOTE: group ASMGRP: updated PST location: disk 0002 (PST copy 0)
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Mon May 27 22:45:59 2019
Received dirty detach msg from inst 1 for dom 2
Mon May 27 22:45:59 2019
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE
379 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Mon May 27 22:45:59 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_smon_36359.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DB01DATE02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: SMON starting instance recovery for group ASMGRP domain 2 (mounted)
NOTE: SMON skipping disk 0 (mode=00000015)
NOTE: SMON skipping disk 1 - no header
NOTE: F1X0 found on disk 3 au 2 fcn 0.0
NOTE: cache initiating offline of disk 1 group ASMGRP
NOTE: PST update grp = 2 completed successfully
NOTE: initiating PST update: grp = 2, dsk = 0/0xe940819c, mask = 0x7e, op = clear
NOTE: PST enabling heartbeating (grp 2)
GMON updating disk modes for group 2 at 9 for pid 33, osid 39832
NOTE: group ASMGRP: updated PST location: disk 0002 (PST copy 0)
NOTE: cache closing disk 0 of grp 2: DISK1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Mon May 27 22:46:00 2019
NOTE: process _b000_+asm2 (40936) initiating offline of disk 1.3913318811 (DISK2) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: PST update grp = 2 completed successfully
GMON checking disk modes for group 2 at 10 for pid 27, osid 40936
NOTE: cache closing disk 0 of grp 2: (not open) DISK1
NOTE: checking PST for grp 2 done.
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: sending set offline flag message 335704780 to 1 disk(s) in group 2
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
WARNING: Disk DISK2 in mode 0x7f is now being offlined
INFO: Instance #1 could not find disk 1 in group 2
NOTE: initiating PST update: grp = 2, dsk = 1/0xe940819b, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 11 for pid 27, osid 40936
NOTE: cache closing disk 0 of grp 2: (not open) DISK1
NOTE: group ASMGRP: updated PST location: disk 0002 (PST copy 0)
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: PST update grp = 2 completed successfully
NOTE: initiating PST update: grp = 2, dsk = 1/0xe940819b, mask = 0x7e, op = clear
GMON updating disk modes for group 2 at 12 for pid 27, osid 40936
NOTE: cache closing disk 0 of grp 2: (not open) DISK1
NOTE: group ASMGRP: updated PST location: disk 0002 (PST copy 0)
NOTE: cache closing disk 1 of grp 2: DISK2
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: starting recovery of thread=1 ckpt=2.268 group=2 (ASMGRP)
NOTE: waiting for instance recovery of group 2
NOTE: PST update grp = 2 completed successfully
NOTE: ASM recovery sucessfully read ACD from one mirror side
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_smon_36359.trc:
ORA-15062: ASM disk is globally closed
ORA-15062: ASM disk is globally closed
NOTE: SMON waiting for thread 1 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 2 (ASMGRP)
NOTE: SMON successfully validated lock domain 2
NOTE: advancing ckpt for group 2 (ASMGRP) thread=1 ckpt=2.268
NOTE: SMON did instance recovery for group ASMGRP domain 2
Mon May 27 22:46:11 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12960) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12960) secs on ASM inst 2
Mon May 27 22:49:14 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12777) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12777) secs on ASM inst 2
Mon May 27 22:52:17 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12594) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12594) secs on ASM inst 2
Mon May 27 22:55:20 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12411) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12411) secs on ASM inst 2
Mon May 27 22:58:23 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12228) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12228) secs on ASM inst 2
Mon May 27 23:01:26 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12045) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12045) secs on ASM inst 2
DB02 crs log:
无结论
CRS STATUS
-rw-rw-r-- 1 grid oinstall 2286 May 3 2014 PatchSearch.xml
[grid@prod01 soft]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMGRP.dg
OFFLINE OFFLINE prod01
ONLINE ONLINE prod02
ora.CRS.dg
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.LISTENER.lsnr
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.asm
ONLINE ONLINE prod01 Started
ONLINE ONLINE prod02 Started
ora.gsd
OFFLINE OFFLINE prod01
OFFLINE OFFLINE prod02
ora.net1.network
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.ons
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.registry.acfs
ONLINE ONLINE prod01
ONLINE ONLINE prod02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE prod01
ora.cvu
1 ONLINE ONLINE prod01
ora.oc4j
1 ONLINE ONLINE prod01
ora.ora.db
1 ONLINE OFFLINE
2 ONLINE ONLINE prod02 Open
ora.prod01.vip
1 ONLINE ONLINE prod01
ora.prod02.vip
1 ONLINE ONLINE prod02
ora.scan1.vip
1 ONLINE ONLINE prod01
[grid@prod01 soft]$ ASM STATUS
+ASM1
[grid@prod01 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Mon May 27 23:06:34 2019
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> @c
DG_NAME DG_STATE TYPE DSK_NO DSK_NAME PATH MOUNT_S FAILGROUP STATE
--------------- ---------- ------ ---------- ---------- -------------------------------------------------- ------- -------------------- --------
ASMGRP DISMOUNTED 0 /dev/oracleasm/disks/DB01DATE02 CLOSED NORMAL
ASMGRP DISMOUNTED 1 /dev/oracleasm/disks/DB01DATE01 CLOSED NORMAL
CRS MOUNTED NORMAL 0 CRS_0000 /dev/asm-diskb CACHED CRS_0000 NORMAL
CRS MOUNTED NORMAL 1 CRS_0001 /dev/asm-diskc CACHED CRS_0001 NORMAL
CRS MOUNTED NORMAL 2 CRS_0002 /dev/asm-diskd CACHED CRS_0002 NORMAL
DISK_NUMBER NAME PATH HEADER_STATUS OS_MB TOTAL_MB FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- -------------------------------------------------- -------------------- ---------- ---------- ---------- ------------ - -------
0 /dev/oracleasm/disks/DB01DATE02 MEMBER 5119 0 0 0 N REGULAR
1 /dev/oracleasm/disks/DB01DATE01 MEMBER 5120 0 0 0 N REGULAR
2 CRS_0002 /dev/asm-diskd MEMBER 5120 5120 22 0 Y REGULAR
0 CRS_0000 /dev/asm-diskb MEMBER 5120 5120 30 0 Y REGULAR
1 CRS_0001 /dev/asm-diskc MEMBER 5120 5120 36 0 Y REGULAR
GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
1 CRS 11.2.0.0.0 10.1.0.0.0 Y
0 ASMGRP 0.0.0.0.0 0.0.0.0.0 N
SQL>
+ASM2
[grid@prod02 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Mon May 27 23:07:33 2019
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> @C
SP2-0310: unable to open file "C.sql"
SQL> @c
DG_NAME DG_STATE TYPE DSK_NO DSK_NAME PATH MOUNT_S FAILGROUP STATE
--------------- ---------- ------ ---------- ---------- -------------------------------------------------- ------- -------------------- --------
ASMGRP MOUNTED NORMAL 0 DISK1 MISSING FG1 NORMAL
ASMGRP MOUNTED NORMAL 1 DISK2 MISSING FG1 NORMAL
ASMGRP MOUNTED NORMAL 2 DISK3 /dev/oracleasm/disks/DB02DATE01 CACHED FG2 NORMAL
ASMGRP MOUNTED NORMAL 3 DISK4 /dev/oracleasm/disks/DB02DATE02 CACHED FG2 NORMAL
CRS MOUNTED NORMAL 0 CRS_0000 /dev/asm-diskcc CACHED CRS_0000 NORMAL
CRS MOUNTED NORMAL 1 CRS_0001 /dev/asm-diskdd CACHED CRS_0001 NORMAL
CRS MOUNTED NORMAL 2 CRS_0002 /dev/asm-diskee CACHED CRS_0002 NORMAL
7 rows selected.
DISK_NUMBER NAME PATH HEADER_STATUS OS_MB TOTAL_MB FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- -------------------------------------------------- -------------------- ---------- ---------- ---------- ------------ - -------
0 /dev/asm-test102 FORMER 3072 0 0 0 N REGULAR
1 /dev/asm-test101 PROVISIONED 3072 0 0 0 N REGULAR
1 DISK2 UNKNOWN 0 5119 4291 11679 N REGULAR
0 DISK1 UNKNOWN 0 5120 4284 11679 N REGULAR
2 CRS_0002 /dev/asm-diskee MEMBER 5120 5120 22 0 Y REGULAR
1 CRS_0001 /dev/asm-diskdd MEMBER 5120 5120 36 0 Y REGULAR
0 CRS_0000 /dev/asm-diskcc MEMBER 5120 5120 30 0 Y REGULAR
3 DISK4 /dev/oracleasm/disks/DB02DATE02 MEMBER 5119 5119 4290 0 N REGULAR
2 DISK3 /dev/oracleasm/disks/DB02DATE01 MEMBER 5120 5120 4285 0 N REGULAR
9 rows selected.
GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
1 CRS 11.2.0.0.0 10.1.0.0.0 Y
2 ASMGRP 11.2.0.0.0 11.2.0.0.0 N
SQL>
切换时间
切断存储:
22:44:04 2019
数据库正常：
22:46:01 2019
