asm normal diskgroup ,strong split test

  1. 云栖社区>
  2. 博客>
  3. 正文

asm normal diskgroup ,strong split test

snowofsummer 2019-05-27 23:20:34 浏览197
展开阅读全文

数据库版本:
p18317531_112040_Linux(建议应用!建议应用该补丁,否则可能所有节点会down.)
11.2.0.4

测试环境:

TEST ASMGRP NORMAL diskgoup.

DG_NAME     DG_STATE   TYPE       DSK_NO DSK_NAME    PATH                           MOUNT_S FAILGROUP        STATE
--------------- ---------- ------ ---------- ---------- -------------------------------------------------- ------- -------------------- --------
ASMGRP        MOUNTED    NORMAL       0 DISK1    /dev/oracleasm/disks/DB01DATE01            CACHED  FG1            NORMAL
ASMGRP        MOUNTED    NORMAL       1 DISK2    /dev/oracleasm/disks/DB01DATE02            CACHED  FG1            NORMAL
ASMGRP        MOUNTED    NORMAL       2 DISK3    /dev/oracleasm/disks/DB02DATE01            CACHED  FG2            NORMAL
ASMGRP        MOUNTED    NORMAL       3 DISK4    /dev/oracleasm/disks/DB02DATE02            CACHED  FG2            NORMAL
CRS        MOUNTED    NORMAL       0 CRS_0000    /dev/asm-diskcc                    CACHED  CRS_0000        NORMAL
CRS        MOUNTED    NORMAL       1 CRS_0001    /dev/asm-diskdd                    CACHED  CRS_0001        NORMAL
CRS        MOUNTED    NORMAL       2 CRS_0002    /dev/asm-diskee                    CACHED  CRS_0002        NORMAL

7 rows selected.


DISK_NUMBER NAME       PATH                          HEADER_STATUS         OS_MB   TOTAL_MB    FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- -------------------------------------------------- -------------------- ---------- ---------- ---------- ------------ - -------
      0           /dev/asm-test102                   FORMER             3072       0          0        0 N REGULAR
      1           /dev/asm-test101                   PROVISIONED             3072       0          0        0 N REGULAR
      2 CRS_0002   /dev/asm-diskee                      MEMBER             5120    5120         22        0 Y REGULAR
      1 CRS_0001   /dev/asm-diskdd                      MEMBER             5120    5120         36        0 Y REGULAR
      0 CRS_0000   /dev/asm-diskcc                      MEMBER             5120    5120         30        0 Y REGULAR
      1 DISK2      /dev/oracleasm/disks/DB01DATE02              MEMBER             5119    5119       4446        0 N REGULAR
      0 DISK1      /dev/oracleasm/disks/DB01DATE01              MEMBER             5120    5120       4443        0 N REGULAR
      3 DISK4      /dev/oracleasm/disks/DB02DATE02              MEMBER             5119    5119       4444        0 N REGULAR
      2 DISK3      /dev/oracleasm/disks/DB02DATE01              MEMBER             5120    5120       4445        0 N REGULAR

9 rows selected.


GROUP_NUMBER NAME    COMPATIBILITY                             DATABASE_COMPATIBILITY                      V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
       1 CRS    11.2.0.0.0                             10.1.0.0.0                           Y
       2 ASMGRP    11.2.0.0.0                             11.2.0.0.0                           N

STRONG LONK DOWN:

pdb01 db log

Mon May 27 22:43:37 2019
Starting background process SMCO
Mon May 27 22:43:37 2019
SMCO started with pid=30, OS id=126181 
Mon May 27 22:43:39 2019
Thread 1 advanced to log sequence 4 (LGWR switch)
  Current log# 2 seq# 4 mem# 0: +ASMGRP/ora/onlinelog/group_2.262.1009406213




Mon May 27 22:46:02 2019
WARNING: Write Failed. group:2 disk:3 AU:675 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/ora/ora1/trace/ora1_ckpt_125688.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
Mon May 27 22:46:02 2019
WARNING: Read Failed. group:2 disk:3 AU:675 offset:16384 size:16384
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group 2 on disk 3 allocation unit 675 
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group [2.3431740168] from disk DISK4  allocation unit 675 reason error; if possible, will try another mirror side
NOTE: process _ckpt_ora1 (125688) initiating offline of disk 3.3916219397 (DISK4) with mask 0x7e in group 2
NOTE: successfully read mirror side 2 of virtual extent 0 logical extent 1 of file 260 in group [2.3431740168] from disk DISK1 allocation unit 677 
Mon May 27 22:46:02 2019
WARNING: Read Failed. group:2 disk:3 AU:677 offset:49152 size:16384
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 260 in group [2.3431740168] from disk DISK4  allocation unit 677 reason error; if possible, will try another mirror side
WARNING: Read Failed. group:2 disk:2 AU:681 offset:524288 size:16384
WARNING: Read Failed. group:2 disk:3 AU:675 offset:65536 size:16384
WARNING: failed to read mirror side 1 of virtual extent 14 logical extent 0 of file 260 in group [2.3431740168] from disk DISK3  allocation unit 681 reason error; if possible, will try another mirror side
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group [2.3431740168] from disk DISK4  allocation unit 675 reason error; if possible, will try another mirror side
NOTE: successfully read mirror side 2 of virtual extent 14 logical extent 1 of file 260 in group [2.3431740168] from disk DISK1 allocation unit 684 
NOTE: successfully read mirror side 2 of virtual extent 0 logical extent 1 of file 260 in group [2.3431740168] from disk DISK1 allocation unit 677 
WARNING: Read Failed. group:2 disk:2 AU:0 offset:0 size:4096
ERROR: cannot read disk header of disk DISK3 (2:3916219398)
WARNING: Read Failed. group:2 disk:3 AU:677 offset:49152 size:16384
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 260 in group [2.3431740168] from disk DISK4  allocation unit 677 reason error; if possible, will try another mirror side
WARNING: failed to offline some disks corresponding to the mirrors of virtual extent 0 of file 260 in group 2 
NOTE: process _lgwr_ora1 (125686) initiating offline of disk 2.3916219398 (DISK3) with mask 0x7e in group 2
Errors in file /u01/app/oracle/diag/rdbms/ora/ora1/trace/ora1_ckpt_125688.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+ASMGRP/ora/controlfile/current.260.1009406211'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15040: diskgroup is incomplete
Errors in file /u01/app/oracle/diag/rdbms/ora/ora1/trace/ora1_ckpt_125688.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+ASMGRP/ora/controlfile/current.260.1009406211'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15040: diskgroup is incomplete
CKPT (ospid: 125688): terminating the instance due to error 221
Mon May 27 22:46:02 2019
System state dump requested by (instance=1, osid=125688 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/ora/ora1/trace/ora1_diag_125656_20190527224602.trc
Dumping diagnostic data in directory=[cdmp_20190527224602], requested by (instance=1, osid=125688 (CKPT)), summary=[abnormal instance termination].
Instance terminated by CKPT, pid = 125688

db01 asm log:

Mon May 27 22:46:02 2019
WARNING: Write Failed. group:2 disk:2 AU:1 offset:1044480 size:4096
WARNING: Waited 127 secs for write IO to PST disk 0 in group 2.
WARNING: Hbeat write to PST disk 2.3916219398 in group 2 failed. [4]
Mon May 27 22:46:02 2019
NOTE: process _user125738_+asm1 (125738) initiating offline of disk 3.3916219397 (DISK4) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
WARNING: Disk DISK1 in mode 0x7f is now being offlined
GMON checking disk modes for group 2 at 16 for pid 32, osid 125738
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 2 done.
NOTE: initiating PST update: grp = 2, dsk = 3/0xe96cc405, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 17 for pid 32, osid 125738
ERROR: no read quorum in group: required 2, found 0 disks
Mon May 27 22:46:02 2019
NOTE: cache dismounting (not clean) group 2/0xCC8C3308 (ASMGRP) 
WARNING: Offline for disk DISK4 in mode 0x7f failed.
ERROR: no read quorum in group: required 2, found 0 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
NOTE: messaging CKPT to quiesce pins Unix process pid: 126328, image: oracle@prod01 (B001)
Mon May 27 22:46:02 2019
NOTE: process _b000_+asm1 (126326) initiating offline of disk 2.3916219398 (DISK3) with mask 0x7e in group 2
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST: grp = 2
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
Mon May 27 22:46:02 2019
NOTE: halting all I/Os to diskgroup 2 (ASMGRP)
Mon May 27 22:46:02 2019
NOTE: LGWR doing non-clean dismount of group 2 (ASMGRP)
NOTE: LGWR sync ABA=2.267 last written ABA 2.267
GMON checking disk modes for group 2 at 18 for pid 28, osid 126326
ERROR: no read quorum in group: required 2, found 0 disks
Mon May 27 22:46:02 2019
kjbdomdet send to inst 2
detach from dom 2, sending detach message to inst 2
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 2 done.
ERROR: Could not read PST for grp 2. Force dismounting the disk group.NOTE: initiating PST update: grp = 2, dsk = 2/0xe96cc406, mask = 0x6a, op = clear

GMON updating disk modes for group 2 at 19 for pid 28, osid 126326
ERROR: no read quorum in group: required 2, found 0 disks
ERROR: no read quorum in group: required 2, found 0 disks
WARNING: Offline for disk DISK3 in mode 0x7f failed.
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
Mon May 27 22:46:02 2019
NOTE: process _user125747_+asm1 (125747) initiating offline of disk 2.3916219398 (DISK3) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
GMON checking disk modes for group 2 at 20 for pid 33, osid 125747
WARNING: Read Failed. group:2 disk:2 AU:1 offset:0 size:4096
WARNING: Read Failed. group:2 disk:2 AU:1 offset:4096 size:4096
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 2 done.
NOTE: initiating PST update: grp = 2, dsk = 2/0xe96cc406, mask = 0x6a, op = clear
WARNING: Read Failed. group:2 disk:2 AU:1 offset:0 size:4096
ERROR: no read quorum in group: required 2, found 0 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
Mon May 27 22:46:02 2019
List of instances:
 1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
GMON updating disk modes for group 2 at 21 for pid 33, osid 125747
ERROR: no read quorum in group: required 2, found 0 disks
ERROR: no read quorum in group: required 2, found 0 disks
WARNING: Offline for disk DISK3 in mode 0x7f failed.
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
 Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE 
 362 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Mon May 27 22:46:02 2019
WARNING: dirty detached from domain 2
NOTE: cache dismounted group 2/0xCC8C3308 (ASMGRP) 
SQL> alter diskgroup ASMGRP dismount force /* ASM SERVER:3431740168 */ 
Mon May 27 22:46:02 2019
NOTE: ASM client ora1:ora disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_125701.trc
Mon May 27 22:46:03 2019
NOTE: cache deleting context for group ASMGRP 2/0xcc8c3308
GMON dismounting group 2 at 22 for pid 30, osid 126328
NOTE: Disk DISK1 in mode 0x7f marked for de-assignment
NOTE: Disk DISK2 in mode 0x7f marked for de-assignment
NOTE: Disk DISK3 in mode 0x7f marked for de-assignment
NOTE: Disk DISK4 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 2
Mon May 27 22:46:33 2019
SUCCESS: diskgroup ASMGRP was dismounted
SUCCESS: alter diskgroup ASMGRP dismount force /* ASM SERVER:3431740168 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group ASMGRP
Mon May 27 22:46:33 2019
NOTE: diskgroup resource ora.ASMGRP.dg is offline
Mon May 27 22:46:34 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_126375.trc:
ORA-17503: ksfdopn:2 Failed to open file +ASMGRP/ora/spfileora.ora
ORA-15001: diskgroup "ASMGRP" does not exist or is not mounted
Mon May 27 22:46:54 2019
SQL> ALTER DISKGROUP ASMGRP MOUNT  /* asm agent *//* {0:9:31} */ 
NOTE: cache registered group ASMGRP number=2 incarn=0xcc8c331d
NOTE: cache began mount (not first) of group ASMGRP number=2 incarn=0xcc8c331d
NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/DB01DATE02)
NOTE: Assigning number (2,0) to disk (/dev/oracleasm/disks/DB01DATE01)
NOTE: cache dismounting (clean) group 2/0xCC8C331D (ASMGRP) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 122630, image: oracle@prod01 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 2/0xCC8C331D (ASMGRP) 
NOTE: cache ending mount (fail) of group ASMGRP number=2 incarn=0xcc8c331d
NOTE: cache deleting context for group ASMGRP 2/0xcc8c331d
Mon May 27 22:46:54 2019
GMON dismounting group 2 at 24 for pid 29, osid 122630
NOTE: Disk DISK1 in mode 0x9 marked for de-assignment
NOTE: Disk DISK2 in mode 0x9 marked for de-assignment
ERROR: diskgroup ASMGRP was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "ASMGRP" cannot be mounted
ORA-15040: diskgroup is incomplete
ERROR: ALTER DISKGROUP ASMGRP MOUNT  /* asm agent *//* {0:9:31} */

db01 crs log

019-05-27 22:46:02.880: 
[/u01/app/11.2.0/grid/bin/oraagent.bin(125610)]CRS-5011:Check of resource "ora" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/prod01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2019-05-27 22:46:02.885: 
[crsd(122432)]CRS-2765:Resource 'ora.ora.db' has failed on server 'prod01'.
2019-05-27 22:46:54.679: 
[crsd(122432)]CRS-2878:Failed to restart resource 'ora.ora.db'
2019-05-27 22:46:54.680: 
[crsd(122432)]CRS-2769:Unable to failover resource 'ora.ora.db'.

db02 db log:

Mon May 27 22:43:13 2019
Thread 2 advanced to log sequence 2 (LGWR switch)
  Current log# 4 seq# 2 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Mon May 27 22:43:36 2019
Thread 2 cannot allocate new log, sequence 3
Checkpoint not complete
  Current log# 4 seq# 2 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Thread 2 advanced to log sequence 3 (LGWR switch)
  Current log# 3 seq# 3 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281

















Mon May 27 22:45:56 2019
WARNING: Read Failed. group:2 disk:0 AU:679 offset:0 size:16384
WARNING: failed to read mirror side 1 of virtual extent 5 logical extent 0 of file 260 in group [2.3430969731] from disk DISK1  allocation unit 679 reason error; if possible, will try another mirror side
NOTE: successfully read mirror side 2 of virtual extent 5 logical extent 1 of file 260 in group [2.3430969731] from disk DISK4 allocation unit 678 
WARNING: Read Failed. group:2 disk:0 AU:0 offset:0 size:4096
ERROR: cannot read disk header of disk DISK1 (0:3913318812)
NOTE: process _ckpt_ora2 (39745) initiating offline of disk 0.3913318812 (DISK1) with mask 0x7e in group 2
Mon May 27 22:45:56 2019
WARNING: Write Failed. group:2 disk:1 AU:765 offset:4096 size:1024
Errors in file /u01/app/oracle/diag/rdbms/ora/ora2/trace/ora2_lgwr_39743.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1024
WARNING: failed to write mirror side 2 of virtual extent 0 logical extent 1 of file 265 in group 2 on disk 1 allocation unit 765 
Mon May 27 22:45:59 2019
NOTE: disk 0 (DISK1) in group 2 (ASMGRP) is offline for reads
Mon May 27 22:45:59 2019
Dumping diagnostic data in directory=[cdmp_20190527224602], requested by (instance=1, osid=125688 (CKPT)), summary=[abnormal instance termination].
Mon May 27 22:46:00 2019
WARNING: Write Failed. group:2 disk:0 AU:677 offset:65536 size:16384
Errors in file /u01/app/oracle/diag/rdbms/ora/ora2/trace/ora2_lmon_39723.trc:
ORA-15081: failed to submit an I/O operation to a disk
WARNING: failed to write mirror side 2 of virtual extent 0 logical extent 1 of file 260 in group 2 on disk 0 allocation unit 677 
NOTE: disk 0 (DISK1) in group 2 (ASMGRP) is offline for writes
NOTE: process _lgwr_ora2 (39743) initiating offline of disk 1.3913318811 (DISK2) with mask 0x7e in group 2
NOTE: successfully read mirror side 2 of virtual extent 5 logical extent 1 of file 260 in group [2.3430969731] from disk DISK4 allocation unit 678 
NOTE: disk 1 (DISK2) in group 2 (ASMGRP) is offline for reads
NOTE: disk 1 (DISK2) in group 2 (ASMGRP) is offline for writes
Reconfiguration started (old inc 4, new inc 6)
List of instances:
 2 (myinst: 2) 
 Global Resource Directory frozen
 * dead instance detected - domain 0 invalid = TRUE 
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Mon May 27 22:46:01 2019
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Mon May 27 22:46:01 2019
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info 
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Mon May 27 22:46:01 2019
Instance recovery: looking for dead threads
Mon May 27 22:46:01 2019
minact-scn: Inst 2 is now the master inc#:6 mmon proc-id:39755 status:0x7
 Submitted all GCS remote-cache requests
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.000e4595 gcalc-scn:0x0000.000e45a4
minact-scn: master found reconf/inst-rec before recscn scan old-inc#:6 new-inc#:6
 Post SMON to start 1st pass IR
 Fix write in gcs resources
Reconfiguration complete
Thread 2 advanced to log sequence 4 (LGWR switch)
  Current log# 4 seq# 4 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Beginning instance recovery of 1 threads
Thread 2 cannot allocate new log, sequence 5
Checkpoint not complete
  Current log# 4 seq# 4 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
 parallel recovery started with 7 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 4, block 10, scn 935405
Recovery of Online Redo Log: Thread 1 Group 2 Seq 4 Reading mem 0
  Mem# 0: +ASMGRP/ora/onlinelog/group_2.262.1009406213
Completed redo application of 0.00MB
Completed instance recovery at
 Thread 1: logseq 4, block 10, scn 955406
 0 data blocks read, 0 data blocks written, 0 redo k-bytes read
Thread 1 advanced to log sequence 5 (thread recovery)
Thread 2 advanced to log sequence 5 (LGWR switch)
  Current log# 3 seq# 5 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
minact-scn: master continuing after IR
Mon May 27 22:46:23 2019
Thread 2 cannot allocate new log, sequence 6
Checkpoint not complete
  Current log# 3 seq# 5 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
Thread 2 advanced to log sequence 6 (LGWR switch)
  Current log# 4 seq# 6 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Thread 2 advanced to log sequence 7 (LGWR switch)
  Current log# 3 seq# 7 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
Thread 2 advanced to log sequence 8 (LGWR switch)
  Current log# 4 seq# 8 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Mon May 27 22:46:38 2019
Thread 2 advanced to log sequence 9 (LGWR switch)
  Current log# 3 seq# 9 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
Thread 2 advanced to log sequence 10 (LGWR switch)
  Current log# 4 seq# 10 mem# 0: +ASMGRP/ora/onlinelog/group_4.266.1009406283
Thread 2 advanced to log sequence 11 (LGWR switch)
  Current log# 3 seq# 11 mem# 0: +ASMGRP/ora/onlinelog/group_3.265.1009406281
Mon May 27 22:47:01 2019
Decreasing number of real time LMS from 2 to 0

db02 asm log:

Mon May 27 22:45:56 2019
NOTE: process _user39832_+asm2 (39832) initiating offline of disk 0.3913318812 (DISK1) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
Mon May 27 22:45:56 2019
GMON checking disk modes for group 2 at 7 for pid 33, osid 39832
NOTE: checking PST for grp 2 done.
NOTE: sending set offline flag message 1221974703 to 1 disk(s) in group 2
WARNING: Disk DISK1 in mode 0x7f is now being offlined
NOTE: initiating PST update: grp = 2, dsk = 0/0xe940819c, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 8 for pid 33, osid 39832
WARNING: Write Failed. group:2 disk:1 AU:1 offset:1044480 size:4096
WARNING: found another non-responsive disk 1.3913318811 (DISK2) that will be offlined 
NOTE: group ASMGRP: updated PST location: disk 0002 (PST copy 0)
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Mon May 27 22:45:59 2019
 Received dirty detach msg from inst 1 for dom 2
Mon May 27 22:45:59 2019
List of instances:
 1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
 Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE 
 379 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Mon May 27 22:45:59 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_smon_36359.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DB01DATE02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: SMON starting instance recovery for group ASMGRP domain 2 (mounted)
NOTE: SMON skipping disk 0 (mode=00000015)
NOTE: SMON skipping disk 1 - no header
NOTE: F1X0 found on disk 3 au 2 fcn 0.0
NOTE: cache initiating offline of disk 1 group ASMGRP
NOTE: PST update grp = 2 completed successfully 
NOTE: initiating PST update: grp = 2, dsk = 0/0xe940819c, mask = 0x7e, op = clear
NOTE: PST enabling heartbeating (grp 2)
GMON updating disk modes for group 2 at 9 for pid 33, osid 39832
NOTE: group ASMGRP: updated PST location: disk 0002 (PST copy 0)
NOTE: cache closing disk 0 of grp 2: DISK1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Mon May 27 22:46:00 2019
NOTE: process _b000_+asm2 (40936) initiating offline of disk 1.3913318811 (DISK2) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: PST update grp = 2 completed successfully 
GMON checking disk modes for group 2 at 10 for pid 27, osid 40936
NOTE: cache closing disk 0 of grp 2: (not open) DISK1
NOTE: checking PST for grp 2 done.
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: sending set offline flag message 335704780 to 1 disk(s) in group 2
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
WARNING: Disk DISK2 in mode 0x7f is now being offlined
INFO: Instance #1 could not find disk 1 in group 2
NOTE: initiating PST update: grp = 2, dsk = 1/0xe940819b, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 11 for pid 27, osid 40936
NOTE: cache closing disk 0 of grp 2: (not open) DISK1
NOTE: group ASMGRP: updated PST location: disk 0002 (PST copy 0)
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: PST update grp = 2 completed successfully 
NOTE: initiating PST update: grp = 2, dsk = 1/0xe940819b, mask = 0x7e, op = clear
GMON updating disk modes for group 2 at 12 for pid 27, osid 40936
NOTE: cache closing disk 0 of grp 2: (not open) DISK1
NOTE: group ASMGRP: updated PST location: disk 0002 (PST copy 0)
NOTE: cache closing disk 1 of grp 2: DISK2
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_36363.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
NOTE: starting recovery of thread=1 ckpt=2.268 group=2 (ASMGRP)
NOTE: waiting for instance recovery of group 2
NOTE: PST update grp = 2 completed successfully 
NOTE: ASM recovery sucessfully read ACD from one mirror side
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_smon_36359.trc:
ORA-15062: ASM disk is globally closed
ORA-15062: ASM disk is globally closed
NOTE: SMON waiting for thread 1 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 2 (ASMGRP)
NOTE: SMON successfully validated lock domain 2
NOTE: advancing ckpt for group 2 (ASMGRP) thread=1 ckpt=2.268
NOTE: SMON did instance recovery for group ASMGRP domain 2
Mon May 27 22:46:11 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12960) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12960) secs on ASM inst 2
Mon May 27 22:49:14 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12777) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12777) secs on ASM inst 2
Mon May 27 22:52:17 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12594) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12594) secs on ASM inst 2
Mon May 27 22:55:20 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12411) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12411) secs on ASM inst 2
Mon May 27 22:58:23 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12228) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12228) secs on ASM inst 2
Mon May 27 23:01:26 2019
WARNING: Disk 0 (DISK1) in group 2 will be dropped in: (12045) secs on ASM inst 2
WARNING: Disk 1 (DISK2) in group 2 will be dropped in: (12045) secs on ASM inst 2

DB02 crs log:

结论

CRS STATUS

-rw-rw-r-- 1 grid oinstall      2286 May  3  2014 PatchSearch.xml
[grid@prod01 soft]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMGRP.dg
               OFFLINE OFFLINE      prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.CRS.dg
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.LISTENER.lsnr
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.asm
               ONLINE  ONLINE       prod01                   Started             
               ONLINE  ONLINE       prod02                   Started             
ora.gsd
               OFFLINE OFFLINE      prod01                                       
               OFFLINE OFFLINE      prod02                                       
ora.net1.network
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.ons
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.registry.acfs
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       prod01                                       
ora.cvu
      1        ONLINE  ONLINE       prod01                                       
ora.oc4j
      1        ONLINE  ONLINE       prod01                                       
ora.ora.db
      1        ONLINE  OFFLINE                                                   
      2        ONLINE  ONLINE       prod02                   Open                
ora.prod01.vip
      1        ONLINE  ONLINE       prod01                                       
ora.prod02.vip
      1        ONLINE  ONLINE       prod02                                       
ora.scan1.vip
      1        ONLINE  ONLINE       prod01                                       
[grid@prod01 soft]$ 

ASM STATUS

+ASM1

[grid@prod01 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Mon May 27 23:06:34 2019

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> @c

DG_NAME     DG_STATE   TYPE       DSK_NO DSK_NAME    PATH                           MOUNT_S FAILGROUP        STATE
--------------- ---------- ------ ---------- ---------- -------------------------------------------------- ------- -------------------- --------
ASMGRP        DISMOUNTED           0        /dev/oracleasm/disks/DB01DATE02            CLOSED            NORMAL
ASMGRP        DISMOUNTED           1        /dev/oracleasm/disks/DB01DATE01            CLOSED            NORMAL
CRS        MOUNTED    NORMAL       0 CRS_0000    /dev/asm-diskb                       CACHED  CRS_0000        NORMAL
CRS        MOUNTED    NORMAL       1 CRS_0001    /dev/asm-diskc                       CACHED  CRS_0001        NORMAL
CRS        MOUNTED    NORMAL       2 CRS_0002    /dev/asm-diskd                       CACHED  CRS_0002        NORMAL


DISK_NUMBER NAME       PATH                          HEADER_STATUS         OS_MB   TOTAL_MB    FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- -------------------------------------------------- -------------------- ---------- ---------- ---------- ------------ - -------
      0           /dev/oracleasm/disks/DB01DATE02              MEMBER             5119       0          0        0 N REGULAR
      1           /dev/oracleasm/disks/DB01DATE01              MEMBER             5120       0          0        0 N REGULAR
      2 CRS_0002   /dev/asm-diskd                      MEMBER             5120    5120         22        0 Y REGULAR
      0 CRS_0000   /dev/asm-diskb                      MEMBER             5120    5120         30        0 Y REGULAR
      1 CRS_0001   /dev/asm-diskc                      MEMBER             5120    5120         36        0 Y REGULAR


GROUP_NUMBER NAME    COMPATIBILITY                             DATABASE_COMPATIBILITY                      V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
       1 CRS    11.2.0.0.0                             10.1.0.0.0                           Y
       0 ASMGRP    0.0.0.0.0                             0.0.0.0.0                              N

SQL> 

+ASM2

[grid@prod02 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Mon May 27 23:07:33 2019

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> @C
SP2-0310: unable to open file "C.sql"
SQL> @c

DG_NAME     DG_STATE   TYPE       DSK_NO DSK_NAME    PATH                           MOUNT_S FAILGROUP        STATE
--------------- ---------- ------ ---------- ---------- -------------------------------------------------- ------- -------------------- --------
ASMGRP        MOUNTED    NORMAL       0 DISK1                               MISSING FG1            NORMAL
ASMGRP        MOUNTED    NORMAL       1 DISK2                               MISSING FG1            NORMAL
ASMGRP        MOUNTED    NORMAL       2 DISK3    /dev/oracleasm/disks/DB02DATE01            CACHED  FG2            NORMAL
ASMGRP        MOUNTED    NORMAL       3 DISK4    /dev/oracleasm/disks/DB02DATE02            CACHED  FG2            NORMAL
CRS        MOUNTED    NORMAL       0 CRS_0000    /dev/asm-diskcc                    CACHED  CRS_0000        NORMAL
CRS        MOUNTED    NORMAL       1 CRS_0001    /dev/asm-diskdd                    CACHED  CRS_0001        NORMAL
CRS        MOUNTED    NORMAL       2 CRS_0002    /dev/asm-diskee                    CACHED  CRS_0002        NORMAL

7 rows selected.


DISK_NUMBER NAME       PATH                          HEADER_STATUS         OS_MB   TOTAL_MB    FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- -------------------------------------------------- -------------------- ---------- ---------- ---------- ------------ - -------
      0           /dev/asm-test102                   FORMER             3072       0          0        0 N REGULAR
      1           /dev/asm-test101                   PROVISIONED             3072       0          0        0 N REGULAR
      1 DISK2                              UNKNOWN            0    5119       4291        11679 N REGULAR
      0 DISK1                              UNKNOWN            0    5120       4284        11679 N REGULAR
      2 CRS_0002   /dev/asm-diskee                      MEMBER             5120    5120         22        0 Y REGULAR
      1 CRS_0001   /dev/asm-diskdd                      MEMBER             5120    5120         36        0 Y REGULAR
      0 CRS_0000   /dev/asm-diskcc                      MEMBER             5120    5120         30        0 Y REGULAR
      3 DISK4      /dev/oracleasm/disks/DB02DATE02              MEMBER             5119    5119       4290        0 N REGULAR
      2 DISK3      /dev/oracleasm/disks/DB02DATE01              MEMBER             5120    5120       4285        0 N REGULAR

9 rows selected.


GROUP_NUMBER NAME    COMPATIBILITY                             DATABASE_COMPATIBILITY                      V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
       1 CRS    11.2.0.0.0                             10.1.0.0.0                           Y
       2 ASMGRP    11.2.0.0.0                             11.2.0.0.0                           N

SQL> 

切换时间

切断存储:
22:44:04 2019
数据库正常:
22:46:01 2019

网友评论

登录后评论
0/500
评论
snowofsummer
+ 关注