asm Normal redundancy storage link is broken(19C)

  1. 云栖社区>
  2. 博客>
  3. 正文

asm Normal redundancy storage link is broken(19C)

snowofsummer 2019-06-25 12:17:06 浏览397
展开阅读全文

测试模拟环境

360_20190625093602067

create diskgroup dg1  normal redundancy 
failgroup aa disk '/dev/oracleasm/disks/VOTE01'  # 存储A
failgroup bb disk '/dev/oracleasm/disks/VOTE02'  # 存储B
quorum failgroup zc disk '/dev/oracleasm/disks/DISK01'    ## 仲裁存储
ATTRIBUTE 'compatible.asm' = '18.0.0.0.0', 'compatible.rdbms' = '18.0.0.0.0';

数据库版本


SQL*Plus: Release 18.0.0.0.0 - Production on Tue Jun 25 12:03:41 2019
Version 18.3.0.0.0

prod04 crs log:

2019-06-25 11:49:19.879 [OCSSD(4103)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE02 will be considered not functional in 99140 milliseconds.
2019-06-25 11:49:23.789 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 103770 msecs.
2019-06-25 11:49:23.791 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 103720 msecs.
2019-06-25 11:49:31.793 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 111780 msecs.
2019-06-25 11:49:31.795 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 111730 msecs.
2019-06-25 11:49:39.798 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 119780 msecs.
2019-06-25 11:49:39.799 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 119730 msecs.
2019-06-25 11:49:47.802 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 127780 msecs.
2019-06-25 11:49:47.803 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 127740 msecs.
2019-06-25 11:49:49.418 [OCSSD(4103)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE02; details at (:CSSNM00059:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc.
2019-06-25 11:49:49.418 [OCSSD(4103)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE02; details at (:CSSNM00060:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc.
2019-06-25 11:49:51.308 [OCSSD(4103)]CRS-1650: Configuration change 1561434591:0 rejected by node 2 prod05; details at (:CSSNM000066:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc
2019-06-25 11:49:51.352 [OCSSD(4103)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:49:53.541 [CRSD(4667)]CRS-2878: Failed to restart resource 'ora.DG1.dg'
2019-06-25 11:49:53.543 [CRSD(4667)]CRS-2769: Unable to failover resource 'ora.DG1.dg'.
2019-06-25 11:49:54.575 [OCSSD(4103)]CRS-1650: Configuration change 1561434594:0 rejected by node 2 prod05; details at (:CSSNM000066:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc
2019-06-25 11:49:54.620 [OCSSD(4103)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:50:09.811 [OCSSD(4103)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE02 will be considered not functional in 49210 milliseconds.
2019-06-25 11:50:39.821 [OCSSD(4103)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE02 will be considered not functional in 19200 milliseconds.
2019-06-25 11:50:59.827 [OCSSD(4103)]CRS-1604: CSSD voting file is offline: /dev/oracleasm/disks/VOTE02; details at (:CSSNM00058:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc.
2019-06-25 11:50:59.830 [OCSSD(4103)]CRS-1672: The number of voting files currently available 2 has fallen to the minimum number of voting files required 2. Further reduction in voting files will result in eviction and loss of functionality

prod05 asm log:

2019-06-25T11:49:50.946702+08:00
WARNING: Write Failed. group:2 disk:1 AU:1 offset:1044480 size:4096
path:/dev/oracleasm/disks/VOTE02
     incarnation:0xf0f0dce6 asynchronous result:'I/O error'
     subsys:System krq:0x7fb6e08d3c38 bufp:0x7fb6df381000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Hbeat write to PST disk 1.4042317030 in group 2 failed. [2]
2019-06-25T11:49:50.987326+08:00
NOTE: checking PST: grp = 2
2019-06-25T11:49:50.988005+08:00
GMON checking disk modes for group 2 at 28 for pid 41, osid 57148
2019-06-25T11:49:50.994499+08:00
NOTE: checking PST for grp 2 done.
2019-06-25T11:49:50.994807+08:00
NOTE: sending set offline flag message (1518616090) to 1 disk(s) in group 2
2019-06-25T11:49:50.995444+08:00
WARNING: Disk 1 (DG1_0001) in group 2 mode 0x7f is now being offlined
2019-06-25T11:49:51.006032+08:00
NOTE: initiating PST update: grp 2 (DG1), dsk = 1/0xf0f0dce6, mask = 0x6a, op = clear mandatory
2019-06-25T11:49:51.006479+08:00
GMON updating disk modes for group 2 at 29 for pid 41, osid 57148
WARNING: Write Failed. group:2 disk:1 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE02
     incarnation:0xf0f0dce6 synchronous result:'I/O error'
     subsys:System krq:0x7fb6df36b668 bufp:0x7fb6df34f000 osderr1:0x534b4950 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.139504+08:00
 Received dirty detach msg from inst 2 for dom 2
2019-06-25T11:49:51.168873+08:00
NOTE: Attempting voting file refresh on diskgroup DG1
NOTE: Refresh completed on diskgroup DG1. Found 3 voting file(s).
NOTE: Voting file relocation is required in diskgroup DG1
2019-06-25T11:49:51.182641+08:00
List of instances (total 2) :
 1 2
My inst 1   
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
 Global Resource Directory partially frozen for dirty detach
dirty detach - domain 2 invalid = TRUE 
 0 GCS resources traversed, 0 cancelled
2019-06-25T11:49:51.235840+08:00
NOTE: Attempting voting file relocation on diskgroup DG1
2019-06-25T11:49:51.249004+08:00
Dirty Detach Reconfiguration complete (total time 0.1 secs) 
2019-06-25T11:49:51.251495+08:00
NOTE: SMON detected lock domain 2 invalid at system inc 4 06/25/19 11:49:51
2019-06-25T11:49:51.372440+08:00
NOTE: Failed voting file relocation on diskgroup DG1
2019-06-25T11:49:51.478897+08:00
NOTE: SMON starting instance recovery of group DG1 domain 2 inc 4 (mounted) at 06/25/19 11:49:51
NOTE: 06/25/19 11:49:51 DG1.F1X0 found on disk 0 au 10 fcn 0.0 datfmt 1
NOTE: SMON skipping disk 1 (mode=00000015)
NOTE: instance recovery of group DG1 will recover thread=2 ckpt=2.0 domain=2 inc#=4 instnum=2
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 2 (DG1)
validate pdb 2, flags x0, valid 0, pdb flags xa4 
* validated domain 2, flags = 0xa0
2019-06-25T11:49:51.515918+08:00
NOTE: SMON successfully validated lock domain 2 inc 4 (DG1)
2019-06-25T11:49:51.575068+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_smon_4494.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/VOTE02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: Write Failed. group:2 disk:1 AU:60 offset:0 size:4096
path:Unknown disk
     incarnation:0xf0f0dce6 asynchronous result:'I/O error'
     subsys:Unknown library krq:0x7fef485bd680 bufp:0x7fef47736000 osderr1:0x434c5344 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
NOTE: advancing ckpt for group 2 (DG1) thread=2 ckpt=2.0 domain inc# 4
NOTE: cache initiating offline of disk 1 group DG1
NOTE: initiating PST update: grp 2 (DG1), dsk = 1/0xf0f0dce6, mask = 0x6a, op = clear mandatory
2019-06-25T11:49:51.620068+08:00
GMON updating disk modes for group 2 at 30 for pid 23, osid 4494
2019-06-25T11:49:51.623397+08:00
NOTE: PST update grp = 2 completed successfully 
NOTE: ospid 4494 initiating cluster wide offline of the following disks from gn 2: 1
2019-06-25T11:49:51.629048+08:00
NOTE: disk 1 (DG1_0001) in group 2 (DG1) is locally offline for writes
2019-06-25T11:49:51.633752+08:00
NOTE: ospid 4494 finished cluster wide offline, gn 2
NOTE: successfully wrote at least one mirror side for diskgroup DG1
2019-06-25T11:49:51.634353+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_smon_4494.trc:
ORA-15080: synchronous I/O operation failed to write block 10752 of disk 1 in disk group DG1
2019-06-25T11:49:51.634652+08:00
NOTE: SMON did instance recovery for group DG1 domain 2
2019-06-25T11:49:52.442330+08:00
NOTE: PST update grp = 2 completed successfully 
NOTE: initiating PST update: grp 2 (DG1), dsk = 1/0xf0f0dce6, mask = 0x7e, op = clear mandatory
2019-06-25T11:49:52.443344+08:00
GMON updating disk modes for group 2 at 31 for pid 41, osid 57148
2019-06-25T11:49:52.449953+08:00
NOTE: cache closing disk 1 of grp 2: DG1_0001
2019-06-25T11:49:52.459736+08:00
NOTE: cache closing disk 1 of grp 2: (not open) DG1_0001
2019-06-25T11:49:52.461747+08:00
NOTE: PST update grp = 2 completed successfully 
2019-06-25T11:49:53.999944+08:00
WARNING: Started Drop Disk Timeout for Disk 1 (DG1_0001) in group 2 with a value 43200
WARNING: Disk 1 (DG1_0001) in group 2 will be dropped in: (43200) secs on ASM inst 1
2019-06-25T11:49:54.475008+08:00
NOTE: Attempting voting file refresh on diskgroup DG1
NOTE: Refresh completed on diskgroup DG1. Found 3 voting file(s).
NOTE: Voting file relocation is required in diskgroup DG1
NOTE: Attempting voting file relocation on diskgroup DG1
NOTE: Failed voting file relocation on diskgroup DG1
2019-06-25T11:59:05.055729+08:00
WARNING: Disk 1 (DG1_0001) in group 2 will be dropped in: (42649) secs on ASM inst 1

prod05 crs log:

2019-06-25 11:49:19.272 [OCSSD(7735)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE01 will be considered not functional in 99990 milliseconds.
2019-06-25 11:49:24.324 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 103750 msecs.
2019-06-25 11:49:24.325 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 104060 msecs.
2019-06-25 11:49:32.329 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 111750 msecs.
2019-06-25 11:49:32.331 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 112070 msecs.
2019-06-25 11:49:40.334 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 119760 msecs.
2019-06-25 11:49:40.336 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 120070 msecs.
2019-06-25 11:49:48.340 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 127760 msecs.
2019-06-25 11:49:48.342 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 128080 msecs.
2019-06-25 11:49:49.811 [OCSSD(7735)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE01; details at (:CSSNM00059:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc.
2019-06-25 11:49:49.812 [OCSSD(7735)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE01; details at (:CSSNM00060:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc.
2019-06-25 11:49:51.313 [OCSSD(7735)]CRS-1641: Rejecting configuration change 1561434591:0 because only 1 of the required 0 voting files of the new configuration were located; details at (:CSSNM00026:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc
2019-06-25 11:49:51.383 [OCSSD(7735)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:49:53.380 [ORAAGENT(8566)]CRS-5017: The resource action "ora.DG1.dg start" encountered the following error: 
2019-06-25 11:49:53.380+ORA-15032: not all alterations performed
ORA-15017: diskgroup "DG1" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DG1"
. For details refer to "(:CLSN00107:)" in "/u01/app/grid_base/diag/crs/prod05/crs/trace/crsd_oraagent_grid.trc".
2019-06-25 11:49:54.596 [OCSSD(7735)]CRS-1641: Rejecting configuration change 1561434594:0 because only 1 of the required 0 voting files of the new configuration were located; details at (:CSSNM00026:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc
2019-06-25 11:49:54.650 [OCSSD(7735)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:49:58.836 [OHASD(2593)]CRS-2771: Maximum restart attempts reached for resource 'ora.storage'; will not restart.
2019-06-25 11:50:10.232 [OCSSD(7735)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE01 will be considered not functional in 49030 milliseconds.
2019-06-25 11:50:40.243 [OCSSD(7735)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE01 will be considered not functional in 19020 milliseconds.
2019-06-25 11:51:00.250 [OCSSD(7735)]CRS-1604: CSSD voting file is offline: /dev/oracleasm/disks/VOTE01; details at (:CSSNM00058:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc.
2019-06-25 11:51:00.251 [OCSSD(7735)]CRS-1672: The number of voting files currently available 2 has fallen to the minimum number of voting files required 2. Further reduction in voting files will result in eviction and loss of functionality
2019-06-25 11:55:29.161 [OHASD(2593)]CRS-2771: Maximum restart attempts reached for resource 'ora.storage'; will not restart.
2019-06-25 12:00:59.804 [ORAROOTAGENT(3154)]CRS-5019: All OCR locations are on ASM disk groups [DG1], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid_base/diag/crs/prod05/crs/trace/ohasd_orarootagent_root.trc".

prod06 asm log:

2019-06-25T11:49:51.029835+08:00
WARNING: Disk 1 (DG1_0001) in group 2 mode 0x7f is now being offlined
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f57466038e0 bufp:0x7f574199a000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f5746603d20 bufp:0x7f5741997000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.051804+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
2019-06-25T11:49:51.093245+08:00
NOTE: cache dismounting (not clean) group 2/0x38008E3F (DG1)
NOTE: messaging CKPT to quiesce pins Unix process pid: 70764, image: oracle@prod05 (B000)
2019-06-25T11:49:51.097051+08:00
NOTE: halting all I/Os to diskgroup 2 (DG1)
2019-06-25T11:49:51.099532+08:00
NOTE: LGWR doing non-clean dismount of group 2 (DG1) thread 2
NOTE: LGWR sync ABA=1.10750 last written ABA 1.10750
2019-06-25T11:49:51.100491+08:00
NOTE: initiating dirty detach from lock domain 2
2019-06-25T11:49:51.169386+08:00
kjbdomdet send to inst 1
detach from dom 2, sending detach message to inst 1
2019-06-25T11:49:51.231898+08:00
List of instances (total 2) :
 1 2
My inst 2   
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
 Global Resource Directory partially frozen for dirty detach
dirty detach - domain 2 invalid = TRUE 
 0 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete (total time 0.0 secs) 
2019-06-25T11:49:51.300226+08:00
WARNING: dirty detached from lock domain 2
NOTE: CKPT released recovery enqueue for thread 2 group 2 (DG1)
2019-06-25T11:49:51.333377+08:00
NOTE: cache dismounted group 2/0x38008E3F (DG1)
2019-06-25T11:49:51.336638+08:00
SQL> alter diskgroup DG1 dismount force /* ASM SERVER:939560511 */ 
2019-06-25T11:49:51.338336+08:00
NOTE: client +ASM2:+ASM:prod-cluster no longer has group 1 (DATA) mounted
2019-06-25T11:49:51.545472+08:00

* instance 1 validates domain 2 
2019-06-25T11:49:51.658363+08:00
NOTE: disk 1 (DG1_0001) in group 2 (DG1) is locally offline for reads
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f57466038e0 bufp:0x7f574199a000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f5746603d20 bufp:0x7f5741997000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.663499+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
NOTE: disk 1 (DG1_0001) in group 2 (DG1) is locally offline for writes
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f57466038e0 bufp:0x7f574199a000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f5746603d20 bufp:0x7f5741997000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.667890+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
2019-06-25T11:49:52.317435+08:00
ERROR: ORA-15130 in COD recovery for diskgroup 2/0x38008e3f (DG1)
ERROR: ORA-15130 thrown in RBAL for group number 2
2019-06-25T11:49:52.328473+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_9706.trc:
ORA-15130: diskgroup "DG1" is being dismounted
2019-06-25T11:49:52.357451+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_9706.trc:
ORA-15130: diskgroup "DG1" is being dismounted
2019-06-25T11:49:52.359170+08:00
NOTE: client +ASM2:+ASM:prod-cluster no longer has group 2 (DG1) mounted
NOTE: cache deleting context for group DG1 2/0x38008e3f
2019-06-25T11:49:52.471944+08:00
GMON dismounting group 2 at 12 for pid 45, osid 70764
2019-06-25T11:49:52.474613+08:00
NOTE: Disk DG1_0000 in mode 0x7f marked for de-assignment
NOTE: Disk DG1_0001 in mode 0x1 marked for de-assignment
NOTE: Disk DISK01 in mode 0x7f marked for de-assignment
2019-06-25T11:49:52.490181+08:00
SUCCESS: diskgroup DG1 was dismounted
NOTE: Notify CRSD to check and update diskgroup resource state
NOTE: diskgroup resource ora.DG1.dg state is update
2019-06-25T11:49:52.556236+08:00
SUCCESS: alter diskgroup DG1 dismount force /* ASM SERVER:939560511 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group DG1
2019-06-25T11:49:52.874415+08:00
SQL> ALTER DISKGROUP DG1 MOUNT  /* asm agent *//* {0:4:10} */ 
2019-06-25T11:49:52.930608+08:00
NOTE: cache registered group DG1 2/0x38008E42
NOTE: cache began mount (not first) of group DG1 2/0x38008E42
NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/VOTE02)
NOTE: Assigning number (2,2) to disk (/dev/oracleasm/disks/DISK01)
2019-06-25T11:49:53.222251+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
2019-06-25T11:49:53.224797+08:00
NOTE: cache dismounting (clean) group 2/0x38008E42 (DG1)
NOTE: messaging CKPT to quiesce pins Unix process pid: 10073, image: oracle@prod05 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 2/0x38008E42 (DG1)
NOTE: cache ending mount (fail) of group DG1 number=2 incarn=0x38008e42
NOTE: cache deleting context for group DG1 2/0x38008e42
2019-06-25T11:49:53.329552+08:00
GMON dismounting group 2 at 14 for pid 41, osid 10073
2019-06-25T11:49:53.330351+08:00
NOTE: Disk DG1_0001 in mode 0x8 marked for de-assignment
NOTE: Disk DISK01 in mode 0x8 marked for de-assignment
ERROR: diskgroup DG1 was not mounted
2019-06-25T11:49:53.372223+08:00
ASM Health Checker found 1 new failures
2019-06-25T11:49:53.375295+08:00
WARNING: Disk Group DG1 containing voting files is not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DG1" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DG1"

2019-06-25T11:49:53.377036+08:00
ERROR: ALTER DISKGROUP DG1 MOUNT  /* asm agent *//* {0:4:10} */

结论,数据库2个节点正常运行,第二个节点磁盘组被umount(alter diskgroup DG1 dismount force / ASM SERVER:939560511 /)

PROD04:

[grid@prod04 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.DATA.GHCHKPT.advm
               OFFLINE OFFLINE      prod04                   STABLE
               OFFLINE OFFLINE      prod05                   STABLE
ora.DATA.dg
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.DG1.dg
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  OFFLINE      prod05                   STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.chad
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.data.ghchkpt.acfs
               OFFLINE OFFLINE      prod04                   STABLE
               OFFLINE OFFLINE      prod05                   STABLE
ora.helper
               ONLINE  ONLINE       prod04                   IDLE,STABLE
               ONLINE  ONLINE       prod05                   IDLE,STABLE
ora.net1.network
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.ons
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.proxy_advm
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       prod04                   STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       prod05                   169.254.10.230 10.16
                                                             8.0.202,STABLE
ora.asm
      1        ONLINE  ONLINE       prod04                   Started,STABLE
      2        ONLINE  ONLINE       prod05                   Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       prod04                   STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       prod05                   Open,STABLE
ora.prod04.vip
      1        ONLINE  ONLINE       prod04                   STABLE
ora.prod05.vip
      1        ONLINE  ONLINE       prod05                   STABLE
ora.qosmserver
      1        ONLINE  ONLINE       prod04                   STABLE
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       prod04                   STABLE
--------------------------------------------------------------------------------
[grid@prod04 ~]$ 
[grid@prod04 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   6a99161549b74fe7bf1ed9c2ac8c32e2 (/dev/oracleasm/disks/VOTE01) [DG1]
 2. OFFLINE  23763a2e80bf4fcabffc35e320ce2e99 (/dev/oracleasm/disks/VOTE02) [DG1]
 3. ONLINE   7c6f5cf2b71c4f5dbfa91b2abfd08d3c (/dev/oracleasm/disks/DISK01) [DG1]
Located 3 voting disk(s).

PROD05:

[grid@prod05 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.DATA.GHCHKPT.advm
               OFFLINE OFFLINE      prod04                   STABLE
               OFFLINE OFFLINE      prod05                   STABLE
ora.DATA.dg
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.DG1.dg
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  OFFLINE      prod05                   STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.chad
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.data.ghchkpt.acfs
               OFFLINE OFFLINE      prod04                   STABLE
               OFFLINE OFFLINE      prod05                   STABLE
ora.helper
               ONLINE  ONLINE       prod04                   IDLE,STABLE
               ONLINE  ONLINE       prod05                   IDLE,STABLE
ora.net1.network
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.ons
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.proxy_advm
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       prod04                   STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       prod05                   169.254.10.230 10.16
                                                             8.0.202,STABLE
ora.asm
      1        ONLINE  ONLINE       prod04                   Started,STABLE
      2        ONLINE  ONLINE       prod05                   Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       prod04                   STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       prod05                   Open,STABLE
ora.prod04.vip
      1        ONLINE  ONLINE       prod04                   STABLE
ora.prod05.vip
      1        ONLINE  ONLINE       prod05                   STABLE
ora.qosmserver
      1        ONLINE  ONLINE       prod04                   STABLE
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       prod04                   STABLE
--------------------------------------------------------------------------------
[grid@prod05 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. OFFLINE  6a99161549b74fe7bf1ed9c2ac8c32e2 (/dev/oracleasm/disks/VOTE01) [DG1]
 2. ONLINE   23763a2e80bf4fcabffc35e320ce2e99 (/dev/oracleasm/disks/VOTE02) [DG1]
 3. ONLINE   7c6f5cf2b71c4f5dbfa91b2abfd08d3c (/dev/oracleasm/disks/DISK01) [DG1]
Located 3 voting disk(s).

网友评论

登录后评论
0/500
评论
snowofsummer
+ 关注