测试模拟环境
create diskgroup dg1 normal redundancy
failgroup aa disk '/dev/oracleasm/disks/VOTE01' # 存储A
failgroup bb disk '/dev/oracleasm/disks/VOTE02' # 存储B
quorum failgroup zc disk '/dev/oracleasm/disks/DISK01' ## 仲裁存储
ATTRIBUTE 'compatible.asm' = '18.0.0.0.0', 'compatible.rdbms' = '18.0.0.0.0';数据库版本
SQL*Plus: Release 18.0.0.0.0 - Production on Tue Jun 25 12:03:41 2019
Version 18.3.0.0.0prod04 crs log:
2019-06-25 11:49:19.879 [OCSSD(4103)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE02 will be considered not functional in 99140 milliseconds.
2019-06-25 11:49:23.789 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 103770 msecs.
2019-06-25 11:49:23.791 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 103720 msecs.
2019-06-25 11:49:31.793 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 111780 msecs.
2019-06-25 11:49:31.795 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 111730 msecs.
2019-06-25 11:49:39.798 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 119780 msecs.
2019-06-25 11:49:39.799 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 119730 msecs.
2019-06-25 11:49:47.802 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 127780 msecs.
2019-06-25 11:49:47.803 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 127740 msecs.
2019-06-25 11:49:49.418 [OCSSD(4103)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE02; details at (:CSSNM00059:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc.
2019-06-25 11:49:49.418 [OCSSD(4103)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE02; details at (:CSSNM00060:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc.
2019-06-25 11:49:51.308 [OCSSD(4103)]CRS-1650: Configuration change 1561434591:0 rejected by node 2 prod05; details at (:CSSNM000066:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc
2019-06-25 11:49:51.352 [OCSSD(4103)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:49:53.541 [CRSD(4667)]CRS-2878: Failed to restart resource 'ora.DG1.dg'
2019-06-25 11:49:53.543 [CRSD(4667)]CRS-2769: Unable to failover resource 'ora.DG1.dg'.
2019-06-25 11:49:54.575 [OCSSD(4103)]CRS-1650: Configuration change 1561434594:0 rejected by node 2 prod05; details at (:CSSNM000066:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc
2019-06-25 11:49:54.620 [OCSSD(4103)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:50:09.811 [OCSSD(4103)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE02 will be considered not functional in 49210 milliseconds.
2019-06-25 11:50:39.821 [OCSSD(4103)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE02 will be considered not functional in 19200 milliseconds.
2019-06-25 11:50:59.827 [OCSSD(4103)]CRS-1604: CSSD voting file is offline: /dev/oracleasm/disks/VOTE02; details at (:CSSNM00058:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc.
2019-06-25 11:50:59.830 [OCSSD(4103)]CRS-1672: The number of voting files currently available 2 has fallen to the minimum number of voting files required 2. Further reduction in voting files will result in eviction and loss of functionalityprod05 asm log:
2019-06-25T11:49:50.946702+08:00
WARNING: Write Failed. group:2 disk:1 AU:1 offset:1044480 size:4096
path:/dev/oracleasm/disks/VOTE02
incarnation:0xf0f0dce6 asynchronous result:'I/O error'
subsys:System krq:0x7fb6e08d3c38 bufp:0x7fb6df381000 osderr1:0x69b5 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Hbeat write to PST disk 1.4042317030 in group 2 failed. [2]
2019-06-25T11:49:50.987326+08:00
NOTE: checking PST: grp = 2
2019-06-25T11:49:50.988005+08:00
GMON checking disk modes for group 2 at 28 for pid 41, osid 57148
2019-06-25T11:49:50.994499+08:00
NOTE: checking PST for grp 2 done.
2019-06-25T11:49:50.994807+08:00
NOTE: sending set offline flag message (1518616090) to 1 disk(s) in group 2
2019-06-25T11:49:50.995444+08:00
WARNING: Disk 1 (DG1_0001) in group 2 mode 0x7f is now being offlined
2019-06-25T11:49:51.006032+08:00
NOTE: initiating PST update: grp 2 (DG1), dsk = 1/0xf0f0dce6, mask = 0x6a, op = clear mandatory
2019-06-25T11:49:51.006479+08:00
GMON updating disk modes for group 2 at 29 for pid 41, osid 57148
WARNING: Write Failed. group:2 disk:1 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE02
incarnation:0xf0f0dce6 synchronous result:'I/O error'
subsys:System krq:0x7fb6df36b668 bufp:0x7fb6df34f000 osderr1:0x534b4950 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.139504+08:00
Received dirty detach msg from inst 2 for dom 2
2019-06-25T11:49:51.168873+08:00
NOTE: Attempting voting file refresh on diskgroup DG1
NOTE: Refresh completed on diskgroup DG1. Found 3 voting file(s).
NOTE: Voting file relocation is required in diskgroup DG1
2019-06-25T11:49:51.182641+08:00
List of instances (total 2) :
1 2
My inst 1
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
Global Resource Directory partially frozen for dirty detach
dirty detach - domain 2 invalid = TRUE
0 GCS resources traversed, 0 cancelled
2019-06-25T11:49:51.235840+08:00
NOTE: Attempting voting file relocation on diskgroup DG1
2019-06-25T11:49:51.249004+08:00
Dirty Detach Reconfiguration complete (total time 0.1 secs)
2019-06-25T11:49:51.251495+08:00
NOTE: SMON detected lock domain 2 invalid at system inc 4 06/25/19 11:49:51
2019-06-25T11:49:51.372440+08:00
NOTE: Failed voting file relocation on diskgroup DG1
2019-06-25T11:49:51.478897+08:00
NOTE: SMON starting instance recovery of group DG1 domain 2 inc 4 (mounted) at 06/25/19 11:49:51
NOTE: 06/25/19 11:49:51 DG1.F1X0 found on disk 0 au 10 fcn 0.0 datfmt 1
NOTE: SMON skipping disk 1 (mode=00000015)
NOTE: instance recovery of group DG1 will recover thread=2 ckpt=2.0 domain=2 inc#=4 instnum=2
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 2 (DG1)
validate pdb 2, flags x0, valid 0, pdb flags xa4
* validated domain 2, flags = 0xa0
2019-06-25T11:49:51.515918+08:00
NOTE: SMON successfully validated lock domain 2 inc 4 (DG1)
2019-06-25T11:49:51.575068+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_smon_4494.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/VOTE02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: Write Failed. group:2 disk:1 AU:60 offset:0 size:4096
path:Unknown disk
incarnation:0xf0f0dce6 asynchronous result:'I/O error'
subsys:Unknown library krq:0x7fef485bd680 bufp:0x7fef47736000 osderr1:0x434c5344 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
NOTE: advancing ckpt for group 2 (DG1) thread=2 ckpt=2.0 domain inc# 4
NOTE: cache initiating offline of disk 1 group DG1
NOTE: initiating PST update: grp 2 (DG1), dsk = 1/0xf0f0dce6, mask = 0x6a, op = clear mandatory
2019-06-25T11:49:51.620068+08:00
GMON updating disk modes for group 2 at 30 for pid 23, osid 4494
2019-06-25T11:49:51.623397+08:00
NOTE: PST update grp = 2 completed successfully
NOTE: ospid 4494 initiating cluster wide offline of the following disks from gn 2: 1
2019-06-25T11:49:51.629048+08:00
NOTE: disk 1 (DG1_0001) in group 2 (DG1) is locally offline for writes
2019-06-25T11:49:51.633752+08:00
NOTE: ospid 4494 finished cluster wide offline, gn 2
NOTE: successfully wrote at least one mirror side for diskgroup DG1
2019-06-25T11:49:51.634353+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_smon_4494.trc:
ORA-15080: synchronous I/O operation failed to write block 10752 of disk 1 in disk group DG1
2019-06-25T11:49:51.634652+08:00
NOTE: SMON did instance recovery for group DG1 domain 2
2019-06-25T11:49:52.442330+08:00
NOTE: PST update grp = 2 completed successfully
NOTE: initiating PST update: grp 2 (DG1), dsk = 1/0xf0f0dce6, mask = 0x7e, op = clear mandatory
2019-06-25T11:49:52.443344+08:00
GMON updating disk modes for group 2 at 31 for pid 41, osid 57148
2019-06-25T11:49:52.449953+08:00
NOTE: cache closing disk 1 of grp 2: DG1_0001
2019-06-25T11:49:52.459736+08:00
NOTE: cache closing disk 1 of grp 2: (not open) DG1_0001
2019-06-25T11:49:52.461747+08:00
NOTE: PST update grp = 2 completed successfully
2019-06-25T11:49:53.999944+08:00
WARNING: Started Drop Disk Timeout for Disk 1 (DG1_0001) in group 2 with a value 43200
WARNING: Disk 1 (DG1_0001) in group 2 will be dropped in: (43200) secs on ASM inst 1
2019-06-25T11:49:54.475008+08:00
NOTE: Attempting voting file refresh on diskgroup DG1
NOTE: Refresh completed on diskgroup DG1. Found 3 voting file(s).
NOTE: Voting file relocation is required in diskgroup DG1
NOTE: Attempting voting file relocation on diskgroup DG1
NOTE: Failed voting file relocation on diskgroup DG1
2019-06-25T11:59:05.055729+08:00
WARNING: Disk 1 (DG1_0001) in group 2 will be dropped in: (42649) secs on ASM inst 1prod05 crs log:
2019-06-25 11:49:19.272 [OCSSD(7735)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE01 will be considered not functional in 99990 milliseconds.
2019-06-25 11:49:24.324 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 103750 msecs.
2019-06-25 11:49:24.325 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 104060 msecs.
2019-06-25 11:49:32.329 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 111750 msecs.
2019-06-25 11:49:32.331 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 112070 msecs.
2019-06-25 11:49:40.334 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 119760 msecs.
2019-06-25 11:49:40.336 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 120070 msecs.
2019-06-25 11:49:48.340 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 127760 msecs.
2019-06-25 11:49:48.342 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 128080 msecs.
2019-06-25 11:49:49.811 [OCSSD(7735)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE01; details at (:CSSNM00059:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc.
2019-06-25 11:49:49.812 [OCSSD(7735)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE01; details at (:CSSNM00060:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc.
2019-06-25 11:49:51.313 [OCSSD(7735)]CRS-1641: Rejecting configuration change 1561434591:0 because only 1 of the required 0 voting files of the new configuration were located; details at (:CSSNM00026:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc
2019-06-25 11:49:51.383 [OCSSD(7735)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:49:53.380 [ORAAGENT(8566)]CRS-5017: The resource action "ora.DG1.dg start" encountered the following error:
2019-06-25 11:49:53.380+ORA-15032: not all alterations performed
ORA-15017: diskgroup "DG1" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DG1"
. For details refer to "(:CLSN00107:)" in "/u01/app/grid_base/diag/crs/prod05/crs/trace/crsd_oraagent_grid.trc".
2019-06-25 11:49:54.596 [OCSSD(7735)]CRS-1641: Rejecting configuration change 1561434594:0 because only 1 of the required 0 voting files of the new configuration were located; details at (:CSSNM00026:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc
2019-06-25 11:49:54.650 [OCSSD(7735)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:49:58.836 [OHASD(2593)]CRS-2771: Maximum restart attempts reached for resource 'ora.storage'; will not restart.
2019-06-25 11:50:10.232 [OCSSD(7735)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE01 will be considered not functional in 49030 milliseconds.
2019-06-25 11:50:40.243 [OCSSD(7735)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE01 will be considered not functional in 19020 milliseconds.
2019-06-25 11:51:00.250 [OCSSD(7735)]CRS-1604: CSSD voting file is offline: /dev/oracleasm/disks/VOTE01; details at (:CSSNM00058:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc.
2019-06-25 11:51:00.251 [OCSSD(7735)]CRS-1672: The number of voting files currently available 2 has fallen to the minimum number of voting files required 2. Further reduction in voting files will result in eviction and loss of functionality
2019-06-25 11:55:29.161 [OHASD(2593)]CRS-2771: Maximum restart attempts reached for resource 'ora.storage'; will not restart.
2019-06-25 12:00:59.804 [ORAROOTAGENT(3154)]CRS-5019: All OCR locations are on ASM disk groups [DG1], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid_base/diag/crs/prod05/crs/trace/ohasd_orarootagent_root.trc".prod06 asm log:
2019-06-25T11:49:51.029835+08:00
WARNING: Disk 1 (DG1_0001) in group 2 mode 0x7f is now being offlined
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
path:/dev/oracleasm/disks/VOTE01
incarnation:0xf0f07ed3 synchronous result:'I/O error'
subsys:System krq:0x7f57466038e0 bufp:0x7f574199a000 osderr1:0x69b5 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE01
incarnation:0xf0f07ed3 synchronous result:'I/O error'
subsys:System krq:0x7f5746603d20 bufp:0x7f5741997000 osderr1:0x69b5 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.051804+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
2019-06-25T11:49:51.093245+08:00
NOTE: cache dismounting (not clean) group 2/0x38008E3F (DG1)
NOTE: messaging CKPT to quiesce pins Unix process pid: 70764, image: oracle@prod05 (B000)
2019-06-25T11:49:51.097051+08:00
NOTE: halting all I/Os to diskgroup 2 (DG1)
2019-06-25T11:49:51.099532+08:00
NOTE: LGWR doing non-clean dismount of group 2 (DG1) thread 2
NOTE: LGWR sync ABA=1.10750 last written ABA 1.10750
2019-06-25T11:49:51.100491+08:00
NOTE: initiating dirty detach from lock domain 2
2019-06-25T11:49:51.169386+08:00
kjbdomdet send to inst 1
detach from dom 2, sending detach message to inst 1
2019-06-25T11:49:51.231898+08:00
List of instances (total 2) :
1 2
My inst 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
Global Resource Directory partially frozen for dirty detach
dirty detach - domain 2 invalid = TRUE
0 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete (total time 0.0 secs)
2019-06-25T11:49:51.300226+08:00
WARNING: dirty detached from lock domain 2
NOTE: CKPT released recovery enqueue for thread 2 group 2 (DG1)
2019-06-25T11:49:51.333377+08:00
NOTE: cache dismounted group 2/0x38008E3F (DG1)
2019-06-25T11:49:51.336638+08:00
SQL> alter diskgroup DG1 dismount force /* ASM SERVER:939560511 */
2019-06-25T11:49:51.338336+08:00
NOTE: client +ASM2:+ASM:prod-cluster no longer has group 1 (DATA) mounted
2019-06-25T11:49:51.545472+08:00
* instance 1 validates domain 2
2019-06-25T11:49:51.658363+08:00
NOTE: disk 1 (DG1_0001) in group 2 (DG1) is locally offline for reads
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
path:/dev/oracleasm/disks/VOTE01
incarnation:0xf0f07ed3 synchronous result:'I/O error'
subsys:System krq:0x7f57466038e0 bufp:0x7f574199a000 osderr1:0x69b5 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE01
incarnation:0xf0f07ed3 synchronous result:'I/O error'
subsys:System krq:0x7f5746603d20 bufp:0x7f5741997000 osderr1:0x69b5 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.663499+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
NOTE: disk 1 (DG1_0001) in group 2 (DG1) is locally offline for writes
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
path:/dev/oracleasm/disks/VOTE01
incarnation:0xf0f07ed3 synchronous result:'I/O error'
subsys:System krq:0x7f57466038e0 bufp:0x7f574199a000 osderr1:0x69b5 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE01
incarnation:0xf0f07ed3 synchronous result:'I/O error'
subsys:System krq:0x7f5746603d20 bufp:0x7f5741997000 osderr1:0x69b5 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.667890+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
2019-06-25T11:49:52.317435+08:00
ERROR: ORA-15130 in COD recovery for diskgroup 2/0x38008e3f (DG1)
ERROR: ORA-15130 thrown in RBAL for group number 2
2019-06-25T11:49:52.328473+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_9706.trc:
ORA-15130: diskgroup "DG1" is being dismounted
2019-06-25T11:49:52.357451+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_9706.trc:
ORA-15130: diskgroup "DG1" is being dismounted
2019-06-25T11:49:52.359170+08:00
NOTE: client +ASM2:+ASM:prod-cluster no longer has group 2 (DG1) mounted
NOTE: cache deleting context for group DG1 2/0x38008e3f
2019-06-25T11:49:52.471944+08:00
GMON dismounting group 2 at 12 for pid 45, osid 70764
2019-06-25T11:49:52.474613+08:00
NOTE: Disk DG1_0000 in mode 0x7f marked for de-assignment
NOTE: Disk DG1_0001 in mode 0x1 marked for de-assignment
NOTE: Disk DISK01 in mode 0x7f marked for de-assignment
2019-06-25T11:49:52.490181+08:00
SUCCESS: diskgroup DG1 was dismounted
NOTE: Notify CRSD to check and update diskgroup resource state
NOTE: diskgroup resource ora.DG1.dg state is update
2019-06-25T11:49:52.556236+08:00
SUCCESS: alter diskgroup DG1 dismount force /* ASM SERVER:939560511 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group DG1
2019-06-25T11:49:52.874415+08:00
SQL> ALTER DISKGROUP DG1 MOUNT /* asm agent *//* {0:4:10} */
2019-06-25T11:49:52.930608+08:00
NOTE: cache registered group DG1 2/0x38008E42
NOTE: cache began mount (not first) of group DG1 2/0x38008E42
NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/VOTE02)
NOTE: Assigning number (2,2) to disk (/dev/oracleasm/disks/DISK01)
2019-06-25T11:49:53.222251+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
2019-06-25T11:49:53.224797+08:00
NOTE: cache dismounting (clean) group 2/0x38008E42 (DG1)
NOTE: messaging CKPT to quiesce pins Unix process pid: 10073, image: oracle@prod05 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 2/0x38008E42 (DG1)
NOTE: cache ending mount (fail) of group DG1 number=2 incarn=0x38008e42
NOTE: cache deleting context for group DG1 2/0x38008e42
2019-06-25T11:49:53.329552+08:00
GMON dismounting group 2 at 14 for pid 41, osid 10073
2019-06-25T11:49:53.330351+08:00
NOTE: Disk DG1_0001 in mode 0x8 marked for de-assignment
NOTE: Disk DISK01 in mode 0x8 marked for de-assignment
ERROR: diskgroup DG1 was not mounted
2019-06-25T11:49:53.372223+08:00
ASM Health Checker found 1 new failures
2019-06-25T11:49:53.375295+08:00
WARNING: Disk Group DG1 containing voting files is not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DG1" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DG1"
2019-06-25T11:49:53.377036+08:00
ERROR: ALTER DISKGROUP DG1 MOUNT /* asm agent *//* {0:4:10} */结论，数据库2个节点正常运行，第二个节点磁盘组被umount(alter diskgroup DG1 dismount force / ASM SERVER:939560511 /)
PROD04:
[grid@prod04 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.DATA.GHCHKPT.advm
OFFLINE OFFLINE prod04 STABLE
OFFLINE OFFLINE prod05 STABLE
ora.DATA.dg
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.DG1.dg
ONLINE ONLINE prod04 STABLE
ONLINE OFFLINE prod05 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.chad
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.data.ghchkpt.acfs
OFFLINE OFFLINE prod04 STABLE
OFFLINE OFFLINE prod05 STABLE
ora.helper
ONLINE ONLINE prod04 IDLE,STABLE
ONLINE ONLINE prod05 IDLE,STABLE
ora.net1.network
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.ons
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.proxy_advm
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE prod04 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE prod05 169.254.10.230 10.16
8.0.202,STABLE
ora.asm
1 ONLINE ONLINE prod04 Started,STABLE
2 ONLINE ONLINE prod05 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE prod04 STABLE
ora.mgmtdb
1 ONLINE ONLINE prod05 Open,STABLE
ora.prod04.vip
1 ONLINE ONLINE prod04 STABLE
ora.prod05.vip
1 ONLINE ONLINE prod05 STABLE
ora.qosmserver
1 ONLINE ONLINE prod04 STABLE
ora.rhpserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE prod04 STABLE
--------------------------------------------------------------------------------
[grid@prod04 ~]$
[grid@prod04 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 6a99161549b74fe7bf1ed9c2ac8c32e2 (/dev/oracleasm/disks/VOTE01) [DG1]
2. OFFLINE 23763a2e80bf4fcabffc35e320ce2e99 (/dev/oracleasm/disks/VOTE02) [DG1]
3. ONLINE 7c6f5cf2b71c4f5dbfa91b2abfd08d3c (/dev/oracleasm/disks/DISK01) [DG1]
Located 3 voting disk(s).PROD05:
[grid@prod05 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.DATA.GHCHKPT.advm
OFFLINE OFFLINE prod04 STABLE
OFFLINE OFFLINE prod05 STABLE
ora.DATA.dg
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.DG1.dg
ONLINE ONLINE prod04 STABLE
ONLINE OFFLINE prod05 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.chad
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.data.ghchkpt.acfs
OFFLINE OFFLINE prod04 STABLE
OFFLINE OFFLINE prod05 STABLE
ora.helper
ONLINE ONLINE prod04 IDLE,STABLE
ONLINE ONLINE prod05 IDLE,STABLE
ora.net1.network
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.ons
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
ora.proxy_advm
ONLINE ONLINE prod04 STABLE
ONLINE ONLINE prod05 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE prod04 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE prod05 169.254.10.230 10.16
8.0.202,STABLE
ora.asm
1 ONLINE ONLINE prod04 Started,STABLE
2 ONLINE ONLINE prod05 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE prod04 STABLE
ora.mgmtdb
1 ONLINE ONLINE prod05 Open,STABLE
ora.prod04.vip
1 ONLINE ONLINE prod04 STABLE
ora.prod05.vip
1 ONLINE ONLINE prod05 STABLE
ora.qosmserver
1 ONLINE ONLINE prod04 STABLE
ora.rhpserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE prod04 STABLE
--------------------------------------------------------------------------------
[grid@prod05 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. OFFLINE 6a99161549b74fe7bf1ed9c2ac8c32e2 (/dev/oracleasm/disks/VOTE01) [DG1]
2. ONLINE 23763a2e80bf4fcabffc35e320ce2e99 (/dev/oracleasm/disks/VOTE02) [DG1]
3. ONLINE 7c6f5cf2b71c4f5dbfa91b2abfd08d3c (/dev/oracleasm/disks/DISK01) [DG1]
Located 3 voting disk(s).
