今天一个HP MSA2312FC的存储发送了离奇的事情，多个VD上的多个磁盘状态变成了LEFTOVER。
造成的结果是多个VD变成QTOF状态如下 :
# show vd
Name Size Free Own Pref RAID Disks Spr Chk Status Jobs
Serial Number
------------------------------------------------------------------------
vd01 3996.7GB 751.5MB A A RAID5 5 0 64k QTOF
00c0ff10386b0000b519384c00000000
vd02 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b10000dd19384c00000000
vd03 3996.7GB 751.5MB A A RAID5 5 0 64k FTOL
00c0ff10386b0000f919384c00000000
vd04 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b100002d1a384c00000000
vd05 3996.7GB 751.5MB A A RAID5 5 0 64k QTOF
00c0ff10386b0000c19f554e00000000
vd06 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b10000ce9f554e00000000
vd07 3996.7GB 751.5MB A A RAID5 5 0 64k QTOF
00c0ff10386b0000da9f554e00000000
vd08 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b10000fc9f554e00000000
------------------------------------------------------------------------
在执行rescan之后，多个磁盘被找回，VD恢复FTOL。但是vd01上面还是有4块盘是leftover的状态。
根据HP工程师的指导，关闭所有的登录MSA2312FC的WEB页面。然后通过命令行连接到这台msa2312fc。
执行
# trust enable
# trust vdisk vd01
报错
Error: Command failed. (vd01) - Vdisk is not online or fault tolerant. Cannot be trusted.
这下搞得比较崩溃了，HP对CASE做了升级。
新的解决方案出来了。先去WEB页面解除VD01的隔离。如下。
右键点击vd01,选择Tools -> Dequarantine Vdisk
按照指示解除vd01的隔离。
然后去命令行看vd01的状态会变成OFFL
# show vd
Name Size Free Own Pref RAID Disks Spr Chk Status Jobs
Serial Number
------------------------------------------------------------------------
vd01 3996.7GB 751.5MB A A RAID5 5 0 64k OFFL
00c0ff10386b0000b519384c00000000
vd02 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b10000dd19384c00000000
vd03 3996.7GB 751.5MB A A RAID5 5 0 64k FTOL
00c0ff10386b0000f919384c00000000
vd04 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b100002d1a384c00000000
vd05 3996.7GB 751.5MB A A RAID5 5 0 64k FTOL VRSC 56%
00c0ff10386b0000c19f554e00000000
vd06 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b10000ce9f554e00000000
vd07 3996.7GB 751.5MB A A RAID5 5 0 64k FTOL VRSC 59%
00c0ff10386b0000da9f554e00000000
vd08 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b10000fc9f554e00000000
------------------------------------------------------------------------
然后再到命令行执行
# trust vdisk vd01
VD恢复为FTOL状态。
# show vd
Name Size Free Own Pref RAID Disks Spr Chk Status Jobs
Serial Number
------------------------------------------------------------------------
vd01 3996.7GB 751.5MB A A RAID5 5 0 64k FTOL
00c0ff10386b0000b519384c00000000
vd02 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b10000dd19384c00000000
vd03 3996.7GB 751.5MB A A RAID5 5 0 64k FTOL
00c0ff10386b0000f919384c00000000
vd04 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b100002d1a384c00000000
vd05 3996.7GB 751.5MB A A RAID5 5 0 64k FTOL VRSC 56%
00c0ff10386b0000c19f554e00000000
vd06 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b10000ce9f554e00000000
vd07 3996.7GB 751.5MB A A RAID5 5 0 64k FTOL VRSC 59%
00c0ff10386b0000da9f554e00000000
vd08 3996.7GB 751.5MB B B RAID5 5 0 64k FTOL
00c0ff1035b10000fc9f554e00000000
------------------------------------------------------------------------
据HP称，这样操作可能导致部分数据丢失，可能指CACHE的脏数据, 或者是leftover掉的磁盘上的坏数据。
另外HP建议观察几天，没有问题的话升级一下固件。
另外如果正常情况下的leftover, 例如硬盘的错误次数达到一定的阈值, 这种情况下建议备份数据,重建vdisk, 再恢复.
卷的WR Policy可能会被修改为write-through, 后续可能需要手工开启.
trust命令 :
# trust
DESCRIPTION
Enables an offline vdisk to be brought online for emergency data collection.
This command must be enabled before each use.
Caution: This command can cause unstable operation and data loss if used
improperly. It is intended for disaster recovery only.
The trust command resynchronizes the time and date stamp and any other metadata
on a bad disk disk. This makes the disk an active member of the vdisk again.
You might need to do this when:
- One or more disks in a vdisk start up more slowly or were powered on after
the rest of the disks in the vdisk. This causes the date and time stamps to
differ, which the system interprets as a problem with the "late" disks.
In this case, the vdisk functions normally after being trusted.
- A vdisk is offline because a disk is failing, you have no data backup, and
you want to try to recover the data from the vdisk. In this case, trust may
work, but only as long as the failing disk continues to operate.
When the "trusted" vdisk is back online, back up its data and audit the data
to make sure that it is intact. Then delete that vdisk, create a new vdisk, and
restore data from the backup to the new vdisk. Using a trusted vdisk is only a
disaster-recovery measure; the vdisk has no tolerance for any additional
failures.
INPUT
To enable the trust command:
trust enable
To trust a vdisk:
trust vdisk <vdisk>
enable
Enables the trust command before use.
vdisk <vdisk>
Name or serial number of the vdisk to trust. For syntax, type "help syntax".
EXAMPLE
Enable the trust command and then trust vdisk VD1:
# trust enable
Success: Command completed successfully.
# trust vdisk VD1
Success: Command completed successfully.
