How to rebuild a degraded vdisk in a HP MSA P2000 / 2012fc; A disk has changed State to “LEFTOVER”; original spare disk showing state: “AVAIL” ; RAID5 become CRITICAL!

We had a power-event and MSA had a power-hiccup. It is possible some disks came-back online AFTER controller came-up. Initially – we could not get to MSA controllers either thru SMU/Web-interface or thru putty, or much worse – not even via the USB cable hooked-up from controller-A to Backup server. After moving the USB cable to controller-B (bottom) now we regained access thru serial port (COM3) via putty. Restarted management interfaces of both controllers, and now we can communicate with the controllers via putty, and via SMU/Web Management Interfaces.

Checking thru SMU, Narrows MSA P2000 indicated VDisk Failure, reporting following messages:
Degraded.
Virtual-disk is not fault tolerant.

Event 55: A disk drive reported a SMART event
Event 314: There is a problem with a FRU.

The Physical Hard Disk Drivelikely not failed, but the vdisk array was in a degraded state. One Physical disk (Enclosure#2, disk#4) was lit AMBER – indicating some sort of failure/malfunction.

The first thing that needs to be done, is themeta-data on the leftover disk needs to be cleared before it can re-join the RAID aseither a regular disk or a dedicated spare.

ClearingDisk Meta Data

Each disk has metadata that identifies whether the disk is a member of a vdisk, and identifies other members of that vdisk. If a disk’s metadata says the disk is a member of a vdisk but other members’ metadata say the disk isn’t a member, the disk becomes a LEFTOVER. The system overview and enclosure overview pages show the disk’s How-Used value as LEFTOVR. A leftover disk’s Fault/UID LED is illuminated amber.

Before you can use that disk in a new vdisk or as a spare, you must clear that disk’s metadata.

To Clear meta data from LEFTOVER disks

1.  In the Configuration View panel, right-click the system and then select Tools > Clear Disk Metadata.

2.  In the main panel, select disks to clear metadata from.

3.  Click Clear Metadata. When processing is complete a success dialog appears.

4.  Click Ok

Re-Join Physical Disk to VDisk RAID

1.  In the HP MSA Storage Management Utility, Right click on the affected vdisk and select Configuration > Manage Dedicated Spares

2.  Your disk should appear in the list ofavailable drives with a state of AVAIL. Tick the drive, then click the Modify Spares button.

3.  The disk will be re-joined to the array. Initially it may be listed as a spare, but the MSA will automatically re-join the disk as an active member of the RAID if this is how it was originally configured.

4.  The array will begin the Reconstruction process automatically. This can take a very long time depending on the size of your drives.

5.  In this situation, original SPARE has now become AVAIL. So, re-add this AVAIL disk as SPARE using above steps. After setting this as spare – this disk will now show a status of: VDISKSP or SPARE


Before State of disks – one disk stuck in LEFTOVER while original SPARE stuck as AVAIL state in enclosure#2 (Tier 2 Storage – RAID5 is now critical):


After performing this rebuilding vdisk after a disk stuck/failed in LEFTOVER state (metadata cleared):

Notice that the disks are now marked as Vdisk under Reconstruction. See RAID rebuild progress below:
# show vdisks

Name Size Free Own Pref RAID Disks Spr Chk Status Jobs Serial Number Drive Spin Down Spin Down Delay

------

ESX Prod Clstr 5995.1GB 0B B B RAID5 11 1 64k FTOL 00c0ffdb2868000084498b4d00000000 Disabled 0

Tier 2 Storage 9991.8GB 775.8GB A A RAID5 11 1 64k CRIT RCON 2% 00c0ffdb2f970000c84a924c00000000 Disabled 0

------

Success: Command completed successfully.

# show vdisks

Name Size Free Own Pref RAID Disks Spr Chk Status Jobs Serial Number Drive Spin Down Spin Down Delay

------

ESX Prod Clstr 5995.1GB 0B B B RAID5 11 1 64k FTOL 00c0ffdb2868000084498b4d00000000 Disabled 0

Tier 2 Storage 9991.8GB 775.8GB A A RAID5 11 1 64k CRIT RCON 19% 00c0ffdb2f970000c84a924c00000000 Disabled 0


After Vdisk/RAID-5 reconstruction finishes, VDISK status shows normal/FTOL:


Events occurred as these process steps performed, Explained in details: