VSAN-Troubleshooting-Reference-Manual

Diagnostics and Troubleshooting Reference Manual – Virtual SAN
2014-08-24T17:00:30.912Z cpu33:7027198)WARNING: LSOM: LSOMEventNotify:4570: VSAN
device 52378176-a9da-7bce-0526-cdf1d863b3b5 is under permanent error.
2014-08-24T17:00:30.912Z cpu33:7027198)WARNING: LSOM: RCVmfsIoCompletion:99:
Throttled: VMFS IO failed. Wake up 0x4136af9a69c0 with status Maximum kernel-level
retries exceeded
2014-08-24T17:00:30.912Z cpu33:7027198)WARNING: LSOM: RCDrainAfterBERead:5070:
Changing the status of child state from Success to Maximum kernel-level retries
exceeded
Eventually, the firmware reset on the controller completed, but by this time, it is too
late and Virtual SAN had already marked the disks as failed:
2014-08-24T17:00:49.279Z cpu21:33542)megasas: FW now in Ready state
2014-08-24T17:00:49.299Z cpu21:33542)megasas:IOC Init cmd success
2014-08-24T17:00:49.320Z cpu36:33542)megaraid_sas: Reset successful.
When a controller ‘wedges’ like this, Virtual SAN will retry I/O for a finite amount of
time. In this case, it took a full 24 seconds (2400ms) for the adapter to come back
online after resetting. This was too long for Virtual SAN, which meant that the
maximum retries threshold had been exceeded. This in turn led to Virtual SAN
marking the disks as DEGRADED.
Virtual SAN is responding as designed here. The problem is that the firmware
crashed. This particular issue was resolved by using recommended versions of
MegaRAID driver and firmware as per the VMware Compatibility Guide.
Storage controller replacement
In general, controller replacement should be for the same make and model and
administrators should not be swapping a pass-through controller with a RAID 0
controller or vice-versa.
Expectations when a drive is reporting errors
In this scenario, a disk drive is reporting errors due to bad blocks. If a read I/O
accessing component data on behalf of a virtual machine fails in such a manner,
Virtual SAN will check other replicas of the component to satisfy the read. If another
mirror can satisfy the read, Virtual SAN will attempt to write the good data onto the
disk that reported the bad read. If this procedure succeeds, the disk does not enter a
DEGRADED state. Note that this is not the behavior if we get a read error when
accessing Virtual SAN metadata. If a read of metadata, or any write fails, Virtual SAN
will mark all components of the disk as DEGRADED. It treats the disk as failed and
the data is no longer usable. Upon entering the DEGRADED state, Virtual SAN will
restore I/O flow immediately (by taking the bad component out of the active set of
the effected object) and try to re-protect the object by creating a new replica of the
component somewhere else in the cluster.
V M W A R E S T O R A G E B U D O C U M E N T A T I O N / 1 6 2