VSAN-Troubleshooting-Reference-Manual

Diagnostics and Troubleshooting Reference Manual – Virtual SAN
7. Understanding expected failure behavior
When doing failure testing with Virtual SAN, or indeed, when failures occur on the
cluster during production, it is important to understand the expected behavior for
different failure scenarios.
Disk is pulled unexpectedly from ESXi host
The Virtual SAN Administrators Guide provides guidelines on how to correctly
decommission a device from a disk group. VMware recommends following tried and
tested procedures for removing devices from Virtual SAN.
In Virtual SAN hybrid configurations, when a magnetic disk is unexpectedly pulled
from an ESXi hosts (where it is being used to contribute storage to Virtual SAN)
without first decommissioning the disk, all the Virtual SAN components that reside
on the disk go ABSENT and are inaccessible. The same holds true for Virtual SAN allflash
configurations when a flash device in the capacity layer is pulled.
The ABSENT state is chosen over DEGRADED because Virtual SAN knows the disk is
not lost, but rather just removed. If the disk gets reinserted into the server before
the 60-minute timeout, no harm is done, and Virtual SAN simply syncs it back up.
Virtual SAN returns to full redundancy without wasting resources on an expensive
rebuild.
Expected behaviors:
If the VM has a policy that includes NumberOfFailuresToTolerate=1 or greater,
the VM’s objects will still be accessible from another ESXi host in the Virtual
SAN cluster.
The disk state is marked as ABSENT and can be verified via vSphere web
client UI.
All in-flight I/O is halted while Virtual SAN reevaluates the availability of the
object without the failed component as part of the active set of components.
All components residing on this disk will be marked as ABSENT in the UI.
If Virtual SAN concludes that the object is still available (based on available
full mirror copy and witness/votes), in-flight I/O is restarted.
The typical time from physical removal of the drive, Virtual SAN processing
this event, marking the component ABSENT halting and restoring I/O flow is
approximately 5-7 seconds.
If the same disk is placed back on the same host within 60 minutes, no new
components will be re-built.
If 60 minutes pass, and the original disk have not been reinserted in the host,
components on the removed disk will be built elsewhere in the cluster, if
V M W A R E S T O R A G E B U D O C U M E N T A T I O N / 62