VSAN-Troubleshooting-Reference-Manual

Diagnostics and Troubleshooting Reference Manual – Virtual SAN
Physical network switch configurations and flow control
There have been situations where misbehaving network switches have led to Virtual
SAN network outages. Symptoms include hosts unable to communicate, and
exceedingly high latency reported for virtual machine I/O in VSAN Observer.
However when latency is examined at the VSAN Disks layer, there is no latency,
which immediately points to latency being incurred at the network layer.
In one case, it was observed that the physical network switch in question was
sending excessive amounts of Pause frames. Pause frames are a flow control
mechanism that is designed to stop or reduce the transmission of packets for an
interval of time. This behavior negatively impacted the Virtual SAN network
performance.
ethtool
There is a command on the ESXi host called ethtool to check for flow control. Here is
an example output:
~ # ethtool -a vmnic4
Pause parameters for vmnic4:
Autonegotiate: on
RX:
off
TX:
off
This output shows that auto-negotiate is set to on, which is recommended for ESXi
host NICs, but that there is no flow control enabled on the switch (RX and TX are
both off).
In the example outage discussed earlier, there were excessive amounts of pause
frames in the RX field, with values in the millions. In this case, one troubleshooting
step might be to disable the flow control mechanisms on the switch while further
investigation into the root cause takes place.
Physical network switch feature interoperability
There have been situations where certain features, when enabled on a physical
network switch, did not interoperate correctly. In one example, a customer
attempted to use multicast with jumbo frames, and because of the inability of the
network switch to handle both these features, it impacted the whole of the Virtual
SAN network. Note that many other physical switches handled this perfectly; this
was an issue with one switch vendor only.
Pay due diligence to whether or not the physical network switch has the ability to
support multiple network features enabled concurrently.
V M W A R E S T O R A G E B U D O C U M E N T A T I O N / 1 2 2