VSAN-Troubleshooting-Reference-Manual

Recommendations

Info

Diagnostics and Troubleshooting Reference Manual – Virtual SAN What is outstanding I/O (OIO)? .................................................................................................................. 208 What is latency standard deviation (stddev)? ..................................................................................... 208 What to look for in the VSAN Client view? ............................................................................................. 208 VSAN CLIENT – FULL SIZE GRAPH .................................................................................................................. 209 NAVIGATING VSAN OBSERVER – VSAN DISKS ............................................................................................ 210 VSAN DISKS – FULL SIZE GRAPHS .................................................................................................................. 211 NAVIGATING VSAN OBSERVER – VSAN DISKS (DEEP-DIVE).................................................................... 212 WriteBuffer Fill ................................................................................................................................................... 213 Evictions................................................................................................................................................................. 213 VSAN DISKS (DEEP-DIVE) – HOST DISK-LAYER AGGREGATE STATS: FULL GRAPHS .............................. 214 Latency, IOPS and Bandwidth ..................................................................................................................... 215 RC Hit Rate ........................................................................................................................................................... 215 RC IOPS breakdown .......................................................................................................................................... 219 Evictions................................................................................................................................................................. 220 Bytes Read from invalidated cache lines/PLOG callback path to RC ........................................ 221 Capacity ................................................................................................................................................................. 221 VSAN DISKS (DEEP-DIVE) – DEVICE-LEVEL STATS: FULL GRAPHS ........................................................... 222 WriteBuffer........................................................................................................................................................... 222 A note on LLOG and PLOG ............................................................................................................................. 222 VSAN DISKS (DEEP-DIVE) – DEVICE LEVEL STATS: PHYSDISK ................................................................. 226 DiskLatency (ms) ............................................................................................................................................... 226 NAVIGATING VSAN OBSERVER – PCPU ........................................................................................................ 227 NAVIGATING VSAN OBSERVER – MEMORY .................................................................................................. 228 MEMORY – FULL SIZE GRAPHS ......................................................................................................................... 229 NAVIGATING VSAN OBSERVER – DOM OWNER .......................................................................................... 230 DOM OWNER – FULL SIZE GRAPHS ................................................................................................................ 232 RecovWrites ......................................................................................................................................................... 233 NAVIGATING VSAN OBSERVER – VM ............................................................................................................ 234 NAVIGATING VSAN OBSERVER – NETWORK ................................................................................................ 235 NETWORK – FULL GRAPHS ................................................................................................................................ 236 vmknic throughput ........................................................................................................................................... 237 vmknic errors ...................................................................................................................................................... 237 Pnic Tput ............................................................................................................................................................... 237 Pnic Errors ............................................................................................................................................................ 237 NAVIGATING VSAN OBSERVER - VSAN (MORE) – CONGESTION ............................................................. 238 VSAN (MORE) – CONGESTION – FULL GRAPHS ........................................................................................... 238 NAVIGATING VSAN OBSERVER - VSAN (MORE) – DISTRIBUTION .......................................................... 239 VSAN (MORE) – DISTRIBUTION – FULL GRAPHS ........................................................................................ 240 GENERATE A VSAN OBSERVER LOG BUNDLE ................................................................................................ 241 EXAMINE THE VSAN OBSERVER OFFLINE LOG BUNDLE ............................................................................. 241 GENERATE A FULL RAW STATS BUNDLE ......................................................................................................... 241 VSAN OBSERVER COMMAND LINE OPTIONS ................................................................................................. 242 16. TROUBLESHOOTING VIRTUAL SAN PERFORMANCE ....................................................... 243 VIRTUAL SAN PERFORMANCE EXPECTATIONS ............................................................................................. 243 I/O FLOW ............................................................................................................................................................ 243 Anatomy of a write in hybrid configurations ....................................................................................... 244 Destaging writes from flash to magnetic disk in hybrid configurations ................................. 245 Anatomy of a read in a hybrid configuration ....................................................................................... 247 Anatomy of a read in all-flash configurations ..................................................................................... 248 V M W A R E S T O R A G E B U D O C U M E N T A T I O N / 8
Diagnostics and Troubleshooting Reference Manual – Virtual SAN Anatomy of a write in all-flash configurations .................................................................................... 248 Virtual SAN caching algorithms ................................................................................................................. 248 Enhancements to caching algorithms in 6.0 ........................................................................................ 248 Latency consideration with distributed cache..................................................................................... 249 HOW VM STORAGE POLICIES IMPACT PERFORMANCE ................................................................................ 249 Where stripe width helps and where it may not ................................................................................. 249 A word of caution on flash read cache reservation ........................................................................... 249 AN OVERVIEW OF DIFFERENT WORKLOAD TYPES ......................................................................................... 250 Single workloads versus multiple workloads in a VM ...................................................................... 250 Single VM versus multiple VMs ................................................................................................................... 250 Steady state versus bursty (averages versus peaks) ......................................................................... 250 Random versus sequential............................................................................................................................. 251 Read versus write .............................................................................................................................................. 252 Where cache helps and where it may not .............................................................................................. 252 Working set size ................................................................................................................................................. 253 Guest file systems can matter ...................................................................................................................... 253 WHAT OPERATIONS CAN IMPACT PERFORMANCE? ...................................................................................... 253 Rebuilding/resyncing operations .............................................................................................................. 253 Observing rebuilding/resyncing operations ......................................................................................... 254 Backup/restore operations ........................................................................................................................... 256 vMotion .................................................................................................................................................................. 256 Virus Scans ............................................................................................................................................................ 257 NOTES ON USING IOMETER WITH VIRTUAL SAN ......................................................................................... 257 USING ESXTOP FOR PERFORMANCE MONITORING ......................................................................................... 259 USING RVC FOR PERFORMANCE MONITORING .............................................................................................. 261 vsan.vm_perf_stats ............................................................................................................................................ 261 USING VSAN OBSERVER FOR PERFORMANCE MONITORING ..................................................................... 261 17. VSAN OBSERVER CASE STUDIES ............................................................................................. 262 I – USING VSAN OBSERVER TO TROUBLESHOOT HIGH LATENCY .............................................................. 262 II – USING VSAN OBSERVER TO TROUBLESHOOT A NETWORK PERFORMANCE ISSUE .......................... 266 18. ENGAGING WITH GLOBAL SUPPORT SERVICES ................................................................ 272 ENSURE SCRATCH AND LOG FILES ARE ON PERSISTENT MEDIA .................................................................. 272 ENSURE VSAN TRACE FILES ARE ON PERSISTENT MEDIA .......................................................................... 272 NETDUMP ............................................................................................................................................................ 273 SYSLOG SERVER .................................................................................................................................................. 273 FILING A SUPPORT REQUEST WITH VMWARE ............................................................................................... 274 WHICH LOGS SHOULD BE GATHERED? ............................................................................................................ 274 HOW TO TRANSFER LOG FILES TO VMWARE? ............................................................................................... 274 APPENDIX A: USEFUL HP UTILITIES ............................................................................................ 275 HPSSACLI UTILITY ............................................................................................................................................... 275 CONTROLLER & DISK DRIVE INFORMATION VIA HPSSACLI ........................................................................ 279 esxcli hpssacli cmd -q "controller all show" .......................................................................................... 279 esxcli hpssacli cmd -q "controller all show status" ............................................................................ 279 esxcli hpssacli cmd -q "controller slot=0 show" .................................................................................. 279 esxcli hpssacli cmd -q "controller slot=0 show config detail" ....................................................... 280 esxcli hpssacli cmd -q "controller slot=0 show config" .................................................................... 284 esxcli hpssacli cmd -q "controller slot=0 Array A logicaldrive 1 show" ................................... 285 esxcli hpssacli cmd -q "controller slot=0 Array G modify led=on” .............................................. 286 V M W A R E S T O R A G E B U D O C U M E N T A T I O N / 9
Page 1 and 2: Diagnostics and Troubleshooting Gui
Page 3 and 4: Diagnostics and Troubleshooting Ref
Page 7: Diagnostics and Troubleshooting Ref
Page 59 and 60:
Diagnostics and Troubleshooting Ref
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121 and 122:
Page 123 and 124:
Page 125 and 126:
Page 127 and 128:
Page 129 and 130:
Page 131 and 132:
Page 133 and 134:
Page 135 and 136:
Page 137 and 138:
Page 139 and 140:
Page 141 and 142:
Page 143 and 144:
Page 145 and 146:
Page 147 and 148:
Page 149 and 150:
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185 and 186:
Page 187 and 188:
Page 189 and 190:
Page 191 and 192:
Page 193 and 194:
Page 195 and 196:
Page 197 and 198:
Page 199 and 200:
Page 201 and 202:
Page 203 and 204:
Page 205 and 206:
Page 207 and 208:
Page 209 and 210:
Page 211 and 212:
Page 213 and 214:
Page 215 and 216:
Page 217 and 218:
Page 219 and 220:
Page 221 and 222:
Page 223 and 224:
Page 225 and 226:
Page 227 and 228:
Page 229 and 230:
Page 231 and 232:
Page 233 and 234:
Page 235 and 236:
Page 237 and 238:
Page 239 and 240:
Page 241 and 242:
Page 243 and 244:
Page 245 and 246:
Page 247 and 248:
Page 249 and 250:
Page 251 and 252:
Page 253 and 254:
Page 255 and 256:
Page 257 and 258:
Page 259 and 260:
Page 261 and 262:
Page 263 and 264:
Page 265 and 266:
Page 267 and 268:
Page 269 and 270:
Page 271 and 272:
Page 273 and 274:
Page 275 and 276:
Page 277 and 278:
Page 279 and 280:
Page 281 and 282:
Page 283 and 284:
Page 285 and 286:
Page 287 and 288:
Page 289 and 290:
Page 291 and 292:
Page 293 and 294:
Page 295:
show all

VSAN-Troubleshooting-Reference-Manual

Create successful ePaper yourself

Delete template?

Save as template?