VSAN-Troubleshooting-Reference-Manual

Recommendations

Info

Diagnostics and Troubleshooting Reference Manual – Virtual SAN Host: 172.32.0.3 ICMPSeq: 0 Received Bytes: 64 Roundtrip Time MS: 372 TTL: 64 Detail: Dup: false Host: 172.32.0.3 ICMPSeq: 1 Received Bytes: 64 Roundtrip Time MS: 186 TTL: 64 Detail: Dup: false Host: 172.32.0.3 ICMPSeq: 2 Received Bytes: 64 Roundtrip Time MS: 232 Checking multicast settings Multicast configurations have been one of the most problematic issues for initial Virtual SAN deployment. One of the simplest ways to verify if multicast is working correctly in your Virtual SAN environment is by using the tcpdump-uw command. This command is available from the command line of the ESXi hosts. tcpdump-uw –i vmk2 udp port 23451 –v This tcpdump command will show if the master is correctly sending multicast packets (port and IP info) and if all other hosts in the cluster are receiving them. On the master, this command will show the packets being sent out to the multicast address. On all other hosts, the same exact packets should be seen (from the master to the multicast address). If they are not seen, multicast is not working correctly. Run the tcpdump-uw command shown here on any host in the cluster and the heartbeats should be seen coming from the master, which in this case is on IP address 172.32.0.4. The “-v” for verbosity is optional. ~# tcpdump-uw -i vmk2 udp port 23451 -v tcpdump-uw: listening on vmk2, link-type EN10MB (Ethernet), capture size 96 bytes 14:17:19.327940 IP truncated-ip - 218 bytes missing! (tos 0x0, ttl 5, id 21137, offset 0, flags [none], proto UDP (17), length 300) 172.32.0.3.30254 > 224.2.3.4.23451: UDP, length 272 14:17:19.791813 IP truncated-ip - 130 bytes missing! (tos 0x0, ttl 5, id 8788, offset 0, flags [none], proto UDP (17), length 212) 172.32.0.4.42569 > 224.2.3.4.23451: UDP, length 184 14:17:20.327813 IP truncated-ip - 218 bytes missing! (tos 0x0, ttl 5, id 28287, offset 0, flags [none], proto UDP (17), length 300) 172.32.0.3.30254 > 224.2.3.4.23451: UDP, length 272 14:17:20.492136 IP truncated-ip - 266 bytes missing! (tos 0x0, ttl 5, id 29439, offset 0, flags [none], proto UDP (17), length 348) 172.32.0.3.30254 > 224.2.3.4.23451: UDP, length 320 14:17:20.493143 IP truncated-ip - 442 bytes missing! (tos 0x0, ttl 5, id 29459, offset 0, flags [none], proto UDP (17), length 524) 172.32.0.3.30254 > 224.2.3.4.23451: UDP, length 496 14:17:20.791810 IP truncated-ip - 130 bytes missing! (tos 0x0, ttl 5, id 26444, offset 0, flags [none], proto UDP (17), length 212) 172.32.0.4.42569 > 224.2.3.4.23451: UDP, length 184 V M W A R E S T O R A G E B U D O C U M E N T A T I O N / 1 0 6
Diagnostics and Troubleshooting Reference Manual – Virtual SAN While this output might seem a little confusing, suffice to say that the output shown here indicates that this host is indeed receiving a heartbeat from the master. This tcpdump-uw command would have to be run on every host to verify that they are all receiving the heartbeat. This will verify that the master is sending the heartbeats, and every other host in the cluster is receiving them, indicating multicast is working. To get rid of the annoying “IP truncated-ip - XX byes missing” messages, one can simply add a –s0 at the end. –s0 will not truncate the packet. # tcpdump-uw -i vmk2 udp port 23451 -v -s0 tcpdump-uw: listening on vmk2, link-type EN10MB (Ethernet), capture size 65535 bytes 21:14:09.093549 IP (tos 0x0, ttl 5, id 61778, offset 0, flags [none], proto UDP (17), length 228) 172.32.0.3.20522 > 224.2.3.4.23451: UDP, length 200 21:14:09.617485 IP (tos 0x0, ttl 5, id 46668, offset 0, flags [none], proto UDP (17), length 316) 172.32.0.4.16431 > 224.2.3.4.23451: UDP, length 288 21:14:10.093543 IP (tos 0x0, ttl 5, id 61797, offset 0, flags [none], proto UDP (17), length 228) If some of the Virtual SAN hosts are not able to pick up the one-second heartbeats from the master, the network admin needs to check the multicast configuration of their switches. The next multicast checking command is related to IGMP (Internet Group Management Protocol) membership. Hosts (and network devices) use IGMP to establish multicast group membership. tcpdump-uw –i vmk2 igmp Each ESXi node in the Virtual SAN cluster will send out IGMP “membership reports” (aka ‘join’) every 90-300 seconds. This tcpdump command will show igmp member reports from a host: ~ # tcpdump-uw -i vmk2 igmp tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode listening on vmk2, link-type EN10MB (Ethernet), capture size 96 bytes 14:35:32.846558 IP 0.0.0.0 > all-systems.mcast.net: igmp query v2 [max resp time 1] 14:35:32.926337 IP 172.32.0.3 > 224.1.2.3: igmp v2 report 224.1.2.3 14:35:32.928899 IP 172.32.0.3 > 224.2.3.4: igmp v2 report 224.2.3.4 14:35:45.757769 IP 172.32.1.253 > all-systems.mcast.net: igmp query v2 14:35:47.528132 IP 172.32.0.3 > 224.2.3.4: igmp v2 report 224.2.3.4 14:35:47.738076 IP 172.32.0.3 > 224.1.2.3: igmp v2 report 224.1.2.3 14:36:45.762795 IP 172.32.1.253 > all-systems.mcast.net: igmp query v2 14:36:51.887170 IP 172.32.0.3 > 224.2.3.4: igmp v2 report 224.2.3.4 14:36:56.207235 IP 172.32.0.3 > 224.1.2.3: igmp v2 report 224.1.2.3 14:37:02.846211 IP 0.0.0.0 > all-systems.mcast.net: igmp query v2 [max resp time 1] The output shows “igmp v2 reports” are taking place, indicating that the ESXi host is regularly updating its membership. If a network administrator has any doubts whether or not Virtual SAN ESXi nodes are doing IGMP correctly, running this command on each ESXi host in the cluster and showing this trace can be used to verify that this is indeed the case. V M W A R E S T O R A G E B U D O C U M E N T A T I O N / 1 0 7
Page 1 and 2:
Diagnostics and Troubleshooting Gui
Page 3 and 4:
Diagnostics and Troubleshooting Ref
Page 5 and 6:
Page 7 and 8:
Page 9 and 10:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17 and 18:
Page 19 and 20:
Page 21 and 22:
Page 23 and 24:
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
Page 37 and 38:
Page 39 and 40:
Page 41 and 42:
Page 43 and 44:
Page 45 and 46:
Page 47 and 48:
Page 49 and 50:
Page 51 and 52:
Page 53 and 54:
Page 55 and 56: Diagnostics and Troubleshooting Ref
Page 105: Diagnostics and Troubleshooting Ref
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185 and 186:
Page 187 and 188:
Page 189 and 190:
Page 191 and 192:
Page 193 and 194:
Page 195 and 196:
Page 197 and 198:
Page 199 and 200:
Page 201 and 202:
Page 203 and 204:
Page 205 and 206:
Page 207 and 208:
Page 209 and 210:
Page 211 and 212:
Page 213 and 214:
Page 215 and 216:
Page 217 and 218:
Page 219 and 220:
Page 221 and 222:
Page 223 and 224:
Page 225 and 226:
Page 227 and 228:
Page 229 and 230:
Page 231 and 232:
Page 233 and 234:
Page 235 and 236:
Page 237 and 238:
Page 239 and 240:
Page 241 and 242:
Page 243 and 244:
Page 245 and 246:
Page 247 and 248:
Page 249 and 250:
Page 251 and 252:
Page 253 and 254:
Page 255 and 256:
Page 257 and 258:
Page 259 and 260:
Page 261 and 262:
Page 263 and 264:
Page 265 and 266:
Page 267 and 268:
Page 269 and 270:
Page 271 and 272:
Page 273 and 274:
Page 275 and 276:
Page 277 and 278:
Page 279 and 280:
Page 281 and 282:
Page 283 and 284:
Page 285 and 286:
Page 287 and 288:
Page 289 and 290:
Page 291 and 292:
Page 293 and 294:
Page 295:
show all

VSAN-Troubleshooting-Reference-Manual

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?