Expert Oracle Exadata - Parent Directory

CHAPTER 12 MONITORING EXADATA PERFORMANCEsubmitted) and once the I/O is successfully reaped, they will check the reaping timestamp against thatI/O’s submit timestamp and know its duration.Regardless of this extra cellsrv-level I/O monitoring, the cells still don’t break the I/O responsetime into the two important components: how much time this I/O request spent uselessly waiting in theOS I/O queue before it was even sent out to the SCSI device, and how long the actual hardware servicetime was once the I/O request was sent out to the hardware. Comparing the I/O service time to waitingtime gives an important clue about whether the storage hardware itself responds slowly, in which casethe service time is higher than you would expect from a modern disk drive. Note: So what is this normal I/O service time to expect from disks? That depends on which disks you have,what their seek latency is, how far the disk read/write head has to seek from its previous location, what therotational latency of the disk is, how fast it spins (RPM), and how fast it can transfer the bits just read over thewires (or fiber). There is no magic in calculating what a good service time should be for a disk; it’s all based on thephysical capabilities of the disk, and these capabilities are documented in the vendor’s disk specs. You can readmore about the various latencies involved from Wikipedia:http://en.wikipedia.org/wiki/Hard_disk_drive#Access_time.Of course, there may be other overhead, like SAN storage network roundtrip times and any extralatency caused by storage controllers, switches, and so on. But remember, Exadata is not connected toSAN storage; the disks are attached directly to the storage cells. Each storage cell has a little LSIMegaRaid controller in it, and the disks are attached to that controller. There are no complex networkcomponents between the disks and storage cells, which might drive up latency or fail. More importantly,there are no “other” databases or applications connected to the storage, which you have no control over,but which could suddenly start saturating the disks without a warning. All you would see is that the diskI/O service times suddenly go up as the storage array gets overloaded.Anyway, a small I/O request (up to 256 KB) against a modern 15000 RPM disk drive should ideallyhave average service time of 5–7 milliseconds per I/O request. This comes from a 3.5ms average seeklatency, 2ms average rotational latency, plus 0.4ms for transferring the 256kB over SAS channels. Intheory this average seek latency is a pessimistic figure, assuming that the whole disk is full and the diskread/write heads have to do random seeks across tracks from end to end of the disk cylinders. But youprobably have configured your disks into hotter and colder regions (the RECO disk group, for example),and the colder regions are visited much less frequently, driving the real average seek times shorter. So,while you may get these low 2–3ms service times when your disk heads don’t have to seek too far, inpractice, 10ms average I/O service times are completely OK once you run real production workloads.407