Expert Oracle Exadata - Parent Directory

CHAPTER 12 MONITORING EXADATA PERFORMANCEOLTP activity doing commits. But is it then doing these commits just after the hour starts? Well, these aredefault Oracle Database installations with AWR snapshots taking place at the full hour. And there areover 10 databases in the cluster, with an AWR snapshot taken and written in each cluster databaseinstance. Each AWR snapshot write involves multiple commits, and apparently the resulting log filewrites have occasionally taken longer than expected. The next step for troubleshooting such low-levelI/O issues in a cell would be to look at the OS-level metrics, like iostat. The cells do perform regularblock I/O against their disks, after all, so all the OS-level I/O monitoring tools still do apply.In the following extract from the iostat archive, you see a few interesting things. The await timeshows that on average, I/O completion time (queuing plus service time) is over 100ms for almost all thedisks. For a couple of disks it is as high as 549 ms and 738 ms on average! The disk’s estimated servicetime (svctm) shows good I/O service times (below 10ms), so the hard disks and lower half of the I/Osubsystem seems to work OK. The high response times must have come from I/O queuing waits, whichhappens when more I/O requests are issued to OS than the storage hardware (and device driverconfiguration) can accept.Time: 06:00:05Device: r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %utilsda 84.60 15.20 162235.20 1038.40 1636.01 61.35 738.49 9.40 93.84sdb 68.60 16.80 124716.80 1375.20 1476.49 20.91 253.98 8.37 71.50sdc 65.00 0.00 130918.40 0.00 2014.13 11.14 171.82 8.94 58.12sdd 60.00 1.20 120358.40 27.60 1967.09 9.74 167.52 8.56 52.38sde 60.00 4.80 122752.00 245.60 1898.11 10.41 172.73 8.32 53.92sdf 1.60 1.20 51.20 28.80 28.57 0.00 0.07 0.07 0.02sdg 58.00 0.00 118272.00 0.00 2039.17 9.51 173.76 8.53 49.46sdh 54.60 2.40 110912.00 48.40 1946.67 77.09 549.34 9.08 51.78sdi 24.20 4.80 133280.00 242.80 1035.06 15.82 125.00 5.24 67.62sdj 64.20 0.40 130566.40 3.40 2021.20 11.07 171.43 8.68 56.10sdk 61.20 0.80 124515.20 25.60 2008.72 12.32 202.59 9.32 57.78sdl 78.40 2.60 131196.80 21.20 1619.98 12.41 153.18 7.55 61.14This case is an example of how low-latency OLTP I/Os (like log file writes for commits) don’t go welltogether with large, massively parallel asynchronous I/O workloads, which drive the disk I/O queuelengths up, possibly making the low-latency I/Os suffer. This is why you should keep the I/O ResourceManager enabled in Exadata cells when running mixed workloads, as IORM is able to throttle the SmartScan I/O and prioritize important background process I/O, such as log file writes, automatically.You can read more about the OS Watcher tool in My Oracle Support article “OS Watcher UserGuide” (Doc ID 301137.1). In fact, you can download the latest OS Watcher from there and use it formonitoring your other servers too. The My Oracle Support version of OS Watcher does not include theExadata cell-specific monitoring scripts, but you don’t need those for your regular servers, anyway. ForExadata storage cells, you’ll have to keep using the original version shipped with the cell software, asOracle doesn’t authorize installation of any additional storage into storage cells if you want to keep thesupported configuration.It pays off to open up and skim through OS Watcher scripts; you might learn some new useful OSlevelmonitoring commands from there and will have a better overview of which OS metrics areavailable. You should drill down to these low-level metrics when the more conveniently accessibleOracle Database and Cellcli-level metrics are not enough.417