Expert Oracle Exadata - Parent Directory

C H A P T E R 9Recovering ExadataYou may have heard the saying “disk drives spin, and then they die.” It’s not something we like to thinkabout, but from the moment you power up a new system, your disk drives begin aging. Disk drives havecome a long way in the past 30 years, and typical life expectancy has improved dramatically. At the endof the day, though, it’s a matter of “when” a disk will fail, not “if.” And we all know that many disk drivesfail long before they should. Knowing how to diagnose disk failures and what to do when they occur hasgenerally been the responsibility of the system administrator or storage administrator. For many DBAs,Exadata is going to change that. Many Exadata systems out there are being managed entirely by the DBAstaff. Whether or not this is the case in your data center, the procedure for recovering from a disk failureon Exadata is going to be a little different than you are used to.Oracle database servers have traditionally required two types of backup: operating system backupsand database backups. Exadata adds storage cells to the mix, and with that comes a whole newsubsystem that must be protected and, on occasion, restored. The storage cell is a fairly resilient piece ofhardware that employs Linux software RAID to protect the operating system volumes. As such, it isunlikely that a disk failure would necessitate an operating system restore. The more likely causes wouldbe human error, a failed patch install, or a bug. Remember that these physical volumes also contain griddisks (database volumes), so a loss one of these disks will most likely mean a loss of database storage aswell. Oracle has engineered several features into Exadata to protect your data and reduce the impact ofsuch failures. This chapter will discuss some of the more common storage failure scenarios, how todiagnose them, and how to recover with minimal downtime.■ Note: One of the most challenging aspects of writing this chapter is the rapidly changing nature of thecommands and scripts we will be discussing. In many cases, recovery tasks will have you working very closelywith the hardware layer of Exadata. So as you read this chapter, keep in mind that with each new version ofExadata hardware and software, the commands and scripts discussed in this chapter may change. Be sure tocheck the Oracle documentation for the latest updates to the commands and scripts discussed here.Exadata Diagnostic ToolsExadata is a highly complex blend of hardware and software that work together to produce an incrediblyresilient delivery platform. The complexity of the platform can be a bit daunting at first. There are simplya lot of moving parts that one must understand in order to maintain the platform effectively. Oracle275