Automatic Performance Diagnosis and Tuning in Oracle

We used various scenarios to gauge the effectiveness ofADDM.Blind tests were performed in the initial testing phase;performance tuning experts were asked to analyse andmake recommendations on running systems. In a majorityof cases ADDM produced comparable results and in somecases ADDM’s recommendations produced greaterbenefit than the experts’ recommendations.As part of the testing performed internal to OracleServer Technologies group we have a number of tests inwhich we introduced known ‘common faults’ andapplication inefficiencies. When ADDM analysis wasperformed on the data captured during these tests ADDMcorrectly identified the fault in all cases.High load stress testing is an integral part of thetesting of the database product at Oracle and the effect ofrunning all of the Oracle 10g manageability features wasmeasured. With both typical customer workloads andhighly tuned industry standard benchmarks the reductionin throughput from enabling all of the manageabilityfeatures was approximately 3%. AWR and ADDM weretwo of the features enabled. ADDM runs performed aftereach AWR snapshot were available in a timely manner,typically under ten seconds.Although Oracle 10g was declared production inJanuary 2004 a number of internal production systemswere running the software for several months before thisdate. ADDM has identified and correctly diagnosed anumber of performance issues in this time on thesesystems.7.1 Experience of QualcommQualcomm Centauri Application was upgraded fromOracle version 8.1.7.4 to 10g RAC. While testing theupgrade they found significant performance problems andtesters reported poor response times. The DBA looked atthe ADDM recommendations which highlighted a SQL(update statement) that was causing over 90% of thesystem load. They then ran SQL Tuning Advisor on thestatement and it recommended the creation of an index. Itwas later found that the recommended index should havebeen in place. The index was missing because a patch tothe application was applied to the production system butnot to the upgraded test system. Identifying the indexmade problem diagnosis easy.7.2 Experience in Oracle’s Bug DatabaseThe Oracle Bug database is used daily by many thousandsof users and was one of the first production systems tomove to Oracle 10g. The system runs on an 8 CPU PA-RISC HP machine. After upgrading to Oracle 10g usersexperienced poor performance. ADDM reported that userswere spending a large proportion of their time waiting forfree buffer waits and recommended examining the I/Osubsystem for write performance (this type of problemhappens when the buffer cache is filled with dirty buffersand faster I/O should solve the problem). When theSystem Administrator looked at the I/O subsystem hefound that the asynchronous I/O was incorrectlyconfigured causing asynchronous I/O requests to fail andthen be performed synchronously leading to extremelyslow IO times.7.3 Experience of Oracle Applications QA TestingAn Applications Development DBA reported that a usersaid that the system was slow. Unfortunately there was notimescale or details given. Investigation was made harderby the fact that the users of the system and the DBAsinvestigating the slowdown where on opposite sides of theworld, 12 time zones apart.Looking at ADDM reports there were a couple of onehour periods in which the time spent in the database wassignificantly higher. Both of these ADDM reports showedthat most of the time was spent in parsing and that theparsing was caused by the application generating largenumbers of SQL statements containing literal stringvalues. (This problem is Oracle’s equivalent of usingmany similar SQL statements instead of a storedprocedure. The cost of such configurations is time spentin parsing, optimizing and compiling the SQL statementsinOracle’s terminology it is called “hard parse”). Therecommendation from ADDM was to modify theapplication to use bind variables rather than literals or tochange a database configuration parameter toautomatically convert the literals into binds. Whenapplication development was approached about the literalusage it was discovered that this QA system was runninga 'known bad' internal build of the application andupgrading to the correct build removed the issue.8. ConclusionIn this paper we described how Oracle 10g offers acomprehensive tuning solution by providing automaticperformance diagnosis and tuning via ADDM. Thisaddresses the ever-increasing performance tuningchallenges currently faced by database administrators.We defined a new measure called Database Time thatallows for comparison of the performance impact ofvarious database components with each other. We alsodescribed what types of performance measurements areneeded for accurate performance diagnosis, as well ashow we obtain them in a manner that has marginal impacton the system. This solution also obviates the need toreproduce a problem in order to diagnose it.ADDM incorporates a holistic view of the system, andby using database time in conjunction with the twodimensionalDBTime-graph it is able to quickly isolatethe root causes of performance bottlenecks affecting thethroughput of the system. ADDM provides specificactionable recommendations with an estimate of thebenefit for alleviating performance problems.