rost_tuning_tools

A Manly Man's Guide toOpen-Source DBTuning Tools:Life without EM12cBjörn Rost@brost

Björn Rost• founder, manager and consultant• president RAC SIG• ACE Director

about us• Software production company founded 2001• mostly J2EE• logistics• telecommunication• media and publishing• customers demand full lifecycle support• hardware resale• datacenter operations• 3rd party software

why does Oracle rock?• a full scan/index scan/join in Oracle is notbetter than the same op in other DB• Oracle has a ton of instrumentation built-in• It is possible to determine (pretty) exactlywhat a query or session is doing at anygiven time

tuning cycle• define a problem and a goal• inspect and measure• understand• change• inspect and measure again

ut really• find• find whatever takes longest time• the lowest hanging fruit• the biggest offender

tuning tools• real-time monitor• historic monitor• session/sql drill-down

EM12c• really good and useful tool• comes with the DB, very integrated• many of the views and methods are common today• coloured AAS view• Top-SQL view• SQL monitor• blocking sessions• AWR reports• ADDM and other advisors

why use CLI tool instead•not licensed for Diag&Tuning••SE or SEONEEE without Diag&Tuning••very low throughput and/or no GUI accesstroubleshooting where EM12c is notavailable•trying to impress potential mating partner

Standard Edition• same codebaseas EE• at drasticallyreduced price• parallel execution•• partitioning• replication• 3rd party vendors orown scriptswrite own parallel jobs• Diag&Tuning• open source tools

lsqlplus• edit sqlplus CLI• history• auto-completion

lsqlplus step 1[root@kickpuppet ~]# yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/rlwrap-0.42-1.el6.x86_64.rpmLoaded plugins: securitySetting up Install ProcessResolving Dependencies--> Running transaction check---> Package rlwrap.x86_64 0:0.41-1.el6 will be installed--> Finished Dependency Resolution================================================================================Package Arch Version Repository Size================================================================================Installing:rlwrap x86_64 0.41-1.el6 epel 92 kTransaction Summary================================================================================Install 1 Package(s)Total download size: 92 kInstalled size: 202 kDownloading Packages:rlwrap-0.41-1.el6.x86_64.rpm | 92 kB 00:00Running rpm_check_debugRunning Transaction TestTransaction Test SucceededRunning TransactionInstalling : rlwrap-0.41-1.el6.x86_64 1/1Verifying : rlwrap-0.41-1.el6.x86_64 1/1Installed:rlwrap.x86_64 0:0.41-1.el6Complete!

lsqlplus step 2• download or generate wordfile• for auto-completion of keywords[oracle@kickpuppet ~]$ curl -o wordfile.txt http://blog.tanelpoder.com/files/scripts/setup/wordfile_11gR2.txthttp://blog.tanelpoder.com/files/scripts/setup/

lsqlplus step 3• create aliases around sqlplus (and rman)• avoid overwriting the original sqlplus namealias rlsqlplus='rlwrap -D2 -irc -b'\''"@(){}[],+=&^%#;|\'\'' -f ~/wordfile.txt sqlplus'alias rlrman='rlwrap -D2 -irc -b'\''"@(){}[],+=&^%#;|\'\'' rman'http://blog.tanelpoder.com/files/scripts/setup/

AAS view• my favourite real-time DB dashboard• one-glance DB activity overview• colour-coded• red -> blocking, app, concurrency• blue -> IO, missing index, etc• green -> CPU, executing too much?• other -> weird, take a look

AAS view

AAS view

web-ash• simple tool by Kyle Hailey• queries v$session by default (ok for SE)• stores data in plain files• simple javascript to read and displayhttp://datavirtualizer.com/web-ash-w-ash/

`• by Adrian Billington and Tanel Poder• install a small script within DB schema• samples v$-views• top-like output that refreshes• excellent overview

MOATSMOATS: The Mother Of All Tuning Scripts v1.0 by Adrian Billington & Tanel Poderhttp://www.oracle-developer.net & http://www.e2sn.com+ INSTANCE SUMMARY ------------------------------------------------------------------------------------------+| Instance: KICKPUPPET | Execs/s: 581.6 | sParse/s: 60.0 | LIOs/s: 4989.5 | Read MB/s: 0.7 || Cur Time: 19-Nov 21:11:09 | Calls/s: 164.2 | hParse/s: 0.0 | PhyRD/s: 86.7 | Write MB/s: 1.6 || History: 0h 9m 45s | Commits/s: 50.3 | ccHits/s: 578.8 | PhyWR/s: 112.7 | Redo MB/s: 0.2 |+------------------------------------------------------------------------------------------------------------++ TOP SQL_ID (child#) -----+ TOP SESSIONS ---------+ + TOP WAITS -------------------------+ WAIT CLASS -+| 29% | 7t0959msvyt5g (0) | 78,65 | | 29% | db file sequential read | User I/O || 14% | () | | | 29% | ON CPU | ON CPU || 14% | 147a57cxq3w5y (0) | 70 | | 14% | oracle thread bootstrap | Other || 14% | gh2g2tynpcpv1 (0) | 65 | | 14% | resmgr:cpu quantum | Scheduler || 14% | 4phvdvx32a3mf (0) | 73 | | 14% | db file parallel read | User I/O |+--------------------------------------------------+ +--------------------------------------------------++ TOP SQL_ID ----+ PLAN_HASH_VALUE + SQL TEXT ---------------------------------------------------------------+| 7t0959msvyt5g | 856749079 | SELECT ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORD || | | ER_TOTAL, SALES_REP_ID, PROMOTION_ID, WAREHOUSE_ID, DELIVERY_TYPE, COST |+ ---------------------------------------------------------------------------------------------------------- +| 147a57cxq3w5y | 0 | BEGIN :1 := orderentry.browseproducts(:2 ,:3 ,:4 ); END; |+ ---------------------------------------------------------------------------------------------------------- +| gh2g2tynpcpv1 | 0 | INSERT INTO CUSTOMERS ( CUSTOMER_ID , CUST_FIRST_NAME , CUST_LAST_NAME || | | , NLS_LANGUAGE , NLS_TERRITORY , CREDIT_LIMIT , CUST_EMAIL , ACCOUNT_MG |+ ---------------------------------------------------------------------------------------------------------- +| 4phvdvx32a3mf | 0 | begin prvt_ilm.stopjobs(-1,true,true,:1); end; |+ ---------------------------------------------------------------------------------------------------------- +

snapper••••great, free CLI tool by Tanel Poderjust one SQL script to callno installation neccessarywill run for sample period•take snapshots of v$session andothers•report on the diffs in snapshotshttp://tech.e2sn.com/oracle-scripts-and-tools/session-snapper

AAS in snapperSQL> @snapper ash=wait_class 5 1 allSampling SID all with interval 5 seconds, taking 1 snapshots...-- Session Snapper v4.11 BETA - by Tanel Poder ( http://blog.tanelpoder.com )-- Enjoy the Most Advanced Oracle Troubleshooting Script on the Planet! :)-------------------------Active% | WAIT_CLASS-------------------------44% | System I/O19% | Commit7% | Scheduler7% | ON CPU5% | User I/O-- End of ASH snap 1, end=2014-11-19 16:23:06, seconds=5, samples_taken=43http://tech.e2sn.com/oracle-scripts-and-tools/session-snapper

snapperSQL> @snapper stats,gather=tw 15 1 sid=70Sampling SID sid=70 with interval 15 seconds, taking 1 snapshots...-- Session Snapper v4.11 BETA - by Tanel Poder ( http://blog.tanelpoder.com ) - Enjoy the Most Advanced Oracle Troubleshooting Script on the Planet! :)--------------------------------------------------------------------------------------------------------------------------------------------------------SID, USERNAME , TYPE, STATISTIC , DELTA, HDELTA/SEC, %TIME, GRAPH , NUM_WAITS, WAITS/SEC, AVERAGES--------------------------------------------------------------------------------------------------------------------------------------------------------70, SOE , TIME, repeated bind elapsed time , 300, 19.8us, .0%, [ ], , ,70, SOE , TIME, parse time elapsed , 9272, 611.99us, .1%, [ ], , ,70, SOE , TIME, PL/SQL execution elapsed time, 230999, 15.25ms, 1.5%, [# ], , ,70, SOE , TIME, DB CPU , 569000, 37.56ms, 3.8%, [@ ], , ,70, SOE , TIME, sql execute elapsed time , 1730975, 114.25ms, 11.4%, [## ], , ,70, SOE , TIME, DB time , 2038543, 134.55ms, 13.5%, [## ], , , .88 % unaccounted time70, SOE , WAIT, log file sync , 235408, 15.54ms, 1.6%, [W ], 183, 12.08, 1.29ms average wait70, SOE , WAIT, db file sequential read , 1123224, 74.14ms, 7.4%, [W ], 397, 26.2, 2.83ms average wait70, SOE , WAIT, db file parallel read , 52763, 3.48ms, .3%, [ ], 3, .2, 17.59ms average wait70, SOE , WAIT, latch: In memory undo latch , 499, 32.94us, .0%, [ ], 1, .07, 499us average wait70, SOE , WAIT, resmgr:cpu quantum , 51794, 3.42ms, .3%, [ ], 3, .2, 17.26ms average wait70, SOE , WAIT, SQL*Net message to client , 1458, 96.23us, .0%, [ ], 393, 25.94, 3.71us average wait70, SOE , WAIT, SQL*Net message from client , 60177, 3.97ms, .4%, [ ], 393, 25.94, 153.12us average wait70, SOE , WAIT, PL/SQL lock timer , 12918348, 852.66ms, 85.3%, [WWWWWWWWW ], 1328, 87.65, 9.73ms average wait70, SOE , WAIT, events in waitclass Other , 960, 63.36us, .0%, [ ], 2, .13, 480us average wait-- End of Stats snap 1, end=2014-11-20 08:04:39, seconds=15.2

TOP SQL in EM• easy to identify low-hanging fruit• top SQL by IO/DB time

TOP SQL in snapperSQL> @snapper ash=sql_id+wait_class 15 1 user=soeSampling SID user=soe with interval 15 seconds, taking 1 snapshots...-- Session Snapper v4.11 BETA - by Tanel Poder ( http://blog.tanelpoder.com )-- Enjoy the Most Advanced Oracle Troubleshooting Script on the Planet! :)-------------------------------------------Active% | SQL_ID| WAIT_CLASS-------------------------------------------14% | | Commit5% | 7hk2m2702ua0g | User I/O4% | 5ckxyqfvu60pj | User I/O3% | 7t0959msvyt5g | User I/O3% | gh2g2tynpcpv1 | User I/O2% | 0w2qpuc6u2zsp | Scheduler2% | g81cbrq5yamf5 | User I/O2% | 7ws837zynp1zv | User I/O2% | f7rxuxzt64k87 | User I/O2% | 1qf3b7a46jm3u | User I/O-- End of ASH snap 1, end=2014-11-20 06:14:42, seconds=15, samples_taken=100PL/SQL procedure successfully completed.http://tech.e2sn.com/oracle-scripts-and-tools/session-snapper

historic performance• moats and snapper only sample when running• EE has ASH and AWR for historic data

AWR -> statspack• good for a high level overview• AWR has some more “stuff”• AWR is setup automatically

statspack• just setup automatic snapshots• a bit painful: keeping “baselines”• and document how to create reportsfrom CLI

statspack install in 12c• not really supported in CDB• perfstat username cannot be common userby default• there is a nasty and unsupportedworkaroundhttps://jonathanlewis.wordpress.com/2013/07/04/12c-statspack-hack/

Pressing will result in STATSPACK's recommended defaulttablespace (identified by *) being used.statspackCDB$ROOT@ORCL> @?/rdbms/admin/spcreate.sqlChoose the PERFSTAT user's password-----------------------------------Not specifying a password will result in the installation FAILINGoracleChoose the Default tablespace for the PERFSTAT user---------------------------------------------------Below is the list of online tablespaces in this database which canstore user data. Specifying the SYSTEM tablespace for the user'sdefault tablespace will result in the installation FAILING, asusing SYSTEM for performance data is not supported.Choose the PERFSTAT users's default tablespace. This is the tablespacein which the STATSPACK tables and indexes will be created.TABLESPACE_NAMECONTENTS STATSPACK DEFAULT TABLESPACE------------------------------ --------- ----------------------------SYSAUX PERMANENT *USERS PERMANENT

statspackSQL> @?/rdbms/admin/spauto.sqlPL/SQL procedure successfully completed.Job number for automated statistics collection for this instance~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Note that this job number is needed when modifying or removingthe job:JOBNO----------1Job queue process~~~~~~~~~~~~~~~~~Below is the current setting of the job_queue_processes init.oraparameter - the value for this parameter must be greaterthan 0 to use automatic statistics gathering:NAME TYPE VALUE------------------------------------ ----------- ------------------------------job_queue_processes integer 1000Next scheduled run~~~~~~~~~~~~~~~~~~The next scheduled run for this job is:JOB NEXT_DATE NEXT_SEC---------- --------- --------------------------------1 14-NOV-14 04:00:00

statspackSQL> select next_date, interval, what from user_jobswhere job=1;NEXT_DATE INTERVALWHAT-------------------------------------------------14-NOV-14 trunc(SYSDATE+1/24,’HH') statspack.snap;

oraSASH• does pretty much what ASH does (but for free)• sample and store information from v$views• and mostly compatible with EE ASH• advisable to store data in repo DB• which means only minimal footprint in DB• views and DB-link is all that is neededhttp://pioro.github.io/orasash/

SQLT• ORACLE supported scripts• needs some installation• will gather *everything* about 1 SQL• option to use Diag and/or Tuningpacks• in one easily exportable zipped htmlMetalink Doc id 215187.1

SQLTSQLT can make extensive use of licensed featuresprovided by the Oracle Diagnostic and the OracleTuning Packs, including SQL Tuning Advisor (STA),SQL Monitoring and Automatic Workload Repository(AWR).To enable or disable access to these featuresfrom the SQLT tool enter one of the followingvalues when asked:"T" if you have license for Diagnostic and Tuning"D" if you have license only for Oracle Diagnostic"N" if you do not have these two licensesOracle Pack license [T]: N

SQLTSQL> @sqlt/run/sqltxtract.sql 7t0959msvyt5g oracle42-3:sqlt brost$ lssqlt_s50512_10053_i1_c1_extract.trc sqlt_s50512_opatch.zip sqlt_s50512_tcb.zipsqlt_s50512_cell_state.zip sqlt_s50512_readme.html sqlt_s50512_tcx.zipsqlt_s50512_driver.zip sqlt_s50512_sqldx.zip sqlt_s50512_trc.zipsqlt_s50512_lite.html sqlt_s50512_tc.zip sqlt_s50512_xpand.sqlsqlt_s50512_log.zip sqlt_s50512_tc_script.sql sqlt_s50512_xtract_7t0959msvyt5g.zipsqlt_s50512_main.html sqlt_s50512_tc_sql.sql

SQLT

eDB360• database scope• no installation neccessary• creates one zipfile for offline viewing• T | D | N options for pack access• convenient: graphshttp://www.enkitec.com/products/edb360

SQLd360• SQL scope• brand new! (feb 2015)• no installation needed• one zipfile for offline browsinghttp://mauro-pagano.com/2015/02/16/sqld360-sql-diagnostics-collection-made-faster/

good old tracing• good (free) analysis tool?• tkprof• generating trace• why not just trace everything?• much better than ASH• overhead vs usefulness

• rlsqlplus• statspack• moats• wash - web ash• sash - simulated ash• snapper• SQLT• eDB360review

thank youtwitter: @brosthttp://portrix-systems.de/blog/b.rost@portrix-systems.de