Performance Tuning Documentum Web Based Applications - EMC ...

Performance Tuning Documentum Web BasedApplicationsCreated by:subir.rastogi@wipro.comCreated on:9-Oct-2008

2Few words about myself ..............................................................................................................3Introduction...................................................................................................................................3Problem ........................................................................................................................................4Solution ........................................................................................................................................6Automatic Database Diagnostic Monitor (ADDM) ...................................................................6Java Management Extension (JMX) and J2SE 5.0 .................................................................9Conclusion..................................................................................................................................11Resources ..................................................................................................................................11

3Few words about myselfI having been developing documentum applications for the past 3 years and currently working asthe Sr. Software Engineer in Wipro Technologies. I have worked on almost all the documentumproducts like Content Server, DFC, WDK/ Webtop, BPM, Application Builder and Installer, etc. Iam also EMC Certified proven professional by passing the exam “Content ManagementFundamentals”.Subir RastogiEMC Proven AssociateIntroductionI have been doing performance testing custom application build on WDK/Webtop Documentum5.3. I used JMeter to simulate load (virtual users) on the application to see how it behaves underload on staging. Below is the high level approach I have followed:‣ Load Test the application with 1 User, 50 Users, 100 Users‣ Analyze the components which are performing slowly.‣ Suggest the new approach/changes in the components to make the performance betterlike DQL tuning, simplifying custom logic, etc.‣ Analyze Application Server, Database and Content Server settings and suggest theappropriate setting which could increase the performance of the application.Below are the details of the environment on which I was performance testing.Application Server Weblogic 8.1 SP6 with Sun JVM 1.4.2, Solaris BoxWebtop, DFC, BPM, Application Builder and Content Server 5.3Documentum Products SP5Database Oracle 9.2

4ProblemDuring performance testing, I encountered following problems initiallyS.NO Problem Problem Description ResolutionDuring load test, http request were gettingqueued and there were no idle threads Increases thread count in1Http request were getting available. This means that the user has to such a way that there arequeuedwait until he is assigned a thread which in no requests getting queuedturn increases the response time for the and no idle threads.user.2345JVM Heap Size usage washigh, around 90%Few SQL were taking lot oftimeDatabase response was slowand DB Server CPU utilizationwas very high- 92%.Database Temporaryconnection error whichdocumentum makes withoracle.JVM heap size was not sufficient for 100Users working concurrently.By analyzing the application logs, I identifiedtop SQL queries which took most time ingetting the responseDatabase server CPU utilization was quitehigh around 92%.Process count in the oracle was less due towhich Content Server was unable to createsessions with the Oracle.Increase the JVM Heapsize to an optimum value.Tune these queries, useDQL hints, create indexes,and tune some DBparameters.Ask DBA to consistentlymonitor the database andsuggest changes.Increase the process countin Oracle.Once the problematic areas were identified, I started consulting respected WeblogicAdministrators, Documentum Administrators and Database Administrators to decide theappropriate values of the settings to resolve the problems. After having few discussions, it wasclear that there are no fixed rules to define most of the settings like Thread Count, Heap Size andprocess count in DB and they are very specific to the application. I did receive the settingsrecommended by Documentum with a warning that these might not be the ideal for your customapplication. So to decide the ideal settings I thought of below approach until we reach the bestresponse time and the optimal value.

5Load Testing ApproachThis turns out to be a time consuming process as I have to do load test multiple times for just onesetting.For e.g. shown below is the graph to find the optimal value for the thread count on the ApplicationServer. For finding the ideal setting, I had to perform at least 6 tests.Thread Count Vs Response TimeResponse Time20151051525507510000 20 40 60 80 100 120Thread Count

6Even though this turns out to be time consuming but we were sure of the output that it will give usthe ideal settings of the problem areas.But while doing all this exercise, I had one more doubt. I am doing this exercise on the stagingenvironment and there is a significant difference in the hardware capabilities b/w the staging andproduction environment. Production environment has more no. of CPU, processors and memory.So these are the ideal settings for the staging environment not for the production and productionmight behave differently with these settings. Since the production environment can not be loadtested, it was decided to proceed with these changes to production and production will bemonitored by the System Administrators and Database administrators.Even though we did not receive any complains from business users but I was very curious toknow that how the production environment is behaving with these settings under actual load.Have these settings actually increased the performance of the application in the production or stillthere is possibility to tune the application further?How is the system behaving under peak load?Even though it is possible to get this data by contacting the respective administrator but that istime consuming and error prone process as it is done manually.SolutionWe need a system which monitors itself automatically, captures all the relevant statistical data ofsignificant event happing within system and helps us decide, validate and analyze the problemsencountered in the system.Automatic Database Diagnostic Monitor (ADDM)As described in the oracle 10g documentation “The Automatic Database Diagnostic Monitor(ADDM) analyzes the data which get saved in Automatic Workload Repository (AWR, describedlater) on a regular basis, then locates the root causes of performance problems, providesrecommendations for correcting any problems, and identifies non-problem areas of the system.Because AWR is a repository of historical performance data, ADDM can be used to analyzeperformance issues after the event, often saving time and resources reproducing a problem.”ADDM provides the following benefits-‣ Automatic performance diagnostic report every hour by default‣ Problem diagnosis based on decades of tuning expertise‣ Time-based quantification of problem impacts and recommendation benefits‣ Identification of root cause, not symptoms‣ Recommendations for treating the root causes of problems‣ Identification of non-problem areas of the system‣ Minimal overhead to the system during the diagnostic processADDM finds the problem using a term “DB Time”. Areas which took significant portion of DB timeare reported as problem-areas. Oracle defines “DB Time” as the cumulative time spent by thedatabase server in processing user requests. It includes wait time and CPU time of all non-idleuser sessions.

7Diagram showing DB TimeThe types of problems that ADDM considers include the following:CPU bottlenecks - Is the system CPU bound by Oracle or some other application?Undersized Memory Structures - Are the Oracle memory structures, such as the SGA,PGA, and buffer cache, adequately sized?I/O capacity issues - Is the I/O subsystem performing as expected?High load SQL statements - Are there any SQL statements which are consumingexcessive system resources?High load PL/SQL execution and compilation, as well as high load Java usageRAC specific issues - What are the global cache hot blocks and objects; are there anyinterconnect latency issues?Sub-optimal use of Oracle by the application - Are there problems with poor connectionmanagement, excessive parsing, or application level lock contention?Database configuration issues - Is there evidence of incorrect sizing of log files, archivingissues, excessive checkpoints, or sub-optimal parameter settings?Concurrency issues - Are there buffer busy problems?Hot objects and top SQL for various problem areasShort-lived performance problemsDegradation of database performance over timeIn addition to problem diagnostics, ADDM recommends possible solutions. When appropriate,ADDM recommends multiple solutions for the DBA to choose from. ADDM considers a variety ofchanges to a system while generating its recommendations. Recommendations include:

8Hardware changes - Adding CPUs or changing the I/O subsystem configurationDatabase configuration - Changing initialization parameter settingsSchema changes - Hash partitioning a table or index, or using automatic segment-spacemanagement (ASSM)Application changes - Using the cache option for sequences or using bind variablesUsing other advisors - Running the SQL Tuning Advisor on high load SQL or running theSegment Advisor on hot objectsReports provided by ADDM1. CPU Utilization2. Average Active Sessions3. Disk I/O4. Instance ThroughputOther Important Features

91. Running the SQL Tuning Advisor on high-load SQL statements or running the SegmentAdvisor on hot objects. Documentum provides the description of how to use it.(Please referthe resources section).2. Automatic Workload Repository (AWR)Database statistics provide information about the type of load on the database and the internaland external resources used by the database. To accurately diagnose performance problemswith the database using ADDM, statistics must be available.Oracle Database generates manytypes of cumulative statistics for the system, sessions, and individual SQL statements. OracleDatabase also tracks cumulative statistics about segments and services. The AutomaticWorkload Repository (AWR) automates database statistics gathering by collecting, processing,and maintaining performance statistics for database problem detection and self-tuning purposes.The database statistics collected and processed by AWR include:Time Model StatisticsTime model statistics are statistics that measure the time spent in the database by operation type.Wait Event StatisticsWait events are statistics that are incremented by a session to indicate that it had to wait for anevent to complete before being able to continue processing. Wait event data reveals varioussymptoms of problems that might be impacting performance, such as latch contention, buffercontention, and I/O contention.Session and System StatisticsA large number of cumulative database statistics are available on a system and session level.Active Session History StatisticsThe Active Session History (ASH) statistics are samples of session activity in the database.Active sessions are sampled every second, and are stored in a circular buffer in the systemglobal area (SGA). Any session that is connected to the database and using CPU, or is waitingfor an event that does not belong to the idle wait class, is considered an active session. Bycapturing only active sessions, a manageable set of data is represented with the size beingdirectly related to the work being performed, rather than the number of sessions allowed on thesystem.High-Load SQL StatisticsSQL statements that are consuming the most resources produce the highest load on the system,based on criteria such as elapsed time and CPU time.Java Management Extension (JMX) and J2SE 5.0The JMX specification defines architecture, the design patterns, the APIs, and the services forapplication and network management and monitoring in the Java programming language.

10JMX technology provides flexible means to instrument Java code, create smart Java agents,implement distributed management middleware and managers, and smoothly integrate thesesolutions into existing management and monitoring systems.Typical uses of the JMX technology include:‣ Consulting and changing application configuration.‣ Collecting statistics about application behavior and making the statistics available.‣ Notification of state changes and erroneous conditions.‣ Managing and monitoring without much load on the Application Server.‣ It provides scalable and dynamic management architecture.J2SE 5.0 has implemented version 1.2 of the JMX specification. J2SE 5.0 includes significantmonitoring and management features, including:1. JVM instrumentation: The JVM is instrumented for monitoring and management providingbuilt-in, out-of-the-box management capabilities for local and remote access.2. Monitoring and Management APIs: The java.lang.management package provides theinterface for monitoring and managing the JVM. It provides access to information such as:number of classes loaded and threads running, memory consumption, garbage collectionstatistics, on-demand deadlock detection, and others.Core Classes of java.lang.management3. Management tools such as JConsole, which is a JMX-compliant monitoring tool that comeswith J2SE 5.0. It uses JMX instrumentation of the JVM to provide information on performanceand resource consumption of applications running on the Java platform.

11ConclusionPerformance Tuning Documentum 5.3 Applications was lacking statistical data of how changedsettings are working in the production environment. With the new Documentum 6.0, since itrequires Oracle 10g and J2SE 5.0, we will have plenty of statistical data to compare and concludethat the changed settings have really actullay increassed the performance of the system andsystem will provide recommendations also if the system can be tuned further.Resources1. Nice article on using JMeter in WDK Applicationhttp://ecmarchitect.com/wpimages/Load_testing_Documentum_WDK/Load_testing_Documentum_WDK.pdf2. http://download.oracle.com/docs/cd/B19306_01/server.102/b14211/toc.htm3. FAQ_Perf_Tuning_With_Oracle_Tuning_Advisor.pdf , document which gives you detailson using Tuning Adviser.4. http://java.sun.com/developer/technicalArticles/J2SE/jmx.html