Eliminating performance bottlenecks in Caché-based apps

Eliminating PerformanceBottlenecks in CachébasedApplicationsAlain Houf

AgendaIntroducing Concepts and MetricsIdentifying Worst OffendersTest for PerformanceTop Tricks to Improve Performance

Plan for Today• Introduce metrics that reflect applicationperformance• Learn about tools to capture and analyze thesemetrics, including identifying the worst offenders• Study some efficient ways to improve these metricsand therefore improve application performance

Addressing Performance ChallengesOptions:1) Identify hardware and software configuration thatwould sustain required load2) Adjust existing application to better utilize existingresourcesOur focus is on #2

System Performance Limiting FactorsCPUInput/Output(Delivering data to/fromCPU)

System Performance Limiting FactorsCPUMemory- Disk- NetworkI/O

System-level MetricsCPUMemory- Disk- NetworkI/OCPU Utilizationiowait/Disk Queue

System Stats – Big Picture• Snapshot (OS – level tools)• Perfmon.exe, Task Manager, Activity Monitor, top, vmstat• Look for CPU utilization, Top CPU consuming processes, Diskqueue/iowait

System Stats – Process ID• OS Process ID = Caché PID ($Job)• Try to match OS process ID with Caché Process inManagement Portal**if you are lucky you‟ll catch your worst offending process right there

Caché-level MetricsCPURoutineCommandsMemoryCache EfficiencyNon DB I/OFiles, NetworkDB I/OCACHE.DAT, WIJ, JournalsGloRefsTotal Execution Time

Key Metrics Caché - side• GloRef• Single access to global (get, set, kill)• Roughly translates to I/O (DB Files and Memory)• Routine Commands• Single COS instruction• Roughly reflects CPU load• Execution Time• Overall measure of performance• Accounts for “not all GloRefs are created equal” andother factors

^GLOSTAT• Caché High-Level performance stats

^GLOSTAT^GLOSTATTo run: DO ^GLOSTAT in %SYS namespace10 to count data for 10 secondsMetrics to watch for:Global references ~ Disk+Memory activityRoutine commands ~ CPU activityCache Efficiency ~ Memory vs. Diskaccess (how good are your global buffers)

Capture Data Over Time• It is important to capture data over representativeperiod of time• OS – Level• logman (Windows)• sar, vmstat, iostat (*nix)• Caché – Level• ^mgstat, SYS.History• Enterprise tools• SNMP, WMI• BMC Patrol

Concepts and Metrics - Summary• I/O and CPU are two primary causes ofperformance problems• GloRefs, RoutineCommands and Execution Timeare most important Caché-side metrics to watch• Decreasing these metrics per task ontest/development system would most likely helpproduction system as well

AgendaIntroducing Concepts and MetricsIdentifying Worst OffendersTest for PerformanceTop Tricks to Improve Performance

^PERFMON• Command – line/API tool for collecting performancestatistics arranged by globals and routines• Can answer questions such as:• What routine responsible for the most lines of codeexecuted?• What global was accessed the most?• Directory block writes per routine…• etc.

DemoPERFMON/%SYS.MONLBL investigationIdentify busiest line[s] of code

^PERFMON^PERFMON%SYS>do ^PERFMON Start Monitor (default) Processes (not important) Routines (we want all of them) Globals (we want all of them) (default) Databases (not important) (default) Network Nodes (notimportant)Wait 2 minutes…

^PERFMON^PERFMON Pause Monitor Report Statistics Custom Category Report All Metrics Sort by Routine Excel format Stop Monitor…

^PERFMON“Freeze Panes” on B2Sort by Global References, Routine Lines…

Understanding PERFMON Report• Most Important Metrics• Global References/Routine Lines – Overall activity• Important Metrics• Data Block Read/Write – What triggers physical diskaccess• Journal Entries, Block Allocation – Adding new data• Lock/Lock Fail – potential lock conflict

^%SYS.MONLBL• Similar in UI and metrics to PERFMON, but collectsdata on line-by-line basisTime – Time spend executing that line of codeTotal Time – Executing line of code and all calls

Monitoring – Other Tools• %Monitor.Process – low footprint in-processmonitoring tool• Collects same set of metrics as PERFMON/MONLBL• No performance impact (safe for production)• Data accessible only within process (need to use API)• ^PROFILE• Unix‟s „top‟ – like UI• “Hybrid” of PERFMON and %SYS.MONLBL• Good for brief first look

^PROFILE

New Stuff!!!• GLOBUFF• Utility to identify Global Bufferusage per-globalNew in 2012.2• BLKCOL• Utility to detect block collisions• Same machinery as cstat –D10,1026

^GLOBUFFNew in 2012.227

^BLKCOLNew in 2012.228

SQL Monitoring - %SYS.PTools• SQL Monitoring tool• Attached to cached queries (requires queries purge)• Works with Class Queries and Embedded SQL(Requires recompile)• Collects data about executed queries (time, glorefs etc.)• SQL interface to results (%SYS_PTools package)

DemoSQL Monitoring with %SYS.PToolsCount # of times executed and total time spend within each SQL queryDO $SYSTEM.SQL.SetSQLStats(2)Purge Queries/Recompile embedded SQLDo $SYSTEM.SQL.Purge()Run some SQL queriesExamine %SYS_PTools.SQLStatsView

%SYS.PTools ExampleSELECT RoutineName, QueryText, COUNT(*) as CountTotal,SUM(TotalTime) as TimeTotalFROM %SYS_PTools.SQLStatsViewWHERE namespace='PERF'GROUP BY RoutineNameORDER BY CountTotal

Caché and OS ToolsSystem activityoverview“Worstoffenders”Process-levelmicroscopeWindows Unix CachéPerfmon.exelogmanvmstat, iostat,sarGLOSTAT, mgstatTask Manager top PERFMONPROFILE%SYS.PTools^BLKCOLProcessExplorertruss, straceOther stats SysInternals Misc. tools cstat%SYS.MONLBL,%Monitor.Process

Identifying Bottlenecks - Summary• Identifying worst offenders? There are apps tools forthat:• PERFMON• %SYS.MONLBL• PROFILE• %SYS.PTools

AgendaIntroducing Concepts and MetricsIdentifying Worst OffendersTest for PerformanceTop Tricks to Improve Performance

What’s Faster? (COS answer)• Collecting metrics inside COS code:• $zh – time with maximum possible precision• %Monitor.Process• ##class(%SYS.ProcessQuery).%OpenId($j)• CommandsExecuted• GlobalReferences

What’s Faster? SQL Answer• SQL – System Management Portal expose metrics fortime and global references:

Query Plan• System Management Portal ->Execute SQL -> Show QueryPlan• Identify what indices and tempstructures are used• Relative cost is relative (nomeaning outside of comparingtwo queries)• You may also look at/parsecached query source

Sample SQL OptimizationBefore:SELECT int1, int2, oneplustwo, str1FROM perf.testWHERE str1 %startswith ‟b'After:SELECT int1, int2, int1+int2 as oneplustwo, str1FROM perf.testWHERE str1 %startswith ‟b'Original Avoid external call Add IndexTime sec. 1.259 0.195 0.157Glorefs 136083 56236 19015

Test for Performance - Summary• Monitor Routine Commands, Global References, Time• Tools to measure:• %SYS.ProcessQuery, %Monitor.Process, $zh in COS• System Management portal for SQL or wrap SQL queryin COS routine to capture metrics• With rare exceptions – if you manage to decrease ANYof these metrics – you’ll improve the overallperformance

AgendaIntroducing Concepts and MetricsIdentifying Worst OffendersTest for PerformanceTop Tricks to Improve Performance

Top 3* COS Performance Optimizations1. Eliminate [External] Calls2. Reduce # of System calls3. Pick effective temp data structures

Avoid External and Internal CallsTime required to execute simple A+B operation 10k times:External Function $$Add^Library 0.018Function $$Add 0.008Macro (no call) $$$Add 0.001Set c=$$Add(a+b)Add(a,b) {Quit a+b}#define Add(%a,%b) (%a+%b)Set c=$$$Add(a,b)

Case Study - Generators• Complex logic, accessing class Metadata. Multiplesuperclasses. Each class implements GeneratorbasedReturnMeta() method, calling ReturnMeta()of superclassesSuperClassesInlineTotal Time 8.762153 0.8962Total Commands 1990989 1290963• Avoiding calls to methods ofsuperclasses ~10x faster10x faster

Reduce # of System CallsSet file=##class(%File).%New("file.txt")Do file.Open("WSN")for i=1:1:1000 {for j=0:1:255 {Do file.Write($c(j))Inefficient}do file.WriteLine()}do file.%Close()

Reduce # of System CallsSet file=##class(%File).%New("file.txt")Do file.Open("WSN")for i=1:1:1000 {set line=""for j=0:1:255 {set line=line_$c(j)}do file.WriteLine(line)}do file.%Close()

Case Study – Minimizing # of System Calls• Original application – reading and processingbunch of small files (3-5 records/file). Using %Fileand %FileCharacterStream to access filesystem• ~7k records/second• Optimized – Low-Level API ($ZSearch, Open, Use,Read) to scan directory and read data• ~56k records/second8x faster

Temp Data• DOs:• Local Variables/Arrays• Starting with 2010.2 super fast even with large # of nodes• More performance-related changes coming in 2013• Special optimization for single integer subscripts• CACHETEMP• Faster than local arrays with large number of nodes• Prone to block contentions• Process-Private globals• Currently CACHETEMP – based, but immune to block contentions• DON’Ts:• “Temporary” objects, tables

Misc. TricksHigh-Performance DataLoad- Direct Global Access/Embedded SQL- $SortBegin()/$SortEnd() for unsorted data- Defer building indices if feasible- Use low-level API to read files

Misc TricksRelationships- One-to-ManyAlways add index on “one” side

Misc TricksBatch Jobs- ^%PRIO to manipulate process priority- $PrefetchOn()/$PrefetchOff()

Misc TricksReducing Lock ConflictsShared vs. Exclusive (default) lockL+ ^global#“S”Object Concurrency.%OpenId(id,0) concurrency level 0-4SQL%NOLOCKTransaction Isolation Level

Misc Tricks$Piece vs. $List$List is usually faster for largestructures$ListFromString() to convert from $P

Misc TricksMisc. COS$Case||, &&$Order, $Data$Bit*()

Top 3* SQL Performance Optimizations1. Add missing index2. Enable/Enforce usage of right index3. Effective use of functions

Identifying Index Usage• System Management Portal ->Execute SQL -> Show QueryPlan• Look at the source code ofgenerated Cached Query, searchfor ‘$o(^’• Check if process statusconstantly at ^TableD global

Case Study – Add Index• Inefficient Client-Server code – (up to 6 JDBCstatements/roundtrips to update single record) ~ 10records/second in a lab• Moving to stored procedure (1 round trip) ~ 150rec/sec in a lab• In production - loading 150k records still takes 8 hours• One of SELECTs missing index – not noticeable in thelab, but extremely slow in production (subject to # ofrows)• Add index8 hours - > 2 min.

Enabling Index Usage• TuneTable() to make sure stats are right• Make sure “where field” is used “as is” and not asargument to a functionSELECT * FROM packagesWHERE dbo.ToUnixDate(starttime)>1262584800AND dbo.ToUnixDate(starttime)

Misc TricksSQL Hints%NOLOCK%NOCHECK%NOINDEX%INORDER%FULL%IGNOREINDICES…See 2012.2 documentation for more

Misc TricksAdvanced SQL IndicesBitmapBitSliceCollection%Text- multi-condition selects- aggregates- list, arrays- full text search

Case Study - Functions• This query searches for “pattern” within URL string:select dpiau.*, %exact(dpias_step) as dpias_step, %exact(dpias_type) as dpias_type, dpicuh.* fromdpi_client_url_history dpicuh left join dpi_application_urls dpiau on dpicuh_appurlid = dpiau_id left joindpi_application_steps on dpiau_apppatternid = dpias_patternid where dpias_step is not null anddpias_patternid = 1 and( dpiau_url like '%?'||dpias_step||'&%'or dpiau_url like '%?'||dpias_stepor dpiau_url like '%&'||dpias_step||'&%'Inefficientor dpiau_url like '%&'||dpias_stepor dpiau_url like '%/'||dpias_step||'/%'or dpiau_url like '%/'||dpias_step||'?%'or dpiau_url like '%/'||dpias_step)

Case Study - Functions• This query searches for “pattern” within URL string:select dpiau.*, %exact(dpias_step) as dpias_step, %exact(dpias_type) as dpias_type, dpicuh.* fromdpi_client_url_history dpicuh left join dpi_application_urls dpiau on dpicuh_appurlid = dpiau_id left joindpi_application_steps on dpiau_apppatternid = dpias_patternid where dpias_step is not null anddpias_patternid = 1 andMyCustomPattern(dpiau_url, dpias_step)• Solution – change combination of conditions to singlefunction• Result ~100x faster100x improvement

Optimizing Performance - Summary• Most Common Optimizations COS:1. Eliminating [External] Calls2. Reducing # of System calls3. Picking effective temp data structures4. …• Most Common Optimizations SQL:1. Add missing index2. Enable/Enforce usage of right index3. Effective use of functions4. …

Upgrade!!!• Recent free-of-charge improvements• Per-process routine-vector caching• Faster object dispatch• C level code for common object methods• New, faster!!! local arrays

AgendaIntroducing Concepts and MetricsIdentifying Worst OffendersTest for PerformanceTop Tricks to Improve Performance

Summary• GloRefs, RoutineCommands, Execution Time• Understand• Learn how to capture• Analyze• Improve