taking advantage of the emc captiva architecture - EMC Community ...

TAKING ADVANTAGE OF 

THE EMC CAPTIVA 

ARCHITECTURE 

Applying Best Practices to Optimize 

Performance 

Christopher Lund 

EMC 

© Copyright 2011 EMC Corporation. All rights reserved. 

1

Agenda 

• How InputAccel works 

• InputAccel 6.5 Benchmark Results 

• Tuning 

– InputAccel Server 

– Batches 

– InputAccel Database 

– Client Modules 

– Capture Workflow 

• Diagnosing Performance Issues 

The EMC ® Captiva ® InputAccel ® and Dispatcher Version 6.5 Performance Sizing 

and Tuning Guide – which is available on PowerLink – provided much of the data for 

this presentation. 


2

How InputAccel Works 

A multi-machine capture application server 

InputAccel System 

Capture 

Capture 

Modules Capture 

Modules 

Modules 

WIP 

& 

Reports Tables 

Processing 

Processing 

Modules Processing 

Modules 

Modules 

InputAccel 

Servers 

Export 

Export 

Modules Export 

Modules 

Modules 

• Server is the data tier (memory 

mapped) 

• Server manages task queues 

• Server is multi-threaded… 

• VBA execution is single-threaded 

• DB writes are queued, but singlethreaded 

• Server uses asynchronous I/O 

• Most work done from a thread 

pool 

• Clients are the executing tier 

(where scaling comes from) 


3


An execution pipeline 

Execution Pipeline 

m1 m2 m3 m4 

t1 

t2 

t3 

t4 

t5 

t6 

t7 

t8 

t9 

B4 

B3 

B4 

B2 B3 B4 

B1 

B2 

B3 

B4 

A4 

B1 

B2 

B3 

A4 

B1 

B2 

A2 

A3 

A4 

B1 

A1 

A2 

A3 

A4 

A1 

A2 

A3 

• Task queues are not FIFO 

• Tasks are scheduled based on 

Priority, then creation date 

• Recovery is through 

reprocessing 

• Not a repository 

• Assume short duration tasks 

• Work is pushed, no polling 

• Tasks may be prefetched 


4


Like a Petri net 

Petri Net 

• A ProcessFlow defines the steps 

and trigger levels. 

B 

B 

A 

D 

C 

C 

• Implicit fire when data is available 

• There is no predefined execution 

order 

• There is no end state – IADone 

triggered implies completed 


5

InputAccel 6.5 Benchmark Results 

• One 8-core IA Server, over 300 Client Modules 

• In-house, ideal conditions, your mileage will vary 

• Performance similar to IA 6.0 SP1 

Performance Level 

50 active batches 

w/reporting disabled 


w/reporting disabled 


w/reporting enabled 

Overall Task 

Processing Rate 

(tasks/hour) 

Processing 

Rate/CPU Core 

(tasks/hour) 

Avg. CPU 

Utilization/C 

PU Core 

2,672,007 324,001 67% 

1,990,892 248,862 53% 

1,384,910 173,114 32% 

VMware ESX 4.x degrades throughput by approx 27% 


6

Tuning – InputAccel Server… 

• It is a server application. It will use all available resources 

• Recommendation: use Windows 2008 R2 for best performance 

• CPU 

– ias.exe is multi-threaded 

– Recommendation: use at least a 4-core CPU for optimum throughput, 8+ is 

better 

• RAM 

– Recommendation: 4-8 GB RAM, no less than 4 GB 

– ias.exe is a 32-bit app, so 4 GB address space max 

– BatchMaxAddressSpaceK controls how much RAM IAS uses 

• 1.5 GB is the default 

• Set to 2 to 2.5 GB on 32-bit Windows with /3G option 

• Set to 3 to 3.5 GB on 64-bit Windows 

– Only as many batches as will fit in BatchMaxAddressSpaceK are kept in RAM, 

when there are more, swapping occurs 

• Have your working set of batches small enough to all fit in RAM 

• Delete batches when you are done with them 


7

…Tuning – InputAccel Server 

• Disk 

– Used heavily for batches and processes 

– RAID 1+0 is best – RAID 5 usually is not fast enough and is not recommended 

• Recommendation: use a caching controller with Read Ahead and Write Back 

– SAN is OK, NAS not recommended 

– Turn off anti-virus scanning on IAS folder 

– IAS folder should be on a dedicated disk drive or array so it is not shared by other 

programs or the Windows swap file 

– #2 cause of slow performance: a slow hard drive 

• Network 

– InputAccel has a “chatty” protocol between client modules and the InputAccel Server 

– For best performance, client modules, the InputAccel Server, Administration Console, and 

DB should be on the same sub-net 

– WANs usually have low bandwidth with high latency, which may make it unsuitable 

• Connecting client modules to InputAccel Server via a WAN is doable with adequate performance – depends on 

the client modules, # batches on IA server, and IPP 

• InputAccel Server and IADB should not be connected by a WAN 


8

Tuning – Batches… 

• Batch size (IAB file) 

– 5 – 20 MB is ideal 

– < 100 MB is OK 

– > 100 MB not recommended, but is allowed 

– 10 – 100 pages per batch recommended 

– 1000 pages per batch degrades throughput by 10% or 

more 

– < 10 pages per batch leads to too many small batches and 

too much batch swapping 


9

…Tuning – Batches 

• Number of Batches 

– Throughput is best when the working set of batches fit in RAM 

• Typically 50 – 500 fit depending on batch size and 

BatchMaxAddressSpaceK 

– When there are more batches than fit in RAM, IA Server swaps 

batches in/out as needed 

• Swapping decreases IA Server throughput 

– Up to 9,000 idle batches is possible with adequate performance 

• Idle means that all tasks within the batch have been processed and 

finished 

– IA Server startup is slower with 1000’s of batches because IAS 

must load all batches into RAM to extract data 

• After startup they are swapped to disk and do not consume too many 

resources 

– #1 cause of slow performance: too many active batches causing 

excessive swapping 


10

Tuning – InputAccel Database 

System Requirements 

• InputAccel Server requires a database to run 

• Only MSSQL is supported, versions 2005 and 2008 

– Express supported only for low volume and no IA Reporting 

– Standard or Enterprise recommended for medium to high volume 

• IADB stores: 

– IA configuration data 

– Reporting data on completed batches/tasks 

– Work In Progress (WIP) status 

– Web services data 


11


System Requirements 

• Recommendation: 64-bit MSSQL server for best performance 

– 2-4 CPU cores 

– 4-8 GB RAM 

– RAID 1+0 with read/write caching controller w/ fast disks (15k RPM) 

• InputAccel Server requires fast, uninterrupted access to the DB 

– If the DB goes offline, IA Server pauses 

• Recommendation: put the DB and InputAccel Server on the 

same low-latency, high-bandwidth subnet 

• Reporting decreases the InputAccel Server throughput by about 

10-30%, although it can be more 


12


Data Volume and Rates 

• IADB File Size and Growth Rates 

– Configuration data - typically < 100 MB 

– WIP data - 1 MB × # batches 

• WIP is transient and grows and shrinks as needed 

– Error/Warning Log data – typically negligible and can be purged as needed 

– Reporting data 

• All reporting log rules off – 0 MB 

• Some or all reporting log rules are on 

Typically data grows 100 MB – 3 GB per hour 

Shrinks only when purged 

But… 

» Growth rate depends on page volume and which log rules are on 

» Overall size depends on # days of data retained 

– Audit Log data 

• All audit log rules off – 0 MB 

• All audit logs rules on – 2× the growth rate of Reporting data 


13


Data Volume and Rates 

• InputAccel Database transaction rates 

Log Rules Enabled 

Estimated Transaction Rate 

None IAS Tasks / Hour × 0.075 

Reports only (no Audit) IAS Tasks / Hour × 2.5 

Audit only (no Reports) IAS Tasks / Hour × 5.0 

Reports + Audit IAS Tasks / Hour × 7.5 


14


Improving Performance 

• Defragment and rebuild indexes 

– up_ReorganizeIndex – defragments all indexes 

– up_RebuildIndex – rebuilds all indexes 

• Purge reporting and auditing data 

– Reporting and Auditing tables grow continuously 

– You must schedule purges via the Admin Console 

• Recommendation: generate reports during non-peak hours 

– Generating Reports runs complex queries that place a heavy load on 

IADB and MSSQL 

• Store MSSQL transaction logs and data files on separate hard 

drives/controllers 


15

Tuning – Client Modules 

• Each client module has its own unique tuning characteristics 

– Example: val2xml slows as more IA Values are exported and is slower when triggered at 

level 1 than level 7 

– See “EMC® Captiva® InputAccel® and Dispatcher Version 6.5 Performance Sizing and 

Tuning Guide” for details 

• Parameters on the client machines can be modified to optimize performance 

– Stored in settings.ini located in %ALLUSERSPROFILE%\EMC\InputAccel 

– PrefetchDefault (default = 2) – the number of additional tasks the InputAccel Server 

sends to each client module 

– FileCacheSize (default = 8) 

– CacheSize (default = 1,048,576) 

– CacheCount (default = 200,000), previously 20,000 – the number of files and IA Values 

the client module caches 

– IAClientDebug (default = 0) – set to 1 to capture the debug log iaclient.log in 

%ALLUSERSPROFILE%\EMC\InputAccel (previously was created in C:\) 


16


• CPU intensive client modules like NuanceOCR & 

ImageEnhancement 

– Require fast CPUs 

– Run one instance for each CPU core. 

• e.g. 4 CPU cores, run 4 instances of NuanceOCR 

• Non-CPU intensive client modules like val2xml & Documentum 

Export 

– Performance is limited by other resources (disk, network) 

– Run at least one instance for each CPU core and possibly more 


17


Using InputAccel over a WAN… 

• InputAccel Server, Database and Administration Console 

– Require high-speed, low-latency connections 

– Must be on the same LAN as each other 

• Unattended client modules 

– Often require high-speed, low-latency connections for best throughput 

– Generally do not need to be remote from the InputAccel Server 

– Should be on the same LAN as the InputAccel Server 

• Attended client modules 

– Performance varies by environment and module 

– Detailed guidance is in the EMC® Captiva® InputAccel® and Dispatcher 

Version 6.5 Performance Sizing and Tuning Guide 

– Recommendation: do benchmark testing to ensure adequate performance 


18


…Using InputAccel over a WAN 

• Attended client module details 

– ScanPlus 

• Not recommended, but may perform acceptably with scanner hardware compression or small 

scanned images 

• Works best when bandwidth is ≥50 Mbps and round trip latency ≤25 ms 

– IndexPlus 

• Generally performs well over a WAN (except for thumbnail display) 

• Displaying the batch list takes about 25% longer 

• Works best when bandwidth is ≥1.5 Mbps and round-trip latency ≤50 ms 

– Dispatcher Classification Edit and Dispatcher Validation 

• Should not be used over a WAN 

• Recommendation: consider using VMware View or Citrix for remote operators 

– Module executes on the LAN, screen display is over the WAN 

– Supports remote scanning 

• ScanPlus is on the LAN, the scanner is on the remote machine 

• Use scanner hardware compression for best results 

– Maximizes InputAccel Server-to-client module throughput 

– EMC OnDemand uses VMware View 


19

Tuning – Capture Workflow 

Trigger Levels 

• Triggering at level 0 or 1 

– Usually gives better throughput than level 7 

– The tasks within a batch can be distributed among many client modules 

– Which results in faster end-to-end processing of any single batch 

• Triggering at level 7 

– Is less work for InputAccel Server as it has fewer tasks to manage 

– Under some circumstances may provide better overall throughput at the 

expense that any single batch may take more time to process 

• Unsupported: accessing external resources in IPP VBA code 

– VBA execution with InputAccel Server is single-threaded 

– The external resource may be slow or not present 

– If InputAccel Server needs to wait for the resource, all other tasks block 

– Put more complicated custom code logic on the client through the .NET Code 

module or client scripting 


20

Diagnosing Performance Issues 

• IA Server Performance Counter 

– Batches loaded and loads/second 

– Connections 

– Disk bytes written & read/second 

– VBA calls/second & queue length 

– Processing Message Count 

– Network bytes written & read/second 

– Packets send & received/second 

– Pending I/O (~ # of asynchronous sends in progress) 

– Event (db) queue length 

• Data Access Layer Performance Counters 

– Data Requests/second 

– % Load Factor 

– Avg. Execution Time Millisec 

– Current connection count 


21

EMC OnDemand 

Captiva Instant Cloud Implementation 


22

IIG Applications on EMC OnDemand 

ENTERPRISE 

CAPTURE 

CONTENT 

MANAGEMENT 

CUSTOMER 

COMMUNICATIONS 

CASE 

MANAGEMENT 

INFORMATION 

GOVERNANCE 

Captiva 

Documentum 

ECM 

Document 

Sciences 

Documentum 

xCP 

SourceOne 

Cloud Management 

Virtualization and Security 

Storage 

Networks 


23

Summary/Key Takeaways 

• Focus on system throughput, not per-task time 

• Use high-speed hardware – disk drives and networks 

• Minimize disk I/O where possible 

– Keep active batches in memory 

– Avoid excessive reporting 

• Parallelize – multi-core CPUs and task granularity 

• Use performance counters to find bottlenecks 


24

Q&A 

Chris Lund 

chris.lund@emc.com 

(858) 320-1215 


25

Learn More About EMC Captiva 

Go to: www.EMC.com/Captiva 


26

Captiva @ Momentum 2011 

Thursday 

10:00 AM AP Automation: Best Practices for Capturing and 

Integrating Paper Invoices into your Accounts Payable 

Processes 

Galileo 705 


27

GET SOCIAL 

DOWNLOAD 

Apple: http://bit.ly/MMTM11 

Other: http://bit.ly/MMTMemc 

Come to the Momentum Lounge today 

to play, win, learn and more. 


28


29

THANK YOU 


30

taking advantage of the emc captiva architecture - EMC Community ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?