Performance Tuning Siebel Software on the Sun Platform

<strong>Performance</strong> <strong>Tuning</strong> 

<strong>Siebel</strong> <strong>Software</strong> 

on the Sun Platform 

Khader Mohiuddin 

Engineering Lead Sun-<strong>Siebel</strong> Alliance 

Market Development Engineering 

Sun Microsystems, Inc. 

June 2006

Copyright 2006 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights 

reserved. 

U.S. Government Rights - Commercial software. Government users are subject to the Sun Microsystems, Inc. standard 

license agreement and applicable provisions of the FAR and its supplements. Use is subject to license terms. 

This distribution may include materials developed by third parties. 

Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a 

registered trademark in the U.S. and in other countries, exclusively licensed through X/Open Company, Ltd. X/Open 

is a registered trademark of X/Open Company, Ltd. 

Sun, Sun Microsystems, the Sun logo, Solaris, Sun Fire, Sun Enterprise, StorEdge, Java, and “The Network 

Is The Computer” are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. 

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, 

Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon architecture developed by 

Sun Microsystems, Inc. 

This product is covered and controlled by U.S. Export Control laws and may be subject to the export or import laws in 

other countries. Nuclear, missile, chemical biological weapons or nuclear maritime end uses or end users, whether 

direct or indirect, are strictly prohibited. Export or reexport to countries subject to U.S. embargo or to entities 

identified on U.S. export exclusion lists, including, but not limited to, the denied persons and specially designated 

nationals lists is strictly prohibited. 

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, 

REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF 

MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE 

DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY 

INVALID. 

<strong>Performance</strong> <strong>Tuning</strong> <strong>Siebel</strong> <strong>Software</strong> on the Sun Platform Page 2

Abstract 

This paper discusses the performance optimization of a complete <strong>Siebel</strong> enterprise 

solution on the Sun platform. The article covers tuning for the Solaris TM Operating 

System, <strong>Siebel</strong> software, Oracle database server, Sun StorEdge TM products, and Sun 

Java TM System Web Server. We also discuss unique features of the Solaris OS that 

reduce risk while helping to improve the performance and stability of <strong>Siebel</strong> applications. 

All of the techniques described here are lessons learned from a series of performance 

tuning studies that were conducted under the auspices of the <strong>Siebel</strong> Platform Sizing and 

<strong>Performance</strong> Program (PSPP). 


1 <strong>Tuning</strong> for Price/<strong>Performance</strong>: Summary...................................................................................7 

2 <strong>Siebel</strong> Application Architecture Overview..................................................................................9 

3 Optimal Sun/<strong>Siebel</strong> Architecture for Benchmark Workload..................................................11 

3.1 Hardware and <strong>Software</strong> Used................................................................................................ 13 

4 Workload Description................................................................................................................. 14 

4.1 OLTP (<strong>Siebel</strong> Web Thin Client End Users)............................................................................14 

4.2 Batch Server Components...................................................................................................... 14 

5 10,000 Concurrent Users: Test Results Summary.................................................................... 15 

5.1 Response Times and Transaction Throughput....................................................................... 15 

5.2 Server Resource Utilization ...................................................................................................16 

6 <strong>Siebel</strong> Scalability on the Sun Platform.......................................................................................17 

7 <strong>Performance</strong> <strong>Tuning</strong>....................................................................................................................20 

7.1 <strong>Tuning</strong> Solaris OS for <strong>Siebel</strong> Server.......................................................................................21 

7.1.1 Solaris MTmalloc <strong>Tuning</strong> for <strong>Siebel</strong>............................................................................... 21 

7.1.2 Solaris Alternate Threads Library Usage.........................................................................22 

7.1.3 The Solaris Kernel and TCP/IP <strong>Tuning</strong> Parameters for <strong>Siebel</strong> Server............................ 23 

7.2 <strong>Tuning</strong> <strong>Siebel</strong> Server for the Solaris OS.................................................................................23 

7.2.1 <strong>Tuning</strong> Call Center, Sales/Service, and eChannel <strong>Siebel</strong> Modules................................. 24 

7.2.2 Workflow .......................................................................................................................26 

7.2.3 Assignment Manager <strong>Tuning</strong>...........................................................................................27 

7.2.4 EAI-MQseries..................................................................................................................28 

7.2.5 EAI-HTTP Adapter..........................................................................................................29 

7.3 <strong>Siebel</strong> Server Scalability Limitations and Solutions.............................................................. 30 

7.3.1 The <strong>Siebel</strong> MaxTasks Upper Limit Problem................................................................... 30 

7.3.2 Bloated <strong>Siebel</strong> Processes (Commonly Mistaken as Memory Leaks)...............................34 

7.4 <strong>Tuning</strong> Sun Java System Web Server......................................................................................36 

7.5 <strong>Tuning</strong> the <strong>Siebel</strong> Web Server Extension (SWSE)..................................................................38 

7.6 <strong>Tuning</strong> <strong>Siebel</strong> Standard Oracle Database and Sun Storage.................................................. 39 

7.6.1 Optimal Database Configuration..................................................................................... 39 

7.6.2 Properly Locating Data on the Disk for Best <strong>Performance</strong>..............................................40 

7.6.3 Disk Layout and Oracle Data Partitioning.......................................................................41 

7.6.4 Solaris MPSS <strong>Tuning</strong> for Oracle Server..........................................................................44 

7.6.5 Hot Table <strong>Tuning</strong> and Data Growth................................................................................ 47 


7.6.6 Oracle Parameters <strong>Tuning</strong>................................................................................................47 

7.6.7 Solaris Kernel Parameters on Oracle Database Server.................................................... 50 

7.6.8 SQL Query <strong>Tuning</strong>...........................................................................................................50 

7.6.9 Rollback Segment <strong>Tuning</strong>............................................................................................... 53 

7.6.10 Database Connectivity Using Host Names Adapter...................................................... 53 

7.6.11 High I/O with Oracle Shadow Processes Connected to <strong>Siebel</strong>......................................54 

7.7 <strong>Siebel</strong> Database Connection Pooling.....................................................................................55 

7.8 <strong>Tuning</strong> Sun Java System Directory Server (LDAP)................................................................56 

8 <strong>Performance</strong> Tweaks with No Gains..........................................................................................57 

9 Scripts, Tips, and Tricks for Diagnosing <strong>Siebel</strong> on the Sun Platform.................................... 59 

9.1 To Monitor <strong>Siebel</strong> Open Session Statistics.............................................................................59 

9.2 To List the Parameter Settings for a <strong>Siebel</strong> Server................................................................ 59 

9.3 To Find Out All OMs Currently Running for a Component.................................................. 59 

9.4 To Find Out the Number of Active Servers for a Component................................................ 60 

9.5 To Find Out the Tasks for a Component................................................................................ 60 

9.6 To Set Detailed Trace Levels on the <strong>Siebel</strong> Server Components (<strong>Siebel</strong> OM).......................60 

9.7 To Find Out the Number of GUEST Logins for a Component...............................................60 

9.8 To Calculate the Memory Usage for an OM..........................................................................61 

9.9 To Find the Log File Associated with a Specific OM.............................................................61 

9.10 To Produce a Stack Trace for the Current Thread of an OM.............................................. 62 

9.11 To Show System-Wide Lock Contention Issues Using lockstat............................................ 62 

9.12 To Show the Lock Statistic of an OM Using plockstat ........................................................ 64 

9.13 To “Truss” an OM............................................................................................................... 65 

9.14 How to Trace the SQL Statements for a Particular <strong>Siebel</strong> Transaction............................. 65 

9.15 Changing Database Connect Stringvi $ODBCINI and Editing the field .ServerName....... 66 

9.16 Enabling/Disabling <strong>Siebel</strong> Components...............................................................................66 

10 Appendix A: Transaction Response Times............................................................................. 68 

11 Appendix B: Database Objects Growth During the Test.......................................................74 

12 Appendix C: Oracle statspack Report ....................................................................................77 

13 References...................................................................................................................................79 


Introduction 

To ensure that the most demanding global enterprise customers can meet their 

deployment requirements, engineers from <strong>Siebel</strong> Systems and Sun are working jointly on 

several engineering projects. Their common goal is to further enhance <strong>Siebel</strong> server 

performance on Sun’s highly-scalable Solaris Operating System. 

This article is an effort to document and spread knowledge of tuning and optimizing 

<strong>Siebel</strong> 7 eBusiness Applications Suite on the Solaris platform. All of the techniques 

discussed here are lessons learned from a series of performance tuning studies 

conducted under the auspices of the <strong>Siebel</strong> Platform Sizing and <strong>Performance</strong> Program 

(PSPP). The tests conducted under this program are based on real-world scenarios 

derived from <strong>Siebel</strong> Systems customers, which reflect some of the most frequently used 

and most critical components of the <strong>Siebel</strong> eBusiness Applications Suite. This article also 

provides tips and best practices--based on our experience--for field staff, benchmark 

engineers, system administrators, and customers who are interested in achieving optimal 

performance and scalability with their <strong>Siebel</strong>-on-Sun installations. The following areas are 

addressed in this paper: 

• What are the unique features of the Solaris Operating System that reduce risk 

while helping to improve the performance and stability of <strong>Siebel</strong> applications? 

• For maximum scalability at a low cost, what is the optimal way to configure <strong>Siebel</strong> 

on the Solaris OS? 

• How does Sun’s Chip Multithreading (CMT) technology based on the UltraSPARC 

IV processor benefit <strong>Siebel</strong> solutions? 

• How can transaction response times be improved for end users in large <strong>Siebel</strong>-on- 

Sun deployments? 

• How can an Oracle database running on Sun StorEdge be tuned for higher 

performance for <strong>Siebel</strong> software? 

The performance and scalability testing was conducted at Sun’s Enterprise Technology 

Center (ETC) in Menlo Park, California, by Sun’s Market Development Engineering 

(MDE) with assistance from <strong>Siebel</strong> Systems. The ETC is a massive, distributed testing 

facility packing more computer power than many Fortune 1000 corporations. A facility of 

this magnitude provides the resources needed to test the limits of software on a much 

greater scale than most enterprises will ever require. 


1 <strong>Tuning</strong> for Price/<strong>Performance</strong>: Summary 

The Solaris Operating System is the cornerstone software technology that enables Sun 

to deliver high performance and scalability for <strong>Siebel</strong> applications; and the OS contains a 

number of features that enable customers to tune for optimal price/performance levels. 

Among the core features of the Solaris OS that contributed to the superior results 

achieved in the tests: 

Solaris MTmalloc: A standard feature with the Solaris system was enabled on the 

<strong>Siebel</strong> application servers. This is an alternate memory allocator module that was built 

specifically for multithreaded applications such as <strong>Siebel</strong>. The MTmalloc routines 

provide a faster, concurrent malloc implementation. This feature resulted in lowering 

CPU consumption by 35%. Though memory consumption doubled as a side effect, 

overall price/performance benefits are positive. There have been improvements made on 

MTmalloc in Solaris 10 OS which reduce the space-inefficiency penalty. More details on 

this topic are available in Section 7.1.1. 

<strong>Siebel</strong> Process Size: For an application process running on the Solaris OS, the default 

setting for stack and data size is unlimited. We found that <strong>Siebel</strong> software running with 

the default Solaris system setting caused bloated stack size and runaway processes that 

compromised scalability and stability of <strong>Siebel</strong> on the Solaris OS. Limiting the stack size 

to 1MB and increasing the data size limit to 4GB resulted in increased scalability and 

higher stability. Both adjustments let a <strong>Siebel</strong> process use its process address space 

more efficiently, fully utilizing the total process address space of 4GB available to a 32-bit 

application process. A significant drop in failure rate of transactions (only eight failures 

out of 1.2 million total transactions!) was observed as a result of these two changes. 

More details on this topic are available in Section 7.3. 

Solaris Alternate Threads Library: Solaris 8 OS provides an alternate threads 

implementation with a one-level model (1x1) in which user-level threads are associated 

one-to-one with lightweight processes (LWPs). This implementation is simpler than the 

standard (MxN) two-level model in which user-level threads are multiplexed over 

(possibly) fewer LWPs. When used on <strong>Siebel</strong> Application Servers, the 1x1 model 

provided good performance improvements to the <strong>Siebel</strong> multithreaded applications. More 

details on this topic are available in Section 7.1.2. 

Solaris Multiple Page Size Support (MPSS): A standard feature available in Solaris 9 

OS and subsequent versions, MPSS gives applications the ability to run on the same OS 

with more than one page size. Use of this Solaris library allows a larger heap size to be 

set for certain applications, which results in better performance due to reduced TLB 

miss%. This feature was enabled on the Oracle server and resulted in performance 

gains. More details on this topic are available in Section 7.6.4. 

Solaris Resource Manager was used on all tiers of the setup to efficiently manage CPU 

resources. This resulted in fewer process migrations and translated to higher cache hits. 


Sun Storage: Oracle database files were laid out on a Sun StorEdge SE6320 system. 

I/O balancing was implemented on the <strong>Siebel</strong> workload to reduce hot spots. Also zone 

bit recording was used on disks to provide higher throughput to the <strong>Siebel</strong> transactions. 

Direct I/O was enabled on certain Oracle files and on the <strong>Siebel</strong> file system. More details 

on this topic are available in Section 7.6.2. 

Connection Pooling: <strong>Siebel</strong>’s database connection pooling was used, providing good 

benefits for CPU and memory. Twenty end users shared a single connection to the 

database. More details on this topic are available in Section 7.7. 

Usage of Appropriate Sun Hardware: Pilot tests were performed to characterize the 

performance of the web, application, and database across the current Sun product line. 

Hardware was chosen based on best price/performance rather than pure performance. 

Details on this topic are available in Section 3. 


2 <strong>Siebel</strong> Application Architecture Overview 

<strong>Siebel</strong> server is a flexible and scalable application server platform that supports a variety 

of services operating on the middle tier of the <strong>Siebel</strong> N-tier architecture, including data 

integration, workflow, data replication, and synchronization service for mobile clients. 

Figure 2.1 provides a high-level view of the <strong>Siebel</strong> application suite architecture. 

Figure 2.1 

<strong>Siebel</strong> server includes business logic and infrastructure for running the different CRM 

modules as well as connectivity interfaces to the back end database. It consists of 

several multithreaded processes commonly referred to as ‘<strong>Siebel</strong> Object Managers’. 

These can be configured so that several instances of it can run on a single Solaris 

machine. The <strong>Siebel</strong> 7.x server makes use of gateway components to track user 

sessions. 

<strong>Siebel</strong> 7.x has a thin client architecture for connected clients. The <strong>Siebel</strong> 7.x thin client 

architecture is enabled through the <strong>Siebel</strong> plug-in (SWSE – <strong>Siebel</strong> Webserver Extension) 

running on the web server. It's the primary interface between the client and the <strong>Siebel</strong> 


application server. For more information on the individual <strong>Siebel</strong> components please refer 

to <strong>Siebel</strong> product documentation at www.siebel.com. 


3 Optimal Sun/<strong>Siebel</strong> Architecture for Benchmark Workload 

Sun offers a wide variety of products ranging from hardware and networks to software 

and storage systems. To obtain the best price/performance from an application, one 

needs to determine which Sun products are appropriate. This selection process can be 

achieved by understanding the application's characteristics, picking Sun products suitable 

to those application characteristics, conducting a series of tests, and then finalizing the 

choice of machines. 

For this project, tests were done to characterize web, application, and database 

performance across the current Sun product line. Hardware was selected based on best 

price/performance rather than pure performance criteria. Figure 3.1 illustrates the 

hardware configuration used in the Sun ETC testing. (Note: "SF" stands for Sun Fire 

server.) 

Load 

Generators 

SF V65x 

Web Servers 

SF V440/SF V880 

<strong>Siebel</strong> App Servers 

<strong>Siebel</strong> Gateway 

Sun Java 

System Directory 

Server 

SF V890 

SF V440 

SF E2900 

SF E2900 

SF V440 

SF V440 

SF 

V480 

Sun StorEdge 

SE 6320 

Key: 

Network traffic between load generators and web servers 

Network packets between web servers to app servers 

Point-Point Gbic between each app server and database server 

Figure 3.1 


After collecting detailed knowledge of <strong>Siebel</strong>'s performance characteristics, the 

hardware/network topology depicted in Figure 3.1 was created. The <strong>Siebel</strong> end user 

workload (OLTP) was distributed across three nodes: Sun Fire TM V890, Sun Fire E2900, 

and Sun Fire V440 servers. Each node ran all the <strong>Siebel</strong> components under test, that is, 

Call Center, eService, eSales, and eChannel. <strong>Siebel</strong> server component jobs (batch tests 

EAI-HTTP, EAI-MQ, WorkFlow, and HTTP-adapter) were distributed across Sun Fire 

V440 and Sun Fire V480 servers. The <strong>Siebel</strong> gateway and the Sun Java System 

Directory Server (LDAP) were intentionally placed on one physical machine (the Sun Fire 

V440 server) because they are very low consumers of CPU and memory resources. With 

the Sun Ethernet GIG connectivity, network throughput was adequate. Each of the 

machines had three network interface cards (NICs), which we used to isolate the main 

categories of network traffic: 

1. End-user (load generator)-to-web-server traffic (shown in green) 

2. Web server-to-gateway-to-<strong>Siebel</strong>-applications-server traffic (shown in black) 

3. <strong>Siebel</strong>-applications-server-to-database-server traffic (shown in red) 

The networking was done using a Cisco router (Catalyst 4000). Two VLANs were created 

to separate network traffic between (1) and (2), while (3) was further optimized with 

individual point-to-point network interfaces from each application server to the database. 

This separation was done to alleviate any possible network bottlenecks at any tier which 

could have occurred as a result of simulating thousands of <strong>Siebel</strong> users. The load 

generators were all Sun Fire V65 servers running Mercury LoadRunner software. The 

load was spread across three web server machines by directing different kinds of users 

(such as Call Center or eService users, and so on) to the three web servers. 

All of the <strong>Siebel</strong> application servers belonged to a single <strong>Siebel</strong> enterprise. A single 

E2900 server hosted the Oracle database; this was connected to a Sun StorEdge TM 

SE6320 system using a fiber channel. 


3.1 Hardware and <strong>Software</strong> Used 

Gateway Server/LDAP: 

• 1 x Sun Fire V440 

o 1 x 1.2 GHz UltraSPARC IIIi 

o 16 GB RAM 

o Solaris 8 OS 2/04 Generic_117350-02 

o Sun Java System Directory Server 

LDAP 4.1 SP9 

Application Servers: 

• 1x Sun Fire V890 

o 8 x 1.2GHz UltraSPARC IV 

o 32 GB RAM 


o <strong>Siebel</strong> 7.5.2 

• 1 x Sun Fire E2900 

o 12 x 1.2 GHz UltraSPARC IV 

o 48 GB RAM 




o 4 x 1.2 GHz UltraSPARC IIIi 

o 16 GB RAM 




o 4 x 1.2 GHz UltraSPARC IIIi+ 

o 16 GB RAM 



o IBM MQ Series 5.2 FP2 

Database Server: 

• 1 x Sun Fire E2900 

o 4 x 1.2 GHz UltraSPARC IV 

o 16 GB RAM 

o Solaris 9 OS 2/04 

Generic_117150-05 

o Oracle 9.2.0.2 32-bit 

o Sun StorEdge SE6320 

Storage Array 4 trays (2+2), 

4x14x36GB 15Krpm FC-AL 

drives. 

Load Runner Drivers: 

• 5 x Sun Fire V65x 

o 4 x 3.02 GHz Xeon 

o 3 GB RAM 

o Windows XP SP1 

o Mercury LoadRunner 7.5.1 

Web Servers: 


o 4 x1.2 GHz UltraSPARC IIIi 

o 16 GB RAM 


Generic_117350-02 

o Sun Java System Web Server 

6.0 SP2 

o <strong>Siebel</strong> 7.5.2 SWSE 


o 2 x 900 MHz UltraSPARC IIIi 

o 16 GB RAM 


Generic_108528-13 

o Sun Java System Web Server 

6.0 SP2 

o <strong>Siebel</strong> 7.5.2 SWSE 


4 Workload Description 

All of the tuning discussed in this document is specific to the PSPP workload as defined 

by <strong>Siebel</strong> Systems. The workload was based on scenarios derived from large <strong>Siebel</strong> 

customers to reflect some of the most frequently used and most critical components of 

the <strong>Siebel</strong> eBusiness Application Suite. At a high level the workload for these tests can 

be categorized into the two categories: (1) OLTP and (2) batch server components. 

4.1 OLTP (<strong>Siebel</strong> Web Thin Client End Users) 

OLTP simulated the real world requirements of a large organization with 10,000 

concurrent users involved in the following tasks and functions (in a mixed ratio): 

• Call Center (sales and service representatives) – 7000 concurrent users 

• Partner Relationship Management (partner organizations) – eChannel, 

1000 concurrent users 

• Web sales (customers) – eSales, 1000 concurrent users 

• Web service (customers) – eService, 1000 concurrent users 

The end users were simulated using LoadRunner version 7.51 SP1 from Mercury 

Interactive, with a think time in the range of 5 to 55 seconds (or an average of 30 

seconds) between user operations. 

4.2 Batch Server Components 

The batch component of the workload consisted of: 

1. <strong>Siebel</strong> Assignment Manager 

2. <strong>Siebel</strong> Workflow 

3. <strong>Siebel</strong> EAI MQ Series Adapter 

4. <strong>Siebel</strong> EAI-HTTP Adapter 

The <strong>Siebel</strong> 7 Assignment Manager processed assignment transactions for sales 

opportunities based on employee positions and territories. <strong>Siebel</strong> 7 Workflow Manager 

executed workflow steps based on inserted service requests. The <strong>Siebel</strong> 7 EAI MQ 

Series Adapter read from and placed transactions into IBM MQ Series queues. The 

<strong>Siebel</strong> 7 EAI-HTTP Adapter executed requests between different web infrastructures. 

All of the tests were conducted by making sure that both the OLTP and batch 

components were run in conjunction for a one hour period (steady state) within the same 

<strong>Siebel</strong> enterprise installation. 


5 10,000 Concurrent Users: Test Results Summary 

The test system demonstrated that <strong>Siebel</strong> 7 architecture on Sun Fire servers and Oracle 

9i database easily scales to 10,000 concurrent users. 

● Vertical scalability. The <strong>Siebel</strong> 7 Server showed excellent scalability within 

an application server. 

● 

● 

● 

Horizontal scalability. The benchmark demonstrates scalability across 

multiple servers without degradation. 

Low network utilization. The <strong>Siebel</strong> 7 Smart Web Architecture and Smart 

Network Architecture efficiently managed the network consuming only 5.5 

kbps per user. 

Efficient use of the database server. The <strong>Siebel</strong> 7 Smart Database 

Connection Pooling and Multiplexing allowed the database to service 

10,000 concurrent users and the supporting <strong>Siebel</strong> 7 Server application 

services with 480 database connections. 

The actual results of the performance and scalability tests conducted at the Sun ETC for 

the <strong>Siebel</strong> workload are summarized in the following section of the article. Chapter 8 

resents specific performance tuning tips and the methodology used to achieve this level 

of performance. 

5.1 Response Times and Transaction Throughput 

Workload 

Number of Average Business Workload 

Business 

Users Operation 

Response 

Time (Sec) 

Transactions 

Throughput/ 

Hour 

Transactions 

Throughput/ 

Hour 

Call Center (Sales and Service) 7,000 .126 34,778 Assignment Manager 11,427 

Partner Relationship Management 1,000 .303 12,540 EAI - HTTP Adapter 278,352 

eSales 

EAI - MQ Series 

1,000 .274 6,775 Adapter 181,319 

eService 1,000 .199 12,870 Workflow Manager 53,439 

Totals 10,000 N/A 66,964 

Table 5.1 


5.2 Server Resource Utilization 

Figure 5.2 

Node Functional Use % CPU 

Utilization 

1 x Sun Fire E2900 (4CPU,16GB) Oracle Database Server 51 1690 

1 x Sun Fire V480 (4CPU,16GB) <strong>Siebel</strong> App Server – EAI-HTTP + WorkFlow 65 2075 

1 x Sun Fire V440 (4CPU, 16GB) <strong>Siebel</strong> App Server – AM + EAI MQ series 37 1742 

1 x Sun Fire V440 (4CPU, 16GB) <strong>Siebel</strong> App Server – 1600 end users 91 10,666 

1 x Sun Fire E2900 (12CPU, 48GB) <strong>Siebel</strong> App Server – 4800 end users 66 38,956 

1x Sun Fire V890 (8CPU,32GB) <strong>Siebel</strong> App Server – 3600 end users 68 25,871 

Sun Java System Web Server – HTTP Adapter, 

8 

1x Sun Fire V440 (4CPU, 16GB) WorkFlow 

126 

1 x Sun Fire V880 (4CPU,16GB) Sun Java System Web Server – Application requests 49 186 

1 x Sun Fire V440 (4CPU, 16GB) Sun Java System Web Server – Application requests 54 225 

1 x Sun Fire V440 (4CPU, 16GB) 

<strong>Siebel</strong> Gateway Server/Sun Java System Directory 

11 81 

Server 

Table 5.2 

Memory 

Utilization (MB) 


6 <strong>Siebel</strong> Scalability on the Sun Platform 

On the Sun platform the <strong>Siebel</strong> application scales to high concurrent users extremely 

well. With <strong>Siebel</strong>’s flexible distributed architecture and Sun’s large server product line, an 

optimal <strong>Siebel</strong>-on-Sun deployment can be achieved either with several small (one to four 

CPU) machines or with a single large Sun machine (such as a E15K, E6900, and so on). 

The following graphs depict <strong>Siebel</strong> scalability on different Sun machine types. All the 

tuning that was applied to achieve these results (as documented in the next chapter in 

this article) can be applied on production systems. 

Figure 6.1 


Figure 6.2 

The V440 server uses UltraSPARC III chips while the V890 and E2900 use UltraSPARC 

IV chips. Figure 6.2 shows the difference in scalability between the classes of Sun 

machines. Customers can this data for the capacity planning and sizing of their real world 

deployments. Keep in mind that the workload used for these results should be identical to 

the customer real-world deployment or appropriate adjustments need to be made to the 

server sizing. For more workload details, please see Section 4. 


Cost per <strong>Siebel</strong> User on Sun Platform 

Figure 6.3 

*Note:$$/user number is based purely on hardware cost and does not include environmental factors, 

facilities, service, or management. 

Sun servers provide the best price/performance for <strong>Siebel</strong> applications. Figure 6.3 

depicts the costs of the typical <strong>Siebel</strong> user on various models of Sun servers tested. 

Price/<strong>Performance</strong> Summary Per Tier of the Sun-<strong>Siebel</strong> Deployment 

 

 

 

 

 

Application tier: 

Sun Fire V440: 440 users/CPU ($17.57/user) 

Sun Fire V890: 662 users/CPU ($23.42/user) 

Sun Fire E2900: 606 users/CPU ($37.54/user) 

Database tier (Sun Fire E2900): 

4902 users/CPU ($5.46/user) 

Web tier (Sun Fire V440): 

2453 users/CPU ($3.15/user) 

Average response time: from 0.126 to 0.303 sec (component-specific) 

Success rate: > 99.999% (8 failures out of ~1.2 million transactions) 


7 <strong>Performance</strong> <strong>Tuning</strong> 

Many people think solving a performance problem is a mysterious talent, however there 

is a particular methodology at work. Figure 8.1 shows the process flow of approaching a 

performance problem or simply tuning the system for best performance. 

Figure 7 

Use of a methodology like the process shown in Figure 7.1 can take the black magic out 

of performance tuning. because there are several books and white papers dedicated to 

the subject of performance tuning methodologies, the subject is not being covered in 

detail in this white paper. The scope of this white paper is to provide the reader with 

specific tuneables for the <strong>Siebel</strong>/Sun platform. The current chapter provides specific 

suggestions for performance and scalability tuning. These suggestions are based on 

lessons learned from the tests conducted at the Sun ETC and through years of 

collaborative engineering between Sun and <strong>Siebel</strong>. 


7.1 <strong>Tuning</strong> Solaris OS for <strong>Siebel</strong> Server 

7.1.1 Solaris MTmalloc <strong>Tuning</strong> for <strong>Siebel</strong> 

The alternate memory allocator module, which is standard to the Solaris OS and was 

built specifically for multithreaded applications such as <strong>Siebel</strong>, was enabled on the <strong>Siebel</strong> 

application servers. The MTmalloc routines provide a faster, concurrent malloc 

implementation. This feature resulted in lowering CPU consumption by 35%. Though 

memory consumption doubled as a side effect, overall price/performance benefits were 

positive. In Solaris 10 OS, improvements have been made on MTmalloc to reduce the 

space-inefficiency cost. 

Effect of MTmalloc on CPU Benefit and Memory Cost 

Figure 7.1.1 

In Figure7.1.1, the blue curve shows the percentage reduction in CPU usage for <strong>Siebel</strong> 

applications running on different Sun Fire SMP machines with MTmalloc enabled . The 

red curve shows the corresponding increase in memory usage due to use of MTmalloc. 

As is apparent from Figure 7.1.1, enabling MTmalloc on a 4 CPU machine will not be 

beneficial; there is 51% increase in memory usage by <strong>Siebel</strong> while CPU utilization 

remains the same. <strong>Performance</strong> gains begin to get better when MTmalloc is used on 

<strong>Siebel</strong> server running on 8 CPU machines and upwards. On an 8 CPU machine (V890) 

one can expect 28% reduction in CPU utilization when using the MTmalloc feature. 

Though memory usage by the <strong>Siebel</strong> application increases when MTmalloc is tuned, 


this is not a big disadvantage, as the standard Sun machine configuration has ample 

memory. 

The v440 has 4 CPUs with 16Gbytes of memory. A typical <strong>Siebel</strong> user on the Sun 

platform uses about 4Mbytes of memory while the CPU use is much higher in 

comparison, so the higher memory footprint is a beneficial trade off. 

How does one enable MTmalloc? 

1. Edit the file $SIEBEL_ROOT/bin/siebmtshw. 

2. Add the line LD_PRELOAD=/usr/lib/libmtmalloc.so. 

3. Save the file and bounce <strong>Siebel</strong> servers. 

4. After <strong>Siebel</strong> restarts, verify that MTmalloc is enabled by executing the 

following command: 

$%pldd -p | grep -i mtmalloc 

7.1.2 Solaris Alternate Threads Library Usage 

Solaris 8 OS provides an alternate threads implementation with a one-level model (1x1). 

User-level threads are associated one-to-one with lightweight processes (LWPs). This 

implementation is simpler than the standard (MxN) two-level model in which user-level 

threads are multiplexed over possibly fewer lightweight processes. The 1x1 model was 

used on <strong>Siebel</strong> Application Servers and provided good performance improvements to the 

<strong>Siebel</strong> multithreaded applications because the <strong>Siebel</strong> 7.5.2 alternate threads library is 

enabled by default. If you are using a version of <strong>Siebel</strong> older than 7.5.2 then the alternate 

threads library is not enabled by default. 

Procedure to enable the alt thread feature: 

1. Update the env variable for the UNIX® user that runs <strong>Siebel</strong> server. 

2. If you are using Korn shell, open up the .profile for the <strong>Siebel</strong> owner 

and add this to the bottom of the file. 

3. Export LD_LIBRARY_PATH=/usr/lib/lwp:$LD_LIBRARY_PATH. 

4. Stop <strong>Siebel</strong>. 

5. Exit from shell and log in again so the change in .profile is in effect. 

6. Start <strong>Siebel</strong>. 

To verify if alt thread is indeed being used: 

pldd -p | grep -i lwp 

The preceding command should return the current Solaris lib, which is 

/usr/lib/lwp/. 


7.1.3 The Solaris Kernel and TCP/IP <strong>Tuning</strong> Parameters for <strong>Siebel</strong> Server 

Parameter Scope Default Tuned Value 

Value 

shmsys:shminfo_shmmax /etc/system 0xffffffffffffffff 

shmsys:shminfo_shmmin /etc/system 100 

shmsys:shminfo_shmseg /etc/system 200 

semsys:seminfo_semmns /etc/system 12092 

semsys:seminfo_semmsl /etc/system 512 

semsys:seminfo_semmni /etc/system 4096 

semsys:seminfo_semmap /etc/system 4096 

semsys:seminfo_semmnu /etc/system 4096 

semsys:seminfo_semopm /etc/system 100 

semsys:seminfo_semume /etc/system 2048 

msgsys:msginfo_msgmni /etc/system 2048 

msgsys:msginfo_msgtql /etc/system 2048 

msgsys:msginfo_msgssz /etc/system 64 

msgsys:msginfo_msgseg /etc/system 32767 

msgsys:msginfo_msgmax /etc/system 16384 

msgsys:msginfo_msgmnb /etc/system 16384 

ip:dohwcksum (for resonate /etc/system 

gbic) 

0 

rlim_fd_max /etc/system 1024 16384 

rlim_fd_cur /etc/system 64 16384 

sq_max_size /etc/system 2 0 

tcp_time_wait_interval ndd /dev/tcp 240000 60000 

tcp_conn_req_max_q ndd /dev/tcp 128 1024 

tcp_conn_req_max_q0 ndd /dev/tcp 1024 4096 

tcp_ip_abort_interval ndd /dev/tcp 480000 60000 

tcp_keepalive_interval ndd /dev/tcp 7200000 900000 

tcp_rexmit_interval_initial ndd /dev/tcp 3000 3000 

tcp_rexmit_interval_max ndd /dev/tcp 240000 10000 

tcp_rexmit_interval_min ndd /dev/tcp 200 3000 

tcp_smallest_anon_port ndd /dev/tcp 32768 1024 

tcp_slow_start_initial ndd /dev/tcp 1 2 

tcp_xmit_hiwat ndd /dev/tcp 8129 32768 

tcp_fin_wait_2_flush_interval ndd/dev/tcp 67500 675000 

tcp_recv_hiwat ndd /dev/tcp 8129 32768 

Table 7.1.3 

7.2 <strong>Tuning</strong> <strong>Siebel</strong> Server for the Solaris OS 

The key factor in tuning <strong>Siebel</strong> server performance is the number of threads or users per 

<strong>Siebel</strong> object manager (OM) process. <strong>Siebel</strong> server architecture consists of multithreaded 

server processes servicing different business needs. Currently, <strong>Siebel</strong> is 


designed such that one thread of a <strong>Siebel</strong> OM services one user session or task. The 

ratio of threads or users/process is configured using the <strong>Siebel</strong> parameters: 

• MinMTServers 

• MaxMTServers 

• MaxTasks 

From several tests conducted it was found that on the Solaris platform with <strong>Siebel</strong> 7.5 the 

following users/OM ratios provided optimal performance: 

• Call Center – 80 users/OM 

• eChannel – 40 users/OM 

• eSales – 50 users/OM 

• eService – 60 users/OM 

As you can see, the optimal ratio of threads/process varies with the <strong>Siebel</strong> OM and the 

type of <strong>Siebel</strong> workload per user. MaxTasks divided by MaxMTServers determines the 

number of users/process. For example, for 300 users the setting would be 

MinMtServers=6, MaxMTServers=6, Maxtasks=300. This would direct <strong>Siebel</strong> to 

distribute the users across the 6 processes evenly, with 50 users on each. The notion of 

anonymous users must also be considered in this calculation, as discussed in the Call 

Center section, the eChannel section, and so on. The prstat –v or the top command 

shows how many threads or users or being serviced by a single multithreaded <strong>Siebel</strong> 

process. 

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 

1880 pspp 504M 298M cpu14 28 0 0:00.00 10% siebmtshmw/69 

1868 pspp 461M 125M sleep 58 0 0:00.00 2.5% siebmtshmw/61 

1227 pspp 687M 516M cpu3 22 0 0:00.03 1.6% siebmtshmw/62 



1246 pspp 681M 509M cpu20 38 0 0:00.03 1.2% siebmtshmw/62 

A thread count of more than 50 threads per process is due to the fact that the count also 

includes some admin threads. If the MaxTasks/MTservers ratio is greater than 100, 

performance degrades in terms of longer transaction response times. The optimal users 

per process setting depends on the workload, that is, how busy each user is. 

7.2.1 <strong>Tuning</strong> Call Center, Sales/Service, and eChannel <strong>Siebel</strong> Modules 

Call Center or sccobjmgr-specific information is presented here. Eighty users/process 

was found to be the optimal ratio. 


In the file $SIEBEL_ROOT/bin/enu/uagent.cfg: 

● 

● 

● 

Set EnableCDA = FALSE to disable invoking CDA functionality. 

Set CommEnable = FALSE for Call Center, to disable downloading the CTI 

bar. 

Set CommConfigManager = FALSE. 

It should be noted that the preceding settings are required to be TRUE for scenarios 

where these functions are required. The benchmark setup required these settings to be 

disabled. 

To decide on the MaxTasks, MTservers, AnonUserPool settings, use the following 

example: 

Target No. of Users 4000 

AnonUserPool 400 

Buffer 200 

Maxtasks 4600 

MaxMTserver=MinMTserver 58 

Table 7.2 

If the target number of users is 4000, then AnonUserPool is 10% of 4000, or 400. Allow 

a 5% buffer, add them all together (4000+400+200=4600), and MaxTasks would be 

4600. Since we want to run 80 users/process, the MaxMTserver value will be 

4600/80=57.5 (rounded to 58). 

The AnonUserPool value is set in the eapps.cfg file on the <strong>Siebel</strong> Web Server 

Engine. If the load is distributed across multiple web server machines or instances, 

simply divide this number. In this example, if two web servers were being used, set 

AnonUserPool to 200 in each web server’s eapps.cfg file. Here is a snapshot of the 

file and the setting for Call Center: 

[/callcenter] 

AnonSessionTimeout = 360 

GuestSessionTimeout = 60 

SessionTimeout = 300 

AnonUserPool = 200 

ConnectString = siebel.tcpip.none.none://19.1.1.18:2320/siebel/SCCObjMgr 

Set the following env variables in the file $SIEBEL_HOME/siebsrvr/siebenv.csh or 

siebenv.sh: 

export SIEBEL_ASSERT_MODE=0 

export SIEBEL_OSD_NLATCH = 7 * Maxtasks + 1000 

export SIEBEL_OSD_LATCH = 1.2 * Maxtasks 

Set Asserts OFF by setting the environment variable SIEBEL_ASSERT_MODE=0. 


OSD LATCH will need to be set higher based on the number of users being run. This is 

the calculation to be followed: 

SIEBEL_OSD_NLATCH = 7 * Maxtasks + 1000 

SIEBEL_OSD_LATCH = 1.2 * Maxtasks 

Restart <strong>Siebel</strong> server after setting these environment variables. 

eChannel 

The same tuning information as Call Center applies for eChannel, except for two 

parameters:The users per process ratio giving the best performance for this workload 

was 40. The optimal AnonUserPool setting found was 30%. 

eService 

The same tuning information as Call Center applies for eService, except for two 

parameters. The users per process ratio giving the best performance for this workload 

was 60. The optimal AnonUserPool setting found was 30%. 

eSales 

Fifty users per <strong>Siebel</strong> object manager process provided better response times and 

throughput. The AnonUserPool setting is 20%. 

7.2.2 Workflow 

<strong>Siebel</strong> 7 Workflow Manager executed workflow steps based on inserted service requests. 

<strong>Siebel</strong> Workflow can be run in two modes: async or sync. Remote async mode is 

discussed here. The workflow test consists of 500 Call Center end users working on 

20,000 service requests. 

Connection pooling of 20:1 was implemented with 10% benefit.Modify uagent.cfg and 

eai.cfg under $SIEBEL_HOME/bin of the Batch application server to have the 

following settings: 

● 

● 

Change EnableCDA=FALSE 

Under [SWE] add: 

 

 

 

 

EnableShuttle=TRUE 

EnableReportsFromToolbar=TRUE 

EnableSIDataLossWarning=TRUE 

EnablePopupInlineQuery=TRUE 


7.2.3 Assignment Manager <strong>Tuning</strong> 

The <strong>Siebel</strong> 7 Assignment Manager (AM) processed assignment transactions for sales 

opportunities based on employee positions and territories. AM was run in batch 

assignment mode. 

For this component the main tunable is the requests parameter. 

Tuneable parameter: Requests 

Default value: 5000 

Value used in benchmark: 20 

Description 

Maximum number of requests read per iteration. This controls the maximum number of 

requests WorkMon reads from the requests queue within one iteration. 

It did not help to change deletesize to 400 (from default of 500) or to add indexes to 

the table S_ESCL_REQ . 

After reducing the number of requests per iteration from 5000 (default value) to 20 for 

component WorkMon, CPU utilization was reduced from 72% to 53% on the <strong>Siebel</strong> 

server node where AM and EAI-MQ tests were run together. 

Through put (txns/sec) Value for the Requests Average CPU utilization 

parameter 

78,511 5000 72.3 

4098 10 52.89 

8146 20 53.31 

Table 7.2.3 


The procedure to list/change the parameter requests for <strong>Siebel</strong> server follows: 

1. Connect to server manager at applications server level. 

2. srvrmgr:siebapp2> list param Requests for comp WorkMon 

PA_ALIAS PA_VALUE PA_DATATYPE PA_SCOPE .. 

PA_NAME 

-------- -------- ----------- --------- 

---------------------- 

Requests 5000 Integer Component .. 

Requests per iteration 

1 row returned. 

3. srvrmgr:siebapp2> change param Requests=20 for comp WorkMon 

Command completed successfully 

Note: To see the entire list of parameters for comp WorkMon, type the following: 

srvrmgr:siebapp2> list param for comp WorkMon 

7.2.4 EAI-MQseries 

The <strong>Siebel</strong> 7.5 EAI MQ Series Adapter read from and placed transactions into IBM MQ 

Series queues. This test is designed to receive 400,000 messages from IBM MQ series 

into the <strong>Siebel</strong> application. Messages are divided into different categories depending on 

the type of operations they perform during receive. This test stresses the file system on 

which the queues reside by performing about 10% database inserts and 10% updates. 

Persistent queues are used. As a result, the database tuning and disk layout explained in 

this document are key for performance. 

To achieve best throughput the following setup was done: 

Minmtservers=maxmtservers=1, Maxtasks=45 for component MqSeriesSrvRcvr on the <strong>Siebel</strong> 

server. 

Moving the following directories to a different disk helped to improve performance by 

alleviating I/O bottlenecks: 

/mqm/qmgrs/PERFQMGR/queues/SENDQUEUE - Sendqueue 

/mqm/qmgrs/PERFQMGR/queues/RECEIVEQUEUE - Receivequeue 

/mqm/qmgrs/PERFQMGR/queues/LOG(ACTIVE) - Active Logs 


MQ series parameter tuning: 

PERFQMGR.OAMPipe_msg=2000000 (changed from default) 

in file /var/mqm/qmgrs/PERFQMGR/qmstatus.ini 

LogBufferPages=512 (changed from default 17) 

in file /var/mqm/qmgrs/PERFQMGR/qm_ini 

logPrimary files=12 (default value 2) 

LogSecondaryfiles=2 (default value 2) 

Logfilepages =1024 (changed to 16384(max)) 

Logtype=CIRCULAR(default CIRCULAR) 

From all of the above tweaks, a performance improvement of 35% in throughput was 

measured. Note: It was discovered during the benchmark that MQ Series version 5.1/5.2 

has a 640,000 message limit, while 5.3 does not. 

7.2.5 EAI-HTTP Adapter 

The <strong>Siebel</strong> 7 EAI-HTTP Adapter executed requests between different web 

infrastructures. <strong>Siebel</strong> EAI-HTTP Adapter Transport Business Service lets one send XML 

messages over HTTP to a target URL (web site). The <strong>Siebel</strong> Web Engine serves as the 

transport to receive XML messages sent over the HTTP protocol to <strong>Siebel</strong>. 

The <strong>Siebel</strong> EAI compgrp is enabled on one appserver and the HTTP driver is run on a 

Sun Fire v65/Windows XP machine. 

To achieve the result of 287,352 business transactions/hour, 4 threads were run 

concurrently, where each thread worked on 70,000 records. The optimum values for the 

number of <strong>Siebel</strong> servers of type EAIObjMgr for this workload were: 

MaxMTServers=2 minmtservers=2 MaxTasks=10 

The other <strong>Siebel</strong> OM parameters for EAIObjMgr are best left at default; changing 

SISPERSISSCON to anything from default of 20 does not help. All of the Oracle database 

optimizations explained under that section in this paper helped achieve the throughput of 

79 objects/sec. 

The HTTP driver machine was maxed out, causing slow performance. The reason for this 

was I/O, as the HTTP test program was generating about 3Gbytes of logs during the test; 

once this was turned, off as shown below, performance improved. 

driver.logresponse=false 

driver.printheaders=false 

The AnonSessionTimeout and SessionTimeout values in the SWSE were set to 

300. 

HTTP adapter (EAIObjMgr_enu) tunable parameters: 

● MaxTasks - Recommended value at the moment: 10 

● MaxMTServers, MinMTServers - Recommended value: 2 


● UpperThreshold - Recommended value: 25 

● LoadBalanced - Recommended value: true (default value, of course) 

● Driver.Count (at client) - Recommended value: 4 

7.3 <strong>Siebel</strong> Server Scalability Limitations and Solutions 

7.3.1 The <strong>Siebel</strong> MaxTasks Upper Limit Problem 

In order to configure <strong>Siebel</strong> server to run over 8000 concurrent users, the <strong>Siebel</strong> 

parameter MaxTasks has to be set to a value of 8800. When this is done, the <strong>Siebel</strong> 

object manager processes (that is, the <strong>Siebel</strong> servers) failed to start and logged error 

messages. This failure was not due to any resource limitation from the Solaris OS, as 

there were ample amounts of CPU, memory, and swap space on the machine. The 

failure to start was due to the E2900 with 12 CPU, 48Gbytes RAM. 

<strong>Siebel</strong> enterprise server logged the following error messages and failed to start: 

GenericLog GenericError 1 2004-07-30 21:40:07 (sissrvr.cpp 47(2617) 

err=2000026 sys=17) SVR-00026: Unable to allocate shared memory 

GenericLog GenericError 1 2004-07-30 21:40:07 (scfsis.cpp 5(57) 


GenericLog GenericError 1 2004-07-30 21:40:07 (listener.cpp 21(157) 


Explanation 

To understand what occurred, it is necessary to review some background on the <strong>Siebel</strong> 

server process siebsvc: 

The siebsvc process runs as a system service that monitors and controls the state of 

every <strong>Siebel</strong> server component operating on that <strong>Siebel</strong> server. Each <strong>Siebel</strong> server is an 

instantiation of the <strong>Siebel</strong> Server System Service (siebsvc) within the current <strong>Siebel</strong> 

Enterprise Server. <strong>Siebel</strong> server runs as a daemon process in a UNIX environment. 

During startup, the <strong>Siebel</strong> Server System Service (siebsvc) performs the following 

sequential steps: 

1. Retrieve configuration information from the <strong>Siebel</strong> Gateway Name Server. 

2. Create a shared memory file located in the "admin" subdirectory of the 

<strong>Siebel</strong> server root directory on UNIX. By default, this file has the name 

Enterprise_Server_Name.<strong>Siebel</strong>_Server_Name.shm. 

The size of this file *.shm is directly proportional to the MaxTask setting. The higher the 

number of concurrent users the higher the MaxTask. 

The <strong>Siebel</strong> Server System Service deletes this .shm file when it shuts down. 


Investigating the Reason for the Failure 

The .shm file that was being created during startup will be memory-mapped and be part 

of the heap size of the siebsvc process. That means the process size of siebsvc 

process grows proportionally with the increase in the size of the .shm file. 

With 9500 MaxTask configured, an .shm file of size 1.15Gbytes was created during 

server startup: 

siebapp6:/tmp/%ls -l /export/siebsrvr/admin/siebel.sdcv480s002.shm 

-rwx------ 1 sunperf other 1212096512 Jan 24 11:44 

/export/siebsrvr/admin/siebel.sdcv480s002.shm 

And the siebsvc had a process size of 1.14Gbytes when the process died abruptly: 


25309 sunperf 1174M 1169M sleep 60 0 0:00:06 6.5% siebsvc/1 

A truss of the process reveals that it is trying to mmap a file 1.15Gbytes in size and fails 

with ENOMEM. 

Here is the truss output: 

8150: brk(0x5192BF78) = 0 

8150: open("/export/siebsrvr/admin/siebel_siebapp6.shm", O_RDWR|O_CREAT|O_EXCL, 0700) 

= 9 

8150: write(9, "\0\0\0\0\0\0\0\0\0\0\0\0".., 1367736320) = 1367736320 

8150: mmap(0x00000000, 1367736320, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) Err#12 

ENOMEM 

If the mmap succeeds, the process may have a process size > 2Gbytes. Because <strong>Siebel</strong> 

is a 32-bit application, it can have a process size up to 4Gbytes (2^32 = 4Gbytes) on the 

Solaris OS. But in our case, the process failed with a size just over 2Gbytes. 


The following were the system resource settings from the failed machine: 

sdcv480s002:/export/home/sunperf/18306/siebsrvr/admin/%ulimit -a 

time(seconds) unlimited 

file(blocks) 

unlimited 

data(kbytes) 

unlimited 

stack(kbytes) 8192 

coredump(blocks) unlimited 

nofiles(descriptors) 256 

vmemory(kbytes) unlimited 

Even though the maximum size of the datasize or heap is reported as unlimited by the 

ulimit command above, the max limit is 2Gbytes by default on the Solaris OS. The 

upper limits seen by an application running on Solaris hardware can be achieved by 

using a simple C program that calls the getrlimit API from sys/resource.h. The 

following program prints the system limits for data, stack, and vmemory: 

#include 

#include 

#include 

#include 

#include 

static void showlimit(int resource, char* str) 

{ 

struct rlimit lim; 

if (getrlimit(resource, &lim) != 0) { 

(void)printf("Couldn't retrieve %s limit\n", str); 

return; 

} 

(void)printf("Current/maximum %s limit is \t%lu / %lu\n", str, lim.rlim_cur, 

lim.rlim_max); 

} 

int main() { 

showlimit(RLIMIT_DATA, "data"); 

showlimit(RLIMIT_STACK, "stack"); 

showlimit(RLIMIT_VMEM, "vmem"); 

} 

return 0; 

Output from the C program on the failed machine is as follows: 

sdcv480s002:/export/siebsrvr/admin/%showlimits 

Current/maximum data limit is 2147483647 / 2147483647 

Current/maximum stack limit is 8388608 / 2147483647 

Current/maximum vmem limit is 2147483647 / 2147483647 

From the output shown, it is clear that the processes were bound to a maximum data limit 

of 2Gbytes on an out-of-the-box Solaris system setup. This limitation is the reason for the 

failure of the siebsvc process as it tried to grow beyond 2Gbytes. 

Solution 

The solution is to increase the default system limit for datasize and reduce the 

stacksize. An increase in datasize creates more room for process address space 

and reduction of stacksize reduces the reserved stack space. Both these adjustments 


let a <strong>Siebel</strong> process use its process address space more efficiently, hence allowing the 

total <strong>Siebel</strong> process size to grow up to 4Gbytes (which is the upper limit for a 32-bit 

application). 

1. What are the recommended values for data and stack sizes on Solaris OS while 

running the <strong>Siebel</strong> application? How does one change the limits of datasize and 

stacksize? 

Set the datasize to 4Gbytes (that is, the maximum address space allowed for a 32- 

bit process) and set the stacksize to any value

file(blocks) unlimited unlimited 

data(kbytes) 4194303 4194303 

stack(kbytes) 512 512 

coredump(blocks) unlimited unlimited 

nofiles(descriptors) 65536 65536 

vmemory(kbytes) unlimited unlimited 

The preceding setting allows the siebsvc to mmap the .shm file and thereby siebsvc 

succeeds in forking the rest of the processes and <strong>Siebel</strong> server processes start up 

successfully. Thus this finding enables one to configure MaxTasks greater than 9000 on 

the Solaris OS. That means one can get around the scalability limit of 9000 users per 

<strong>Siebel</strong> server, but how much higher can one go? We determined the limit to be 18,000. If 

MaxTasks was configured to > 18000, <strong>Siebel</strong> calculates the *.shm file size to be about 

2Gbytes. siebsvc tries to mmap this file and fails, as it has hit the limit for a 32-bit 

application process. 

The following listing shows the <strong>Siebel</strong> .shm file with maxtask set to 18,000: 

sdcv480s002:/export/siebsrvr/admin/%ls -l *.shm ; ls -lh *.shm 

-rwx------ 1 sunperf other 2074238976 Jan 25 13:23 siebel.sdcv480s002.shm* 

-rwx------ 1 sunperf other 1.9G Jan 25 13:23 siebel.sdcv480s002.shm* 

sdcv480s002:/export/siebsrvr/admin/%plimit 12527 

12527: siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002 

resource current maximum 

time(seconds) unlimited unlimited 

file(blocks) unlimited unlimited 

data(kbytes) 4194303 4194303 

stack(kbytes) 512 512 

coredump(blocks) unlimited unlimited 

nofiles(descriptors) 65536 65536 

vmemory(kbytes) unlimited unlimited 

sdcv480s002:/export/siebsrvr/admin/%prstat -s size -u sunperf -a 



12565 sunperf 2111M 116M sleep 59 0 0:00:08 0.1% siebmtsh/8 

12566 sunperf 2033M 1159M sleep 59 0 0:00:08 3.1% siebmtshmw/10 

12564 sunperf 2021M 27M sleep 59 0 0:00:01 0.0% siebmtsh/12 

12563 sunperf 2000M 14M sleep 59 0 0:00:00 0.0% siebproc/1 


In the rare case there is a need to run more than 18,000 <strong>Siebel</strong> users on a single <strong>Siebel</strong> 

server node, the way to do it is to install another <strong>Siebel</strong> server instance on the same 

node. This works well and is a supported configuration on <strong>Siebel</strong>/Sun platform. It is not 

advisable to configure more MaxTasks than needed, as this could affect the 

performance of the overall <strong>Siebel</strong> enterprise. 

7.3.2 Bloated <strong>Siebel</strong> Processes (Commonly Mistaken as Memory Leaks) 

Symptom 

During some of the high concurrent user tests, it was observed that hundreds of users 

suddenly failed. 


Description 

The reason for this symptom was that some of the <strong>Siebel</strong> server processes servicing 

these users had either hung or crashed. Close observation with repeated replays 

revealed that the Siebmtshmw process memory shot up from 700M to 2GB. 

The UltraSPARC IV-based E2900 server can handle up to 30,000 processes and about 

87,000 LWPs (threads), so there was no resource limitation from the machine here. The 

<strong>Siebel</strong> applications are running into 32-bit process space limits -- but why is the memory 

per process going from 700Mbytes to 2GBytes? Further debugging lead us to the 

problem: the stack size. 

The total memory size of a process is made up of heap size + stack size + data 

and anon segments. In this case, it was found that the bloating in size was at the stack, 

which grew from 64Kbytes to 1Gbyte, which was abnormal. pmap is a utility on the 

Solaris system that provides a detailed breakdown of memory sizes per process. 

Here is an output of pmap that shows that the stack segment bloated to about 1Gbyte: 

siebapp6@/export/pspp> grep stack pmap.4800mixed.prob.txt (pmap output unit is 

in Kbytes) 

FFBE8000 32 32 - 32 read/write/exec [ stack ] 





C012A000 1043224 1043224 - 1043224 read/write/exec [ stack ] 

We reduced the stack size to 128Kbytes and this fixed the problem. Now a single <strong>Siebel</strong> 

server configured with 10,000 MaxTasks started up successfully. It was failing to startup 

before breaking at the mmap call. The stack size of 8Mytes is a default setting on the 

Solaris system. However, limiting the stack size to 128Kbytes removed a lot of the 

instability in our high load tests and we were able to run 5000 user tests without errors, 

pushing up to 80% to 90% CPU usage on the 12-way E2900 server. 

Changing the mainwin address MW_GMA_VADDR=0xc0000000 to other values did not 

seem to make a big difference. Please do not vary this parameter. 

Solution 

Change the stack size hard limit of the <strong>Siebel</strong> process from unlimited to 512 bytes. 

Stack size on the Solaris OS has a hard limit and a soft limit. The default values for these 

two limits are unlimited and 8Mbytes, respectively. This means that an application 

process on the Solaris OS can have its stack size anywhere up to the hard limit. Since 

the default hard limit on the Solaris system is unlimited the <strong>Siebel</strong> application 

processes could grow their stack size all the way up to 2Gbytes. When this occurs, the 

total memory size of the <strong>Siebel</strong> process hits the max limit of memory addressable by a 

32-bit process and bad things happen (such as a hang or crash). Setting the limit for 

stack to 1Mbytes or a lower value resolved the issue. 


How is this done? 

sdcv480s002:/export/home/sunperf/18306/siebsrvr/admin/%limit stack 

stacksize 8192 kbytes 

sdcv480s002:/export/siebsrvr/admin/%limit stacksize 1024 

sdcv480s002:/export/siebsrvr/admin/%limit stacksize 

stacksize 512 kbytes 

Please note that a large stack limit can inhibit the growth of the data segment, because 

the total process size upper limit is 4Gbytes for a 32-bit application. Also even if the 

process stack hasn't grown to a large extent, the virtual memory space will be reserved 

for it according to the limit value. While the recommendation to limit stack size to 512 

bytes worked well for the workload defined in this paper, this setting may have to be 

tweaked for different <strong>Siebel</strong> deployments and workloads. The range could be from 512 to 

1024 bytes. 

7.4 <strong>Tuning</strong> Sun Java System Web Server 

The three main files where tuning can be done are obj.conf, server.xml and 

magnus.conf. 

Edit magnus.conf: 

1. Set the RqThrottle=4028 in the magnus.conf file under the web server 

root directory. 

2. ListenQ 16000. 

3. ConnQueueSize 8000. 

4. KeepAliveQueryMeanTime 50. 

Edit server.xml: 

1. Replaced host name with ip address in server.xml. 

Edit obj.conf: 

1. Turned off access logging. 

2. Turned off cgi, jsp, servlet support. 

3. Removed the following lines from obj.conf: ###PathCheck fn="checkacl" 

acl="default" and ###PathCheck fn=unix-uri-clean, since 

these are not being used by <strong>Siebel</strong>. 

<strong>Tuning</strong> parameters used for high user load with Sun Java System Web Servers are 

listed in the following table. 


Parameter Scope Default Tuned Value 

Value 

shmsys:shminfo_shmmax /etc/system 0xfffffffffffff 

fff 












msgsys:msginfo_msgssz /etc/system 64 

msgsys:msginfo_msgseg /etc/system 32767 





sq_max_size /etc/system 2 0 

tcp_time_wait_interval ndd /dev/tcp 240000 60000 

tcp_conn_req_max_q ndd /dev/tcp 128 1024 

tcp_conn_req_max_q0 ndd /dev/tcp 1024 4096 

tcp_ip_abort_interval ndd /dev/tcp 480000 60000 

tcp_keepalive_interval ndd /dev/tcp 720000 900000 

0 

tcp_rexmit_interval_initial ndd /dev/tcp 3000 3000 

tcp_rexmit_interval_max ndd /dev/tcp 240000 10000 

tcp_rexmit_interval_min ndd /dev/tcp 200 3000 

tcp_smallest_anon_port ndd /dev/tcp 32768 1024 

tcp_slow_start_initial ndd /dev/tcp 1 2 

tcp_xmit_hiwat ndd /dev/tcp 8129 32768 

tcp_fin_wait_2_flush_interv ndd/dev/tcp 67500 675000 

al 

tcp_recv_hiwat ndd /dev/tcp 8129 32768 

Table 7.4 


7.5 <strong>Tuning</strong> the <strong>Siebel</strong> Web Server Extension (SWSE) 

1. In the <strong>Siebel</strong> Web Plugin installation directory, go to the bin directory. Edit the 

eapps.cfg file and make the following changes: 

• Set AnonUserPool to 15% of . 

• Set the following settings in the “default” section: 

5. GuestSessionTimeout to 60; this is required for scenarios where the 

user is browsing without logging in. 

6. AnonSessionTimeout = 300 

7. SessionTimeout = 300 

• Set appropriate AnonUser name/passwords 

SADMIN/SADMIN for eChannel and Call Center (Database login) 

GUEST1/GUEST1 for eService and eSales (LDAP login) 

GUESTERM/GUESTERM for ERM (Database login) 

The AnonUserPool setting can vary on different types of <strong>Siebel</strong> users (call center, 

sales, and so on). 

The table below summarizes the individual eapps.cfg settings for each type of <strong>Siebel</strong> 

application: Call Center, eChannel, eSales and eService, used in the 10,000 users test. 

Parameter callcenter_enu prmportal_enu esales_enu eservice_enu 

AnonUserName SADMIN GUESTCP eApps2 eApps3 

AnonPassword SADMIN GUESTCP eApps2 eApps3 

AnonUserPool 

CC1=420,CC2=525, 

CC3=315, CC4=210 

320 160 360 

AnonSessionTimeout 360 360 360 360 

GuestSession 

Timeout 

60 60 60 60 

SessionTimeout 300 300 300 300 

Table 7.5 

The SWSE stats web page is a very good resource for tuning <strong>Siebel</strong>. 


7.6 <strong>Tuning</strong> <strong>Siebel</strong> Standard Oracle Database and Sun Storage 

The size of the database used was approximately 140 Gbytes. The database was built to 

simulate customers with large transaction volumes and data distributions that 

represented the most common customer data shapes. Table 7.6 shows a sampling of 

record volumes and sizes in the database for key business entities of the standard <strong>Siebel</strong> 

volume database. 

Business Entity 

Database Table 

Name 

Number of 

Records 

Size in Kbytes 

Accounts S_ORG_EXT 1,897,161 3,145,728 

Activities S_EVT_ACT 8,744,305 6,291,456 

Addresses S_ADDR_ORG 3,058,666 2,097,152 

Contacts S_CONTACTS 3,366,764 4,718,592 

Employees S_EMPLOYEE_ATT 21,000 524 

Opportunities S_OPTY 3,237,794 4,194,304 

Orders S_ORDER 355,297 471,859 

Products S_PROD_INT 226,000 367,001 

Quote Items S_QUOTE_ITEM 1,984,099 2,621,440 

Quotes S_QUOTE_ATT 253,614 524 

Service Requests S_SRV_REQ 5,581,538 4,718,592 

Table 7.6 

7.6.1 Optimal Database Configuration 

Creating a well-planned database to begin with will require less tuning/reorganizing 

during runtime. While many resources are available to facilitate creation of highperformance 

Oracle databases, most tuning engineers would find themselves tweaking a 

database consisting of thousands of tables and indexes, piece by piece. This is both time 

consuming and prone to mistakes. Eventually one would end up rebuilding the entire 

database from scratch. 

The following approach provides an alternative to tuning a pre-existing, pre-packaged 

database: 

1. Measure the exact space used by each object in the schema. The 

dbms_space packages provide the accurate space used by an index or a table. 

Other sources like dba_free_space only tell you “how much is free” from the 

total allocated space, which is always more. Next, run the benchmark test and 

again measure the space used. The difference will result in an accurate report of 

“how much each table/index grows during the test.” One can use this data to rightsize 

all of the tables, such as the capacity planned for growth during the test. Also 

the data can be used to figure out and concentrate on only the hot tables used by 

the test. 

2. Create a new database with multiple index and data tablespaces. The idea is 

to place all equi-extent-sized tables into their own tablespace. Keeping the data 

and index objects in their own tablespace reduces contention and fragmentation, 


and also provides for easier monitoring. Keeping tables with equal extent sizes in 

their own tablespace reduces fragmentation as old and new extent allocations are 

always of the same size within a given tablespace, leaving no room for empty oddsized 

pockets in between. This leads to compact data placement which reduces 

the number of I/Os done. 

3. Build a script to create all of the tables and indexes. This script should have 

the tables being created in their appropriate tablespaces with the appropriate 

parameters like freelists, freelist_groups, pctfree, pctused, and so on. 

Use this script to place all of the tables in their tablespaces and then import the 

data. This will result in a clean, defragmented, optimized, and right-sized 

database. 

The tablespaces should also be built to be locally managed. This allows the space 

management to be done locally within the tablespace, unlike default (dictionary 

managed) tablespaces that write to the system tablespace for every extent change. The 

list of hot tables for <strong>Siebel</strong> is available in Appendix B. 

7.6.2 Properly Locating Data on the Disk for Best <strong>Performance</strong> 

To achieve the incredible capacity on current disk drives, disk manufacturers have 

implemented zone bit recording. This means that the outer edge of the disk drive has 

more available storage area than the inside edge of the disk drive; that is, the number of 

sectors per track decreases as you move toward the center of the disk. Disk drive 

manufactures take advantage of this situation by recording more data on the outer 

edges. Since the disk drive rotates at a constant speed, the outer tracks have faster 

transfer rates than the inner tracks. For example, a Seagate 36 Gbyte Cheetah 1 drive has 

a data transfer speed range of 57 MBytes/sec on the inner tracks to 86 Mbytes/sec on 

the outer tracks -- a 50% improvement in transfer speed. 

For benchmarking purposes, it is desirable to: 

1. Place active large block transfers on the outer edges of the disk to minimize data 

transfer time. 

2. Place active random small block transfers on the outer edges of the disk drive only 

if active large block transfers are not in the benchmark. 

3. Place inactive random small block transfers on the inner sections of disk drive to 

minimize the impact of the data transfer speed discrepancies. 

Further, if the benchmark only deals with small block I/Os, like SPC Benchmark-1 2 

benchmarking, the priority is to put the most active LUNs on the outer edge and the less 

active LUNs on the inner edge of the disk drive. 

1 The Cheetah 15 KB RPM disk drive datasheet can be found at www.seagate.com. 

2 Further information about SPC Benchmark-1 TM can be found at www.Storage<strong>Performance</strong>.org. 


86 MB/second 

57 MB/second 

Figure 7.6.2 

Figure 7.6.2 shows a zone bit recording example with five zones. The outer edge holds 

the most data and has the fastest transfer rate. 

7.6.3 Disk Layout and Oracle Data Partitioning 

An I/O subsystem with less contention and high throughput is key for obtaining high 

performance with Oracle. After analyzing the <strong>Siebel</strong> workload an appropriate design was 

made. 

The I/O subsystem consisted of Sun StorEdge SE6320 connected to the E2900 

database server via fiber channel. The SE6320 has 2 base + 2 expansion arrays driven 

through two controllers. Each tray consists of 14x36Gbyte disks at 15,000 RPM. That 

means there are 56 total disks providing over 2 Tbytes of total storage. Each tray has a 

cache of 1Gbyte. All of the trays were formatted in the RAID 0 mode and two LUNs per 

tray were created. Eight striped volumes of 300 Gbytes each were carved. Each volume 

was striped across seven physical disks with stripe sizes of 64KB. Eight filesystems (ufs) 

were built on top of these striped volumes: T4disk1, T4disk2, T4disk3, T4disk4, 

T4disk5, T4disk6, T4disk7, T4disk8. 

The following example shows the tuning done at the disk array level. Please note that 

cache writebehind was turned off and disk scrubber was turned off. 


array00:/:sys list 

controller : 2.5 

blocksize 

: 64k 

cache 

: auto 

mirror 

: auto 

mp_support 

: none 

naca 

: off 

rd_ahead 

: off 

recon_rate 

: med 

sys memsize : 256 MBytes 

cache memsize : 1024 MBytes 

fc_topology : auto 

fc_speed 

: 2Gb 

disk_scrubber : off 

ondg 

: befit 

array00:/:

DATA_51200K 

INDX_512K 

DATA_5120K 

TEMP 

TOOLS 

DATA_512K 

SYSTEM 

Tablespace to hold the medium <strong>Siebel</strong> tables. 

Tablespace for all the indexes on small tables. 

Tablespace for the small <strong>Siebel</strong> tables. 

Oracle temporary segments. 

Oracle performance measurement objects. 

Tablespace for <strong>Siebel</strong> small tables. 

Oracle system tablespace. 

Tablespace to Logical Volume Mapping 

DATA_512000K 

/t3disk2/oramst/oramst_data_512000K.01 










DATA_51200K 

DATA_5120K 

DATA_512K 

INDX_51200K 

INDX_5120K 

INDX_512K 

RBS 







/t3disk5/oramst/oramst_indx_51200K.02 




















/t3disk2/oramst/oramst_rbs.01 










t3disk5/oramst/oramst_rbs.04 









SYSTEM 

TEMP 

TOOLS 

/t3disk2/oramst/oramst_system.01 

/t3disk7/oramst/oramst_temp.12 

/t3disk2/oramst/oramst_tools.01 






With the preceding setup of Oracle using the hardware level stripping, and placing Oracle 

objects in different tablespaces, an optimal setup was reached with no I/O waits noticed 

and no single disk being more than 20% occupied during the tests. Veritas was not used 

as this setup provided the required I/O throughput. 

Following is the Iostat output: two snapshots at five-second intervals, taken during 

steady state of the test on the database server. There are minimal reads (r/s), and 

writes are balanced across all volumes. With the exception of c7t1d0, this is the 

dedicated T4+ array for Oracle redologs. It is quite normal to see high writes/sec on 

redologs, this just indicates that transactions are getting done rapidly in the database. 

The reads/sec is abnormal. This volume is at 27% busy, which is considered borderline 

high. Fortunately, service times are very low. 

Wed Jan 8 15:25:20 2003 

extended device statistics 

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 

0.0 1.0 0.0 5.6 0.0 0.0 0.0 1.0 0 1 c0t3d0 

1.0 27.6 8.0 220.8 0.0 0.0 0.0 1.3 0 2 c2t7d0 

0.0 26.0 0.0 208.0 0.0 0.0 0.0 0.5 0 1 c3t1d0 

0.6 37.4 4.8 299.2 0.0 0.0 0.0 0.8 0 2 c4t5d0 

0.0 23.4 0.0 187.2 0.0 0.0 0.0 0.6 0 1 c5t1d0 

0.0 10.2 0.0 81.6 0.0 0.0 0.0 0.5 0 0 c6t1d0 

3.8 393.0 1534.3 3143.8 0.0 0.2 0.0 0.6 0 22 c7t1d0 

0.0 28.2 0.0 225.6 0.0 0.0 0.0 0.6 0 1 c8t1d0 

Wed Jan 8 15:25:25 2003 

extended device statistics 

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 

1.4 19.8 11.2 158.4 0.0 0.0 0.0 1.0 0 2 c2t7d0 

0.0 18.2 0.0 145.6 0.0 0.0 0.0 0.5 0 1 c3t1d0 

0.8 39.0 6.4 312.0 0.0 0.0 0.0 0.7 0 2 c4t5d0 

0.0 18.8 0.0 150.4 0.0 0.0 0.0 0.6 0 1 c5t1d0 

0.0 17.4 0.0 139.2 0.0 0.0 0.0 0.5 0 1 c6t1d0 

5.2 463.2 1844.8 3705.6 0.0 0.3 0.0 0.7 0 27 c7t1d0 

0.0 29.4 0.0 235.2 0.0 0.0 0.0 0.5 0 1 c8t1d0 


7.6.4 Solaris MPSS <strong>Tuning</strong> for Oracle Server 

Available as a standard feature since Solaris 9 OS, Multiple Page Size Support (MPSS) 

allows a program to use any hardware-supported page sizes to access portions of virtual 

memory. MPSS improves virtual memory performance by allowing applications to use 

large page sizes, therefore improving resource efficiency and reducing overhead, and 

accomplishes this without recompiling or recoding applications. 

Enable MPSS (Multiple Page Size Support) for Oracle Server and shadow processes on 

Solaris 9 OS or later versions to reduce the TLB miss% rate. It is recommended to use 

the largest available page size if the TLB miss% is high. 

Enabling MPSS for Oracle Processes 

1. Enable the kernel cage if the machine is not an E10K or F15K, and reboot the 

system. Kernel cage an be enabled by the following setting in /etc/system: 

set kernel_cage_enable=1 

Why do we need the kernel cage? To address a problem with memory 

fragmentation. Immediately after a system boot, a sizeable pool of large pages are 

available and the applications can get all of their mmap() memory allocated from 

large pages. This can be verified using pmap -xs . If the machine has 

been in use for a while, the application may not get the desirable large pages until 

the machine is rebooted. This is mainly due to the fragmentation of physical 

memory. 

We can vastly minimize the fragmentation by enabling the kernel cage. With the 

kernel cage enabled, the kernel will be allocated from a small contiguous range of 

memory, minimizing the fragmentation of other pages within the system. 

2. Find out all possible hardware address translation (HAT) sizes supported by the 

system with pagesize -a. 

$ pagesize -a 

8192 

65536 

524288 

4194304 

3. Run trapstat -T. The value shown in the ttl row and %time column is the 

percentage of time the processor(s) spent in virtual-to-physical memory address 

translations. Depending on %time, make a wise choice of a pagesize that will help 

reduce reducing the iTLB/dTLB miss rate. 

4. Create a simple config file for MPSS as follows: 

oracle*:: 

Desirable heap and stack size must be one of the supported HAT sizes. By 

default, 8Kbytes is the page size for heap and stack on all Solaris releases. 


5. Set the environment variables MPSSCFGFILE and MPSSERRFILE. MPSSCFGFILE 

should point to the config file that was created in Step 3. MPSS writes any errors 

during runtime to file. 

6. Preload MPSS interposing library mpss.so.1, and bring up Oracle server. It is 

recommended to put the MPSSCFGFILE, MPSSERRFILE, and LD_PRELOAD 

environment variables in the Oracle startup script. 

With all the env variables mentioned above, a typical startup script may look like 

the following: 

echo starting listener 

lsnrctl start 

echo preloading mpss.so.1 .. 

MPSSCFGFILE=/tmp/mpsscfg 

MPSSERRFILE=/tmp/mpsserr 

LD_PRELOAD=/usr/lib/mpss.so.1:$LD_PRELOAD 

export MPSSCFGFILE MPSSERRFILE LD_PRELOAD 

echo starting oracle server processes .. 

sqlplus /nolog

<strong>Siebel</strong> Object Name Type 

Growth in 

Bytes 

S_DOCK_TXN_LOG TABLE 1,177,673,728 

S_EVT_ACT TABLE 190,341,120 

S_DOCK_TXN_LOG_P1 INDEX 96,116,736 

S_DOCK_TXN_LOG_F1 INDEX 52,600,832 

S_ACT_EMP TABLE 46,202,880 

S_SRV_REQ TABLE 34,037,760 

S_AUDIT_ITEM TABLE 29,818,880 

S_OPTY_POSTN TABLE 28,180,480 

S_ACT_EMP_M1 INDEX 25,600,000 

S_EVT_ACT_M1 INDEX 23,527,424 


S_ACT_EMP_U1 INDEX 21,626,880 

S_ACT_CONTACT TABLE 21,135,360 

S_EVT_ACT_U1 INDEX 18,391,040 




S_AUDIT_ITEM_M2 INDEX 16,277,504 

S_ACT_EMP_P1 INDEX 15,187,968 


Table 7.6.5 

7.6.6 Oracle Parameters <strong>Tuning</strong> 

Here are the key Oracle init.ora tunables that were tuned. 

db_cache_size=3048576000 

The preceding parameter determines the size for Oracle’s SGA (Shared Global Area). 

Database performance is highly dependent on available memory. In general, more 

memory increases caching, which reduces physical I/O to the disks. Oracle’s SGA is a 

memory region in the application that caches database tables and other data for 

processing. With 32-bit Oracle software on a 64-bit Solaris OS the SGA is limited to 4 

Gbytes. 

Oracle comes in two basic architectures: 64-bit and 32-bit. The number of address bits 

determines the maximum size of the virtual address space. 

32-bits = 2 32 = 4 GB maximum 

64-bits = 2 64 = 16777216 TB maximum 

For the <strong>Siebel</strong> 10,000 concurrent users' PSPP workload, the 4Gbyte SGA was sufficient. 

As a result, the 32-bit Oracle server version was used. 


db_block_max_dirty_target=0 

db_writer_processes=4 

Fast_start_io_target=0 

The default value of db_block_max_dirty_target was changed from 4294967294 to 

0. Setting this value to 0 disables writing of buffers for incremental checkpointing 

purposes. This stops deferred checkpointing and achieves CPU and bus improvements. 

The default value for db_writer_processes is 1. Changing db_writer_processes 

to 4 starts up four dbwr processes. These three parameter changes drastically reduced 

wait times in the database and thus improved <strong>Siebel</strong> overall throughput. 

Db_block_size=8K – default value is 2K. An 8K value for <strong>Siebel</strong> is optimal. 

DB_BLOCK_LRU_LATCHES=48 – Specifies the upper bound of the number of LRU 

latch sets. Set this parameter to a value equal to the desired number of LRU latch sets. 

Oracle decides whether to use this value or reduce it based on a number of internal 

checks. If the parameter is not set, Oracle calculates a value for the number of sets 

based on the number of CPUs. The value calculated by Oracle is usually adequate; 

increase this value only if misses are higher than 3% in V$LATCH. For <strong>Siebel</strong> 10,000 

users run on a four CPU machine, setting the value to 48 was optimal. 

distributed_transactions=0 – Setting this value to 0 disables the Oracle 

background process called reco. <strong>Siebel</strong> does not use distributed transactions, meaning 

that we can regain CPU and bus by having one less Oracle background process. The 

default value is 99. 

replication_dependency_tracking=FALSE 

<strong>Siebel</strong> does not use replication so it is safe to turn this off by setting it to false. 

transaction_auditing=FALSE 

Writes less redo for every commit. It is the nature of <strong>Siebel</strong> OLTP to do many small 

transactions with frequent commits. Setting transaction_auditing to false buys 

back CPU and bus. 

Here is the complete listing of the init.ora file used with all the parameters set for 

10,000 <strong>Siebel</strong> users. 

Oracle 9.2.0.2 init.ora file 

# Oracle 9.2.0.2 init.ora for solaris 9, running up to 10000 <strong>Siebel</strong> 7.5.2 user 

benchmark. 

db_block_size=8192 

db_cache_size=3048576000 

db_domain="" 

db_name=oramst 

background_dump_dest=/export/pspp/oracle/admin/oramst/bdump 

core_dump_dest=/export/pspp/oracle/admin/oramst/cdump 

timed_statistics=FALSE 

user_dump_dest=/export/pspp/oracle/admin/oramst/udump 

control_files=("/export/pspp/oracle/admin/oramst/ctl/control01.ctl", 

"/export/pspp/oracle/admin/oramst/ctl/control02.ctl", 

"/export/pspp/oracle/admin/oramst/ctl/control03.ctl") 


instance_name=oramst 

job_queue_processes=0 

aq_tm_processes=0 

compatible=9.2.0.0.0 

hash_join_enabled=TRUE 

query_rewrite_enabled=FALSE 

star_transformation_enabled=FALSE 

java_pool_size=0 

large_pool_size=8388608 

shared_pool_size=503316480 

processes=2500 

pga_aggregate_target=25165824 

rollback_segments = 

(rb_001,rb_002,rb_003,rb_004,rb_005,rb_006,rb_007,rb_008,rb_009,rb_010,rb_011,rb 

_012,rb_013,rb_014,rb_015,rb_016,rb_017,rb_018,rb_019,rb_020,rb_021,rb_022,rb_02 

3,rb_024,rb_025,rb_026,rb_027,rb_028,rb_029,rb_030,rb_031,rb_032,rb_033,rb_034,r 

b_035,rb_036,rb_037,rb_038,rb_039,rb_040,rb_041,rb_042,rb_043,rb_044,rb_045,rb_0 

46,rb_047,rb_048,rb_049,rb_050,rb_051,rb_052,rb_053,rb_054,rb_055,rb_056,rb_057, 

rb_058,rb_059,rb_060,rb_061,rb_062,rb_063,rb_064,rb_065,rb_066,rb_067,rb_068,rb_ 

069,rb_070,rb_071,rb_072,rb_073,rb_074,rb_075,rb_076,rb_077,rb_078,rb_079,rb_080 

,rb_081,rb_082,rb_083,rb_084,rb_085,rb_086,rb_087,rb_088,rb_089,rb_090,rb_091,rb 

_092,rb_093,rb_094,rb_095,rb_096,rb_097,rb_098,rb_099,rb_100) 

log_checkpoint_timeout=10000000000000000 

nls_sort=BINARY 

sort_area_size = 10485760 

sort_area_retained_size = 10485760 

nls_date_format 

= "MM-DD-YYYY:HH24:MI:SS" 

transaction_auditing 

= false 

replication_dependency_tracking = false 

session_cached_cursors = 8000 

open_cursors=4048 

cursor_space_for_time 

= TRUE 

db_file_multiblock_read_count = 8 # stripe size is 64K and not 1M 

db_block_checksum=FALSE 

log_buffer = 10485760 

optimizer_mode=RULE 

filesystemio_options=setall 

pre_page_sga=TRUE 

fast_start_mttr_target=0 

db_writer_processes=6 

distributed_transactions = 0 

transaction_auditing 

= FALSE 

replication_dependency_tracking = falsE 

#timed_statistics 

max_rollback_segments = 120 

job_queue_processes = 0 

java_pool_size = 0 

db_block_lru_latches = 48 

db_writer_processes = 4 

session_cached_cursors = 8000 

FAST_START_IO_TARGET = 0 

= TRUE # turned off 

DB_BLOCK_MAX_DIRTY_TARGET = 0 

pre_page_sga 

= TRUE 

7.6.7 Solaris Kernel Parameters on Oracle Database Server 

Parameter Scope Default 

Value 

Tuned 

Value 

shmsys:shminfo_shmmax /etc/system 0xffffffffffffffff 











semsys:seminfo_semvmx /etc/system 32767 

semsys:seminfo_semaem /etc/system 16384 







Table 7.6.7 

7.6.8 SQL Query <strong>Tuning</strong> 

During the course of the test, the most resource-intensive and long-running queries were 

tracked. In general, the best way to tune a query is to change the SQL statement 

(keeping the result set the same). The other method is to add or drop indexes so the 

execution plan changes. The latter method is the only option in most benchmark tests. 

We added four additional indexes to the <strong>Siebel</strong> schema which helped performance. With 

<strong>Siebel</strong> 7.5 there is no support for CBO (cost-based optimization) with Oracle database. 

CBO support is available in <strong>Siebel</strong> 7.7. 

The following example shows one of the resource-consuming queries. 

Buffer Gets Executions Gets per Exec % Total Hash Value 

--------------- ------------ -------------- ------- ------------ 

220,402,077 35,696 6,174.4 33.2 2792074251 

This query was responsible for 33% of the total buffer gets from all queries during the 

benchmark tests. 

SELECT 

T4.LAST_UPD_BY, 

T4.ROW_ID, 

T4.CONFLICT_ID, 

T4.CREATED_BY, 

T4.CREATED, 

T4.LAST_UPD, 

T4.MODIFICATION_NUM, 

T1.PRI_LST_SUBTYPE_CD, 

T4.SHIP_METH_CD, 

T1.PRI_LST_NAME, 

T4.SUBTYPE_CD, 

T4.FRGHT_CD, 

T4.NAME, 

T4.BU_ID, 

T3.ROW_ID, 

T2.NAME, 

T1.ROW_ID, 

T4.CURCY_CD, 

T1.BU_ID, 

T4.DESC_TEXT, 

T4.PAYMENT_TERM_ID 

FROM 

ORAPERF.S_PRI_LST_BU T1, 

ORAPERF.S_PAYMENT_TERM T2, 

ORAPERF.S_PARTY T3, 


ORAPERF.S_PRI_LST T4 

WHERE 

T4.PAYMENT_TERM_ID = T2.ROW_ID (+) 

AND 

T1.BU_ID = :V1 

AND 

T4.ROW_ID = T1.PRI_LST_ID 

AND 

T1.BU_ID = T3.ROW_ID 

AND 

((T1.PRI_LST_SUBTYPE_CD != 'COST LIST' 

AND 

T1.PRI_LST_SUBTYPE_CD != 'RATE LIST') 

AND 

(T4.EFF_START_DT = TO_DATE(:V3,'MM/DD/YYYY HH24:MI:SS')) 

AND 

T1.PRI_LST_NAME LIKE :V4 

AND 

T4.CURCY_CD = :V5)) 

ORDER BY T1.BU_ID, T1.PRI_LST_NAME; 

Executing plan and statistics before the new index was added: 

Execution Plan 

---------------------------------------------------------- 

0 SELECT STATEMENT Optimizer=RULE 

1 0 NESTED LOOPS (OUTER) 

2 1 NESTED LOOPS 


4 3 TABLE ACCESS (BY INDEX ROWID) OF 'S_PRI_LST_BU' 

5 4 INDEX (RANGE SCAN) OF 'S_PRI_LST_BU_M1' (NON-UNIQUE) 

6 3 TABLE ACCESS (BY INDEX ROWID) OF 'S_PRI_LST' 

7 6 INDEX (UNIQUE SCAN) OF 'S_PRI_LST_P1' (UNIQUE) 

8 2 INDEX (UNIQUE SCAN) OF 'S_PARTY_P1' (UNIQUE) 

9 1 TABLE ACCESS (BY INDEX ROWID) OF 'S_PAYMENT_TERM' 

10 9 INDEX (UNIQUE SCAN) OF 'S_PAYMENT_TERM_P1' (UNIQUE) 

Statistics 

---------------------------------------------------------- 

364 recursive calls 

1 db block gets 

41755 consistent gets 

0 physical reads 

0 redo size 

754550 bytes sent via SQL*Net to client 

27817 bytes received via SQL*Net from client 

341 SQL*Net roundtrips to/from client 

4 sorts (memory) 

0 sorts (disk) 

5093 rows processed 

New Index created 

create index S_PRI_LST_X2 on S_PRI_LST 

(CURCY_CD, EFF_END_DT, EFF_START_DT) 

STORAGE(INITIAL 512K NEXT 512K 

MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 FREELISTS 7 FREELIST 

GROUPS 7 BUFFER_POOL DEFAULT) TABLESPACE INDX NOLOGGING PARALLEL 4 ; 

As shown from the difference in statistics, the new index caused consistent reads to 

reduce by about 100%. 


Execution Plan 

---------------------------------------------------------- 

0 SELECT STATEMENT Optimizer=RULE 

1 0 SORT (ORDER BY) 



4 3 NESTED LOOPS (OUTER) 

5 4 TABLE ACCESS (BY INDEX ROWID) OF 'S_PRI_LST' 

6 5 INDEX (RANGE SCAN) OF 'S_PRI_LST_X2' (NON-UNIQUE 

7 4 TABLE ACCESS (BY INDEX ROWID) OF 'S_PAYMENT_TERM' 

8 7 INDEX (UNIQUE SCAN) OF 'S_PAYMENT_TERM_P1' (UNIQUE) 

9 3 TABLE ACCESS (BY INDEX ROWID) OF 'S_PRI_LST_BU' 

10 9 INDEX (RANGE SCAN) OF 'S_PRI_LST_BU_U1' (UNIQUE) 

11 2 INDEX (UNIQUE SCAN) OF 'S_PARTY_P1' (UNIQUE) 

Statistics 

---------------------------------------------------------- 

0 recursive calls 

0 db block gets 

27698 consistent gets 

0 physical reads 

0 redo size 

754550 bytes sent via SQL*Net to client 

27817 bytes received via SQL*Net from client 

341 SQL*Net roundtrips to/from client 

1 sorts (memory) 

0 sorts (disk) 

5093 rows processed 

Similarly, the three other new indexes added were: 

• create index S_CTLG_CAT_PROD_F1 on ORAPERF.S_CTLG_CAT_PROD (CTLG_CAT_ID ASC) PCTFREE 10 

INITRANS 2 

MAXTRANS 255 STORAGE (INITIAL 5120K NEXT 5120K MINEXTENTS 2 MAXEXTENTS UNLIMITED 

PCTINCREASE 0 FREELISTS 47 FREELIST GROUPS 47 BUFFER_POOL DEFAULT) TABLESPACE 

INDX_5120K NOLOGGING; 

• create index S_PROG_DEFN_X1 on ORAPERF.S_PROG_DEFN 

(NAME, REPOSITORY_ID) 


MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 

BUFFER_POOL DEFAULT) TABLESPACE INDX_512K NOLOGGING; 

• create index S_ESCL_OBJECT_X1 on ORAPERF.S_ESCL_OBJECT 

(NAME, REPOSITORY_ID, INACTIVE_FLG) 


MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 

BUFFER_POOL DEFAULT) TABLESPACE INDX_512K NOLOGGING; 

The last two indexes are for assignment manager tests. No inserts occurred on the base 

tables during these tests. 

7.6.9 Rollback Segment <strong>Tuning</strong> 

Incorrect number and size of rollback segments will cause bad performance. The right 

number and size of rollback segments depends on the application workload. Most OLTP 

workloads require several small-sized rollback segments. The number of rollback 

segments should be equal or more than the number of concurrent active transactions in 


the database during peak load. In the <strong>Siebel</strong> tests this was about 80 during a 10,000 user 

test. 

The size of each rollback segment should be approximately equal to the size in bytes of a 

user transaction. For the <strong>Siebel</strong> workload 100 rollback segments of 20Mbytes with extent 

size of 1Mbyte was found suitable. Note: If the rollback segment sizing is larger than 

required by the application, one would end up wasting valuable space in the database 

cache. 

The new feature from oracle UNDO Segments, which can be used instead of Rollback 

Segments, was not tested during this project with <strong>Siebel</strong> application. 

7.6.10 Database Connectivity Using Host Names Adapter 

It has been observed that high-end <strong>Siebel</strong> test runs using Oracle as the backend have 

performed better when client-to-Oracle server connectivity is done via the hostnames 

adapter feature. This is an Oracle feature that allows for an alternate method of 

connecting to the Oracle database server without using the tnsnames.ora file. 

Set the GLOBAL_DBNAME to something other than the Oracle_SID in the 

Listener.ora file on the database server as shown below. Bounce the Oracle listener 

after making this change. 

LISTENER = 

(DESCRIPTION_LIST = 

(DESCRIPTION = 

(ADDRESS_LIST = 

(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) 

) 

(ADDRESS_LIST = 

(ADDRESS = (PROTOCOL = TCP)(HOST = dbserver)(PORT = 1521)) 

) 

) 

(DESCRIPTION = 

(PROTOCOL_STACK = 

(PRESENTATION = GIOP) 

(SESSION = RAW) 

) 

(ADDRESS = (PROTOCOL = TCP)(HOST = dbserver)(PORT = 2481)) 

) 

) 

SID_LIST_LISTENER = 

(SID_LIST = 

(SID_DESC = 

(SID_NAME = PLSExtProc) 

(ORACLE_HOME = /export/pspp/oracle) 

(PROGRAM = extproc) 

) 

(SID_DESC = 

(GLOBAL_DBNAME = oramst.dbserver) 

(ORACLE_HOME = /export/pspp/oracle) 

(SID_NAME = oramst) 


On the <strong>Siebel</strong> applications server, go into the Oracle client installation and delete (or 

rename) the tnsnames.ora and sqlnet.ora files. You no longer need these as 

Oracle now connects to the database by resolving the name from the /etc/hosts file. 

As root edit the file /etc/hosts on the client machine (that is, the <strong>Siebel</strong> applications 

server in this case) and add an entry like the following: 

 

oramst.dbserver 

The name oramst.dbserver should be whatever you have provided as the 

GLOBAL_DBNAME in the listener .ora file. This becomes the connect string to connect to 

this database from any client. 

7.6.11 High I/O with Oracle Shadow Processes Connected to <strong>Siebel</strong> 

The disk on which Oracle binaries were installed was close to 100% occupied during the 

peak load of 1000 concurrent users. This problem was diagnosed to be the well-known 

oraus.msb problem. On Oracle clients that make use of OCI (Oracle Call Interface), the 

OCI driver makes thousands of calls to translate messages from the oraus.msb file. 

This problem has been documented by Oracle under bug ID 2142623. 

The Sun workaround for this problem is to cache the oraus.msb file in memory and 

translate the file access and system calls to user calls and memory operations. The 

caching solution is dynamic, and code changes are not needed. With <strong>Siebel</strong> this 

workaround helped in reducing the calls. The 100% occupied I/O went away and a 4% 

reduction in CPU was observed. also, the response times for transactions got a 11% 

boost. 

This problem is supposed to have been fixed in Oracle 9.2.0.4. Details on how to 

implement the Sun workaround by LD_PRELOAD of an interpose library are available at 

http://developers.sun.com/solaris/articles/oci_cache.html. 

7.7 <strong>Siebel</strong> Database Connection Pooling 

The database connection pooling feature built into the <strong>Siebel</strong> server software provides 

better performance. A users:database connection ratio of 20:1 has been proven to 

provide good results with <strong>Siebel</strong>7.5/Oracle9202. This reduced about 3% CPU at the 

<strong>Siebel</strong> server, as fewer connections were made from <strong>Siebel</strong> server to database during a 

2000 user Call Center test. <strong>Siebel</strong> memory/user was 33% lower and Oracle memory/user 

was 79% lower, because 20 <strong>Siebel</strong> users share the same Oracle connection. 

<strong>Siebel</strong> anonymous users do not use connection pooling. If the anonuser count is set 

too high (that is, higher than the recommended 10 to 20%) you would end up wasting the 

tasks, as MaxTasks is a number inclusive of real users. Also the anon sessions do not 


use connection pooling, so you would have lots of one-to-one connections and use 

increased memory and CPU on both the database server and the applications server. 

How to Enable Connection Pooling 

Set the following <strong>Siebel</strong> parameters at the server level via the <strong>Siebel</strong> thin client GUI or 

svrmgr: 

MaxSharedDbConns integer full 

MinSharedDbConns integer full 

MaxTrxDbConns integer full 

Then bounce the <strong>Siebel</strong> server. For example, if you are set up to run 1000 users, the 

value for is 1000/20=50. Set all of three 

parameters to the same value (50) to direct <strong>Siebel</strong> to share a single database connection 

for 20 <strong>Siebel</strong> users or tasks. 

srvrmgr:SIEBSRVR> change param MaxTasks=1100, MaxMTServers=20, 

MinMTServers=20, MinSharedDbConns=50, MaxSharedDbConns=50, 

MinTrxDbConns=50 for comp esalesobjmgr_enu 

How to Check if Connection Pooling is Enabled 

During the steady state, log in to dbserver and ps -eaf | grep NO | wc -l 

This should return around 50 for this example; if this action returns 1000 then connection 

pooling is not in effect. 

7.8 <strong>Tuning</strong> Sun Java System Directory Server (LDAP) 

In order to prevent connections from timing out, change the following parameter on the 

LDAP directory server: idletimeout = 15 secs. Also increase the cache entries 

parameter for LDAP from the default to 25,000. To change these parameters: 

1. Stop the LDAP Server. 

2. Change the value associated with cache entries in slapd.conf. 

3. Change the value associated with idletimeout in slapd.conf to 15. 

4. Restart the LDAP Server. 


8 <strong>Performance</strong> Tweaks with No Gains 

This section is most important as it sheds light on various myths of performance tuning. 

Sometimes a tunable that you think is sure to result in a performance gain, does not – but 

it ends up being counted as if it did. These are tunables that mostly help other 

applications in a different scenario or are default settings already in effect. This section 

lists out some of the tuning parameters that provided no gain when tested. 

Please note: These observations are specific to the workload, architecture, software versions, 

and so on used during the project at SUN ETC labs. The workload is described in Chapter 4. 

The outcome of certain tuneables may vary when implemented with a different workload on 

a different architecture/configuration. 

Changing the mainwin address MW_GMA_VADDR=0xc0000000 

Changing this to other values did not seem to make a big difference to performance. This 

value is set in the siebenv file. 

Solaris Kernel Parameter stksize 

The default value for this is 16k, or 0x4000 on Sun4U (Ultra) architecture machines 

booted in 64 bit mode (which is always the default). Increasing this to 24k, 0x6000 by the 

below settings in /etc/system file did not provide any gains in performance during the 

tests. 

set rpcmod:svc_default_stksize=0x6000 

set lwp_default_stksize=0x6000 

<strong>Siebel</strong> Server Parameters 

1. Recycle Factor. Enabling this did not provide any performance gains. Default is 

disabled. 

2. SISSPERSISSCONN. This is the parameter that changes the multiplexing ratio 

between <strong>Siebel</strong> server and web server; its default value is 20. Varying this value 

did not result in any performance gains for the specific modules tested in this 

project with the PSPP standard workload (defined in Chapter 4). 

Resonate: Cache size -- RES_PERSIST_BLOCK_SIZE 

This is an environment variable which is set inside the resonate startup script. This 

was changed to as high as 240Kbytes and there was no difference in behavior. 

Sun Java System Web Server maxprocs 

This parameter, when changed from the default of 1, starts up more than one ns-httpd 

(web server process). No gain was measured with a value greater than 1. It was better to 

use a new web server instance. 

Database Connection Pooling with <strong>Siebel</strong> Server Components 

Enabling this for the server component batch workload caused performance to degrade 

such that server processes could not start. Some of the server component modules 

connect to the database using ODBC, which does not support connection pooling. 


Oracle Database Server 

1. Larger than required sharedpoolsize. For a 10,000 <strong>Siebel</strong> user benchmark, a 

value of 400Mbytes was more than sufficient. Too large a value for 

sharedpoolsize wastes valuable database cache memory. 

2. Large SGA. A larger than required SGA size is not going to improve performance, 

whereas too small an SGA size will degrade performance. 

3. Large RBS. Being larger than required by an application would waste space in the 

database cache. It is better to make the application commit more often. 


9 Scripts, Tips, and Tricks for Diagnosing <strong>Siebel</strong> on the Sun 

Platform 

We found the following tips to be very helpful for diagnosing performance and scalability 

issues while running <strong>Siebel</strong> on the Sun platform. 

9.1 To Monitor <strong>Siebel</strong> Open Session Statistics 

You can use the following URLs to monitor the amount of time that a <strong>Siebel</strong> end user 

transaction is taking within your <strong>Siebel</strong> enterprise. This data is updated near realtime. 

Call 

Center 

eSales 

eService 

eChannel 

http://webserver:port/callcenter_enu/_stats.swe?verbose=high 

http://webserver:port/esales_enu/_stats.swe?verbose=high 

http://webserver:port/eservice_enu/_stats.swe?verbose=high 

http://webserver:port/prmportal_enu/_stats.swe?verbose=high 

Table 10.1 

The stat page provides a lot of diagnostic information. Watch out for any rows that are in 

bold. They represent requests that have been waiting for over 10 seconds. 

9.2 To List the Parameter Settings for a <strong>Siebel</strong> Server 

Use the following server command to list all parameter settings for <strong>Siebel</strong> server: 

srvrmgr> list params for server servername comp component show PA_ALIAS, 

PA_VALUE 

Parameters of interest are MaxMTServers, MinMTSServers, MaxTasks, 

MinSharedDbConns, MaxSharedDbConns, MinTrxDbConns. 

9.3 To Find Out All OMs Currently Running for a Component 

Use the following server command to find out all OMs that are running in a <strong>Siebel</strong> 

enterprise, sorted by <strong>Siebel</strong> component type: 

$> srvrmgr /g gateway /e enterprise /u sadmin /p sadmin -c "list tasks for comp 

component show SV_NAME, CC_ALIAS, TK_PID, TK_DISP_RUNSTATE" | grep Running | sort 

-k 3 | uniq -c | sort -k 2,2 -k 1,1 

The following output came from one of the runs: 

47 siebelapp2_1 eChannelObjMgr_enu 19913 Running 





118 siebelapp2_1 eSalesObjMgr_enu 19943 Running 






169 siebelapp2_1 eServiceObjMgr_enu 19873 Running 





45 siebelapp2_1 SCCObjMgr_enu 19696 Running 




Run States are Running, Online, Shutting Down, Shutdown, and Unavailable. 

Run Tasks should be evenly distributed across servers and close to MaxTasks. 

9.4 To Find Out the Number of Active Servers for a Component 

Use the following server command to find out the number of active MTS Servers for a 

component: 

srvrmgr> list comp component for server servername show SV_NAME, CC_ALIAS, 

CP_ACTV_MTS_PROCS, CP_MAX_MTS_PROCS 

The number of active MTS servers should be close to the number of Max MTS servers. 

9.5 To Find Out the Tasks for a Component 

Use the following server command to find the tasks for a component: 

srvrmgr> list task for comp component server servername order by TK_TASKID 

Ordering by task id places the most recently started task at the bottom. It is a good sign 

if the most recently started tasks (that is, those started in the past few seconds or 

minutes) are still running. Otherwise, further investigation is required. 

9.6 To Set Detailed Trace Levels on the <strong>Siebel</strong> Server Components 

(<strong>Siebel</strong> OM) 

Log in to svrmgr and execute the following commands: 

Change evtloglvl taskcounters=4 for comp sccobjmgr 

change evtloglvl taskcounters=4 for comp eserviceobjmgr 

change evtloglvl taskevents=3 for comp sccobjmgr 

change evtloglvl taskevents=3 for comp eserviceobjmgr 

change evtloglvl mtwaring=2 for comp sccobjmgr 

change evtloglvl mtwaring=2 for comp eserviceobjmgr 

change evtloglvl set mtInfraTrace = True 

9.7 To Find Out the Number of GUEST Logins for a Component 

Use the following server command to find out the number of GUEST login for a 

component: 


$ srvrmgr /g gateway /e enterprise /s server /u sadmin /p sadmin /c “list task 

for comp component” | grep Running | grep GUEST | wc -l 

9.8 To Calculate the Memory Usage for an OM 

1. We use the following script, pmem_sum.sh, to calculate the memory usage for an 

OM: 

#!/bin/sh 

if [ $# -eq 0 ]; then 

echo "Usage: pmem_sum.sh " 

fi 

WHOAMI=`/usr/ucb/whoami` 

PIDS=`/usr/bin/ps -ef | grep $WHOAMI" " | grep $1 | grep -v "grep $1" | grep -v 

pmem_sum | \ 

awk '{ print $2 }'` 

for pid in $PIDS 

do 

echo 'pmem process :' $pid 

pmem $pid > `uname -n`.$WHOAMI.pmem.$pid 

done 

pmem $PIDS | grep total | awk 'BEGIN { FS = " " } {print $1,$2,$3,$4,$5,$6} 

{tot+=$4} {shared+=$5} {private+=$6} END {print "Total memory used:", tot/1024 "M by 

"NR" procs. Total Private mem: "private/1024" M Total Shared mem: " shared/1024 "M 

Actual used memory:" ((private/1024)+(shared/1024/NR)) "M"}' 

2. To use it, type the following: 

pmem_sum.sh siebmtshmw 

9.9 To Find the Log File Associated with a Specific OM 

1. Check the server log file for the creation of the multithreaded server process: 

ServerLog Startup 1 2003-03-19 19:00:46 <strong>Siebel</strong> Application Server 

is ready and awaiting requests 

… 

ServerLog ProcessCreate 1 2003-03-19 19:00:46 Created 

multithreaded server process (OS pid = 24796) for Call Center Object Manager 

(ENU) with task id 22535 

… 

2. The log file associated with the preceding OM is SCCObjMgr_enu_24796.log. 

1021 2003-03-19 19:48:04 2003-03-19 22:23:20 -0800 0000000d 001 001f 0001 09 

SCCObjMgr_enu 24796 24992 111 

/export/pspp/siebsrvr/enterprises/siebel2/siebelapp1/log/SCCObjMgr_enu_24796.log 

7.5.2.210 [16060] ENUENU 

… 


9.10 To Produce a Stack Trace for the Current Thread of an OM 

1. Find out the current thread number that the OM is running (assuming the pid is 

24987): 

% > cat SCCObjMgr_enu_24987.log 

1021 2003-03-19 19:51:30 2003-03-19 22:19:41 -0800 0000000a 001 001f 0001 09 

SCCObjMgr_enu 24987 24982 93 

/export/pspp/siebsrvr/enterprises/siebel2/siebelapp1/log/SCCObjMgr_enu_24987.log 

7.5.2.210 [16060] ENUENU 

… 

2. The thread number for the preceding example is 93. 

3. Use pstack to produce the stack trace: 

$ > pstack 24987 | sed –n ’/lwp# 93/,/lwp# 94/p’ 

----------------- lwp# 93 / thread# 93 -------------------- 

7df44b7c lwp_mutex_lock (c00000c0) 

7df40dc4 mutex_lock_kernel (4ea73a00, 0, 7df581b8, 7df56000, 0, c00000c0) + c8 

7df41a64 mutex_lock_internal (4ea73a00, 7df581ac, 0, 7df56000, 1, 0) + 44c 

7e3c430c CloseHandle (11edc, 7e4933a8, c01f08c8, 7ea003e4, c1528, 4ea73a98) + a8 

7ea96958 __1cKCWinThread2T6M_v_ (7257920, 2, c1538, 1d, 0, 0) + 14 

7ea97768 __SLIP.DELETER__B (7257920, 1, 7ebc37c0, 7ea00294, dd8f8, 4a17f81c) + 4 

7ea965f0 __1cMAfxEndThread6FIi_v_ (7257ab8, 7257920, 0, 1, 1, 0) + 58 

7edd2c6c __1cVOSDSolarisThreadStart6Fpv_0_ (7aba9d0, 1, c01f08c8, 51ecd, 1, 0) + 

50 

7fb411bc __1cUWslThreadProcWrapper6Fpv_I_ (7aba9e8, 7e4933a8, c01f08c8, 

c01f08c8, 0, ffffffff) + 48 

7ea9633c __1cP_AfxThreadEntry6Fpv_I_ (51ecc, ffffffff, 1, 7ea9787c, 4000, 

4a17fe10) + 114 

7e3ca658 __1cIMwThread6Fpv_v_ (1, 7e4a6d00, 7e496400, c0034640, c01f0458, 

c01f08c8) + 2ac 

7df44970 _lwp_start (0, 0, 0, 0, 0, 0) 

----------------- lwp# 94 / thread# 94 -------------------- 

9.11 To Show System-Wide Lock Contention Issues Using lockstat 

1. You can use lockstat to find out a lot of things about lock contentions. One 

interesting question is, “What is the most contended lock in the system?” 

2. The following shows the system locks contention during one of the 4600 user runs 

with large latch values and double ramp up time. 

# lockstat sleep 5 

Adaptive mutex spin: 17641 events in 4.998 seconds (3529 events/sec) 

Count indv cuml rcnt spin Lock Caller 

------------------------------------------------------------------------------- 

3403 19% 19% 1.00 51 0x30017e123e0 hmestart+0x1c8 

3381 19% 38% 1.00 130 service_queue background+0x130 

3315 19% 57% 1.00 136 service_queue background+0xdc 

2142 12% 69% 1.00 86 service_queue qenable_locked+0x38 

853 5% 74% 1.00 41 0x30017e123e0 hmeintr+0x2dc 


… 

1 0% 100% 1.00 5 0x300267b75f0 lwp_unpark+0x60 

1 0% 100% 1.00 18 0x3001d9a79c8 background+0xb0 

---------------------------------------------------------------------------- 

Adaptive mutex block: 100 events in 4.998 seconds (20 events/sec) 

Count indv cuml rcnt nsec Lock Caller 

---------------------------------------------------------------------------- 

25 25% 25% 1.00 40179 0x30017e123e0 hmeintr+0x2dc 

8 8% 33% 1.00 765800 0x30017e123e0 hmestart+0x1c8 

6 6% 39% 1.00 102226 service_queue background+0xdc 

5 5% 44% 1.00 93376 service_queue background+0x130 

… 

1 1% 100% 1.00 74480 0x300009ab000 callout_execute+0x98 

---------------------------------------------------------------------------- 

Spin lock spin: 18814 events in 4.998 seconds (3764 events/sec) 


---------------------------------------------------------------------------- 

2895 15% 15% 1.00 2416 sleepq_head+0x8d8 cv_signal+0x38 

557 3% 18% 1.00 1184 cpu[10]+0x78 disp_getbest+0xc 

486 3% 21% 1.00 1093 cpu[2]+0x78 disp_getbest+0xc 

… 

1 0% 100% 1.00 1001 turnstile_table+0xf68 turnstile_lookup+0x50 

1 0% 100% 1.00 1436 turnstile_table+0xbf8 turnstile_lookup+0x50 

1 0% 100% 1.00 1618 turnstile_table+0xc18 turnstile_lookup+0x50 

---------------------------------------------------------------------------- 

Thread lock spin: 33 events in 4.998 seconds (7 events/sec) 


---------------------------------------------------------------------------- 

2 6% 6% 1.00 832 sleepq_head+0x8d8 setrun+0x4 

2 6% 12% 1.00 112 cpu[3]+0xb8 ts_tick+0xc 

2 6% 18% 1.00 421 cpu[8]+0x78 ts_tick+0xc 

… 

1 3% 97% 1.00 1 cpu[14]+0x78 turnstile_block+0x20c 

1 3% 100% 1.00 919 sleepq_head+0x328 ts_tick+0xc 

---------------------------------------------------------------------------- 

R/W writer blocked by writer: 73 events in 4.998 seconds (15 events/sec) 


---------------------------------------------------------------------------- 

8 11% 11% 1.00 100830 0x300274e5600 segvn_setprot+0x34 

5 7% 18% 1.00 87520 0x30029577508 segvn_setprot+0x34 

4 5% 23% 1.00 96020 0x3002744a388 segvn_setprot+0x34 

… 



---------------------------------------------------------------------------- 

R/W writer blocked by readers: 40 events in 4.998 seconds (8 events/sec) 


---------------------------------------------------------------------------- 


3 8% 18% 1.00 55733 0x3002744a388 segvn_setprot+0x34 

3 8% 25% 1.00 102240 0x3001c729668 segvn_setprot+0x34 

… 

1 2% 98% 1.00 48720 0x3002759b500 segvn_setprot+0x34 


---------------------------------------------------------------------------- 

R/W reader blocked by writer: 52 events in 4.998 seconds (10 events/sec) 



---------------------------------------------------------------------------- 

5 10% 10% 1.00 131488 0x300274e5600 segvn_fault+0x38 

3 6% 15% 1.00 111840 0x3001a62b940 segvn_fault+0x38 

3 6% 21% 1.00 139253 0x3002792f2a0 segvn_fault+0x38 

… 



---------------------------------------------------------------------------- 

Lockstat record failure: 5 events in 4.998 seconds (1 events/sec) 

Count indv cuml rcnt Lock Caller 

---------------------------------------------------------------------------- 

5 100% 100% 0.00 lockstat_lock lockstat_record 

---------------------------------------------------------------------------- 

9.12 To Show the Lock Statistic of an OM Using plockstat 

1. The syntax is: plockstat [ -o outfile ] -p pid. The program grabs a 

process and shows the lock statistics upon exit or interrupt. 

2. The following shows the lock statistics of an OM during one of the 4600 user runs 

with large latch values and double ramp up time: 

$> plockstat -p 4027 

^C 

----------- mutex lock statistics ----------- 

lock try_lock sleep avg sleep avg hold location: 

count count fail count time usec time usec name 

2149 0 0 1 5218 142 siebmtshmw: __environ_lock 

2666 0 0 0 0 3 [heap]: 0x9ebd0 

948 0 0 0 0 1 [heap]: 0x9f490 

312 0 0 2 351 88 [heap]: 0x9f4c8 

447 0 0 0 0 2 [heap]: 0x9f868 

237 0 0 0 0 101 [heap]: 0x9f8a0 

2464 0 0 1 4469 2 [heap]: 0xa00f0 

1 0 0 0 0 11 [heap]: 0x17474bc0 

… 

219 0 0 0 0 2 libsscassmc: m_cacheLock+0x8 

41 41 0 0 0 2 0x79a2a828 

152295 0 0 15 11407 1 libthread: tdb_hash_lock 

2631 0 0 10 297603 468 libc: _time_lock 

1807525 0 0 16762 59752 14 libc: __malloc_lock 

----------- condvar statistics ----------- 

cvwait avg sleep tmwait timout avg sleep signal brcast location: 

count time usec count count time usec count count name 

0 0 41 40 4575290 0 0 [heap]: 0x2feec30 

8 16413463 0 0 0 8 0 [heap]: 0x305fce8 

20 7506539 0 0 0 20 0 [heap]: 0x4fafbe8 

16 6845818 0 0 0 16 0 [heap]: 0x510a8d8 

… 

12 8960055 0 0 0 12 0 [heap]: 0x110f6138 

13 10375600 0 0 0 13 0 [heap]: 0x1113e040 

----------- readers/writer lock statistics ----------- 

rdlock try_lock sleep avg sleep wrlock try_lock sleep avg sleep avg 

hold location: 

count count fail count time usec count count fail count time usec time 

usec name 


382 0 0 0 0 0 0 0 0 0 

0 [heap]: 0x485c2c0 

102100 0 0 0 0 0 0 0 0 0 

0 libsscfdm: g_CTsharedLock 

9.13 To “Truss” an OM 

1. Modify siebmtshw: 

#!/bin/ksh 

. $MWHOME/setmwruntime 

MWUSER_DIRECTORY=${MWHOME}/system 

LD_LIBRARY_PATH=/usr/lib/lwp:${LD_LIBRARY_PATH} 

#exec siebmtshmw $@ 

truss -l -o /tmp/$$.siebmtshmw.trc siebmtshmw $@ 

2. After you start up the server, this wrapper creates the truss output in a file named 

pid.siebmtshmw.trc in /tmp. 

9.14 How to Trace the SQL Statements for a Particular <strong>Siebel</strong> 

Transaction 

If response times are high, or if you think that the database is a bottleneck, you can 

check how long the SQL queries are taking to execute by running a SQL trace on a 

LoadRunner script. The SQL trace is run through <strong>Siebel</strong> and tracks all of the database 

activity and how long things take to execute. If execution times are too high, there is a 

problem with the database configuration, which most likely is contributing to high 

response times. To run a SQL Trace on a particular script, follow these instructions: 

1. Configure the <strong>Siebel</strong> environment to the default settings (that is, the component 

should have only one OM, and so on). 

2. Open the LoadRunner script in question in the Virtual User Generator. 

3. Place a breakpoint at the end of Action 1 and before Action 2. This will stop 

the user at the breakpoint. 

4. Run a user. 


5. Once the user has stopped at the breakpoint, enable SQL tracing via srvrmgr: 

change evtloglvl ObjMgrSqlLog=4 for comp 

6. Press Play, which will resume the user. 

7. Wait until the user has finished. 

8. Under the $SIEBEL_SERVER_HOME/enterprises// 

directory, there will be a log for the component that is running. This log contains 

detailed information on the database activity, including how long the SQL queries 

took to execute. Search for high execution times (greater than 0.01 seconds). 

9. Once you are done, disable SQL tracing via srvrmgr: 

change evtloglvl ObjMgrSqlLog=0 for comp 

9.15 Changing Database Connect Stringvi $ODBCINI and Editing the 

field .ServerName. 

1. srvrmgr /g siebgateway /e siebel /u sadmin /p sadmin /s 

2. At srvrmgr prompt: Verify and change the value of DSConnectString 

parameter: 

i. List params for named subsystems serverdatasrc 

ii. Change param DSConnectString= for named subsystems 

serverdatasrc. 

9.16 Enabling/Disabling <strong>Siebel</strong> Components 

Disabling a Component 

1. Bring up srvrmgr console: 

srvrmgr /g siebgateway /e siebel /u sadmin /p sadmin /s 

2. Disable the component: 

disable comp 

3. List the components and verify their status: 

list components 

Enabling a Component 

Disabling a component may disable the component definition. We may need to 

enable the component definition 

1. Bring up srvrmgr console at enterprise level (just do not use "/s" switch). 

2. srvrmgr /g siebgateway /e siebel /u sadmin /p sadmin 

3. Enable the component definition: 

enable compdef 

(Note: This enables the component definition at all active servers; you may need to 

disable the component at servers where you don't need the component.) 


4. Bring up srvrmgr console at server level: 

srvrmgr /g siebgateway /e siebel /u sadmin /p sadmin /s 

5. Enable the component definition at server level: 

enable compdef 

6. Bounce gateway and all active servers. 

Note: Sometimes the component may not be enabled even after following these steps. In 

such cases you may need to enable the component group at enterprise level before 

enabling the actual component: 

enable compgrp 


10 Appendix A: Transaction Response Times 

These are the different transactions executed by the OLTP workload of 10,000 <strong>Siebel</strong> 

users. The collection here is an average of all 10,000 users in steady state of 1 hour, 

executing multiple iterations with 30 second average think time between each 

transaction. The average of all of the average transactions was 0.167 seconds. 

Transaction Name Min Average Max 

eSvc2_Query_Branch 0.094 0.14 3.703 

eSvc2_Mail_And_Fax 0.063 0.085 3.156 

eSvc2_Login_Page 0.203 0.324 4.578 

eSvc2_Email 0.078 0.193 5.563 

eSvc2_Contact_Us_3 0.063 0.126 3.859 

eSvc2_Contact_Us 0.047 0.064 4.313 

eSvc2_Branch_Locator 0.031 0.042 3.422 

Eservice2 Avg 0.13914286 

eSvc1_Query_Product 0.031 0.052 2.672 

eSvc1_Product_Button 0.016 0.033 0.125 

eSvc1_OK_Product 0.016 0.025 0.266 

eSvc1_My_SR 0.047 0.084 4.031 

eSvc1_My_Account 0.063 0.088 3.813 

eSvc1_Logout 0.031 0.093 3.609 

eSvc1_Login_Page 0.203 0.351 3.984 

eSvc1_Login 0.266 0.446 3.75 

eSvc1_Enter_Info_And_Submit 0.047 0.098 3.453 

eSvc1_Drilldown_Submit_SR 0.063 0.168 4.344 

eSvc1_Drilldown_SR 0.078 0.115 2.969 

eSvc1_Continue 0.047 0.086 3.359 

eSrevice2_Contact_Us_2 0.031 0.06 3.797 

0.13069231 

eSls3_338LogOut 0.063 0.1 0.563 

eSls3_337UserProfile 0.031 0.049 0.156 

eSls3_336MyAccount 0.094 0.13 1.438 

eSls3_335DDOrder 0.141 0.239 2.922 

eSls3_334MyOrders 0.094 0.128 0.203 


eSls3_332ConfirmOrder 0.531 0.676 1.625 

eSls3_331Continue 0.266 0.428 2.484 

eSls3_330EditShippingDeatils 0.078 0.113 2.016 

eSls3_329EditShipping 0.078 0.098 0.25 

eSls3_328EnterCreditInfo 0.406 0.589 2.875 

eSls3_327CheckOut 0.219 0.341 2.578 

eSls3_326GotoCart 0.203 0.243 0.625 

eSls3_325ChangeQuantAddtocart 0.281 0.453 3.172 

eSls3_324DDproduct 0.406 0.503 3.328 

eSls3_323BackFromcompare 0 0 0 

eSls3_322Pick4ProdCompare 0.344 0.413 3.125 

eSls3_321Color_RedSize_MedManuf_Acme 0.531 0.6 0.797 


eSls3_320SelectProdFamily 0.109 0.181 2.281 

eSls3_319ParamSearch 0.078 0.101 0.328 

eSls3_318Home 0.188 0.251 3.203 

eSls3_317AddtoCart 0.203 0.283 3.438 

eSls3_316DDProdBabyHoodPolice 0.125 0.148 0.219 

eSls3_315NextRecs 0.094 0.097 0.172 

eSls3_314NextRecs 0.094 0.141 2.828 

eSls3_313DDSubCategory1.1 0.172 0.226 3.031 

eSls3_312DDCategory1 0.125 0.178 2.609 

eSls3_311CatalogTab 0.141 0.17 0.359 


eSls3_309DDProdBabyHood 0.109 0.134 0.266 

eSls3_308NextRecs 0.078 0.097 0.188 

eSls3_307NextRecs 0.094 0.1 0.516 




eSls3_302Login 0.406 0.595 3.922 

eSls3_301ClickLogin 0.047 0.073 0.156 

eSls3_300StartApp 0.25 0.373 3.313 

0.24378947 

eSls2_224LogOut 0.047 0.119 2.734 

eSls2_223EmptyCart 0.063 0.129 2.203 

eSls2_222GotoCart 0.141 0.237 2.734 


eSls2_220DDProdBabyHood 0.109 0.147 1.453 

eSls2_219NextRecs 0.094 0.111 2.719 

eSls2_218NextRecs 0.094 0.106 2.375 




eSls2_214Addressbook 0.031 0.054 2.031 


eSls2_212UserProfile 0.031 0.056 2.563 


eSls2_210MyOrders 0.078 0.119 2.719 



eSls2_207DDProdBing 0.125 0.176 3.672 

eSls2_206NextRecs 0.078 0.104 3.672 

eSls2_205NextRecs 0.078 0.105 3.578 



eSls2_202Login 0.375 0.536 3.641 

eSls2_201ClickLogin 0.047 0.077 2.125 

eSls2_200StartApp 0.234 0.331 4.063 

0.18456 

eSls1_118DDProdBingle 0.125 0.163 3.297 

eSls1_117NextRecs 0.078 0.096 2.922 

eSls1_116NextRecs 0.078 0.101 3.109 





eSls1_112DDProdBinge 0.125 0.161 3.859 

eSls1_111NextRecs 0.078 0.099 3.469 

eSls1_110NextRecs 0.078 0.106 3.438 




eSls1_106DDProdBing 0.125 0.159 3.266 

eSls1_105NextRecs 0.078 0.106 3.297 

eSls1_104NextRecs 0.078 0.103 3.203 




eSls1_100StartApp 0.219 0.348 3.594 

0.14910526 

eChannel3_ResetStates 0 0.006 1.969 

eChannel3_GoToSRSolutionView 0.094 0.152 5.766 

eChannel3_GoToServiceTab 0.156 0.246 6.328 

eChannel3_ExecuteQueryBySRNum 0.078 0.152 4.547 

eChannel3_EnterDetailsAndSave 0.219 0.388 6.656 

eChannel3_DrilldownSR 0.125 0.218 6.469 

eChannel3_CreateNewActivity 0.078 0.124 4.875 

eChannel3_ClickQueryButton 0.094 0.166 4.313 

eChannel3_ChangeStatusSubStatusAndSave 0.188 0.28 6.188 

0.19244444 

eChannel2_132_SaveQuote 0.156 0.296 8.422 

eChannel2_131_NewQuote 0.125 0.185 4.172 

eChannel2_130_OpportunityQuoteView 0.125 0.2 3.813 

eChannel2_129_OpportunityAttachmentView 0.109 0.175 4.125 

eChannel2_128_SaveActivity 0.125 0.257 4.25 

eChannel2_127_NewActivity 0.109 0.15 2.141 

eChannel2_126_OpportunityActivityView 0.125 0.192 4.25 

eChannel2_125_PickSalesTeam 0.141 0.271 5.828 

eChannel2_124_NewSalesTeam 0.047 0.085 3.875 

eChannel2_123_OpportunitySalesTeamView 0.109 0.193 3.781 

eChannel2_122_SaveProduct2 0.203 0.345 4.641 

eChannel2_121_PickProduct2 0.094 0.132 2.203 

eChannel2_120_QueryForProduct2 0.031 0.117 2.969 

eChannel2_119_ClickOnProductField2 0.016 0.021 0.484 

eChannel2_118_NewProduct2 0.094 0.153 4.703 

eChannel2_117_SaveProduct1 0.203 0.364 6.141 

eChannel2_116_PickProduct1 0.094 0.137 1.688 

eChannel2_115_QueryForProduct1 0.031 0.061 2.5 

eChannel2_114_ClickOnProductField 0.031 0.037 1.656 

eChannel2_113_NewProduct1 0.094 0.139 4.125 

eChannel2_112_OpptyProductView 0.109 0.175 3.984 

eChannel2_111_SaveContact 0.156 0.293 6.469 

eChannel2_110_NewContact 0.156 0.305 5.578 


eChannel2_109_NewContactOppty 0.078 0.118 2.719 

eChannel2_108_DrilldownOnOppty 0.156 0.267 5.094 

eChannel2_107_SaveOppty 0.172 0.341 5.891 

eChannel2_106_PickAccount 0.125 0.22 5.516 

eChannel2_105_QueryForAccount 0.031 0.071 2.813 

eChannel2_104_ClickOnQueryForAccount 0.016 0.02 0.578 

eChannel2_103_ClickOnAccountField 0.109 0.189 5.453 

eChannel2_102_NewOppty 0.391 0.706 8.313 

eChannel2_101_OpportunityScreen 0.156 0.264 5.844 

eChannel2_100_ResetStates 0 0.014 0.359 

0.19675758 

eChannel1_SearchForContact 0.047 0.087 2.953 

eChannel1_SearchForAccount 0.047 0.09 4.234 

eChannel1_SaveSR 0.094 0.19 4.078 

eChannel1_SaveContact 0.109 0.216 4.031 

eChannel1_SaveAddress 0.078 0.174 5.234 

eChannel1_SaveActivity 0.109 0.217 4.828 

eChannel1_SaveAccount 0.141 0.276 4.797 

eChannel1_ResetStates 0 0.005 0.469 

eChannel1_LookInContact 0.016 0.029 0.703 

eChannel1_LookInAccount 0.031 0.042 1.719 

eChannel1_GoContacts 0.219 0.326 7.375 

eChannel1_GoAccounts 0.156 0.25 6.547 

eChannel1_DrilldownOnAccount 0.141 0.303 6.172 

eChannel1_CreateSR 0.094 0.153 4.563 

eChannel1_CreateContact 0.063 0.118 3.156 

eChannel1_CreateAddress 0.063 0.085 2.359 

eChannel1_CreateActivity 0.094 0.128 2.766 

eChannel1_CreateAccount 0.078 0.115 5.734 

eChannel1_CloseFind 0.078 0.114 2.328 

eChannel1_ClickNewInContactMVG 0.109 0.236 5.25 

eChannel1_ClickFind 0.109 0.168 4.578 

eChannel1_AccountTeamView 0.109 0.15 3.109 

eChannel1_AccountSRView 0.109 0.169 3.531 

eChannel1_AccountRevenueView 0.109 0.169 4.547 

eChannel1_AccountQuoteView 0.109 0.186 4.391 

eChannel1_AccountOrderView 0.109 0.19 3.953 

eChannel1_AccountAssetView 0.094 0.16 4.5 

eChannel1_AccountAddressView 0.094 0.152 4.234 

eChannel1_AccountActivityView 0.109 0.165 3.375 

0.1607931 

CC3_2solutionView 0.047 0.092 4.844 

CC3_2solutionSRview 0.031 0.058 2.781 

CC3_2setStatusAndSaveSR 0.156 0.284 5.938 

CC3_2setStatusAndSaveActivity 0.094 0.132 4.844 

CC3_2serviceScreen 0.078 0.123 4.172 

CC3_2saveSolution 0.063 0.136 4.438 

CC3_2ResetStates 0 0.005 0.641 

CC3_2relatedSRview 0.063 0.116 5.063 

CC3_2newSolutionInMVG 0.047 0.115 5.156 


CC3_2newSolution 0.047 0.136 3.281 

CC3_2goSolution 0.047 0.096 4.609 

CC3_2backToActivitiesView 0.047 0.075 3.688 

CC3_2activitiesView 0.078 0.121 4.781 

0.11453846 

CC2_2serviceScreen 0.172 0.259 5.391 

CC2_2searchSR 0.047 0.088 4.25 

CC2_2saveSR 0.172 0.339 6.563 

CC2_2saveActivityPlan 0.563 0.931 7.984 

CC2_2ResetStates 0 0.004 0.469 

CC2_2queryContact 0.047 0.067 3.422 

CC2_2productPicklist 0.047 0.134 2.156 

CC2_2OpenBinocular 0.031 0.034 1.078 

CC2_2okProduct 0.016 0.031 2.859 

CC2_2okContact 0.047 0.098 5.328 

CC2_2newSR 0.031 0.064 4.5 

CC2_2newActivityPlan 0.031 0.054 3.688 

CC2_2goProduct 0.063 0.088 3.984 

CC2_2goContact 0.063 0.121 5.375 

CC2_2contactMVG 0.109 0.226 5.734 

CC2_2closeBinocular 0 0.002 0.594 

CC2_2activityPlanView 0.047 0.1 4.5 

0.15529412 

CC1_ResetStates 0 0.005 0.594 

CC1_183_NavigateBackToOpptyQuoteView 0.141 0.213 5.453 

CC1_182_SaveSalesOrder 0.063 0.135 4.141 

CC1_181_NewSalesOrder 0.359 0.617 6.344 

CC1_180_QuoteOrderView 0.25 0.351 6.766 

CC1_179_UpdateOppty 0.141 0.258 6.359 

CC1_178_Reprice 0.031 0.055 3.234 

CC1_177_SelectDiscountAndSaveQuote 0.172 0.288 5.516 

CC1_176_PickPriceList 0.031 0.067 3.328 

CC1_175_GoQueryForPriceList 0.063 0.106 4.141 

CC1_174_QueryForPriceList 0.047 0.076 2.609 

CC1_173_BringUpPriceListMVG 0.297 0.423 4.203 

CC1_172_DrilldownOnQuote 0.281 0.403 5.797 

CC1_171_SaveQuote 0.094 0.208 6.406 

CC1_170_AutoQuote 0.141 0.293 6.313 

CC1_169_OpptyQuoteView 0.063 0.12 4.391 

CC1_168_SaveProduct2 0.063 0.131 3.781 

CC1_167_PickProduct2 0.016 0.033 1.016 

CC1_166_GoQueryForProduct2 0.063 0.112 4.172 

CC1_165_QueryForProduct2 0.031 0.116 1.25 

CC1_164_NewProduct2 0.016 0.035 1.281 

CC1_163_SaveProduct1 0.063 0.137 3.875 

CC1_162_PickProduct1 0.016 0.028 3.406 

CC1_161_GoQueryForProduct1 0.063 0.234 5 

CC1_160_QueryForProduct1 0.047 0.145 2.266 

CC1_159_NewProduct1 0.016 0.041 3.156 

CC1_158_OpptyProductView 0.047 0.093 3.969 


CC1_157_DrilldownOnOppty 0.172 0.256 4.797 

CC1_156_SaveOppty 0.109 0.236 4.656 

CC1_155_PickAccount 0.047 0.082 3.797 

CC1_154_GoQueryForAccount 0.078 0.137 5.125 

CC1_153_QueryForAccount 0.25 0.381 5.25 

CC1_152_BringUpAccountMVG 0.109 0.229 2.891 

CC1_151_NewInOpptyMVG 0.359 0.599 7.375 

CC1_150_NewOppty 0.078 0.209 4.063 

CC1_149_ContactOpptyView 0.063 0.115 3.438 

CC1_148_SaveContact 0.078 0.176 4.969 

CC1_147_NewContact 0.016 0.031 4.328 

CC1_146_ContactScreen 0.125 0.204 5.141 

CC1_145_CloseFind 0 0.002 0.547 

CC1_144_SearchForContact 0.031 0.062 3.75 

CC1_143_ClickFind 0.031 0.04 1.328 

cc1 avg 0.17814286 

total avg 0.16775095 


11 Appendix B: Database Objects Growth During the Test 

GROWTH IN 

SIEBEL OBJECT NAME TYPE BYTES 

S_DOCK_TXN_LOG TABLE 

1,177,673,72 

8 

S_EVT_ACT TABLE 190,341,120 

S_DOCK_TXN_LOG_P1 INDEX 96,116,736 

S_DOCK_TXN_LOG_F1 INDEX 52,600,832 

S_ACT_EMP TABLE 46,202,880 

S_SRV_REQ TABLE 34,037,760 

S_AUDIT_ITEM TABLE 29,818,880 

S_OPTY_POSTN TABLE 28,180,480 




S_ACT_EMP_U1 INDEX 21,626,880 

S_ACT_CONTACT TABLE 21,135,360 

S_EVT_ACT_U1 INDEX 18,391,040 





S_ACT_EMP_P1 INDEX 15,187,968 


S_EVT_ACT_F9 INDEX 13,852,672 

S_ACT_CONTACT_U1 INDEX 13,361,152 

S_ORDER_ITEM TABLE 13,066,240 

S_REVN TABLE 12,943,360 

S_CONTACT TABLE 12,779,520 

S_SRV_REQ_M7 INDEX 12,492,800 


S_ACT_EMP_F1 INDEX 11,804,672 

S_SRV_REQ_U2 INDEX 10,731,520 

S_SRV_REQ_U1 INDEX 10,444,800 

S_DOC_QUOTE TABLE 10,321,920 

S_QUOTE_ITEM TABLE 9,666,560 


S_ACT_CONTACT_F2 INDEX 8,716,288 

S_OPTY TABLE 8,396,800 

S_ACT_CONTACT_P1 INDEX 8,183,808 

S_AUDIT_ITEM_F2 INDEX 7,987,200 

S_ORDER TABLE 7,987,200 

S_SRV_REQ_F13 INDEX 7,905,280 

S_SRV_REQ_P1 INDEX 7,872,512 


S_RESITEM TABLE 7,798,784 

S_AUDIT_ITEM_P1 INDEX 7,634,944 

S_REVN_U1 INDEX 7,454,720 






S_REVN_M1 INDEX 7,004,160 

S_OPTY_U1 INDEX 6,676,480 


S_EVT_ACT_F11 INDEX 6,602,752 

S_OPTY_TERR TABLE 6,569,984 



S_DOC_ORDER TABLE 4,972,544 

S_SR_RESITEM TABLE 4,972,544 

S_ORG_EXT TABLE 4,833,280 

S_ACCNT_POSTN TABLE 4,341,760 

S_OPTY_CON TABLE 4,136,960 

S_DOC_QUOTE_BU TABLE 4,096,000 

S_PARTY TABLE 4,096,000 


S_POSTN_CON TABLE 3,932,160 



S_OPTY_BU TABLE 3,194,880 


S_CONTACT_M13 INDEX 3,153,920 

S_OPTY_U2 INDEX 3,072,000 


S_OPTY_BU_M9 INDEX 2,990,080 

S_OPTY_BU_P1 INDEX 2,949,120 

S_PARTY_M2 INDEX 2,949,120 

S_ORDER_BU_M2 INDEX 2,867,200 

S_OPTY_BU_U1 INDEX 2,744,320 

S_ORG_EXT_F1 INDEX 2,629,632 





S_CONTACT_F6 INDEX 2,580,480 

S_OPTY_V2 INDEX 2,580,480 

S_RESITEM_M4 INDEX 2,547,712 

S_REVN_F6 INDEX 2,539,520 

S_ORDER_M5 INDEX 2,498,560 



S_POSTN_CON_M1 INDEX 2,498,560 









S_ORDER_POSTN TABLE 2,334,720 



S_CONTACT_X TABLE 2,293,760 



S_RESITEM_U2 INDEX 2,138,112 



S_ORDER_BU TABLE 2,088,960 


S_DOC_QUOTE_U1 INDEX 2,048,000 

S_RESITEM_U1 INDEX 2,023,424 


S_REVN_P1 INDEX 1,966,080 


S_DOC_QUOTE_U2 INDEX 1,925,120 

S_SR_RESITEM_U1 INDEX 1,892,352 

S_POSTN_CON_M2 INDEX 1,843,200 


12 Appendix C: Oracle statspack Report 

This is one of the Oracle statspack reports collected during the one-hour steady state 

of the test. This report provides an efficient way to identify where the time is spent inside 

the Oracle database during the <strong>Siebel</strong> tests. 

STATSPACK report for 

DB Name DB Id Instance Inst Num Release Cluster Host 

------------ ----------- ------------ -------- ----------- ------- ------------ 

ORAMST 3597051609 oramst 1 9.2.0.2.0 NO siebdb 

Snap Id Snap Time Sessions Curs/Sess Comment 

------- ------------------ -------- --------- ------------------- 

Begin Snap: 7 14-Aug-04 00:29:10 1,043 57.2 5000-and-server-use 

End Snap: 8 14-Aug-04 01:52:31 639 77.8 5000-and-server-use 

Elapsed: 

83.35 (mins) 

Cache Sizes (end) 

~~~~~~~~~~~~~~~~~ 

Buffer Cache: 2,912M Std Block Size: 8K 

Shared Pool Size: 480M Log Buffer: 10,240K 

Load Profile 

~~~~~~~~~~~~ Per Second Per Transaction 

--------------- --------------- 

Redo size: 874,784.17 7,640.12 

Logical reads: 95,274.57 832.10 

Block changes: 5,344.32 46.68 

Physical reads: 1,632.28 14.26 

Physical writes: 632.79 5.53 

User calls: 3,171.71 27.70 

Parses: 1,243.43 10.86 

Hard parses: 0.10 0.00 

Sorts: 385.85 3.37 

Logons: 0.92 0.01 

Executes: 1,662.82 14.52 

Transactions: 114.50 

% Blocks changed per Read: 5.61 Recursive Call %: 22.22 

Rollback per transaction %: 0.22 Rows per Sort: 18.68 

Instance Efficiency Percentages (Target 100%) 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

Buffer Nowait %: 99.99 Redo NoWait %: 100.00 

Buffer Hit %: 98.64 In-memory Sort %: 100.00 

Library Hit %: 100.02 Soft Parse %: 99.99 

Execute to Parse %: 25.22 Latch Hit %: 99.95 

Parse CPU to Parse Elapsd %: 95.76 % Non-Parse CPU: 97.30 

Shared Pool Statistics Begin End 

------ ------ 

Memory Usage %: 67.68 68.34 

% SQL with executions>1: 37.08 33.60 

% Memory for SQL w/exec>1: 62.79 58.60 

Top 5 Timed Events 

~~~~~~~~~~~~~~~~~~ % Total 

Event Waits Time (s) Ela Time 

-------------------------------------------- ------------ ----------- -------- 

CPU time 11,550 72.04 

log file sync 573,210 2,143 13.37 

direct path read 280,947 627 3.91 

direct path write 154,781 533 3.32 


log file parallel write 550,101 515 3.21 

------------------------------------------------------------- 

Wait Events for DB: ORAMST Instance: oramst Snaps: 7 -8 

-> s - second 

-> cs - centisecond - 100th of a second 

-> ms - millisecond - 1000th of a second 

-> us - microsecond - 1000000th of a second 

-> ordered by wait time desc, waits desc (idle events last) 

Avg 

Total Wait wait Waits 

Event Waits Timeouts Time (s) (ms) /txn 

---------------------------- ------------ ---------- ---------- ------ -------- 

log file sync 573,210 283 2,143 4 1.0 

direct path read 280,947 0 627 2 0.5 

direct path write 154,781 0 533 3 0.3 

log file parallel write 550,101 546,565 515 1 1.0 

db file sequential read 6,451,104 0 294 0 11.3 

db file parallel write 27,046 0 152 6 0.0 

SQL*Net more data to client 2,006,251 0 152 0 3.5 

control file parallel write 1,639 0 38 23 0.0 

enqueue 8,971 0 10 1 0.0 

buffer busy waits 37,083 0 9 0 0.1 

latch free 5,753 5,685 6 1 0.0 

log file switch completion 51 0 3 54 0.0 

db file scattered read 3,064 0 1 0 0.0 

LGWR wait for redo copy 13,244 0 1 0 0.0 

control file sequential read 743 0 0 0 0.0 

SQL*Net break/reset to clien 28 0 0 1 0.0 

log file single write 4 0 0 1 0.0 

buffer deadlock 211 211 0 0 0.0 

log file sequential read 4 0 0 0 0.0 

SQL*Net message from client 13,127,091 0 4,983,503 380 22.9 

SQL*Net more data from clien 2,197,829 0 285 0 3.8 

SQL*Net message to client 13,126,685 0 22 0 22.9 

------------------------------------------------------------- 

Background Wait Events for DB: ORAMST Instance: oramst Snaps: 7 -8 

-> ordered by wait time desc, waits desc (idle events last) 

Avg 

Total Wait wait Waits 

Event Waits Timeouts Time (s) (ms) /txn 

---------------------------- ------------ ---------- ---------- ------ -------- 

log file parallel write 550,105 546,569 515 1 1.0 

db file parallel write 27,046 0 152 6 0.0 

control file parallel write 1,639 0 38 23 0.0 

log file sync 825 0 2 2 0.0 

LGWR wait for redo copy 13,244 0 1 0 0.0 

db file scattered read 176 0 0 2 0.0 

SQL*Net more data to client 3,320 0 0 0 0.0 

db file sequential read 995 0 0 0 0.0 

control file sequential read 691 0 0 0 0.0 

enqueue 12 0 0 1 0.0 

direct path write 54 0 0 0 0.0 

log file single write 4 0 0 1 0.0 

buffer busy waits 24 0 0 0 0.0 

direct path read 54 0 0 0 0.0 

log file sequential read 4 0 0 0 0.0 

rdbms ipc message 1,186,256 554,100 28,351 24 2.1 

SQL*Net message from client 9,948 0 4,869 489 0.0 

smon timer 16 16 4,501 ###### 0.0 

SQL*Net more data from clien 3,845 0 1 0 0.0 

SQL*Net message to client 9,948 0 0 0 0.0 


13 References 

Solaris 8 <strong>Software</strong> Developer Collection: Multithreaded Programming Guide 

http://docs.sun.com/app/docs/doc/806-5257?q=multithreading 

This book is very useful for understanding the Solaris OS threads implementation. 

Developers must read this to take advantage of the various features. The book is also 

helpful for performance tuning engineers. 

Solaris 8 Reference Manual Collection: mallocctl(3MALLOC) - MT hot memory 

allocator 

http://docs.sun.com/app/docs/doc/806-0627/6j9vhfn1i?q=mtmalloc&a=view 

Details on the MTmalloc library that is part of the standard Solaris OS. 

Sun Java System (formerly known as iPlanet) Web Server 6.0 <strong>Performance</strong> <strong>Tuning</strong>, 

Sizing, and Scaling Guide 

http://docs.sun.com/source/816-5690-10/perf6.htm 

Solaris Tunable Parameters Reference Manual 

http://docs.sun.com/app/docs/doc/816-0607?q=kernel+tuning 

<strong>Siebel</strong> PSPP (Platform Sizing and <strong>Performance</strong> Program) Benchmark Web Site 

http://www.siebel.com/crm/performance-benchmark.shtm 

This is the official repository of certified benchmark results from all hardware vendors. 

Sun Fire E2900-E25K Servers Benchmarks 

http://www.sun.com/servers/midrange/sunfire_e2900/benchmarks.html 

Industry leading benchmark results. 

<strong>Siebel</strong> SupportWeb (<strong>Siebel</strong> product documentation) 

http://supportweb.siebel.com/default.asp?lf=support_search.asp&rf=search/seasearch.asp&tf=tsupport_search.asp 


Acknowledgements 

Thanks to <strong>Siebel</strong> Systems' Mark Farrier, Francisco Casas, Farinaz Farsai, Vikram Kumar, 

Santosh Hasani, Sanjay Agarwal, Harsha Gadagkar, and others from <strong>Siebel</strong> Systems for 

working with Sun. Thanks to Scott Anderson, George Drapeau, and Lester Dorman for 

their role as management and sponsors from Sun. Thanks to Diane Kituara, Sherill Ellis, 

and George Drapeau for their valuable feedback on content for this paper. Thanks to Sun 

MDE Team Kesari Mandyam, Giri Mandalika, and Devika Gollapudi for their 

contributions. Thanks to the various engineers across Sun Microsystems for providing 

subject matter expertise: Ravindra Talashikar (PAE), Stephen Johnson (Network 

Storage), and Dileep Kumar (Sun Java System Web Server). 

About the Author 

Khader Mohiuddin works as a Staff Engineer in the Market Development Engineering 

organization at Sun Microsystems, Inc. His role as the engineering lead for the Sun- 

<strong>Siebel</strong> Alliance involves optimizing <strong>Siebel</strong> CRM applications on the Sun platform, joint 

technology adoption projects, and so on. He has been at Sun for four years empowering 

ISVs to adopt Sun’s latest technologies. Prior to that he worked at Oracle as a developer 

and Senior <strong>Performance</strong> Engineer for five years, and at AT&T Bell Labs, New Jersey, for 

three years. 

About Sun Microsystems, Inc. 

Since its inception in 1982, a singular vision — “The Network Is The Computer” — has 

propelled Sun Microsystems, Inc. (Nasdaq: SUNW) to its position as a leading provider of 

industrial-strength hardware, software, and services that make the Net work. Sun can be 

found in more than 170 countries and on the World Wide Web at http://sun.com/.

Performance Tuning Siebel Software on the Sun Platform

Create successful ePaper yourself

Delete template?

Save as template?