Performance Tuning Siebel Software on the Sun Platform

More documents

Recommendations

Info

file(blocks) unlimited unlimited data(kbytes) 4194303 4194303 stack(kbytes) 512 512 coredump(blocks) unlimited unlimited nofiles(descriptors) 65536 65536 vmemory(kbytes) unlimited unlimited The preceding setting allows the siebsvc to mmap the .shm file and thereby siebsvc succeeds in forking the rest of the processes and <strong>Siebel</strong> server processes start up successfully. Thus this finding enables one to configure MaxTasks greater than 9000 on the Solaris OS. That means one can get around the scalability limit of 9000 users per <strong>Siebel</strong> server, but how much higher can one go? We determined the limit to be 18,000. If MaxTasks was configured to > 18000, <strong>Siebel</strong> calculates the *.shm file size to be about 2Gbytes. siebsvc tries to mmap this file and fails, as it has hit the limit for a 32-bit application process. The following listing shows the <strong>Siebel</strong> .shm file with maxtask set to 18,000: sdcv480s002:/export/siebsrvr/admin/%ls -l *.shm ; ls -lh *.shm -rwx------ 1 sunperf other 2074238976 Jan 25 13:23 siebel.sdcv480s002.shm* -rwx------ 1 sunperf other 1.9G Jan 25 13:23 siebel.sdcv480s002.shm* sdcv480s002:/export/siebsrvr/admin/%plimit 12527 12527: siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002 resource current maximum time(seconds) unlimited unlimited file(blocks) unlimited unlimited data(kbytes) 4194303 4194303 stack(kbytes) 512 512 coredump(blocks) unlimited unlimited nofiles(descriptors) 65536 65536 vmemory(kbytes) unlimited unlimited sdcv480s002:/export/siebsrvr/admin/%prstat -s size -u sunperf -a PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 12527 sunperf 3975M 3966M sleep 59 0 0:00:55 0.0% siebsvc/3 12565 sunperf 2111M 116M sleep 59 0 0:00:08 0.1% siebmtsh/8 12566 sunperf 2033M 1159M sleep 59 0 0:00:08 3.1% siebmtshmw/10 12564 sunperf 2021M 27M sleep 59 0 0:00:01 0.0% siebmtsh/12 12563 sunperf 2000M 14M sleep 59 0 0:00:00 0.0% siebproc/1 26274 sunperf 16M 12M sleep 59 0 0:01:43 0.0% siebsvc/2 In the rare case there is a need to run more than 18,000 <strong>Siebel</strong> users on a single <strong>Siebel</strong> server node, the way to do it is to install another <strong>Siebel</strong> server instance on the same node. This works well and is a supported configuration on <strong>Siebel</strong>/Sun platform. It is not advisable to configure more MaxTasks than needed, as this could affect the performance of the overall <strong>Siebel</strong> enterprise. 7.3.2 Bloated <strong>Siebel</strong> Processes (Commonly Mistaken as Memory Leaks) Symptom During some of the high concurrent user tests, it was observed that hundreds of users suddenly failed. <strong>Performance</strong> <strong>Tuning</strong> <strong>Siebel</strong> <strong>Software</strong> on the Sun Platform Page 34
Description The reason for this symptom was that some of the <strong>Siebel</strong> server processes servicing these users had either hung or crashed. Close observation with repeated replays revealed that the Siebmtshmw process memory shot up from 700M to 2GB. The UltraSPARC IV-based E2900 server can handle up to 30,000 processes and about 87,000 LWPs (threads), so there was no resource limitation from the machine here. The <strong>Siebel</strong> applications are running into 32-bit process space limits -- but why is the memory per process going from 700Mbytes to 2GBytes? Further debugging lead us to the problem: the stack size. The total memory size of a process is made up of heap size + stack size + data and anon segments. In this case, it was found that the bloating in size was at the stack, which grew from 64Kbytes to 1Gbyte, which was abnormal. pmap is a utility on the Solaris system that provides a detailed breakdown of memory sizes per process. Here is an output of pmap that shows that the stack segment bloated to about 1Gbyte: siebapp6@/export/pspp> grep stack pmap.4800mixed.prob.txt (pmap output unit is in Kbytes) FFBE8000 32 32 - 32 read/write/exec [ stack ] FFBE8000 32 32 - 32 read/write/exec [ stack ] FFBE8000 32 32 - 32 read/write/exec [ stack ] FFBE8000 32 32 - 32 read/write/exec [ stack ] FFBE8000 32 32 - 32 read/write/exec [ stack ] C012A000 1043224 1043224 - 1043224 read/write/exec [ stack ] We reduced the stack size to 128Kbytes and this fixed the problem. Now a single <strong>Siebel</strong> server configured with 10,000 MaxTasks started up successfully. It was failing to startup before breaking at the mmap call. The stack size of 8Mytes is a default setting on the Solaris system. However, limiting the stack size to 128Kbytes removed a lot of the instability in our high load tests and we were able to run 5000 user tests without errors, pushing up to 80% to 90% CPU usage on the 12-way E2900 server. Changing the mainwin address MW_GMA_VADDR=0xc0000000 to other values did not seem to make a big difference. Please do not vary this parameter. Solution Change the stack size hard limit of the <strong>Siebel</strong> process from unlimited to 512 bytes. Stack size on the Solaris OS has a hard limit and a soft limit. The default values for these two limits are unlimited and 8Mbytes, respectively. This means that an application process on the Solaris OS can have its stack size anywhere up to the hard limit. Since the default hard limit on the Solaris system is unlimited the <strong>Siebel</strong> application processes could grow their stack size all the way up to 2Gbytes. When this occurs, the total memory size of the <strong>Siebel</strong> process hits the max limit of memory addressable by a 32-bit process and bad things happen (such as a hang or crash). Setting the limit for stack to 1Mbytes or a lower value resolved the issue. <strong>Performance</strong> <strong>Tuning</strong> <strong>Siebel</strong> <strong>Software</strong> on the Sun Platform Page 35
Page 1 and 2: Performance <stron
Page 3 and 4: Abstract This paper discusses the p
Page 5 and 6: 7.6.6 Oracle Parameters Tun
Page 7 and 8: 1 Tuning for Price
Page 9 and 10: 2 Siebel Applicati
Page 11 and 12: 3 Optimal Sun/Siebel</stron
Page 13 and 14: 3.1 Hardware and Software</
Page 15 and 16: 5 10,000 Concurrent Users: Test Res
Page 17 and 18: 6 Siebel Scalabili
Page 19 and 20: Cost per Siebel Us
Page 21 and 22: 7.1 Tuning Solaris
Page 23 and 24: 7.1.3 The Solaris Kernel and TCP/IP
Page 25 and 26: In the file $SIEBEL_ROOT/bin/enu/ua
Page 27 and 28: 7.2.3 Assignment Manager Tu
Page 29 and 30: MQ series parameter tuning: PERFQMG
Page 31 and 32: Investigating the Reason for the Fa
Page 33: let a Siebel proce
Page 37 and 38: Parameter Scope Default Tuned Value
Page 39 and 40: 7.6 Tuning <strong
Page 41 and 42: 86 MB/second 57 MB/second Figure 7.
Page 43 and 44: DATA_51200K INDX_512K DATA_5120K TE
Page 45 and 46: 7.6.4 Solaris MPSS Tuning</
Page 47 and 48: Siebel Object Name
Page 49 and 50: instance_name=oramst job_queue_proc
Page 51 and 52: ORAPERF.S_PRI_LST T4 WHERE T4.PAYME
Page 53 and 54: the database during peak load. In t
Page 55 and 56: use connection pooling, so you woul
Page 57 and 58: Oracle Database Server 1. Larger th
Page 59 and 60: 160 siebelapp2_1 eSalesObjMgr_enu 1
Page 61 and 62: 9.10 To Produce a Stack Trace for t
Page 63 and 64: Count indv cuml rcnt nsec Lock Call
Page 65 and 66: 5. Once the user has stopped at the
Page 67 and 68: 10 Appendix A: Transaction Response
Page 69 and 70: eSls1_115DDSubCategory1.3 0.125 0.1
Page 71 and 72: CC3_2newSolution 0.047 0.136 3.281
Page 73 and 74: 11 Appendix B: Database Objects Gro
Page 75 and 76: S_OPTY_BU_M8 INDEX 2,334,720 S_ORDE
Page 77 and 78: log file parallel write 550,101 515
Page 79: Acknowledgements Thanks to

Performance Tuning Siebel Software on the Sun Platform

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?