Performance Tuning Siebel Software on the Sun Platform

Description
The reason for this symptom was that some of the <strong>Siebel</strong> server processes servicing
these users had either hung or crashed. Close observation with repeated replays
revealed that the Siebmtshmw process memory shot up from 700M to 2GB.
The UltraSPARC IV-based E2900 server can handle up to 30,000 processes and about
87,000 LWPs (threads), so there was no resource limitation from the machine here. The
<strong>Siebel</strong> applications are running into 32-bit process space limits -- but why is the memory
per process going from 700Mbytes to 2GBytes? Further debugging lead us to the
problem: the stack size.
The total memory size of a process is made up of heap size + stack size + data
and anon segments. In this case, it was found that the bloating in size was at the stack,
which grew from 64Kbytes to 1Gbyte, which was abnormal. pmap is a utility on the
Solaris system that provides a detailed breakdown of memory sizes per process.
Here is an output of pmap that shows that the stack segment bloated to about 1Gbyte:
siebapp6@/export/pspp> grep stack pmap.4800mixed.prob.txt (pmap output unit is
in Kbytes)
FFBE8000 32 32 - 32 read/write/exec [ stack ]
FFBE8000 32 32 - 32 read/write/exec [ stack ]
FFBE8000 32 32 - 32 read/write/exec [ stack ]
FFBE8000 32 32 - 32 read/write/exec [ stack ]
FFBE8000 32 32 - 32 read/write/exec [ stack ]
C012A000 1043224 1043224 - 1043224 read/write/exec [ stack ]
We reduced the stack size to 128Kbytes and this fixed the problem. Now a single <strong>Siebel</strong>
server configured with 10,000 MaxTasks started up successfully. It was failing to startup
before breaking at the mmap call. The stack size of 8Mytes is a default setting on the
Solaris system. However, limiting the stack size to 128Kbytes removed a lot of the
instability in our high load tests and we were able to run 5000 user tests without errors,
pushing up to 80% to 90% CPU usage on the 12-way E2900 server.
Changing the mainwin address MW_GMA_VADDR=0xc0000000 to other values did not
seem to make a big difference. Please do not vary this parameter.
Solution
Change the stack size hard limit of the <strong>Siebel</strong> process from unlimited to 512 bytes.
Stack size on the Solaris OS has a hard limit and a soft limit. The default values for these
two limits are unlimited and 8Mbytes, respectively. This means that an application
process on the Solaris OS can have its stack size anywhere up to the hard limit. Since
the default hard limit on the Solaris system is unlimited the <strong>Siebel</strong> application
processes could grow their stack size all the way up to 2Gbytes. When this occurs, the
total memory size of the <strong>Siebel</strong> process hits the max limit of memory addressable by a
32-bit process and bad things happen (such as a hang or crash). Setting the limit for
stack to 1Mbytes or a lower value resolved the issue.
<strong>Performance</strong> <strong>Tuning</strong> <strong>Siebel</strong> <strong>Software</strong> on the Sun Platform Page 35