Broker Performance & Tuning using EntireX v8.2 - Software AG

Broker Performance & Tuning 

using EntireX v8.2 

Miguel Pujol 

Mainframe Integration Team 

Global Support Services 

September 2012

Agenda 

- Basic Broker Concepts and Broker Stubs 

- Tracing Tips 

- Performance &Tuning General Tips 

- Monitor resources via SMH v8.2 

- EXX v8.2 Enhancements 

- DWM and DMM 

- Command Logging 

- Deferred Tracing 

- EntireX Trace Utility 

- Miscellaneous 

- Contacting Support 

- Premium Support Services 

27 September 2012 Software AG - Get There Faster 2

Snoozing / leaving soon? 

• Important Items to Take Away 

• Upgrade to EntireX v8.2.2 as soon as possible 

• Lots of new functionality in SMH v8.2 

• Run Broker with Dynamic Memory Management and Dynamic 

Worker Management 

• Access tracing techniques in the EntireX documentation, at: 

http://documentation.softwareag.com/webmethods/wmsuites/wmsuite8-2_ga/EntireX/8-2- 

SP1_EntireX/adminWin/tracing.htm 


EntireX Broker Architecture 


In simple terms: A traffic cop! 


Is it that simple, though? 


Broker: What exactly is it? 

Example: A Travel Agency 

• Travel Agent 

• Customers 

• Services 

• Types of travel 


A Travel Agency 

Customer 

Customer 

Customer 

Customer 

EntireX Broker 

Travel 

Agent 

Whew! Lots 

of traffic 

today!!! 

Inquire 

Book 

Issue 

Airplane 

Cruise 

Train 

27 September 2012 Software AG - Get There Faster 8 

Pay 

Bus 

Clients Class Server Service

How do you talk to Broker? 

You need a Server and a Client � Conversation 

The Server: 

1) LOGON 

2) REGISTER 

Repeat 

3) RECEIVE 

… 

4) SEND 

End-Repeat 

5) DEREGISTER 

6) LOGOFF 


How do you talk to Broker? 

The Client: 

1) LOGON 

2) SEND 

3) RECEIVE 

4) LOGOFF 

You need a Server and a Client � Conversation 


Tracing Broker Components 

• Broker Kernel (Attribute File) 

Different sections, different levels 

• Broker Stubs 

Depends on Client/Server type & platform 


Broker Kernel Standard Tracing 

Broker Attribute File Sections – Recomm. trace levels 

(Development might request higher trace levels) 

• DEFAULTS=BROKER 

� TRACE-LEVEL=1 

• DEFAULTS=SECURITY 


• DEFAULTS=ADABAS / NET / TCP / SSL 


• DEFAULTS=SERVICE / TOPIC 

� Recommended: CONVERSION=(SAGTCHA,TRACE=3) 

� Older Approach: TRANSLATION=SAGTCHA 

�Only a minimal set of special characters is converted properly 

�If characters are not supported, they are turned into a blank; no error reported 


Tracing EntireX Components 

“Table Summarizing Tracing for webMethods EntireX 

Components” – see documentation, at: 

http://documentation.softwareag.com/webmethods/wmsuites/wmsuite8- 

2_ga/EntireX/8-2-SP1_EntireX/adminWin/tracing.htm 


Tracing EntireX Components – Cont. 

The “EntireX Trace Utility” – see documentation, at: 


2_ga/EntireX/8-2- 

SP1_EntireX/adminWin/brokerTraceUtility.htm#brokerTraceUtility 


Broker Kernel Special Tracing 

�Command Logging (Independent from Trace) 

• Startup JCL 

� DD Cards: 

//DDCLOGR1 DD DISP=SHR,DSN=.CMDLOG1 

//DDCLOGR2 DD DISP=SHR,DSN=.CMDLOG2 

• DEFAULTS=BROKER section of the Attribute File 

� CMDLOG=YES 

� CMDLOG-FILE-SIZE=5317 (KB) 

� NUM-CMDLOG-FILTER=12 

� Add filters in SMH or using ETBCMD 

Documentation Link: 


2_ga/EntireX/8-2-SP1_EntireX/adminWin/commandLogging.htm#commandLogging 


Broker Kernel Special Tracing – Cont. 

�Deferred Tracing 

• Triggered when a specific condition occurs 

• All information leading up to the specified event is traced 

• If event does not occur, it is like TRACE-LEVEL=0 

• DEFAULTS=BROKER section of the Attribute File 

• TRAP-ERROR=0007 (Error code, not error class) 

• TRBUFNUM=4 (Trace buffer size, e.g. 4 * 64KB) 

• TRMODE=WRAP (Only possible value – trace written when event occurs) 

• Gotcha – No output in the trace? 

• Deferred Tracing is ON but no TRAP-ERROR is specified 

• See KB 1707748 

Documentation Link: 


2_ga/EntireX/8-2-SP1_EntireX/adminZos/setupBroker.htm#deferredTracing 


Performance & Tuning – Considerations 

� The architecture of Broker is the same on all systems 

� Development recommends the dynamic approach (DYNAMIC-MEMORY- 

MANAGEMENT and DYNAMIC-WORKER-MANAGEMENT turned on), unless a 

stable configuration has been found for a specific scenario and it does not 

change 

� The static approach (both DMM and DWM set to NO) is still good for a well 

configured environment 

� Rule of Thumb #1 

� Run Broker Kernel on the platform with the highest “horsepower” 


� Run Servers “as close” to the data as possible 


� Apply the latest EntireX Hotfixes on all platforms �� Empower > Fix Explorer 


Performance & Tuning – Cont. 

� z/OS Operating System upgrades 

(SHARE Atlanta 2012) 

� The next release of z/OS will be in the 

second half of 2013 instead of Sep. 2012 

� Future releases will be every two years 

instead of every year 

� Support for concurrent releases will be 

adjusted 

� IBM found only 5% of customers migrate 

every year - majority migrate every two 

years (or even less frequently) 



� Workload Manager (WLM) - prioritization of workload 

� Compatibility Mode (old way) 

� Specify priorities 

� Goal Mode (new way) 

� Specify goals for a workload, e.g. response time thresholds 

� When goal is not achieved, WLM will raise the priority 

� When goal is exceeded, WLM reduces the priority to give other 

workload better access to the system 

� Velocity parameter 

� Velocities tell WLM what percentage of time a workload can use 

resources and to what extent it can be delayed 

� Adabas requires a very high velocity to run well: 99 is best 

� Specify a high velocity for EntireX also 

� Ensure that the WLM never thinks the goal for EntireX has been met when 

there are still requests to be processed 



� Linux/UNIX/Windows (LUW) 

� In EXX 8.2, usage of semaphores and shared memory was 

removed from Broker 

� ETBADM running as local service on Unix controls all local 

Brokers 

� ETBADM is also a Broker (ETBNUC) running in RUN-MODE=ADMIN 

with a front-end 

� Broker is a single process on Unix machines with a varying 

number of POSIX threads (“pthreads”) according to the settings 

� PSTORE-TYPE=ADABAS or CTREE works on Unix, with ADABAS 

being the fastest method on all systems even though IPC is 

involved 



� Applications 

� Servers 

� Load balancing (RPC/ACI Server replicates) 

� Broker Services’ Attach Server or “embedded” in XML RPC 

Server (or write your own Attach Server, as documented) 

� Natural subprograms (e.g. READ instead of complex FIND, use 

SETTIME and *TIMD, etc.) 

� Typical reasons for long-running requests: 

� too much work to do 

� waits for external resources (e.g. database records) 

� priority of server address space too low 

� Clients 

� Number of sockets (Java) – see Technical Paper “Understanding 

and Tuning the POOLSIZE and POOLTIMEOUT Parameters” 

� LOGOFF explicitly to release resources (otherwise, wait for the 

Timeout Manager, ETBTOM) 



� Other Technical Considerations 

� SVC vs. TCP/IP 

� Compression / Decompression 

� Security – to what extent? 

� SSL Encryption / Decryption 

� EntireX Security 

� Natural Security (NSC, NATSAF) 

� Adabas (ADASCR, ADASAF) 

� Entire Net-Work Security (NETSAF) 


Security - Protecting Broker Against Attacks 

� Denial-of-Service Attacks 

� If Broker runs with SECURITY=YES and a process tries to 

logon using invalid credentials repeatedly, the whole 

system could be affected 

� With PARTICIPANT-BLACKLIST = YES, after 10 

consecutive security errors within 30 seconds, the 

Blacklist Handler puts participant on the blacklist 

� All subsequent requests from this participant are 

blocked until BLACKLIST-PENALTY-TIME has elapsed 

See documentation, at: 


SP1_EntireX/adminZos/setupBroker.htm#denialOfService 


New Features in EntireX v8.2 


Dynamic Memory Management (DMM) 

Motivation 

• Broker v7 and below used a Static Memory model 

• Used to share memory between multiple processes 

• Required configuration of all necessary resources 

• It was difficult to determine reasonable maximum values 

• After Broker initialization, no changes to memory organization were 

possible 

• Memory consumption could not be accommodated 

• Changing memory requirements regarding number of specific resources 

could only be adapted after reconfiguration and restart of the Broker 


DMM – Old Design 

Static Broker memory pool 

ETBMAIN ETBTOM ETBXWKR 

ETBXWKR 

ETBXWKR 

TCPCOM TCPPROXY 

SSLCOM SSLPROXY 

NETCOM 

ETBOPER 


DMM – New Design 

Broker memory pools 

dynamically allocated 

and released 

ETBMAIN ETBTOM ETBXWKR 

ETBXWKR 

ETBXWKR 

TCPCOM TCPPROXY 

SSLCOM SSLPROXY 

NETCOM 

ETBOPER 


DMM – Some Design Details 

• Introduce concept of resource-specific Memory Pools 

• Each Memory Pool provides a defined number of resources of one 

particular resource type, e.g. Participants, Conversations, etc. 

• Each Memory Pool is divided into blocks of a size, which is 

required by its related resource 

• The amount of objects per resource type is still configurable 

• All NUM-* Broker attributes can be omitted 


DMM - Pool Control Blocks 

Pool control 

struct _POOL_CONTROL 

{ 

SMP_LOCK_T lock; 

HEADER* pPoolHeaderTop; 

HEADER* pPoolHeaderBot; 

SMBLK* pSemaResponse; 

CE_BOOL fAllocatePool; 

CE_BOOL fDeallocatePool; 

}; 

One POOL_CONTROL per resource type 

One POOL_HEADER per memory pool 

Pool header 

Pool header 

next next 

struct _POOL_HEADER 

{ 

HEADER header; 

CE_VOID* pMemoryPool; 

CE_BYTE* pBitList; 

CE_BYTE* pDataLow; 

CE_BYTE* pDataHigh; 

CE_LONG nBytes; 

CE_LONG nSlots; 

CE_LONG nSlotsFree; 

CE_LONG sizeSlot; 

time_t lastActivityTime; 

}; 


NULL

DMM - Memory Pool Layout 

0 0 1 0 0 …. 

Bit list, one bit per slot 

Slot 1 Slot 2 

Slot 3 

Slot n-2 Slot n-1 Slot n 

Administrative part 

Data part 


DMM - Design Details (Cont.) 

• Pool allocation/de-allocation based on 

ticket system 

• Every task requiring new resource issues 

order through ticket system 

• Dedicated task (ETBMAIN, Main Task) 

receives ticket, performs requested 

operation 

• Dedicated task (ETBTOM, Timeout 

Manager) controls all pools, issues order 

to release unused pool 

• As a result, the method is portable and 

can operate on different underlying 

Operating Systems 

• Applicable to systems supporting 

multithreading and systems supporting 

only multitasking 

http://www.freepatentsonline.com/20090172336.pdf 


DMM - New global Broker Attribute 

DYNAMIC-MEMORY-MANAGEMENT 

NO 

YES 

NO | YES 

All memory is allocated at Broker startup based on the calculation from the 

defined NUM-* attributes. Size of memory cannot be changed. This is the 

known behavior of EXX 7.3. 

An initial portion of memory is allocated at Broker startup based on defined 

NUM-* attributes or internal default values if no NUM-* attributes have been 

defied. More memory is allocated without Broker restart if there is a need to 

use more storage. Unused memory is deallocated. The upper limit of memory 

consumption can be defined using attribute MAX-MEMORY. 

If you run Broker with DYNAMIC-MEMORY-MANAGEMENT=YES, the following attributes are not needed: 

NUM-CLIENT / NUM-CMDLOG-FILTER / NUM-COMBUF / NUM-CONV[ERSATION] 

NUM-LONG[-BUFFER] / NUM-PUBLICATION / NUM-PUBLISHER / NUM-SERVER / NUM-SERVICE 

NUM-SERVICE-EXTENSION / NUM-SHORT[-BUFFER] / NUM-SUBSCRIBER / NUM-SUBSCRIBER-TOTAL 

NUM-TOPIC / NUM-TOPIC-EXTENSION / NUM-TOPIC-TOTAL 

NUM-UOW|MAX-UOWS|MUOW / NUM-WQE 

Caution: 

However, if one of these attributes is defined, it determines the allocation size of that 

particular Broker resource. 


O 

z 

u 

w


MAX-MEMORY 

0, UNLIM 

others 

0 | n | nK | nM | nG | UNLIM 

Defines the upper limit of memory allocated by Broker if 

DYNAMIC-MEMORY-MANAGEMENT has been defined. 

No memory limit. 

Defines the maximum limit of allocated memory. If limit is 

exceeded, error 671 Requested allocation exceeds MAX-MEMORY 

is generated. 


O 

z 

u 

w


• If you want statistics on allocation and deallocation operations in 

Broker, you can configure Broker to create a storage report with the 

attribute STORAGE-REPORT. 

STORAGE-REPORT 

NO | YES 

Create a storage report about Broker memory usage. 

NO 

YES 

Do not create the storage report. 

Create the storage report 


O 

z 

u 

W

DMM - Enhanced Broker Reporting 

• Broker provides a STORAGE REPORT to display memory management activities 

EntireX 8.0.0.00 STORAGE Report 2007-11-11 19:34:23 Page 1 

Identifier Address Length Date Time Action 

COMMUNICATION POOL 0xb6a8e010 16781312 bytes 2007-11-11 19:34:23.934 Allocated 

CONVERSATION POOL 0xb6a28010 409856 bytes 2007-11-11 19:34:23.935 Allocated 

CONNECTION POOL 0xb69f0010 225408 bytes 2007-11-11 19:34:23.938 Allocated 

LONG MESSAGES POOL 0xb65b9010 4415616 bytes 2007-11-11 19:34:23.940 Allocated 

SHORT MESSAGES POOL 0xb6208010 3867648 bytes 2007-11-11 19:34:23.951 Allocated 

PARTICIPANT POOL 0xb61e5010 141344 bytes 2007-11-11 19:34:23.962 Allocated 

PARTICIPANT EXTENSION POOL 0xb61d9010 47168 bytes 2007-11-11 19:34:23.963 Allocated 

PROXY QUEUE POOL 0xb61d3010 21536 bytes 2007-11-11 19:34:23.964 Allocated 

SERVICE ATTRIBUTES POOL 0xb61b2010 133648 bytes 2007-11-11 19:34:23.965 Allocated 

SERVICE POOL 0xb619f010 74784 bytes 2007-11-11 19:34:23.966 Allocated 

SERVICE EXTENSION POOL 0xb6194010 43072 bytes 2007-11-11 19:34:23.967 Allocated 

TIMEOUT QUEUE POOL 0xb6178010 112800 bytes 2007-11-11 19:34:23.968 Allocated 

TRANSLATION POOL 0xb615b010 116768 bytes 2007-11-11 19:34:23.970 Allocated 

UNIT OF WORK POOL 0xb612a010 196736 bytes 2007-11-11 19:34:23.971 Allocated 

WORK QUEUE POOL 0xb60c8010 397344 bytes 2007-11-11 19:34:23.972 Allocated 

COMMUNICATION POOL 0xb6a8e010 16781312 bytes 2007-11-11 19:35:40.142 Deallocated 

CONVERSATION POOL 0xb6a28010 409856 bytes 2007-11-11 19:35:40.143 Deallocated 

CONNECTION POOL 0xb69f0010 225408 bytes 2007-11-11 19:35:40.144 Deallocated 


DMM - Creating a Storage Report 

• Set Broker global attribute STORAGE-REPORT = YES. 

• UNIX and Windows 

• Broker creates a file with the name STORAGE.REPORT in the current 

working directory 

• If the environment variable ETB_STORAGE_REPORT is supplied, the file 

name specified in the environment variable will be used 

• If Broker receives the command-line argument -r, the token following 

argument -r will be used as the file name 

• z/OS 

• DDNAME ETBSREP assigns the report file 

• Format RECFM=FB, LRECL=121 


Broker Dynamic Memory Management in SMH 

Pool Usage 



Resource Usage 


Worker Task Handling in Broker v1 – v8.0 

Broker Task configuration defined in Attribute File 

• Global Attribute NUM-WORKER 

• Possible values: 1 – 10 

• The number of Worker Tasks was static 

• No operator intervention or CIS command was available to change it 


Worker Task Handling in Broker v8.1 and above 

Broker Task configuration still defined in Attribute File 

• Global attribute NUM-WORKER 

• Possible values 1 – 10 

but 

Worker task handling in Broker v.1 – v.7 

• DYNAMIC-WORKER-MANAGEMENT=YES 

can be set to let Broker adjust the number of running Worker Tasks 

according to a fluctuating workload 

• The initial portion of Worker Tasks started at Broker startup is still 

determined by NUM-WORKER 


Worker Task Handling in Broker v8.1 and above 

Optional attributes to optimize overall processing: 

• WORKER-MAX, WORKER-MIN, WORKER-NONACT, WORKER-QUEUE- 

DEPTH, WORKER-START-DELAY 

• These are very sensitive attributes and depend on the environment, 

especially the underlying Operating System and the hardware 

• The goal is to achieve high-performance user request processing 

without starting too many Worker Tasks 

• A good starting point to achieve high-performance is not to change 

the attributes and observe the performance of application programs 

after activating Dynamic Worker Management 



Worker Usage 



Statistics 


More Broker Information 

� Shutdown Stats 

• After Broker execution, shutdown statistics and related 

information are produced 

• Output written to Trace file in following sequence: 

� Diagnostic message ETBD0444 

� Statistics, internals and user-specified parameters 

• Table of Shutdown Statistics – see documentation, at: 


SP1_EntireX/adminZos/shutdownStats.htm#Table_of_Shutdown_Statistics 


EntireX Broker Reporting 

� Configuration Report 

(ETBCREP) 

� Load Module Report 

(ETBMREP) 

� Storage Report 

(ETBSREP) 

� Persistent Store Report 

(ETBPREP) 

� License Report 


Miscellaneous 

� Documentation Perspectives 

http://documentation.softwareag.com/webmethods/wmsuites/wmsuite8-2_ga/EntireX/8-2-SP1_EntireX/overview.htm 

� Variable Definition File 

• Broker Attribute File: Configuration of one Broker instance 

• Sharing attribute files among different Brokers is possible 

1. Identify the attributes that are unique 

2. Move them to a variable definition file 

3. Startup JCL 

//ETBVARS DD DISP=SHR,DSN= 

• Unique attributes for each machine: 

BROKER-ID (in DEFAULTS=BROKER) 

NODE (in DEFAULTS=NET) 

PORT (in DEFAULTS=SSL and DEFAULTS=TCP) 


Contacting Global Support 

Services 


Contacting Global Support Services 

� Empower � eService 

� Contact Support Web page 

https://empower.softwareag.com/ContactSupport/default.asp 

� Global Support Contact Directory 

https://empower.softwareag.com/ContactSupport/Directory/default.asp 


Information needed by Support and/or Development 

• Environment & Problem Description 

• What is the error? 

• Where/when is the error occurring? 

• For each Component 

�Platform 

�Component Type 

�Configuration Details 

• Traces, traces, traces 

• Broker Kernel 

• Servers 

• Clients 


Environment / Problem Description 

� What is the error? 

� Most error messages have the format 

ccccnnn[n/s], where: 

cccc = Message Class 

nnnn = Message Number from 0000-9999 

or 

nnns = Message Number from 000-999, and severity of the 

error: 

E = Error; function cannot be performed 

I = Information 

W = Warning; potential problem exists 


Premium Support Services 


Software AG Premium Support Services (PSS) 

Multiple support options, delivered by Global Support or Global Consulting Services 


Software AG Standard Support 

• Use of Empower for 

research and Incident 

management 

• Phone/web assistance from 

Global Support 

• New software versions and 

fixes 

• Access to communities 

Highly-rated support services for day-to-day business activities 


Software AG Premium Support Services 

A senior technician at 

your service for critical 

projects 

Special support for go-live situations reduces risk 

Please contact your Sales team, or email premium.support@softwareag.com 



Centralized management 

of your Support Incidents 

A designated engineer understands –your- environment 




Take advantage of the 

opportunity for an onsite 

analysis of your 

environment 

A health check helps identify strengths and opportunities in your implementation 




Use our DBA expertise for 

short or long-term coverage 

Our DBAs have years of experience supporting production systems 




We can help you 

configure new and 

changing environments 

PSS brings expertise in the latest technologies and product versions 




Enhanced proactive and 

reactive support for missioncritical 

environments 

CSMs offer the right combination of technical and management skills 




Coaching offers a way to build repeatable skill in your own staff 


Enabling self-sufficiency, 

and implementing best 

practices 



Specific product tuning delivers effective utilization and potential cost reduction 


Helping to tune your 

systems for best 

performance 


Broker Performance & Tuning 

using EntireX v8.2

Broker Performance & Tuning using EntireX v8.2 - Software AG

Create successful ePaper yourself

Delete template?

Save as template?