Peter Berggren - OneSAF Public Site

Peter Berggren
OneSAF Production
September, 2012
Distribution A – Approved for Public Release – Distribution is Unlimited

Performance Analysis (Version 5.1.1)
‭ Introduction
‭ OneSAF V5.1
‭ Performance Improvement Strategy
LR Compositions
Threading
HLA Interop Performance and Memory Usage
General Memory Usage
‭ Fort Stewart Experiment
September 2012
OneSAF Technical Exchange
2

Introduction
‭
‭
Background
Performance of OneSAF V5.1 Released in February 2011 was
Significantly Improved from Previous Versions
Yet Because the TEMO Assessment Planned for Later that Year Would
be Conducted on Very Poor Hardware, Performance was (Still) a
Concern
• (HP 8400, 4GB RAM)
Thus, in Preparation for the TEMO Assessment, a Significant Effort Was
Undertaken to Analyze and Improve OneSAF Performance
Results Were Incorporated into OneSAF V5.1.1 Released in December
2011
Purpose
The Purpose of this Briefing is to Describe the Performance Analysis
and Improvement Effort Carried out Between March and June 2011
September 2012
OneSAF Technical Exchange
3

V5.1 Performance Update
‭ Where We Were as of V5.0
‭ Changes to Line-On-Line Benchmark
‭ V5.1 Performance
‭ Details Between V5.0 and V5.1
‭ PTR 51570
September 2012 OneSAF Technical Exchange
4

Where We Were As of V5.0
‭ Line-On-Line Benchmark
‭ Common Hardware Platform
‭ AcquireLC Model with Operational Sensor Ranges per OneSAF V3.0
450
400
350
300
250
200
150
100
50
0
V3.0 V4.0 V5.0
Entities 320 340 400
September 2012 OneSAF Technical Exchange 5

Benchmark Changes Post-V5.0
‭ PTR 53136
Integrated Code Changes to MR Mobility Model for Benchmark;
Controlled via Property
‭ CR 52879
• [Keeps Entities from Getting Stuck]
Added Property to Control Use of Operational Sensor Ranges
‭ CR 54531
Created‭“temo”‭Extension‭and‭“temo-configuration”‭Script‭to‭Configure‭
with AcquireLC and Operational Sensor Ranges
• [Recommended for Best Performance]
September 2012 OneSAF Technical Exchange 6

V5.1 Performance (As of 2011-02-09)
900
810
800
700
600
500
400
400
320 340
300
200
100
0
V3.0 V4.0 V5.0 V5.1
Entities 320 340 400 810
‭
Why the Big Improvement?
September 2012
OneSAF Technical Exchange
7

Details Between V5.0 and V5.1
900
800
700
670 690 710
800 810
600
500
400
300
200
100
400 400
PTR 51570
0
V5.0
Release
trunk
09/07
trunk
10/01
trunk
11/17
trunk
12/08
trunk
12/22
V5.1
02/09
Entities 400 400 670 690 710 800 810
‭ Big Improvement Due to PTR 51570
‭
Also Steady Improvement Throughout V5.1 Development
September 2012
OneSAF Technical Exchange
8

PTR 51570
‭
SE Core: RWA Running Attack ends prematurely
‭ Integrated Into Trunk on 9/20
‭
‭
Fixed error in Grid 3D implementation of area of interest (AOI) service
In theory, benefit of this fix increases with number of entities in
exercise
Warrants additional study
September 2012 OneSAF Technical Exchange
9

Performance Improvement Strategy
‭
‭
Target Issues Relevant to TEMO Assessment
Very Large Scale (~40k Entities)
HLA Interop
Tight Memory (4GB)
Pursue Multiple, Complementary Approaches
LR and ULR Compositions (Use Less CPU)
Threading
Memory Usage in Battlemaster, Simcore, and HLA Interop
HLA Interop Internal Design
September 2012
OneSAF Technical Exchange
10

Reduce Computation Cost of Actor
Compositions for TEMO Use Case
‭
‭
Tickets:
PTR 56727 Performance: actor compositions contain incorrect or
unnecessary components
• Updated over 4000 OneSAF unit and entity compositions to:
– Correct efficiency errors
– Enable on-demand C2 modeling capability
CR 57544 Need Model that uses C4I proximity-based sensor
• Add low-cost sensor model to ULR actors
CR 57102 Create an LR Driver component with OA and formation
keeping
Results:
• Upgrade mobility of LR entities in OneSAF to increase opportunities
to replace MR actors with LR actors
LR and ULR Actors are Much More Capable Than in Previous Versions
Vast Majority of Actors in Final TEMO Assessment Scenario Were LR
or ULR
September 2012
OneSAF Technical Exchange
11

Performance PTRs (Page 1 of 2)
id Subject Integrated
56534 Performance: OneSAF inefficient on platforms with large numbers of CPU cores 4/12/2011
56658 Performance: OneSAF should have more than 75% of memory available for heap on 64 bit machines 4/19/2011
56660 Performance: Compressed OOPS should be enabled by default 4/19/2011
55973 Performance: UniqueIDs used as keys in ordered collections 4/20/2011
56763 Performance: AOI queries use intermediate collections 4/21/2011
56830 Performance: Memory usage in PVD 4/21/2011
56849 Performance: OA wasting time filtering entities 4/25/2011
56906 Performance: SnapToLinear locking up MCT 4/28/2011
56871 Performance: Event delivery optimizations 4/29/2011
56941 Performance: Thread contention & liveliness issues 5/2/2011
56944 Performance: Snap To Linear always replans all segments 5/3/2011
56909 Performance: Munition Selection process consumes too much CPU time 5/5/2011
57141 Performance: Default OneSAF heap size is too large 5/10/2011
57152 Performance: MatrixTable inefficient when units are removed from the distribution 5/11/2011
56662 Performance: OneSAF should display units by default for large scenarios 5/12/2011
56727 Performance: Actor compositions contain incorrect or unnecessary components 5/13/2011
56914 Performance: Massive performance hit when editing Relationships in a large scenario 5/13/2011
56996 Performance: ULR entities have command agent, can be assigned orders in MissionEditor 5/13/2011
57329 Performance: Sim Event interest evaluation is slow 5/18/2011
September 2012
OneSAF Technical Exchange
12

Performance PTRs (Page 2 of 2)
id Subject Integrated
56182 Performance: Unnecessary ODB updates generated in HLA adapter(s) 5/19/2011
57340 Performance: Blackboard trigger delivery optimization 5/19/2011
57127 Performance: ODB should be filter unneeded property changes where possible 5/24/2011
56026 SECore: Performance: ICs spend too much time checking whether they are in buildings 5/25/2011
57359 Performance: Grid3D position manager is not efficient in large exercises 5/26/2011
57186 Performance: Sensor Range Fans on PVD lock up node 5/27/2011
57447 Performance: PositionManagerGrid3D memory opitimizations for entry storage 5/27/2011
57448 Performance: AOI Visitor Pattern optimizations 5/27/2011
57486 Performance: AOI services called in redundant or incorrect cases 5/27/2011
57391 Performance: BDE: Interop performance with large scenarios. 6/2/2011
57169 Performance: DamagedComponent objects are redundant 6/7/2011
57613 Performance: LR fuel consumption calculations are inefficient 6/8/2011
57847 Performance: Stewart11: ODB StatusPanel is ticking pruneDeadObjects 6/28/2011
57860 Performance: Stewart11: Performance improvements in HLA to prevent entity timeout during 6/28/2011 join
57633 Performance: False Positives in GASP Detection Algorithm 6/29/2011
58042 Performance: Clean up proxy object handling of getUniqueID methods 7/1/2011
58101 Performance: ODB addAll() method inefficient with new concurrent collections 7/6/2011
58136 Performance: Algorithm for Pruning ODB Objects Clashes with CopyOnWrite Data Structure 7/13/2011
September 2012
OneSAF Technical Exchange
13

MR Entities *
Line-On-Line Benchmark Trend
‭
Default Configuration
3500
3000
2500
2000
1500
1000
500
0
18-Mar 7-Apr 27-Apr 17-May 6-Jun 26-Jun 16-Jul
High end developer box
Older CHP HP 8400
Z800, 6-core (LAB 002)
Newest CHP Z800 , 4-core (Estimated)
September 2012
OneSAF Technical Exchange
14

Threading
‭
‭
‭
‭
‭
Used YourKit to Identify Points of Thread Contention
Reduced Thread Contention By:
Moving Variables into Thread-Local Storage
Using Concurrent Collections
Changing Logic to Avoid Need for Synchronization
Line on Line Benchmark Shows Nearly 100% Utilization of a Four-Core
Machine
Utilization Tops out at Roughly 4-6 Cores
Thus, To Fully Utilize Machines with >4 Cores, Run Multiple OneSAF
Processes
See‭“Deploying‭OneSAF‭in‭Your‭Environment”
September 2012
OneSAF Technical Exchange
15

Memory Improvements
‭
‭
Identified (with YourKit) Areas of Excessive Memory Consumption
PTRs (Subset of Those Listed Earlier):
56658 Performance: OneSAF should have more than 75% of memory
available for heap on 64 bit machines
56660 Performance: Compressed OOPS should be enabled by default
56830 Performance: Memory usage in PVD
57359 Performance: Grid3D position manager is not efficient in large
exercises
57447 Performance: PositionManagerGrid3D memory opitimizations for
entry storage
58042 Performance: Clean up proxy object handling of getUniqueID
methods
September 2012
OneSAF Technical Exchange
16

Memory Model
‭
‭
Collected Memory Usage Data During Large-Scale Exercises with
Varying:
Number of Actors Internal / External
Number of Simcores
Used Memory Data to Construct a Predictive Model of Memory Usage
Implemented as Excel Spreadsheet
• https://dev.onesaf.net/devsite/Development/SystemArchitecture/trun
k/Performance/large-scale-memory.xlsx
Works with Battlemaster, Simcore, and Interop Nodes
September 2012
OneSAF Technical Exchange
17

Java Heap Calculations
‭
‭
For a Given Quantity of Physical Memory on a Machine, How Much is
Available for Use by the Java Heap in a OneSAF Process?
As‭Per‭the‭OneSAF‭“runtimeloader”‭Script:
• Multiply by 0.9 (This is Done to Allow Space for the OS and Other
Processes on the Box)
• Subtract 800 MB for ERC (Native) Memory
• Subtract 256 MB for Java PERMGEN Memory
Multiply by 0.5 to Allow For:
Example:
• Spikes in Memory Usage
• Garbage
((4,096 MB physical memory * 0.9) - 800 - 256) * 0.5 = 1,315 MB Max
Safe Java Heap Usage
September 2012
OneSAF Technical Exchange
18

Memory Model - Battlemaster
3000
Battlemaster Java Heap Usage
2500
2000
Java Heap Usage (MB)
4 GB
Total
Physical
RAM
1500
1000
500
0
0 10000 20000 30000 40000 50000 60000 70000 80000
Entities
May
June
Linear (May)
Linear (June)
September 2012
OneSAF Technical Exchange
19

Simcore Java Heap Usage at Final as a Function of Entities
and Simcores
2500
2000
1500
Java Heap Usage (MB)
4 GB Total
Physical
RAM
1000
500
10
20
30
Simcores
0
10000 20000 30000 40000 50000 60000
Entities
September 2012 OneSAF Technical Exchange 20

Capacity Trend
Total Entities by Role and Month Based
Exclusively on 4 GB Memory Limit
70,000
60,000
50,000
40,000
30,000
20,000
33,106
47,274
42,366
57,148
53,189
May
June
10,000
-
Battlemaster Simcore Interop
Assumes 4GB physical RAM, 30 simcores, and 10,000 external entities.
September 2012
OneSAF Technical Exchange
21

HLA Interop Internal Design
‭
PTRs (Subset of Those Listed Earlier):
57391 Performance: BDE: Interop performance with large scenarios.
• Increased default number of heartbeat buckets when running with
TEMO extension
• Used simple (no argument) version of RTI tick() method
57860 Performance: Stewart11: Performance improvements in HLA to
prevent entity timeout during join
• Multi-threading of RTIMessageManager tested in BDE lab the week
before Ft. Stewart.
• Fix to not timeout entities when TranslationManager queue has
reached a certain threshold.
• Put heartbeat events straight onto the RTI Queue instead of passing
through the TranslationManager queue.
September 2012
OneSAF Technical Exchange
22

Ft. Stewart “Expected” Scenario
‭
‭
The “30k + 10k” Configuration Viewed from the OneSAF (30k) Federate
After HLA Join
Ghost Entities (From 10k Federate) Are Associated with
“EXTERNAL…” Sides
September 2012
OneSAF Technical Exchange
23

Ft. Stewart Configuration
‭
‭
Machine Types
8400 Client: HP 8400 Dual-Core with 4 GB RAM
• Typical (JCATS) Client Machine at NSC
8400 Server: HP 8400 Dual Dual-Core with 8 GB RAM
• Typical (JCATS) Server Machine at NSC
Monster: High-Powered Machines Borrowed from OneSAF Lab
• Not on Critical Path, Excluded from Memory and CPU Reports
Allocations
Battlemaster: One (1) 8400 Server as Primary, One (1) Monster as
Backup
Simcore: Twenty-Eight (28) 8400 Client; Two (2) 8400 Client Held in
Reserve
HLA Interop Adapter: One (1) 8400 Server
C2 Adapter: One (1) 8400 Client
MCT: Twenty-Eight (28) 8400 Clients, Connected via Pub-Sub
September 2012
OneSAF Technical Exchange
24

Ft. Stewart Representative Exercise
‭ Representative Exercise Carried Out Morning of Wednesday, 6/22/2011
‭
‭
‭
Operators Were Active for 3.5 Hours of Run Time
All Machines Were Within Acceptable Limits for Memory and CPU
Load Throughout the Exercise
Memory Measurements Were Taken at the Following Points:
07:00 : Initialized
07:17 : Joined
07:19 : Running
08:49 : Elapsed 1.5 Hours Sim Time
10:54 : Elapsed 3.5 Hours Sim Time
September 2012
OneSAF Technical Exchange
25

4
3.5
3
CPU
Load Average
2.5
2
1.5
1
battlemaster
simcore
interop_ict
c2Core
0.5
Java Heap
Usage (MB)
0
1400
1300
1200
1100
1000
900
800
700
600
500
400
Joined and Running
Running 3.5 Hours
6:43 7:12 7:40 8:09 8:38 9:07 9:36 10:04 10:33 11:02
battlemaster
simcore
interop
c2 adapter
Initialized
Running 1.5 Hours
September 2012
OneSAF Technical Exchange
26

Back Up Slides