DARPA ULTRALOG Final Report - Industrial and Manufacturing ...

A DI value greater than one implies that the particular
parameter can help in discriminating between the situations
(more discrimination power). We calculated DI values for
38 behavioral parameters and selected those parameters with
DI values larger than one. This resulted in 10 parameters,
comprising of eight statistical and two deterministic
parameters, as shown in Table III. ‘# of events’ from ‘Task
Arrival’ and ‘Time to Solution’ was the most discriminating
behavioral parameter. Note that ‘# of events’ is the same for
both time series’ as arrived tasks are processed.
TABLE III
DISCRIMINABILITY INDEX OF BEHAVIORAL PARAMETERS
Rank DI Time Series Parameter
1 2477.5 Task Arrival/Time to Solution # of events
2 5.7 Time to Solution(G) Variance
3 5.1 Time to Solution(G) Radius
4 4.4 Time to Solution(G) Average
5 4.2 Time to Solution(G) Maximum
6 2.9 Queue length # of events
7 2.8 Queue length Maximum
8 2.2 Queue length AMI
9 1.2 Queue length Average
10 1.1 Time to Solution(C) L_Expo
(G): Generation, (C): Completion
V. SITUATION IDENTIFICATION
Results from preliminary experimentation shows that 10
of the 38 behavioral parameters have better discriminating
power in the stress space. By using them as features we
identify the stress situations using k-nearest neighbor
classification algorithm.
A. k-nearest neighbor algorithm
k-nearest neighbor algorithm, one of the instance-based
learning methods, is conceptually straightforward. The
summary reported here is based on [14]. In this algorithm
learning is simply storing the training instances, in which
each instance corresponds to a point in the n-dimensional
feature space. Given a new query instance k nearest
neighbors are retrieved from memory and used to classify
the new query instance. The nearest neighbors of an instance
are defined in terms of the standard Euclidean distance. One
problem in this algorithm is the sensitivity to noise axes in
high dimensional problems. One possible solution would be
to normalize each feature. However, normalization does not
resolve this problem because Euclidean distance can become
very noisy for high dimensional problems where only a few
of the features carry classification information. The solution
to this problem is to modify the Euclidean metric by a set of
weights that represents the information content or goodness
of each feature. Therefore, given a set of weights w the
distance between two normalized instances x i and x j with m
features can be calculated as in (2).
m
∑
d( x , x ) = w[
k](
x [ k]
− x [ k])
(2)
i
j
k = 1
B. Empirical results
i
We performed 200 experiments with the same
experimental configuration as in Fig. 2 to construct the
database of training instances. Each training instance is
represented with the 10 behavioral parameters. In this
experimentation each agent’s OpTempo (Low/High) and the
number of agent 1 (0/1/2) are randomly chosen. Given a
new instance we select 20 nearest neighbors (10% of the
population of training instances) from the database and
estimate the stress situations of the agents in the society by
using those neighbors. To address the effectiveness of DI we
use 12 different sets of weights to calculate the distance. In
the first 10 sets only one parameter is considered with other
parameters’ weights equal to zero. We weight the
parameters equally in the 11th set and proportionally to DI
in the 12th set.
We estimated the stress situations for 100 new instances
using those 12 different sets of weights. To eliminate the
noise from those agents that has no significant effect on the
behavioral parameters, we removed the agents from our
analysis that we cannot identify more than 2/3 by using any
weight set. Through this procedure only 8 agents are
selected. The results of correct estimation from different
weight sets are shown in Fig. 4.
Correctness (%)
75
70
65
60
55
50
1 2 3 4 5 6 7 8 9 10
On the whole, performance of behavioral parameters to
identify the stress situations is quite correlated with DI of
the parameters. As a parameter’s DI ranked high its
estimation accuracy is also high. And, when weighted
equally the performance is in the middle. But, when we
weight proportionally to DI the performance becomes the
highest. This result demonstrates the effectiveness of DI to
get the goodness of the behavioral parameters for situation
identification. Fig. 5 shows the performance for each agent
when we weight proportionally to DI. As agents located
farther from the TAO the performance becomes degraded.
j
DI Rank
2
Weighted w ith DI
Weighted equally
Fig. 4. Correct estimation using different weight sets