Artificial Intelligence and Soft Computing: Behavioral ... - Arteimi.info

Sensitivity Analysis: ‘Sensitivity’ in electrical engineering means change
in response of a system, when there is even a small change in its input
excitation. Thus an electrical system M1 is more sensitive than the system M2,
if M1 when compared to M2 responds to smaller changes in the input.
Sensitivity in expert systems, however, has a wider meaning. It corresponds to
a change in inferences for a change in input variables or parameters of the
expert system. Fig. 21.5 describes the scheme for validating an expert system
performance through sensitivity analysis.
Let the value of the input variables supplied by the problem be v1 = v 1,
v2 = v2 and v3 = v3 and the internal parameters (like certainty factors,
conditional probabilities, etc.) of the expert system be p1 = p1, p2 = p2, p3 =p3.
Suppose, the inferences derived by the expert system match the inferences
drawn by the expert for the above settings of input variables and parameters.
The performance evaluator now adjusts each variable/parameter one by one
by a small amount over their current value and observes the change in
response of the system and the expert. If the responses are closer (w.r.t. some
evaluator), then the design of the expert system is satisfactory; otherwise the
system has to be updated with new knowledge or its inference engine has to
be re-modeled. Sensitivity analysis, though a most powerful qualitative
method for performance evaluation, has not unfortunately been exploited to
any commercial expert systems. We hope that in the coming decade, it will
play a significant role in performance evaluation of commercial systems.
21.2.2 Quantitative Methods for Performance
Evaluation
When the responses of an expert system can be quantified (numerically), we
may employ quantitative (statistical) tools for its performance evaluation.
Generally, for quantitative evaluation of performance, a confidence interval
for one or more measure is considered, and the performance of the expert
system w.r.t. the expert is ascertained for a given level of confidence (95% or
99%). If the confidence interval obtained, for an expert system for a given
level of confidence, is satisfactory the system requires no adjustment;
otherwise we need to retune the knowledge base and the inference engine of
the system. An alternative way of formulation is to define a hypothesis H0 and
test whether we would accept or reject the hypothesis w.r.t. a predefined
performance range. The hypothesis H0 may be of the following form.
H0: The expert system is valid for the acceptable performance
range under the prescribed input domain.
Among the quantitative methods, the most common are paired t-test [4] and
Hotelling’s one sample T 2 test.