pdf download - Software and Computer Technology - TU Delft

3.6 Evaluation of Approaches to Fault Diagnosis Automated Fault Diagnosis
this evaluation. The technique that is referred to as ’current approach’ is a combination of the approaches
PMS-1, PMS-2, PMS-3 and PMS-4 that are characterized by Table 3.1. The values of
all scores range from very bad (−−) to very good (++). The assessment of all scores is based on
own opinion, experience and interviews with experts. The remainder of this section motivates the
assessment of Table 3.4.
The estimation I made is that the current approach achieves better accuracy than an approach
that uses automated induction on black box information, but worse than the three other automated
approaches. This is because the other these three automated approaches use information of experts,
while the current approach uses information of service engineers, and ’Automated Induction (BB)’
only uses black box information. The speed of diagnosis is higher for an automated approach
than for manual approaches. MBD is faster than the other automated approaches. This is in the
domain of milliseconds, thus only essential when produced diagnoses are used for automated system
recovery. Like accuracy, the diagnostic resolution is best when experts use their knowledge about
the system. Either by applying the inference off-line (called ’Off-line inference’ in Table 3.4), or
by online inference of an automated entity that uses a model made by experts (MBD). Intuitively,
inference of white box information scores better on diagnostic resolution than inference of black
box information. The current approach is estimated having better diagnostic resolution as inference
on black box information.
The context independency (denoted by ’independence’ in Table 3.4) of techniques is worse for
both the current approach and the inference of black box information. As said before, the current
approach depends on the skills and experience of currently employed people. ’Automated induction
(BB)’ requires much data. This data must be produced by systems that have the same characteristics
as systems that are diagnosed by the using the inductively constructed LUT. Furthermore,
a manually inferred LUT of ’Off-line inference’ scores less on context independency than ’Automated
induction (WB)’ and MBD. This is because these consistency-based models contain more
knowledge. This is useful in case employees change jobs.
There has been no research of the development costs of the various diagnostic techniques. For
PMS, the technique ’Automated induction (BB)’ is cheapest, because the remote monitoring tools
already provide most of the needed functionality. The runtime costs of the approaches has not been
investigated as well. The only estimation made is that automated approaches require less runtime
costs, because the labor costs are likely to be less.
The approaches that use a consistency-based model score highest on adaptability. This is because
changes in the design do not require time consuming and error-prone inferencing of a new
LUT (’Off-line inference’ and ’Current approach’). The inductive process of ’Automated induction
(BB)’ and ’Current approach’ require that systems, that implemented the new design, already
produced sufficient data. The explanation facility is best for the ’Off-line inference’, ’Automated
Induction (WB)’ and MBD approaches. In these approaches, the diagnoses can be explained in
terms of the consistency-based model, while the current approach and ’Automated induction (BB)’
do not have such a model. MBD scores higher than ’Off-line inference’ and ’Automated induction
(BB)’, because diagnoses made by ’Off-line inference’ can only be explained by the experts that implemented
it, and ’Automated induction (WB)’ uses a consistency-based model that only specifies
structure.
As a consequence of the evaluation given by Table 3.4, MBD is chosen as a proposed approach
to fault diagnosis. It is likely to improve the fault diagnosis process at PMS, because it has equal
score or higher score than the current approach. Also, it scores best of all automated approaches.
The following chapter introduces model-based fault diagnosis.
30