Probabilistic Performance Analysis of Fault Diagnosis Schemes

lating between the achieved performance values. (By Fact 3.7, the points (0,0) and (1,1)
should also be included in this lower bound.) However, as mentioned above, the set V is
too abstract to make this approach practical. Therefore, in the next section, we consider an
extended definition of the roc that applies to more concrete sets of tests.
Extending the ROC to Specific Families of Tests
In Definition 3.9, the roc is defined with respect to the set of all tests, including randomized
tests. This definition allowed us to characterize the roc in terms of likelihood ratio tests, via
the Neyman–Pearson Lemma (Lemma 3.10), or in terms of Pareto optimality. In practice,
however, we want to be able to evaluate the performance of a given test or a given family of
tests. For example, consider the parameterized family of fault detection schemes
ˆ V = { V ε ∈ V : V ε = (F,δ ε ) and ε > 0 } , (3.18)
where the residual generator F is fixed and δ ε is a threshold function defined as
⎧
⎨0, if |r | < ε,
δ ε (r ) :=
⎩1, otherwise.
Clearly, V ε → V yes as ε → 0, regardless of the choice of F . Similarly, V ε → V no as ε → ∞.
Hence, the set of achievable performance points is a curve passing through (0,0) and (1,1)
(see Figure 3.4). Using randomization, as in Example 3.5, the tests in V ˆ can be used to
achieve any performance point between this curve and the diagonal (i.e., any point in the
convex hull of the curve). Hence, we have the following natural extension of the definition
of the roc.
W ˆ k ⊂ W k to be the set of performance
points that are achieved by some test in V ˆ . The upper boundary of the set Wˆ
k is
Definition 3.11. Let ˆ V ⊂ V be some subset of tests. Define
called the receiver operating characteristic (roc) for the class of tests ˆ V at time k.
3.5 Certifying and Visualizing Performance
3.5.1 Bounds on Performance Metrics
Given a fault detection scheme V , the system G θ is said to be available at time k if no fault
has occurred and no false alarm has been issued. Hence, the probability of availability is
given by the performance metric P tn,k . In a physical system affected by wear and deterioration,
Q 1,k → 1 as k → ∞, so P tn,k → 0 as k → ∞. Therefore, any bound on P tn,k can only
be enforced over a specified time window. Given N ∈ N and a > 0, one criterion for system
35