Principles of Modern Radar - Volume 2 1891121537

14.3 Metrics and Performance Prediction 637Again, the less the probability density functions (pdfs) overlap, the smaller the probabilityof error.14.3.2.3 Comparison of Algorithms Via the Wald Sequential TestThe Wald test can be used to compare any two systems that produce binary responses,rendering it suitable to compare ATR systems in a pair-wise fashion [3]. By theoreticaldesign, this test requires fewer samples than a confidence interval or ranking and selectionapproach to choose the superior system [3]. Furthermore, unlike other tests, it remainsvalid even when independent sampling from two processes cannot be performed [3].The basic Wald test can be performed using the ratio of successes to failures foreach process, or the efficiency (k) of the process, risk tolerances (α and β), and countingvariables (t) [3]. The ratio of efficiencies between the first and second process is definedas u; u 0 and u 1 are set such that errors yielding values of u ≤ u 0 or u ≥ u 1 are consideredsignificant [3]. The counting variable, t, is equal to t 1 plus t 2 , where t 1 is the number oftimes an image is classified correctly by the first ATR process but not the second and t 2 isthe corollary for the second ATR process [3]. Lower and upper bounds are defined byandβloglog 1 + u 1lower bound:1 − α1 + u+ t0(14.9)log u 1 − log u 0 log u 1 − log u 0log 1 − β log 1 + u 1upper bound: α1 + u+ t0(14.10)log u 1 − log u 0 log u 1 − log u 0If t 2 is below the lower bound for any value of t, then process 1 is better than 2 [3].Similarly, if t 2 is above the upper bound, then process 2 is better [3]. If t 2 is between thebounds, testing continues [3].14.3.3 Using Performance Prediction Tools in the Design ProcessPerformance prediction techniques have great utility in the design phase of an ATR scheme.To demonstrate this, consider the following list of questions likely to be considered whendesigning an ATR scheme.• Have enough features been selected in the feature set?• Is another feature going to improve overall results enough to justify the cost ofcollecting it?• Can likely ATR errors be predicted to inform design of subsequent processing steps,hence improving overall system robustness?• How much will variations in target articulation affect the robustness of the ATR answer?• What are the performance ramifications if the features are corrupted to varying degrees?• How long must the features be collected to achieve the desired level of reliability?While high-fidelity models could be developed and exercised to address these questions,extensive high-fidelity modeling is not likely to be computationally efficient orcost effective. Performance prediction techniques, in contrast, can be leveraged to addressthese questions in a computationally efficient, cost effective manner. For example, to