User's Guide to the Model Validation Kit - Harmonisation within ...

Note that a useful extension to FB and MG has been implemented in
the current BOOT package. FB is of limited value because
overpredictions and underpredictions compensate each other.
However, FB and MG can be separated into overpredicting and
underpredicting components. Thus, FB fn
(false negative) only
considers underpredictions, while FB fp
only considers
overpredictions.
Bootstrap resampling is used to estimate the confidence limits of a
performance measure – hence the name BOOT of the package.
Consult the BOOT User's Guide for a detailed discussion of the
interpretation of results from BOOT. The User’s Guide (Chang and
Hanna, 2005) is part of the Model Validation Kit.
(1): File with statistics
Proceed as follows in order to generate a file containing statistics for
data of quality 3 with the use of BOOT.
As noted above, we assume that the tools reside in C:\MvK\Tools.
BOOT can be launched in various ways. E.g., you can use Explorer to
show C:\MvK\Tools and then double-click on the icon for BOOT.
Alternatively (assuming some familiarity with DOS), BOOT can also
be launched by these steps:
• Open a Command Prompt window (explained in Section 6.2.5)
Table 12 Dialogue when using the BOOT programme.
Name of input file:
Run1\ki3.inp
Name of output file:
Run1\ki3.boo
Select one from the following options:
(1) straight Co and Cp comparison
(4) consider ln(Co) and ln(Cp)
1
Use ASTM procedure? (y/)
n
Print out original data? (y/)
n
Use E- or F-format for mean, sigma, and bias? (/e)
f
Calculate partial correlation? (y/)
That is, the influence from a certain model is removed.
n
Do the bootstrap resampling? (/n)
y
Print out detailed information on confidence limits? (y/)
n
Create files containing FB (with its 95% confidence limits) and NMSE that
can later be plotted? (/n)
n
Reading data
Calculating performance measures...
Start resampling procedure...
Computing c.l. of performance measures for each model...
Make another run? (y/)
N
43