PhD thesis - School of Informatics - University of Edinburgh

Appendix A
Evaluation Metrics and Notation
This appendix explains the evaluation metrics and statistical significance tests used for
the experiments presented in Chapters 3, 4 and 5 and explains how they are calculated.
It also specifies various notations in order to avoid confusion.
A.1 System Evaluation Metrics
In the broad sense, English inclusion detection can be regarded as an information ex-
traction task, where the aim is to identify all English inclusions occurring in text that
is written primarily in a different language. The English inclusion classifier’s per-
formance is evaluated intrinsically on seen and unseen evaluation data against gold
standard annotation. The evaluation measures used for this intrinsic evaluation are ac-
curacy and F-score which are calculated using the conlleval script written by Erik
Tjong Kim Sang. 1 The identification of English inclusions is therefore evaluated in a
similar way to named entity recognition (NER), but for single tokens. A useful way
to illustrate how accuracy and F-score are computed is via a contingency table of the
gold standard annotation and the system output (see Table A.1). The positive and
negative annotations of the gold standard are compared against those produced by the
system. The positive and negative labels which are correctly predicted by the system
with respect to the gold standard are called true positives (TP) and true negatives
(TN), respectively. A wrongly predicted positive label is called false positive (FP) and
1 This script is freely available at: http://www.cnts.ua.ac.be/conll2000/chunking/
conlleval.txt
177