PhD thesis - School of Informatics - University of Edinburgh

Chapter 3. Tracking English Inclusions in German 97
F-score
100
90
80
70
60
50
40
30
Statistical tagger
English inclusion classifier
10000 20000 30000 40000 50000 60000 70000 80000
Amount of training data (in tokens)
Figure 3.4: Learning curve of a supervised ML classifier versus the performance of the
annotation-free English inclusion classifier.
sion classifier does not rely on annotated data, it can be tested and evaluated once for
this entire corpus. It yields an overall F-score of 85.43 (see Figure 3.4).
In order to determine the machine learner’s performance over the entire data set,
and at the same time investigate the effect of the quantity of annotated training data
available, a 10-fold cross-validation test was conducted whereby increasingly larger
sub-parts of training data are provided when testing on each held out fold. At first, the
pooled data is randomised and split into a 90% large training and 10% large test set.
This randomisation and split is done on the document level, i.e the training set contains
131 newspaper articles and the test set 14. The training sub-sets are also increased on
the document level by batches of 6 newspaper articles at each step. The increasingly
larger sub-sets of the training data are then used to train the classifier and subsequently
evaluate it on the test set. This procedure is then repeated for each of the 10 held out
folds and scores are averaged. Each point in the resulting learning curve presented in
Figure 3.4 shows the average F-score of the ML classifier when trained on the selected
sub-set of articles and evaluated on the held out set. Average F-scores are plotted