PhD thesis - School of Informatics - University of Edinburgh

Chapter 2. Background and Theory 44
expression matching and a combination thereof to automatically extract anglicisms in
Norwegian text. The test set, a random sub-set of 10,000 tokens from a neologism
archive (Wangensteen, 2002), was manually annotated by the author for anglicisms.
For this binary classification, anglicisms were defined as either English words or com-
pounds containing at least one element of English origin. Based on this annotation, the
test data contained 563 tokens classified as anglicisms.
Using lexicon lookup only, Andersen determines that exact matching against a lex-
icon undergenerates in detecting anglicisms, resulting in low recall (6.75%). Con-
versely, fuzzy matching overgenerates, resulting in low precision (8.39%). The char-
acter n-gram matching is based on a chargram list of 1,074 items constisting of 4-6
characters which frequently occur in the British National Corpus (BNC). Being typ-
ical English letter sequences, any word in the test set containing such a chargram is
classified as English. This method leads to a higher precision of 74.73% but a rela-
tively low recall of 36.23%. Finally, regular expression matching based on English
orthographic patterns results in a precision of 60.6% and a recall of 39.0%.
On the 10,000 word test set of the neologism archive (Wangensteen, 2002), the
best method of combining character n-gram and regular expression matching yields an
accuracy of 96.32%. Simply assuming that the data does not contain any anglicisms
yields an accuracy of 94.47%. Andersen’s reported accuracy score is therefore mis-
leadingly high. In fact, the best F-score, which is calculated based on the number of
recognised and target anglicisms only, amounts to only 59.4 (P = 75.8%, R = 48.8%).
However, this result is unsurprisingly low as no differentiation is made between full-
word anglicisms and tokens with mixed-lingual morphemes in the gold standard.
A shortcoming of Andersen’s work, and other reviewed studies, is that the meth-
ods are not evaluated on unseen test data. The knowledge of previous evaluations
could have affected the design of later algorithms. This could easily be tested on an-
other set of data that was not used during the development stage. It would also be
interesting to investigate how the methods devised by Andersen perform on running
text instead of a collection of neologisms extracted from text. While Andersen’s work
is already applied in a language identification module as part of a classification tool
for neologisms, language identification on running text could exploit knowledge of
the surrounding text. Applied in such a way, anglicism detection would also allow
lexicographers to examine the use of borrowings in context.