PhD thesis - School of Informatics - University of Edinburgh

Chapter 2. Background and Theory 33
by calculating the sum of all absolute rank distances for each n-gram ti. The language
l resulting in the smallest distance D signals the language of the test document.
D =
N
∑
i=1
|rank(ti,text) − rank(ti,l)| (2.1)
TextCat provides corpus profiles for 69 languages each containing 400 n-grams per
language. If an n-gram does not occur in the profile, it is given a maximum distance
score of 400. Cavnar and Trenkle (1994) report an accuracy of 99.8% for input strings
of at least 300 bytes and an accuracy of 98.3% for strings of less that 300 bytes using
their LID approach. No information is given regarding its performance for really small
strings. In the experiments presented in Sections 3.4.1 and 3.4.3.1, the collection of
corpus profiles was limited to the required languages, namely German and English.
Most automatic LID systems are successful in identifying the base language of a
document but are not designed to deal with mixed-lingual text to identify the origin
of individual foreign words and proper names within a given sentence. Initial work
on this issue has been carried out by Font-Llitjós and Black (2001) who use Cavnar’s
n-gram statistics approach to estimate the origin of unseen proper names as a means of
improving the pronunciation accuracy for TTS synthesis. While the LID performance
(for 26 languages) is not evaluated separately, the pronunciation accuracy of proper
names increases by 7.6% from the baseline of 54.08% when adding language origin
probabilities as a feature to the CART-based decision tree model. LID has also been
applied in the field of name transliteration. Qu and Grefenstette (2004) used LID as
a way of determining language-specific transliteration methods for Japanese, Chinese
and English named entities (NEs) written in Latin script. Their language identifier is
based on tri-gram frequencies, whereby the language of a word is that for which the
sum of the normalised tri-grams is highest. On the training data, LID yields accura-
cies of 92% for Japanese, 87% for Chinese and 70% for English names. A further
study is that of Lewis et al. (2004) who implemented a character n-gram classifier to
differentiate between the 10,000 most common English words, 3,143 unique transliter-
ated Arabic and 20,577 Russian names. Each of the three data sets is divided into 80%
training and 20% test data, a process which is repeated 4 times resulting in overlapping
training and test sets. Average precision values amount to 81.1% for Russian names,