PhD thesis - School of Informatics - University of Edinburgh

More documents

Recommendations

Info

Chapter 2. Background and Theory 31 Excuse Me... Kyaa Re Excuse me...What’s it? Meraa Dil Tere Pe Fidaa Re... I am smitten by you Bus Stop Pe Dekhaa Tujhe Pehli Baar I saw you first at the bus stop JhaTke MeN Ho Gayaa Tere Se Pyaar I fell in love as the bus lurched Excuse Me... aaN Bolnaa Excuse me...yes? MaiN Pehle Se Shaadi Shudaa Re... I am already married. Abhi To HooN Saalaa Roadpati I am the master of street Ladki ChaahuuN KaroRpati... I want a billionaire girl Race Course MeN Dekha Tujhe Pehli Baar I saw you at the Race Course Counter Pe Ho Gayaa Tere Se Pyaar I fell in love at the counter Excuse Me...Kyaa Re Excuse me...What’s it? Police MeN Hai Meraa MiyaaN Re My husband is in the police ...Excuse Me...Yes Please Excuse me...Yes please Ban Jaa Mera Bhaiyaa Re Better become my brother Figure 2.5: Hinglish song lyrics, example taken from Kachru (2006) of a bouncer outside a club when deciding who can enter or not (Dunn, forthcoming). It can be concluded that English is influencing many languages. The anglicisa- tion of those languages is a complex process with various reasons and motivations. Foreign words or proper names pose substantial difficulties to NLP applications, not only because they are hard to process but also because, theoretically, they are infinite in number. Moreover, it is impossible to predict which foreign words will enter a language, let alone to create an exhaustive gazetteer of them. Therefore, the increasing use of English forms in different languages presents a challenge to NLP systems that are conventionally designed to handle monolingual text. The task of recognising English inclusions is the main focus of the work presented in this thesis. Before introducing an automatic classifier able to detect English language portions in otherwise monolingual text (Chapters 3 and 4), this chapter will review previous work on automatic, specifically mixed-lingual, language identification (Section 2.2).
Chapter 2. Background and Theory 32 2.2 Automatic Language Identification Written language identification (LID) is the task of automatically determining the language of an electronic document. This is an important issue in many areas of multilingual information processing such as indexing or text classification. In our increasingly multilingual society, automatic LID is vital to enable accurate processing of natural language in the case where a document’s language is not apparent for example from its metadata, or from background information. Whilst first research initiatives in written LID began 30 years ago, the 1990s saw increasing efforts in this field as the first multilingual corpora were published. Today, a variety of off-the-shelf LID systems are available. Presented with several lines of text, they will attempt to identify its language. This chapter will first give a brief overview of conventional approaches to LID and then report specifically on LID systems designed to deal with mixed-lingual text. The majority of existing state-of-the-art LID systems rely on word-level information such as diacritics and special characters (Newman, 1987), common short words (Johnson, 1993), character-level language models (Dunning, 1994) or methods based on character-based n-gram frequencies (Beesley, 1988; Cavnar and Trenkle, 1994; Damashek, 1995; Ahmed et al., 2004). Other automatic LID programs function by determining the character encoding of a document (Kikui, 1996). Comparisons of different techniques (Grefenstette, 1995; Capstik et al., 1999; Padró and Padró, 2004; Kranig, 2005) demonstrate that it is difficult to determine the one best LID method. 11 Their results largely depend on the type and number of languages involved, the amount of training data and the number of input words. This means that LID accuracy increases with the length of the test data and is not satisfactory for individual words. Cavnar and Trenkle (1994)’s approach of using character n-gram frequency lists to determine the language of a new piece of text is the underlying algorithm of TextCat, an automatic LID system developed by Gertjan van Noord. 12 . It is used for comparison in some of the experiments described in Chapter 3. It creates n-gram frequency lists (1 to 5-grams) for various language corpora and the text to be identified (n-gram profiles) which are sorted by frequency counts in ascending order. The n-gram profile of the text is then compared to each of the language corpus profiles in terms of n-gram ranks 11 For a summary of different LID methods see also Cole et al. (1997). 12 http://www.let.rug.nl/˜vannoord/TextCat
Page 1 and 2: Automatic Detection of English Incl
Page 3 and 4: these parsers with the annotation-f
Page 5 and 6: Declaration I declare that this the
Page 7 and 8: 3.3.5 Post-processing Module . . .
Page 9 and 10: A.2.2 Kappa Coefficient . . . . . .
Page 11 and 12: 5.6 Average relative token frequenc
Page 13 and 14: 3.16 Most frequent English inclusio
Page 15 and 16: Chapter 1. Introduction 2 siderable
Page 17 and 18: Chapter 1. Introduction 4 Chapter 3
Page 19 and 20: Chapter 1. Introduction 6 1.1 Relat
Page 21 and 22: Chapter 2. Background and Theory 8
Page 43: Chapter 2. Background and Theory 30
Page 59 and 60: Chapter 3 Tracking English Inclusio
Page 61 and 62: Chapter 3. Tracking English Inclusi
Page 95 and 96:
Chapter 3. Tracking English Inclusi
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Chapter 4 System Extension to a New
Page 115 and 116:
Chapter 4. System Extension to a Ne
Page 117 and 118:
Page 119 and 120:
Page 121 and 122:
Page 123 and 124:
Page 125 and 126:
Page 127 and 128:
Page 129 and 130:
Chapter 5 Parsing English Inclusion
Page 131 and 132:
Chapter 5. Parsing English Inclusio
Page 133 and 134:
Page 135 and 136:
Page 137 and 138:
Page 139 and 140:
Page 141 and 142:
Page 143 and 144:
Page 145 and 146:
Page 147 and 148:
Page 149 and 150:
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
Page 157 and 158:
Page 159 and 160:
Chapter 6 Other Potential Applicati
Page 161 and 162:
Chapter 6. Other Potential Applicat
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185 and 186:
Page 187 and 188:
Chapter 7 Conclusions and Future Wo
Page 189 and 190:
Chapter 7. Conclusions and Future W
Page 191 and 192:
Appendix A. Evaluation Metrics and
Page 193 and 194:
Page 195 and 196:
Page 197 and 198:
Page 199 and 200:
Appendix B. Guidelines for Annotati
Page 201 and 202:
Page 203 and 204:
Page 205 and 206:
Appendix C TIGER Tags and Labels C.
Page 207 and 208:
Appendix C. TIGER Tags and Labels 1
Page 209 and 210:
Appendix C. TIGER Tags and Labels 1
Page 211 and 212:
Bibliography 198 Andersen, G. (2005
Page 213 and 214:
Bibliography 200 Bresnan, J. (2001)
Page 215 and 216:
Bibliography 202 Damashek, M. (1995
Page 217 and 218:
Bibliography 204 Finkel, J., Dingar
Page 219 and 220:
Bibliography 206 Hachey, B., Alex,
Page 221 and 222:
Bibliography 208 Kirkness, A. (1984
Page 223 and 224:
Bibliography 210 and Technology (In
Page 225 and 226:
Bibliography 212 Poplack, S. (1988)
Page 227 and 228:
Bibliography 214 Sokol, D. K. (2000
Page 229:
Bibliography 216 Yang, W. (1990). A
show all

PhD thesis - School of Informatics - University of Edinburgh

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?