Dictionary Alignment for Context-sensitive Word Glossing

More documents

Recommendations

Info

Figure 5: The output of word-translating Japanese definition sentences to EnglishWordNet sense pairing, we extract the hypernymsof the respective senses and expand the definitionsentences with the definition sentences from the hypernyms.The term vectors are then based on thisexpanded term set, similar to query expansion in informationretrieval.5 Experimental Setup5.1 Gold-standard dataTo evaluate the performance of our system, werandomly selected 100 words from Lexeed, extractedout the Lexeed–WordNet sense pairings asdescribed above, and manually selected the goldstandardalignments from amongst them. The 100words were associated with a total of 268 Lexeedsenses and 772 WordNet senses, creating a total of206,896 possible alignment pairs. Of these, 259alignments were selected as our gold-standard.We encountered a number of partial matches thatwere caused by the Japanese word being more specificthan its English counterparts (as identified byour WordNet matching method). For example,kakkazan is translated as “active volcano”. SinceWordNet does not have any entry for active volcano,the longest right word substring that matchesin WordNet is simply volcano. The definition sentencesreturned by Lexeed describe kakkazan as “avolcano which still can erupt” and “a volcano thatwill soon erupt”, while volcano is described as “afissure in the earth’s crust (or in the surface of someother planet) through which molten lava and gaseserupt” and “a mountain formed by volcanic material”.Although there is some similarity betweenthese definitions (namely key words such as eruptand volcano), we do not include this pairing in ourgold-standard alignment data.5.2 BaselineAs a baseline, we take the most-frequent sense ofeach of the 100 random words from Lexeed, andmatch it with the synset with the highest SemCorfrequency count out of all the candidate synsets.5.3 ThresholdingAll our calculations are based on cosine similarity,which returns a similarity between 0 and 1, with 1being an exact match. In its simplest form, we wouldidentify the unique WordNet sense with highest similarityto each Lexeed sense, irrespective of the magnitudeof the similarity. This has the dual disadvantageof allowing only one WordNet sense for eachLexeed sense, and potentially forcing alignments tobe made on low similarity values. A more reasonableapproach is to apply a threshold x, and treatall WordNet senses with similarity greater than x asbeing aligned with the Lexeed sense. Thresholdingalso gives us more flexibility in terms of tuning theperformance of our method: at higher threshold values,we can hope to increase precision at the expenseof recall, and at lower threshold values, we can hopeto increase recall at the expense of precision.5.4 Evaluation metricsTo evaluate the performance of our system, we useprecision, recall and F-score. In an alignment context,precision is defined as the proportion of correctalignments to all alignments returned by the system,and recall is defined as the proportion of the correctalignments returned by our system to all the align-130
Basic Stopping POS filtering Lemmatisation Stemming NormalisationBasic model 0.228 0.334 0.256 0.240 0.243 0.221Basic+TF-IDF 0.292 0.335 0.295 0.344 0.330 0.288Table 1: Best system F-score of combination of all features using the basic model vs. the basic model withTF-IDF weightingments in our gold-standard. F-score is the harmonicmean of precision and recall, and provides a singlefigure-of-merit rating of the balance between thesetwo factors. We evaluate our system using unbiased(β =1) F-score.6 ResultsThroughout our experimentation, we evaluate relativeto the 100 manually sense-aligned Japanesewords.Our baseline method predicts 100 alignments (asit is guaranteed to produce a unique alignment persource-language word), of which 60 are correct.Hence, the precision is 60100=0.600, the recall is60259=0.231, and the F-score is 0.334.With the basic alignment model, the highest F-score achieved with thresholding is 0.228 at a thresholdvalue of 0.19, well below the baseline F-score.The recall and precision value at this threshold are0.263 and 0.202, respectively.The basic model with TF-IDF weighting performedconsiderably better, scoring the highest F-score of 0.292 (recall = 0.382 and precision = 0.236)at a threshold value of 0.04, but is still well belowthe baseline F-score. To confirm that TF-IDFterm weighting is always beneficial to overall alignmentperformance, we took the unweighted modelcombined with each of the proposed extensions, andcompared it with the same extension but with the inclusionof TF-IDF (without lexical relations at thispoint). The result of these experiments can be foundin Table 1. As we can see, TF-IDF weighting constantlyimproves alignment performance. Also notethat, with the exception of simple (punctuation) normalisation,all extensions improve over the basicmodel both with and without TF-IDF weighting.We extended our experiments by considering allpossible combinations of 2 or more proposed extensions(excluding lexical relations for the time being)with TF-IDF weighting. The purpose of this ex-Method Precision Recall F-scoreBaseline 0.600 0.231 0.334WS+PF+L+N 0.298 0.494 0.372WS+PF+L 0.326 0.428 0.370WS+L 0.305 0.455 0.365WS+PF+L+S+N 0.275 0.540 0.364WS+L+S 0.301 0.455 0.363Table 2: Top-5 combinations of extensions, excludinglexical relations (WS = Word stopping, PF =POS filtering, L = Lemmatisation, S = Stemming,N = Normalisation)periment is to investigate whether the proposed extensionsare complementary in improving alignmentperformance. The 5 top-performing combinationsare presented in Table 2.The best result is achieved by combining all theproposed extensions, at an F-score of 0.364, whichis significantly above baseline. It is also interestingto see that not all methods are fully complementary.By excluding stemming, e.g., the system actuallyperforms better, producing a higher F-score of0.372.We then experimented with the addition of lexicalrelations to the different combinations of extensionsexplored above. The 5 top-performing combinationsare presented in Table 3. The best F-score of0.408 is achieved with the combination of all the extensionsproposed. When lexical relations are usedexclusively or combined with less than three of theproposed extensions, the performance tends to decline.In our best performing combination, we outperformedthe baseline F-score by 22%. 349 alignmentswere returned for this F-score, of which 124matched the gold-standard. The precision and recallscores are 0.355 and 0.478, respectively.We carried out more detailed analysis of theprecision–recall trade-off. While we expect the pre-131
Page 1: Proceedings of the Australasian Lan
Page 4 and 5: Figure 3: Example of normalisation
Page 8 and 9: Method Precision Recall F-scoreBase

Dictionary Alignment for Context-sensitive Word Glossing

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?