Effectiveness of Stemming and n-grams String Similarity Matching ...

More documents

Recommendations

Info

Issue 3, Volume 5, 2011 209be computed by using Dice Coefficient or the Overlap Coefficient asshown in Table-1 below.Table-1: Similarity Measure of Bigram and Trigram for wordsconstruct and destruct.Unique N-gram ofWord 1Unique N-gram ofWord 2A = Unique N-gramof Word 1B = Unique N-gramof Word 2C = Shared UniqueBigramDice Coefficient=(2C)/(A+B)Overlap Coefficient= C/min(A,B)Note : * denotes a spaceExperiments on N-gram MatchingBigram Trigram*c co on ns st tr **c *co con ons nstru uc ct t* str tru ruc uct ct* t***d de es st tr tu **d *de des est struc ct t* tru ruc uct ct* t**10 119 106 60.631579 0.5714290.666667 0.600000Again there are much research on n-gram string-similarity measuresare done on English. Adamson and Boreham [14] used intersimilarity coefficient to cluster a small group of mathematical titleson the basis of their constituent digrams and form the basis for adocument retrieval system; but the same procedure gave poor resultsusing Cransfield test collection [15]. Using the same collectionLenon et al [4] represented all unique terms occuring in thedocument titles and queries by lists of constituent digrams andtrigrams. Index terms with a similarity coefficient greater than somethreshold were considered to be variants of the query term and areused for searching relevant documents.Freund and Willett [2] performed online query expansion usinginverted file structure constituted of digrams on Evans and Vasmanitest collections. They found that using the digram inverted fileretrieved more non-related terms at lower similarity thresholdcompared to terms retrieved using the trigram inverted file. Theythen used arbitrary truncation of the query terms and retrieved higherproportion of related words and still maintain an acceptable level ofprecision.Robertson and Willett [16] used n-gram matching, both digram andtrigram, phonetic and non-phonetic coding, and dynamicprogramming methods to identify words in the Historical TextDatabasea using query words which are modern English. Theyconcluded that digram string-similarity is the appropriate method tobe implemented in an operational environment where a largedictionary was to be serached.Other sudies on different languages include Turkish and Malay.Ekmekcioglu et al.[17] had performed n-gram string-similaritymatching experiments similar to that of Lenon et al [4] on sixTurkish databases and found that trigrams performed slightly betterthan bigrams in most of the text corpora. They performedexperiments by using stemmed queries and nonstemmed dictionaryand both stemmed queries and dictionary. Results show that thestemmed versions performed significantly better than thenonstemmed.Sembok et al. [18] performed experiment using n-gram stringsimilaritymeasure, digram and trigram on Malay dictionary andqueries and found that overlap coefficient results are better than dicecoefficients and that digram perform significantly better than thetrigrams. They too found that stemming both queries and thedictionary performs significantly better than just stemming only thequeries. However, no conclusion is made between the conventionalstemmed-Boolean approach and incorporating-stemming in n-gramsapproach since both approaches are based on different paradigms.Other applications using n-gram include work done byYannakoudakis and Angelidakis [19]. They used n-grams differentlythat is to show distribution of n-grams in the Shorter Oxford EnglishDictionary for values of n from 2 to 5, from bigram to pentragram.They shown that the corresponding redundancy of n-grams increasesfrom 0.1067 to 0.3409.Experimental DetailObjectivesThe objectives of the experiments are to investigate the inherentcharacteristics of n-gram in Malay text and to evaluate in terms ofretrieval effectiveness of n-gram matching techniques, bigram andtrigram, applied to Malay text.Experimental SetupThe Malay test collection used in the experiments is the collection ofAhmad et al. [18] which is based on Malay translation of the Quran.There are 6236 documents and 36 natural language queries.Two types of index term dictionary (hereafter abbreviated toITD) are created; index nonstemmed term dictionary (hereafterabbreviated to INTD) and index stemmed term dictionary (hereafterabbreviated to ISTD). Stop words are removed before the terms areindexed. From INTD, the number of unique index nonstemmed termsis 5525, and from ISTD, the number of unique index stemmed termsis 2101.Characteristics of N-gram on Malay TextThere are 728 distinct bigrams and 18980 distinct trigrams bothinclude leading and trailing space(s). The sizes in bytes for each ofthe bigram and trigram files are shown in Table 2:Table 2. File sizes of Bigrams and TrigramsBigramTrigramINTD 517358 bytes 858580 bytesISTD 145278 bytes 444340 bytesBigram and trigram created from INTD herafter are known asnonstemmed- bigram and nonstemmed-trigram. And those createdfrom ISTD are known as stemmed-bigram and stemmed-trigram.BigramINTERNATIONAL JOURNAL OF APPLIED MATHEMATICS AND INFORMATICS
Issue 3, Volume 5, 2011 210The theoritical maximum number of distinct bigrams that can befound is 26x26=676 (without spaces) and 27x27=729 (with spaces),less 1 bigram of the form (space,space) which gives a total of 728. Asin English these theoretical maxima can never be reached in practisebecause certain bigrams such as QQ and XY [19] simply do notoccur in Malay text too. In this Malay text the maximum number ofnon-zero nonstemmed-bigrams is 377, only 52%, and fromstemmed-bigrams 355, only 49%. Table-3 contains the top 100bigrams for both nonstemmed and stemmed bigrams. In order to get ageneral idea about the whole range of both bigrams, their rankfrequencydistribution and zipfian distribution were plotted (Figures1 and 2).TrigramsThe maximum possible number of distinct trigrams is26x26x26=17576 (without spaces) and 27x27x27=19683 (withspaces), less 702 bigrams with a space in between, less 1 trigram ofthe form (space,space,space) which gave a total of 19890.Appropriate storage arrays were used to hold both nonstemmed andstemmed trigrams as indicated in Table-2 above. The total maximumnumber of non-zero nonstemmed-trigrams is 2214 which utilisedonly 12% and for stemmed-trigrams is 1892 which utilised only 10%.Thus during run time the size of array allocation can greatly bereduced.The most 50 frequently occuring trigrams are listed in Table-4 above.The rank-frequency distribution and zipfian distribution for bothtrigrams are presented in Figures 3 and 4 above.Zipfian DistributionThe curves demonstrating the zipfian distributions in Figures1,2,3and 4, of the n-gram for Malay text have a similar hyperbolic curvecomplying to the Zipf's law [20] which states that the frequency ofwords and the rank order is approximately constant. Such analysisshould not be restricted to just words [1]. Luhn [21] used Zipf's lawto specify upper and a lower cut-offs. The words below the upper cutoffwere considered to be common and those below the lower cut-offto be rare, leaving the rest to be significant words. These cut-offswere established by trial and error by estimating in both directionfrom the peak of the rank-order position. Thus by removing the mostfrequently n-grams should not change the result of retrievaleffectiveness of the documents, as carried out later.Experimental Evaluation ProcedureThe experiments performed involved the ranking and the calculationof string similarity measures of each unique terms in the ITD to aspecified query term. This procedure is the same as automatic queryexpansion approach as set by Lennon et al [4]. Following are theevaluation procedures that are carried out.Table-3. The Most Frequently Occurring Bigrams (with spaces)Nonstemmed StemmedRank Bigra Frequency Bigra Frequency1 m an 2567 m an 4012 ka 1472 a 3413 n 1391 ng 3304 a 1356 s 2425 ng 1136 g 2096 er 1111 b 2087 ya 1104 t 2078 m 1067 h 2049 ny 1062 k 20010 me 917 i 19811 en 909 ah 19512 ah 868 ra 19113 la 826 la 19014 d 731 ka 18815 di 707 n 18416 h 674 at 17517 b 632 ar 17418 k 594 t 17419 pe 581 r 17120 p 578 m 16721 ak 563 ta 16622 be 549 p 16023 em 530 er 15824 at 522 ak 15625 ta 509 ba 15226 ra 498 ma 15027 in 471 k 14428 ar 462 sa 13929 u 461 in 13530 ga 442 u 13031 ba 441 al 12932 i 430 am 12833 ke 428 en 12434 sa 419 un 12135 t 417 as 12036 s 414 pa 11937 nn 395 l 11738 ku 391 da 11539 ha 385 ha 11340 ma 382 a 10941 mu 358 ur 10642 al 349 ga 10343 un 349 s 9944 am 341 ya 9745 pa 335 ri 9546 ik 320 na 9447 se 316 d 9348 tu 310 se 8949 na 308 el 8850 as 299 h 87INTERNATIONAL JOURNAL OF APPLIED MATHEMATICS AND INFORMATICS
Page 1: Issue 3, Volume 5, 2011 208Effectiv
Page 6 and 7: Issue 3, Volume 5, 2011 213stemming
Page 8: Issue 3, Volume 5, 2011 215APPENDIX

Effectiveness of Stemming and n-grams String Similarity Matching ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?