A Wordnet from the Ground Up

90 Chapter 3. Discovering Semantic Relatednessdrawback of ROCK is that it sometimes produces a very deep and unbalanced hierarchy.On the other hand, GHSOM assigned pairs of documents into one cluster which didnot appear together in any manually created category more often then ROCK.We wanted to label with representative words document groups clustered in a hierarchicaltree. Words describing groups of documents closer to the root of the treeshould be more general than words used for the documents in the leaves. Ideally, wewould obtain a basic hypernymy structure for plWordNet (or at least instances of is-arelation) out of the assigned labels.Keyword extraction can be supervised or unsupervised. Supervised algorithm requiresample manually constructed resources. We applied only such unsupervisedmethods that try to capture statistical properties of words occurrences to identifywords which best describe the given document. The statistics can be counted locally,using data from a single document only, or estimated from a large body of text.To benefit from both local and global strategies, we extended the method proposed byIndyka-Piasecka (2004) with the algorithm of Matsuo and Ishizuka (2004) into a hybridkeyword extraction method.Indyka-Piasecka (2004) assigns a weight w to every lemma l that occurs in eachdocument of a group. Additionally, lemmas are filtered on the basis of their documentfrequency df l , that is a number of documents in which lemma l occurred. Both rare andfrequent lemmas are not good discriminator of document content (cf. Indyka-Piasecka,2004). The weight w is calculated by using two weighting schemes:and cue validitytf.idf l,d = tf l,d × log N df l(3.6)cv = tf group(3.7)tfwhere tf and df denote term frequency and document frequency.Matsuo and Ishizuka (2004) used a three–step–process to assign a weight to everylemma. First, all words in a document are reduced to their lemmas (basic morphologicalforms) and filtered on the basis of term frequency and a stoplist. Then, theycluster lemmas in a document using two algorithms. If two lemmas have similar distributions,it means that they belong to the same group. As a measure of the probabilitydistribution similarity between two lemmas l 1 and l 2 (Matsuo and Ishizuka, 2004)used the Jensen–Shannon divergence 22 . Lemmas are also clustered when they followsimilar co-occurrence pattern with other lemmas. This can be measured using MutualInformation.22 Jensen–Shannon divergence is a symmetrised and smoothed version of Kullback–Leibler divergence(Manning and Schütze, 2001).