Design and development of a concept-based multi ... - Citeseer

Shiyan Ou, Christopher S.G. Khoo and Dion H. Goh
Table 6
Precision and recall for the system-extracted contextual relations, research methods
and relationships in the 50 structured abstracts
Information piece Precision Recall
Relationships 81.02% 54.86%
Contextual relations 85.71% 90.00%
Research methods 97.20% 71.65%
From the two human codings, a ‘gold standard’ was constructed by taking the agreements in the codings.
Differences in the codings were resolved through discussion. The average precision and recall
for the system-extracted contextual relations, research methods and relationships in the 50 structured
abstracts are given in Table 6.
The system obtained a high precision of 97.2% for extracting research methods and a little lower
precision of 85.7% for extracting contextual relations. This indicates that it is effective to use cue
phrases to identify these two kinds of information. However, the recall of 90.0% for extracting contextual
relations is much higher than that of 71.7% for extracting research methods. This is because
research methods can be expressed in various ways. Thus the list of cue phrases for research methods
used in the summarization system was not complete since it was only derived from the 300 sample
abstracts. Moreover, the research methods expressed in other grammatical forms, such as verb,
adverb, and the whole sentence, cannot be identified by the system. In contrast, contextual relations
are very specific information. It is easy to derive the cue words for contextual relations exhaustively
from the 300 sample abstracts.
The system obtained a high precision of 81.0% for extracting relationships between research concepts,
but a low recall of 54.9%. The list of relationship patterns derived from the 300 sample
abstracts appears to be incomplete. Moreover, the system can only identify relationships that are
located within sentences and with clear cue phrases. Cross-sentence relationships and implied relationships
that do not contain clear cue phrases and need inferring cannot be identified with the current
pattern matching method.
4.3. Evaluation of information integration
For evaluating the quality of clusters, two types of measures – internal quality measures and external
quality measures – have been used [33]. Internal measures calculate the internal quality of a set
of clusters without reference to external knowledge, e.g. overall internal similarity based on the pairwise
similarity between members within each cluster. External measures compare how closely a set
of clusters matches a set of known reference clusters. In this study, we adopted an external measure –
F-measure from the field of information retrieval – to calculate the similarity between the set of
system-generated clusters and the reference clusters. Two sets of human codings were each used as
reference clusters.
In the evaluation, 15 research topics in the sociology domain were haphazardly selected. For each
topic, a set of dissertation abstracts was retrieved from the database using the topic as search query.
But only five abstracts were selected from the retrieved abstracts to form a document set. In addition,
for five of the topics (i.e. document set 11–15), an additional five abstracts were selected for
each of them and combined with the previously chosen five abstracts to form a second bigger document
set. Thus 20 document sets in total were used in the evaluation. The bigger document sets
were used to examine the difference in concept clustering between small sets (five documents) and
bigger sets (10 documents). For each abstract, the important concepts were automatically extracted
by the system from the research objectives and research results sections of the abstract.
Human coders were asked to identify similar concepts across abstracts from the list of concepts
extracted from each document set and group them into clusters. Each cluster had to contain two
concepts at least and was assigned a label by the coders. Thus some of the concepts in the concept
Journal of Information Science, XX (X) 2007, pp. 1–19 © CILIP, DOI: 10.1177/0165551507084630 14