Evaluation of Automatic Text Summarization - KTH

16 CHAPTER 1. SUMMARIES AND THE PROCESS OF SUMMARIZATION
texts collected were news flashes and press releases from large Swedish newspapers
like Svenska Dagbladet, Dagens Nyheter and Aftonbladet, business and computer
magazines as well as press releases from humanitary organizations like Amnesty
International, RFSL 16 and authorities like Riksdagen. 17
The KTH News Corpus has since been used to train a Named Entity tagger
(Dalianis and Åström 2001), that also is used in the SweSum text summarizer (see
Paper 4). The KTH News Corpus has also been used to evaluate the robustness
of the SweSum summarizer by summarizing Swedish news text collected by the
Business Intelligence tool NyhetsGuiden, “NewsGuide” (see Hassel 2001b), that
actually is an interface to the KTH News Corpus tool.
Furthermore, the corpus is currently used in experiments concerning the training
of a Random Indexer (see Karlgren and Sahlgren 2001, Sahlgren 2001) for experiments
where “synonym sets”, or rather semantic sets, are used to augment the
frequency count. Gong and Liu (2001) have carried out similar experiments, but
using LSA for creating their sets, where they used the semantic sets to pin point
the topically central sentences, using each set only once in an attempt to avoid
redundancy. LSA, however, is a costly method for building these types of sets, and
if RI performs equally or better there would be much benefit.
1.4.2 Paper 2.
Improving Precision in Information Retrieval for Swedish using Stemming
(Carlberger, Dalianis, Hassel, and Knutsson, 2001)
An early stage of the KTH News Corpus 18 , as from October 2000, was used to evaluate
a stemmer for Swedish and its inpact on the performance of a search engine,
SiteSeeker 19 . The work was completely manual in the sense that three persons first
had to each annotate ≈33 20 randomly selected texts from the KTH News Corpus
each, and for each text construct a relevant/central question with corresponding
answer, in total 100 texts and 100 answers. The next step was to use the search engine
formulating queries to retrieve information, i.e to find answers to our allotted
questions, with and without stemming in hope that one would find answers to the
questions. Here the questions where swapped one step so that no one formulated
queries concerning their own questions. The last step was to assess the answers of
the partner, again a swap was made, to find out the precision and recall. This time
consuming and manual evaluation schema showed us that precision and relative
recall improved with 15 respectively 18 percent for Swedish in information retrieval
using stemming.
16Riksförbundet För Sexuellt Likaberättigande; A gay, lesbian, bisexual and transgendered
lobby organization.
17The Swedish Parliament.
18The corpus at this stage contained about 54,000 texts.
19http://www.euroling.se/siteseeker/ 20One person actually had to do 34 questions/queries/evaluations at each corresponding stage,
but this “extra load” was shifted with each stage.