W10-09

moreappropriately viewedasthepurpose oftraveling).
The other effect clauses also appear to be
incorrect. Thisshouldnotcomeasmuchofasurprisebecausetherankingwasgeneratedsoleyfrom
thematchscorebetweentheinputdescription and
thecausesin Rd,whicharequiterelevant.
One potential problem with the naïve selection
method is that it ignores information contained in
therankedlist R ′ d ofclausesthatareassociatedwith
the clauses in Rd. In our experiments, we often
observedredundancies in R ′ d thatcapturedgeneral
propertiesofthedesiredinference. Intuitively,contentthatissharedacrosselementsof
R ′ d couldrepresentthecoremeaningofthedesiredinferenceresult.
In what follows, we describe various re-rankings
of R ′ d using this shared content. For each model
described, thefinalinferencepredictionisthetoprankedelementof
R ′ d .
Centroidsimilarity To approximate the shared
contentofdiscourseunitsin R ′ d ,wetreateach
discourseunitasavectorofTFscores.Wethen
computetheaveragevectorandre-rankalldiscourseunitsin
R ′ d basedontheircosinesimilaritywiththeaveragevector.Thisfavorsinferenceresultsthat“agree”withmanyalternative
hypotheses.
Descriptionscorescaling Inthisapproach,weincorporatethescorefrom
Rd intothecentroid
similarityscore,multiplyingthetwoandgiving
equalweighttoeach. Thiscaptures theintuitionthatthetop-rankedelementof
R ′ d should
represent the general content of the list but
shouldalsobelinkedtoanelementof Rdthat
bearshighsimilaritytothegivenstateorevent
description d.
Log-lengthscaling When working with the centroid
similarity score, weoften observed topranked
elements of R ′ d that were only a few
words in length. This was typically the case
when components from sparse TF vectors in
R ′ d matched well with components from the
centroid vector. Ideally, wewould like more
lengthy (but not too long) descriptions. To
achievethis, wemultiplied thecentroid similarityscorebythelogarithmofthewordlength
ofthediscourseunitin R ′ d .
47
Descriptionscore/log-lengthscaling In this approach,wecombinethedescriptionscorescalingandlog-lengthscaling,multiplyingthecentroidsimilaritybybothandgivingequalweight
toallthreefactors.
4.2 Evaluatingthegeneratedtextualinferences
To evaluate the inference re-ranking models described
above, we automatically generated forward/backward
causal andtemporal inferences for
five documents (265 sentences) drawn randomly
from the story corpus. For simplicity, we generated
an inference for each sentence in each document.
Each inference re-ranking model is able to
generatefourtextualinferences(forward/backward
causal/temporal) foreachsentence. Inourexperiments,weonlykeptthehighest-scoringofthefour
inferencesgeneratedbyamodel.Oneoftheauthors
thenmanuallyevaluatedthefinalpredictionsforcorrectness.
Thiswasasubjectiveprocess,butitwas
guidedbythefollowingrequirements:
1. Thegeneratedinferencemustincreasethelocalcoherenceofthedocument.
Asdescribed
byGraesseretal.(1994),readersaretypically
requiredtomakeinferencesaboutthetextthat
leadtoacoherent understanding thereof. We
requiredthegeneratedinferencestoaidinthis
task.
2. The generated inferences must be globally
valid. Todemonstrateglobalvalidity,consider
thefollowingactualoutput:
(4) Ididn’tevenneedajacket(untilIgot
there).
In Example 4, the system-generated forward
temporal inference is shown in parentheses.
Theinferencemakessensegivenitslocalcontext;
however, it is clear from the surroundingdiscourse(notshown)thatajacketwasnot
neededatanypointintime(ithappenedtobe
awarmday). Asaresult,thispredictionwas
taggedasincorrect.
Table 2 presents the results of the evaluation. As
showninthetable, thetop-performing modelsare
thosethatcombinecentroid similarity withoneor
bothoftheotherre-rankingheuristics.