W10-09

Recommendations

Info

strengthsandweaknessesofthesysteminaneffort toguidefuturework. Westructureourpresentationasfollows: inSection2,wepresentpreviousresearchthathasinvestigatedtheuseoflargewebcorporafornaturallanguageprocessing(NLP)tasks.InSection3,wedescribeanefficientmethodofautomaticallyparsing weblogstoriesfordiscoursestructure. InSection4, wepresentasetofinferencemechanismsthatuse theextracted discourse relations togenerate opendomaintextualinferences. Weconclude,inSection 5,withinsightsintostory-based envisionment that wehopewillguidefutureworkinthisarea. 2 Relatedwork Researchers have made many attempts to use the massive amount of linguistic content created by usersoftheWorldWideWeb. Progressandchallengesinthisareahavespawnedmultipleworkshops (e.g.,thosedescribedbyGurevychandZesch(2009) andEvertetal.(2008)) thatspecifically target the useofcontentthatiscollaborativelycreatedbyInternetusers. Ofparticularrelevancetothepresent workistheweblogcorpusdevelopedbyBurtonet al. (2009), which was used for the data challenge portionoftheInternationalConferenceonWeblogs andSocialMedia(ICWSM).TheICWSMweblog corpus(referredtohereasSpinn3r)isfreelyavailableandcomprisestensofmillionsofweblogentriespostedbetweenAugust1st,2008andOctober 1st,2008. Gordon et al. (2009) describe an approach to knowledgeextractionovertheSpinn3rcorpususing techniquesdescribedbySchubertandTong(2003). Inthisapproach,logicalpropositions(knownasfactoids) are constructed via approximate interpretationofsyntacticanalyses.Asanexample,thesystemidentifiedafactoidglossedas“doorstoaroom maybeopened”. Gordonetal.(2009) found that theextractedfactoidscoverroughlyhalfofthefactoidspresentinthecorresponding Wikipedia 2 articles. We used a subset of the Spinn3r corpus in ourwork,butfocusedondiscourseanalysesofentiretextsinsteadofsyntacticanalysesofsinglesentences. Ourgoalwastoextractgeneralcausaland temporal propositions instead of the fine-grained 2 http://en.wikipedia.org 44 propertiesexpressedbymanyfactoidsextractedby Gordonetal.(2009). Clark and Harrison (2009) pursued large-scale extraction ofknowledge fromtextusing asyntaxbasedapproachthatwasalsoinspiredbythework ofSchubertandTong(2003). Theauthorsshowed how the extracted knowledge tuples can be used toimprovesyntacticparsingandtextualentailment recognition. Bar-Haimetal.(2009)presentanefficient method of performing inference with such knowledge. Ourworkisalsorelated totheworkofPersing and Ng (2009), in which the authors developed a semi-supervisedmethodofidentifyingthecausesof events described in aviation safety reports. Similarly, our system extracts causal (as well as temporal)knowledge; however,itdoesthisinanopen domainanddoesnotplacelimitationsonthetypes of causes to be identified. This greatly increases thecomplexityoftheinferencetask,andourresults exhibit acorresponding degradation; however, our evaluationsprovideimportantinsightsintothetask. 3 Discourseparsingacorpusofstories Gordon and Swanson (2009) developed a supervised classification-based approach for identifying personal stories within the Spinn3r corpus. Their methodachieved75%precisiononthebinarytask of predicting story versus non-story on a held-out subsetoftheSpinn3rcorpus. Theextracted“story corpus”comprises960,098personalstorieswritten by weblog users. Due to its large size and broad domaincoverage,thestorycorpusoffersuniqueopportunitiestoNLPresearchers.Forexample,SwansonandGordon(2008)showedhowthecorpuscan beusedtosupportopen-domaincollaborativestory writing. 3 As described by Gordon and Swanson (2008), storyidentificationisjustthefirststeptowardscommonsensereasoningusingpersonalstories.Weaddressed the second step - knowledge extraction - byparsingthecorpususingaRhetorical Structure Theory(CarlsonandMarcu,2001)parserbasedon the one described by Sagae (2009). The parser performsjointsyntactic anddiscourse dependency 3 The system (called SayAnything) is available at http://sayanything.ict.usc.edu
parsingusingastack-based, shift-reducealgorithm withruntimethatislinearintheinputlength. This lightweight approach is very efficient; however, it maynotbequiteasaccurateasmorecomplex,chartbased approaches (e.g., the approach of Charniak andJohnson(2005)forsyntacticparsing). We trained the discourse parser over the causal andtemporalrelationscontainedintheRSTcorpus. Examplesoftheserelationsareshownbelow: (1) [causePackagesoftengetburiedintheload] [resultandaredeliveredlate.] (2) [beforeThreemonthsaftershearrivedinL.A.] [aftershespent$120shedidn’thave.] The RST corpus defines many fine-grained relations that capture causal and temporal properties. Forexample, thecorpusdifferentiates betweenresultandreasonforcausationandtemporal-afterand temporal-beforefortemporalorder. Inordertoincreasetheamountofavailabletrainingdata,wecollapsed all causal and temporal relations into two generalrelationscausesandprecedes. Thissteprequired normalization ofasymmetric relations such astemporal-beforeandtemporal-after. Toevaluatethediscourseparserdescribedabove, wemanuallyannotated100randomlyselectedweblogstoriesfromthestorycorpusproducedbyGordonandSwanson(2009). Forincreasedefficiency, welimitedourannotationtothegeneralizedcauses and precedes relations described above. We attempted to keep our definitions of these relations inlinewiththoseusedbyRST.Followingprevious discourseannotationefforts,weannotatedrelations over clause-level discourse units, permitting relationsbetweenadjacentsentences. Intotal, weannotated770instancesofcausesand1,009instances ofprecedes. Weexperimented withtwoversions oftheRST parser, one trained on the fine-grained RST relationsandtheothertrainedonthecollapsedrelations.Attestingtime,weautomaticallymappedthefinegrained relations to their corresponding causes or precedesrelation. Wecomputedthefollowingaccuracystatistics: Discoursesegmentationaccuracy For each predicteddiscourseunit,welocatedthereference 45 discourseunitwiththehighestoverlap. Accuracyforthepredicteddiscourseunitisequaltothepercentagewordoverlapbetweenthereferenceandpredicteddiscourseunits. Argumentidentificationaccuracy For each discourse unit of a predicted discourse relation, welocatedthereferencediscourseunitwiththe highestoverlap. Accuracyisequaltothepercentageoftimesthatareferencediscourserelation(ofanytype)holdsbetweenthereferencediscourseunitsthatoverlapmostwiththepredicteddiscourseunits. Argumentclassificationaccuracy For the subset ofinstancesinwhichareferencediscourserelationholdsbetweentheunitsthatoverlapmost withthepredicteddiscourseunits,accuracyis equaltothepercentage oftimesthatthepredicteddiscourserelationmatchesthereference discourserelation. Completeaccuracy For each predicted discourse relation, accuracy is equal to the percentage wordoverlap withareference discourse relationofthesametype. Table1showsthe accuracy results for thefinegrainedandcollapsedversionsoftheRSTdiscourse parser. AsshowninTable1,thecollapsedversion of the discourse parser exhibits higher overall accuracy. Bothparsers predicted thecauses relation muchmoreoftenthantheprecedesrelation,sothe overallscoresarebiased towardthescores forthe causesrelation.Forcomparison,Sagae(2009)evaluatedasimilarRSTparseroverthetestsectionof theRSTcorpus, obtaining precision of42.9%and recallof46.2%(F1 = 44.5%). Inadditiontotheautomaticevaluationdescribed above,wealsomanuallyassessedtheoutputofthe discourse parsers. One of the authors judged the correctnessofeachextracteddiscourserelation,and we found that the fine-grained and collapsed versions of the parser performed equally well with a precisionnear33%;however,throughoutourexperiments,weobservedmoredesirablediscoursesegmentationwhenworkingwiththecollapsedversion ofthediscourseparser.Thisfact,combinedwiththe resultsoftheautomaticevaluationpresentedabove,
Page 1 and 2:
NAACL HLT 2010 First International
Page 3: Introduction It has been a long ter
Page 7: Table of Contents Machine Reading a
Page 10 and 11: Sunday, June 6, 2010 (continued) Se
Page 12 and 13: "coherent", based on criteria such
Page 14 and 15: (SRL). While there are a number of
Page 16 and 17: epeat select a clause chain Cu of
Page 18 and 19: even if those words reflect somethi
Page 20 and 21: Building an end-to-end text reading
Page 22 and 23: found to be wrong or correct (by su
Page 24 and 25: Text Queue Parser 4 Summary Text Mi
Page 26 and 27: formalized procedure to attach elem
Page 28 and 29: Steve_Walsh:throw:pass Steve_Walsh:
Page 30 and 31: NVNPN 2 'person':'intercept':'pass'
Page 32 and 33: 6 Related Work To build the knowled
Page 34 and 35: Large Scale Relation Detection ∗
Page 36 and 37: 1000 900 800 700 600 500 400 300 20
Page 38 and 39: to run against a web-scale corpus a
Page 40 and 41: Relation Prec Rec F1 Tuples Seeds i
Page 42 and 43: of the pattern space for any given
Page 44 and 45: Mining Script-Like Structures from
Page 46 and 47: e1 [ enter, nsubj, {customer, John}
Page 48 and 49: To identify such pairs, the topic s
Page 50 and 51: ≈ 57% 2. More words (bold) were j
Page 52 and 53: References Sergey Brin and Lawrence
Page 56 and 57: Fine-grainedRSTparser CollapsedRSTp
Page 58 and 59: Inference accuracy 0.3 0.25 0.2 0.1
Page 60 and 61: wehaveintroducedinferenceprocedures
Page 62 and 63: Semantic Role Labeling for Open Inf
Page 64 and 65: inference. Pruning involves using a
Page 66 and 67: TEXTRUNNER SRL-IE P R F1 P R F1 Bin
Page 68 and 69: that maximizes information gain div
Page 70 and 71: References Eugene Agichtein and Lui
Page 72 and 73: 1. Patterns based on words vs. pred
Page 74 and 75: state of the art information extrac
Page 76 and 77: ence of an attack. For instance, th
Page 78 and 79: manually annotated examples, which
Page 80 and 81: Towards Learning Rules from Natural
Page 82 and 83: tions of the learner suit the obser
Page 84 and 85: Accuracy Accuracy (Aggressive−Nov
Page 86 and 87: Accuracy Accuracy 100 95 90 85 80 7
Page 88 and 89: Unsupervised techniques for discove
Page 90 and 91: to categorize them. The article on
Page 92 and 93: ized label) or the most specialized
Page 94 and 95: Similarity Function k (L=2) F (L=2)
Page 96 and 97: References Sören Auer, Christian B
Page 98 and 99: a wide spectrum of solutions to the
Page 100 and 101: tion from the Web corpus at large s
Page 102 and 103: TextRunner, Kylin, KOG, WOE, WPE).
Page 104 and 105:
Doug Downey, Matthew Broadhead, and
Page 106 and 107:
Analogical Dialogue Acts: Supportin
Page 108 and 109:
Heat flows from one place to anothe
Page 110 and 111:
Source Text Translation* QRG-CE Tex
Page 112 and 113:
1987). We simplified the syntax of
Page 114 and 115:
Gentner, D. (1983). Structure-Mappi
Page 116 and 117:
sources can help in tasks like name
Page 118 and 119:
werset in the previous step and bas
Page 120 and 121:
we were able to construct a list of
Page 122 and 123:
found at threshold of 2. There were
Page 124 and 125:
Supporting rule-based representatio
Page 126 and 127:
the domain of the spatial argument
Page 128 and 129:
the time of the movement. This link
Page 130 and 131:
don’t seem to be plausible candid
Page 132 and 133:
PRISMATIC: Inducing Knowledge from
Page 134 and 135:
Figure 1: System Overview by a suit
Page 136 and 137:
ferent dimensions. Continuing with
Page 139:
Author Index Barbella, David, 96 Ba
show all

W10-09

Create successful ePaper yourself

Delete template?

Save as template?