Full proceedings volume - Australasian Language Technology ...

More documents

Recommendations

Info

Krippendorff’s Alpha () value achieving the least value of 0.76 and a maximum value of 0.907, respectively. As seen in Table 3, the number of sentences annotated by annotators varied from a minimum of 44 to a maximum of 347. This is due to the number of articles annotated by individual annotators. The annotators in our study were requested to annotate any number of articles depending on their availability. While some chose to annotate a single article, three of the annotators (Annotators 1, 2 and 3 – shown in grey in Table 3) annotated three articles. The value for these annotators was of the order 0.85 to 0.90. This shows that the increase in annotated sentences resulted in better agreement indicating the ease of applying the labels to sentences by these annotators. The agreement achieved for each article between three of the annotators is tabulated in Table 4. 7 Conclusion We presented in this paper an annotation scheme of context types for scientific articles, considering the persuasive characteristic of citations. We described the application of the Toulmin model for developing an argumentation framework for scientific articles, which was used for defining our context types. We discussed the results of the inter-rater reliability study carried out for establishing the reliability of our scheme. As we mentioned in Section 1, studies have successfully used this annotation scheme for developing tools that provide intelligent citation context based information services, indicating the usefulness of the annotation scheme. Our future work involves examining the application of annotation schemes across other disciplines. We also intend to focus on using our context types for analyzing sentiments associated with citation contexts. Article A B C D E Article 3 82.83 82.17 90.09 76.23 0.75 Article 2 84.73 86.74 90.36 77.10 0.77 Article 6 89.09 85.45 97.27 54.54 0.78 Article 7 90.47 95.23 90.47 58.71 0.82 Article 8 87.87 88.63 93.18 81.81 0.83 Article 4 90.21 85.32 91.74 93.57 0.84 Article 9 89.43 97.80 95.12 85.36 0.85 Article 5 93.93 91.81 85.45 94.54 0.87 Article 1 95.02 96.31 96.31 92.63 0.91 A – Average Pairwise percent agreement; B – Agreement between Annotator A and Annotator 2; C – Agreement between Annotator A and Annotator 1; D – Agreement between Annotator 1 and Annotator 2; E - Krippendorff’s Alpha () Table 4: Agreement for Articles Classification Results Confusion Matrix Label* P R F BGR CWIG CWO IRCW CWOG DES GAPS ISSUE FW RWCW UOCW TOTAL BGR 0.50 1.00 0.66 BGR 3 0 0 0 0 0 0 0 0 0 0 3 CWIG 1.00 1.00 1.00 CWIG 0 4 0 0 0 0 0 0 0 0 0 4 CWO 0.87 1.00 0.93 CWO 0 0 14 0 0 0 0 0 0 0 0 14 IRCW 0.92 1.00 0.96 IRCW 0 0 0 52 0 0 0 0 0 0 0 52 CWOG 1.00 0.66 0.80 CWOG 0 0 0 4 8 0 0 0 0 0 0 12 DES 0.85 0.87 0.86 DES 2 0 1 0 0 126 1 14 0 0 0 144 GAPS 0.95 0.87 0.91 GAPS 0 0 0 0 0 3 21 0 0 0 0 24 ISSUE 0.39 0.31 0.34 ISSUE 1 0 1 0 0 18 0 9 0 0 0 29 FW 1.00 1.00 1.00 FW 0 0 0 0 0 0 0 0 3 0 0 3 RWCW 1.00 1.00 1.00 RWCW 0 0 0 0 0 0 0 0 0 10 0 10 UOCW 1.00 1.00 1.00 UOCW 0 0 0 0 0 0 0 0 0 0 5 5 6 4 16 56 8 147 22 23 3 10 5 300 * Labels CCW and SCCW are not shown in the table as none of the sentences were labeled with these labels. Captions: P – Precision; R – Recall; F – F-Score Table 1: Results of Pilot Study for each Label 12
References Angrosh, M A, Cranefield, S., and Stanger, N. (2012a). Contextual Information Retrieval in Research Articles: Semantic Publishing Tools for the Research Community. Semantic Web Journal. Accepted for publication. Angrosh, M.A., Cranefield, S., and Stanger, N. (2012b). Context Identification of Sentences in Research Articles: Towards Developing Intelligent Tools for the Research Community. Natural Language Engineering. DOI: 10.017/S1351324912000277. Artstein, R., and Poesio, M. (2008). Survey Article Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4), 555-596. Baldi, S. (1998). Normative versus Social Constructivist Processes in the Allocation of Citations: A Network-Analytic Model. American Sociological Review, 63(6), 829. doi:10.2307/2657504 Bird, S., Loper, E. and Klein, E. (2009). Natural Language Processing with Python. O’Reilly Media Inc. Brooks, T. A. (1985). Private acts and public objects: An investigation of citer motivations. Journal of the American Society for Information Science, 36(4), 223-229. doi:10.1002/asi.4630360402 Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37(1), 34-36. DOI:10.1002/asi.4630370106 Chubin, D. E., and Moitra, S. D. (1975). Content Analysis of References: Adjunct or Alternative to Citation Counting? Social Studies of Science, 5(4), 423-441. Cozzens, S. E. (1989). What do citations count? The rhetoric-first model. Scientometrics, 15(5- 6), 437-447. DOI:10.1007/BF02017064 Frost, C. O. (1979). The Use of Citations in Literary Research: A Preliminary Classification of Citation Functions. The Library Quarterly, 49(4), 399-414. Gao, L., Tang, Z., and Lin, X. (2009). CEBBIP: A Parser of Bibliographic Information in Chinese Electronic Books. ACM/IEEE Joint Conference on Digital Libraries, JCDL 2009 (pp. 73-76). ACM Press. Garfield, E. (1964). Science Citation Index - A new dimension in indexing. Science, 144(3619), 649-654. Gilbert, G. N. (1977). Referencing as Persuasion. Social Studies of Science, 7(1), 113-122. Hyland, K. (2002). Directives: Argument and Engagement in Academic Writing. Applied Linguistics, 23(2), 215-239. DOI:10.1093/applin/23.2.215 Ibekwe-sanjuan, F., Chen, C., and Pinho, R. (2007). Identifying Strategic Information from Scientific Articles through Sentence Classification. Journal of Applied Linguistics, 1518-1522. Krippendorff, K. (2011). Computing Krippendorff’s Alpha Reliability. Annenberg School of Communication. Departmental Papers (ASC). University of Pennsylvania. http://www.asc.upenn.edu/usr/krippendorff/mw ebreliability4.pdf Langer, H., Lüngen, H., and Bayerl, P. S. (2004). Text type structure and logical document structure. Proceedings of the 2004 ACL Workshop on Discourse Annotation - DiscAnnotation’04 (pp. 49-56). Morristown, NJ, USA: Association for Computational Linguistics. DOI:10.3115/1608938.1608945 Le, M.H., Ho, T.B., and Nakamori, Y. (2006). Detecting Citation Types Using Finite-State. PAKDD 2006, Lecture Notes in Artificial Intelligence 3918 (pp. 265-274). Springer- Verlag, Berlin Heidelberg. Liakata, M. (2010). Zones of Conceptualisation in Scientific Papers: a Window to Negative and Speculative Statements. Proceedings of the Workshop on Negation and Speculation in Natural Language Processing (pp. 1-4). Lipetz, B.-A. (1965). Improvement of the selectivity of citation indexes to science literature through inclusion of citation relationship indicators. American Documentation, 16(2), 81-90. DOI:10.1002/asi.5090160207 Majeed, A., Razak, S., Abu-Ghazaleh, N. B., & Harras, Kh. A. (2009). TCP over Multi-Hop Wireless Networks: The Impact of MAC Level Interactions. ADHOC-NOW 2009, Lecture Notes in Computer Science 5793 (pp. 1-15). Springer-Verlag, Berlin Heidelberg. Mizuta, Y., and Collier, N. (2004). An Annotation Scheme for a Rhetorical Analysis of Biology Articles. Proceedings of the Fourth International Conference on Language Resource and Evaluation (LREC 2004) (pp. 1737-1740). 13
Page 1 and 2: Australasian Language Technology As
Page 3 and 4: ALTA 2012 Workshop Committees Works
Page 5 and 6: ALTA 2012 Programme The proceedings
Page 7 and 8: Contents Invited talks 1 Using a la
Page 9 and 10: Invited talks 1
Page 11 and 12: Diverse Words, Shared Meanings: Sta
Page 13 and 14: A Citation Centric Annotation Schem
Page 15 and 16: The structure of this paper is as f
Page 17 and 18: understanding of sentences surround
Page 19: henceforth referred to as Annotator
Page 23 and 24: Semantic Judgement of Medical Conce
Page 25 and 26: 2012). The TE model provides a form
Page 27 and 28: Correlation coefficient with human
Page 29 and 30: Pair # Concept 1 Doc. Freq. Concept
Page 31 and 32: Active Learning and the Irish Treeb
Page 33 and 34: 2.2 Sources of annotator disagreeme
Page 35 and 36: complement (csubj) 2 . See Figure 4
Page 37 and 38: top Y trees from this ordered set a
Page 39 and 40: References Ron Artstein and Massimo
Page 41 and 42: Unsupervised Estimation of Word Usa
Page 43 and 44: not lend itself to determining unsu
Page 45 and 46: T 0: think, want, thing, look, tell
Page 47 and 48: correlation is much stronger than t
Page 49 and 50: References Eneko Agirre and Philip
Page 51 and 52: eaders with sentences with a negati
Page 53 and 54: Question Which word is closest in m
Page 55 and 56: Acceptability and negativity: conce
Page 57 and 58: MORE NEGATIVE / LESS NEGATIVE MR Δ
Page 59 and 60: Julie Elizabeth Weeds. 2003. Measur
Page 61 and 62: cation produced by a supervised mac
Page 63 and 64: understood to carry a lower weight
Page 65 and 66: Figure 1: Average classification ac
Page 67 and 68: 0.8 0.75 0.7 AuthorA4 AuthorB4 Auth
Page 69 and 70: Segmentation and Translation of Jap
Page 71 and 72:
tomatosōsu “tomato sauce”, rev
Page 73 and 74:
“markup” and “Mach”, so the
Page 75 and 76:
MWE Segmentation Possible Translati
Page 77 and 78:
English Term Pairs from Search Engi
Page 79 and 80:
techniques used are, of necessity,
Page 81 and 82:
ation metrics robust, if they are v
Page 83 and 84:
Figure 2: Methodologies of human as
Page 85 and 86:
an optimal method? Machine translat
Page 87 and 88:
Towards Two-step Multi-document Sum
Page 89 and 90:
archical and non-hierarchical relat
Page 91 and 92:
We use the MetaMap 7 tool to automa
Page 93 and 94:
T T & C CC z -1.5 -1.27 -1.33 p-val
Page 95 and 96:
References Alan R. Aronson. 2001. E
Page 97 and 98:
techniques and results are shown. F
Page 99 and 100:
Rock Hip-Hop Electronic Punk Metal
Page 101 and 102:
Rank Frequency Frequency (filtered)
Page 103 and 104:
Ex. 1 Ex. 2 Ex. 3 Score Song Title
Page 105 and 106:
Free-text input vs menu selection:
Page 107 and 108:
consistent with the much earlier ed
Page 109 and 110:
3.2 Dialogue system architecture Th
Page 111 and 112:
Control Free-text Menu-based n=119
Page 113 and 114:
tions in analyzing algebra example
Page 115 and 116:
langid.py for better language model
Page 117 and 118:
documents of MIME type text/html wi
Page 119 and 120:
Acknowledgments NICTA is funded by
Page 121 and 122:
LaBB-CAT: an Annotation Store Rober
Page 123 and 124:
elationship between the coincidence
Page 125 and 126:
Communication 33 (special issue on
Page 127 and 128:
matically extracting location infor
Page 129 and 130:
Classifier DBp:1R Geo:1R D+G:1R DBp
Page 131 and 132:
ALTA Shared Task papers 123
Page 133 and 134:
All Struct. Unstruct. Total - Abstr
Page 135 and 136:
System Population Intervention Back
Page 137 and 138:
tence positions (absolute and relat
Page 139 and 140:
sitional attributes, and sequential
Page 141 and 142:
ordering labels, All includes BOW,
Page 143 and 144:
3 Software Used All experimentation
Page 145 and 146:
Combination Output Public Private C
Page 147 and 148:
Experiments with Clustering-based F
Page 149 and 150:
3 Results For the initial experimen
show all

Full proceedings volume - Australasian Language Technology ...

Create successful ePaper yourself

Delete template?

Save as template?