Full proceedings volume - Australasian Language Technology ...

them to Aim, Method, Results and Conclusions
in this order. Using sentence ordering labels for
unstructured abstracts is the main difference
compared to earlier methods (Kim et al., 2011;
Verbeke et al., 2012).
We tried 6 combinations of features which will
be discussed in Results section.
Kim et al. Verbeke et al. Our System
Class S U S U S U
Background 81.84 68.46 86.19 76.90 95.55 95.02
Intervention 20.25 12.68 26.05 16.14 23.71 50.79
Outcome 92.32 72.94 92.99 77.69 95.24 99.04
Population 56.25 39.8 35.62 21.58 42.11 60.36
Study Design 43.95 4.40 45.5 6.67 0.67 3.57
Other 69.98 24.28 87.98 24.42 83.71 91.76
Overall 80.9 66.9 84.29 67.14 81.7 89.2
Table 3: F-score per class for structured (S) and unstructured
(U) abstracts (bold states improvement over
other systems)
4.2 Algorithm
Our sentence classifier uses CRF learning algorithm
4 . We have also executed few experiments
using SVM and observed CRF performed better
over this dataset with our choice of features. Due
to space constraints in this paper we are not comparing
CRF versus SVM results.
For feature selection, we used Fselector 5 package
from R-system 6 . From the pool of features,
we select the ”meaningful” features based on the
selecting criteria. We have tested several criteria
including (1) information gain (2) oneR (3) chisquare
test (4) spearman test. Among them, information
gain outperformed the others. We select
the 700 best features from our pool of features
based on information gain score.
Other technique we used for this shared task
is ”bootstrapping”. Our system performed very
well on training data but did not perform well on
test data, perhaps it suffered over-fitting. To overcome
this, we ran our current best model on test
data (without using gold-standard labels) and then
merge the result with train data to get the new
train. In that way, under ROC evaluation, we improved
our final scores by 3%. In addition, we
also pre-process the data. Since the heading such
as ”AIM,OUTCOME,INTRODUCTION etc.” are
always classified as ”other” in train data, when we
4 We used open-source CRF++ tool.
http://crfpp.googlecode.com
5 http://cran.r-project.org/web/packages/FSelector/index.html
6 http://www.r-project.org/
find sentence which has less than 20 characters
and all in upper case (our notion of heading), we
directly classify it as ”other” in test data.
5 Results
Features B I O P SD Oth All
BOW 9.1 3.2 68.8 2.9 0 31.7 38.4
+lexical 18.2 7.0 71.6 11.1 0 65.2 55.3
+struct 60.7 8.3 87.7 17.1 0.6 57.4 62.2
+ordering 93.7 23.7 96.6 41.0 1.3 80.9 80.8
All 95.2 23.7 95.2 42.1 0.6 83.7 81.7
All+seq 95.5 23.7 94.9 44.2 0.6 82.9 81.4
Table 4: Analysis of structured abstracts: microaverage
f-score, best performance per column is given in
bold
Features B I O P SD Oth All
BOW 13.0 0.7 79.1 1.8 0 14.3 38
+lexical 34.2 1.5 68.0 2.2 0 13.3 40.0
+struct 58.1 5.0 72.1 12.3 1.2 26.9 52.6
+ordering 93.7 40.2 99.2 52.4 1.2 96.6 88.0
All 95.0 50.7 99.0 60.3 3.5 91.7 89.2
All+seq 94.9 50.7 98.7 60.1 3.5 90.8 89.0
Table 5: Analysis of unstructured abstracts: microaverage
f-score, best performance per column is given in
bold
In this section we present the analysis of results
on structured and unstructured abstracts separately.
In all our experiments, we performed 10-
fold cross validation on the given dataset. Shared
task organisers have used Receiver operating
characteristic (ROC) to evaluate the scores. According
to ROC our best system scored 93.78%
(public board) and 92.16% (private board). However,
we compare our results with (Kim et al.,
2011) and (Verbeke et al., 2012) using the microaveraged
F-scores as in Table 3. Our system
outperformed previous works in unstructured
abstracts (22% higher than state-of-the-art).
Our system performed well in classifying Background,
Outcome and Other for both structured
and un-structured data. However, our system performed
poor in classifying study design as very
few instances of it is available in both test and
train.
We present the results of 6 systems learned
using different feature sets: Table 4 for structured
abstracts and Table 5 for unstructured abstracts.
We choose bag-of-words (BOW) as the
base features, +lexical includes BOW and lexical
features, +struct include BOW and structural
features, +ordering includes BOW and sentence
132