Automatic Mapping Clinical Notes to Medical - RMIT University

formation was determined using the Connexor
parser. This information was used to
identify the main utterance subject where dependency
information was not available.
• Morphology: Morphological tags, also generated
by Connexor, were used to distinguish
between first and third person pronouns, as
well as between singular and plural forms of
first person pronouns. Additionally, we used
morphological tags from Connexor to identify
imperative verbs.
• Hand-constructed word lists: Several of the
features used relate to closed sets of common
lexical items (e.g., verbs of permission,
interrogative words, variations of “yes” and
“no”). For these features, we employ handconstructed
simple lists, using online thesauri
to expand our lists from an initial set of seed
words. While some of the lists are not exhaustive,
they seem to help our results and
involved only a small amount of effort; none
took more than an hour to construct.
Feature Likely VRM
1st person singular subject D,Q
1st person plural singular
subject
D,C
3rd person subject E,Q
2nd person subject A,Q,I,R
Inverted subject-verb order Q
Imperative verb A
Verbs of permission, prohibition,
obligation
A
Interrogative words Q
Non-lexical content K
Yes/No variants K
Terms of evaluation I
Utterance length all
First word all
Last token all
Bi-grams all
Table 5: Features used in VRM Classifier
6 Results
Our classification results using several different
learning algorithms and variations in feature sets
are summarised in Table 6. We experimented
with using only the linguistic features suggested
37
by Stiles, using only the additional features we
identified, and using a combination of all features
shown in Table 5. All our results were validated
using stratified 10-fold cross validation.
We used supervised learning methods implemented
in Weka (Witten and Frank, 2005) to train
our classifier. Through experimentation, we found
that Weka’s Support Vector Machine implementation
(SMO) provided the best classification performance.
Encouragingly, other relatively simple
approaches, such as a Bayesian Network classifier
using the K2 hill-climbing search algorithm, also
performed reasonably well.
The baseline against which we compare our
classifier’s performance is a OneR (one rule) classifier
using an identical feature set. This baseline
system is a one-level decision tree, (i.e., based on
a set of rules that test only the single most discriminative
feature). As shown in Table 6, the accuracy
of this baseline varies from 42.76% to 49.27%, depending
on the exact features used. Regardless of
features or algorithms, our classifier performs significantly
better than the baseline system.
Mean
Algorithm Feature Set Accuracy Abs
Error
SVM All 79.75% 0.19
SVM Only Stiles’ 60.82% 0.20
SVM No Stiles’ 74.49% 0.19
Bayes Net All 78.51% 0.06
Bayes Net Only Stiles’ 60.16% 0.12
Bayes Net No Stiles’ 75.68% 0.07
Baseline All 49.27% 0.36
Baseline Only Stiles’ 49.27% 0.36
Baseline No Stiles’ 42.76% 0.38
Table 6: VRM classifier results
Another tunable parameter was the level of
pruning of n-grams from our feature set according
to their frequency of occurrence. Heuristically, we
determined that a cut-off of 5 (i.e., only n-grams
that occur five or more times in our corpus of utterances
were included as features) gave us the highest
accuracy for the learning algorithms tested.
7 Discussion
This work appears to be the first attempt to automatically
classify utterances according to their
literal meaning with VRM categories. There are
thus no direct comparisons to be easily drawn for