Automatic Mapping Clinical Notes to Medical - RMIT University

it is even used as a black box that is not fine-tuned
to the task. In this paper we perform a step towards
such a formal study of the ideal characteristics
of a NER for the task of QA. In particular,
section 2 comments on the desiderata of a NER for
QA. Next, section 3 describes the QA system used
in the paper, while section 4 describes the NER
and its modifications for its use for QA. Section 5
presents the results of various experiments evaluating
variations of the NER, and finally Section 6
presents the concluding remarks and lines of further
research.
2 Named Entity Recognition for
Question Answering
Most QA systems gradually reduce the amount of
data they need to consider in several phases. For
example, when the system receives a user question,
it first selects a set of relevant documents,
and then filters out irrelevant pieces of text of these
documents gradually until the answer is found.
The NER is typically used as an aid to filter out
strings that do not contain the answer. Thus, after
a question analysis stage the type of the expected
answer is determined and mapped to a list of entity
types. The NER is therefore used to single out
the entity types appearing in a text fragment. If a
piece of text does not have any entity with a type
compatible with the type of the expected answer,
the text is discarded or heavily penalised. With
this in mind, the desiderata of a NER are related
with the range of entities to detect and with the
recall of the system.
2.1 Range of Entities
Different domains require different types of answers.
Typically, the question classification component
determines the type of question and the
type of the expected answer. For example, the
questions used in the QA track of past TREC conferences
can be classified following the taxonomy
shown in Table 1 (Li and Roth, 2002).
The set of entity types recognised by a standalone
NER is typically very different and much
more coarse-grained. For example, a typical set
of entity types recognised by a NER is the one
defined in past MUC tasks and presented in Table
2. The table shows a two-level hierarchy and
the types are much more coarse-grained than that
of Table 1. Within each of the entity types of Table
2 there are several types of questions of Ta-
50
ABBREVIATION
abb, exp
ENTITY
animal, body, color, creative, currency, dis.med., event,
food, instrument, lang, letter, other, plant, product, religion,
sport, substance, symbol, technique, term, vehicle,
word
DESCRIPTION
definition, description, manner, reason
HUMAN
group, ind, title, description
LOCATION
city, country, mountain, other, state
NUMERIC
code, count, date, distance, money, order, other, period,
percent, speed, temp, size, weight
Table 1: Complete taxonomy of Li & Roth
Class Type
ENAMEX Organization
Person
Location
TIMEX Date
Time
NUMEX Money
Percent
Table 2: Entities used in the MUC tasks
ble 1.
A QA system typically uses both a taxonomy
of expected answers and the taxonomy of named
entities produced by its NER to identify which
named entities are relevant to a question. The
question is assigned a type from a taxonomy such
as defined in Table 1. This type is then used to filter
out irrelevant named entities that have types as
defined in Table 2.
A problem that arises here is that the granularity
of the NEs provided by a NER is much coarser
than the ideal granularity for QA, as the named entity
types are matched against the types the question
requires. Consequently, even though a question
classifier could determine a very specific type
of answer, this type needs to be mapped to the
types provided by the NER.
2.2 Recall
Given that the NER is used to filter out candidate
answers, it is important that only wrong answers
are removed, while all correct answers stay in the
set of possible answers. Therefore, recall in a NER
in question answering is to be preferred above precision.
Generally, a NER developed for a generic
NE recognition task (or for information extraction)
is fine-tuned for a good balance between recall
and precision, and this is not necessarily what