Swiss Medical Informatics - SGMI

Typical recommendationsare useful to prescribe themost
appropriate antibiotic, according to different parameters
such aspatient situation, clinical assessment, but also
costs, benefits, adverse effects, and the risk of resistance
development. As acase study, we started investigating
guidelines for geriatrics from the University Hospitals of
Geneva (HUG). Totransform such verbose documents into
machine-readable data, the guidelines were transformed
intodatabase tuples. The translation from French
to English was performed manually, assisted by Frenchto-English
translation tools (e.g. http://eagl.unige.ch/
EAGLm/), and aSNOMEDcategoriser [10].For some of the
queries several answers were possible –three on average
–asshown in table 1, where severe diverticulitis caused
by enterobacteriaceae can be treated by three different antibiotics:
ceftriaxone, metronidazole and piperacillintazobactam;
each of them is unambiguously associated
with aunique terminological identifier.
Further, two search engines corresponding to different
search models were tested: easyIR (a relevance-driven
searchengine well known for outperforming other search
methods on MEDLINE searchtasks [11])and PubMed (the
NCBI’s Boolean search instrument). In addition, different
combinations of the outputs of the two engines (PubMed
and easyIR) were tested to combine the strengths of both
engines.
All targets werenormalised using standardterminologies.
Three terminologies were tested to normalise the antibiotic
type of target: alist of 70 SNOMED CT terms, corresponding
chiefly to the available antibiotics at the University
Hospitals of Geneva, alist of MeSH terms, including
synonymous terms, corresponding to the UMLS Semantic
Type T195, and alist of 70 MeSH terms, including synonymous
terms, corresponding mostly to the available antibiotics
at the University Hospitals of Geneva, associated
with WHO-ATC identifiers. The disease type of target was
normalised using alist of MeSH terms corresponding to
disease, corresponding tothe following UMLS Semantic
Types T020, T190, T049, T019, T047, T050, T033, T037,
T047, T191, T046 and T184. The pathogen type of target
was normalised using asubsetofthe NEWTtaxonomy corresponding
to the bacterial taxonomy.
Furthermore,tofine-tune thequestion-answering module,
several descriptors, in particular generic ones, needed
to be removed. Thus, infectious diseases or cross-infection
were removed from the descriptor list for the disease
type of target. Finally, specific keywords were used tore-
PROCEEDINGS ANNUAL MEETING 2009
fine the search equation. Thus, we added context-specific
descriptors such as geriatrics, elderly, etc. The impact of
general keywords such asrecommended antibiotic, antibiotherapy,
etc. was also tested.
Results and discussion
The evaluation of our results isdone with TrecEval, a
program developed to evaluate TREC (Text Retrieval Conferences)
results using NIST(US NationalInstitute of Standards
and Technologies) evaluation procedures. Fine-tuning
of the engine was based on the TREC Genomics
competitions: see e.g.[11]. As usual in information retrieval
[12] and factoid question-answering tasks, we focus on precision-oriented
metrics. In particular, the so-called precision
at recall 0, i.e. the precision of the top-returned answer,isused
to evaluate the effectiveness of our approach.
To complement this metrics, which provides the precision
of the system used without user interaction, we also measure
the recall of the system achieved by the top five answers.
Thus, we try to estimate how useful such asystem
would be when used by an expert able to validate the guideline
generator’s ranked output.
The 64 triplets generated manually are used as the gold
standard. Each rule/query concerns a specific disease
caused by aspecific pathogen and isrepresented by atuple
of four columns(table 1). Diseases, pathogens and antibiotics
wereentered for each entry.Optionally, conditions
were also addeddepending on the entries, suchasweight
or age. Evaluation is done with the focus on the precision
of the first retrieved antibiotic, which corresponds to the
top precision, noted P0 in the following.
Three terminological targets have been tested. The best results
are obtained using asubsetofthe MeSHincluding 70
antibioticentities. P0 is of 0.58 for the PubMed engine and
of 0.52 for the easyIR engine. Eachantibioticentity is mentioned
by several terms, allowing more results to be retrieved.
Thus, amoxycillinwith clavulanatepotassiumcan
also be mentioned as amoxicillin-clavulanic acid or augmentin.
Moreover, using alimited number of antibiotic
entities avoids returninggeneral terms, suchasAnti-Bacterial
Agents. Using keywords provides no significant improvement
in top precision compared to the baseline system,
but aslight decline in top precision.
From figure 1weobserve that thetwo engines, which tend
to perform very similarly on average, seem not tobehave
Table 1
Example of manually-generated rules: terminological identifiers are provided in parenthesis. In this example the infection can be treated by five different
antibiotics. The use of ceftriaxone requires the precondition severe.
Pathologies Pathogenic agents Antibiotics Other conditions
Diverticulitis (D004238) Enterobacteriaceae (543) Amoxicillin-potassium clavulanate combination (D019980);
ciprofloxacin (D002939);
metronidazole (D008795)
Diverticulitis (D004238) Enterobacteriaceae Ceftriaxone (D002443); Severe
(543) metronidazole (D008795);
piperacillin-tazobactam combination product (C085143)
Swiss Medical Informatics 2009; n o 67
37