INTERNATIONAL WORKSHOP Patient Safety - MedicalNet-BG

INTERNATIONAL WORKSHOP 

Patient Safety through Intelligent 

Procedures in Medication (PSIP) 

Related to the FP7 ICT project PSIP grant 216130 

PROCEEDINGS 

Edited by 

Régis Beuscart, Dimitar Tcharaktchiev and Galia Angelova 

Sofia, Bulgaria 

23 June 2011

International Workshop 

Patient Safety through Intelligent Procedures in Medication 

(related to the FP7 ICT project PSIP, grant 216130) 


23 June 2011 

PROCEEDINGS 

ISBN 978-954-452-015-1 

Designed and printed by INCOMA Ltd. 

Shoumen, Bulgaria

Organisers and Sponsors 

The International Workshop “Patient Safety through Intelligent 

Procedures in Medication” is organised by: 

Linguistic Modelling Department, 

Institute for Information and Communication Technologies (IICT), 

Bulgarian Academy of Sciences (BAS) 

and 

Medical Informatics Department, 

University Specialised Hospital for Active Treatment of Endocrinology (USHATE) 

“Acad. I. Penchev”, 

Medical University- Sofia, Bulgaria 

The International Workshop “Patient Safety through Intelligent 

Procedures in Medication” is partially supported by: 

The European Commission via the project PSIP, ICT FP7 grant 216130 

Bulgarian Academy of Sciences 

Medical University - Sofia, Bulgaria 

Foundation Europartners 2000 

Association PROREC – Bulgaria

Editors’ Foreword 

Adverse Drug Events (ADE) due to product safety problems and medication errors due to human 

factors (HF) are a major Public Health issue. They endanger the patients’ safety and instigate 

considerable extra hospital costs. The project Patient Safety through Intelligent Procedures in 

Medication (PSIP) develops and validates healthcare ICT applications which help reducing the 

risks of preventable ADEs. PSIP facilitates the systematic production of epidemiological 

knowledge on ADE and provides healthcare professionals and patients with relevant knowledge 

(guidelines, recommendations, etc.). The approach of PSIP is validated in various settings which 

ameliorate the entire medication cycle in a hospital environment. 

The integrated project PSIP is funded via the European Community’s Seventh Framework 

Programme (FP7) under grant agreement 216130 in the period January 2008 – July 2011. In May 

2010, two new partners from Bulgaria have joined the PSIP consortium: the University 

Specialised Hospital for Active Treatment of Endocrinology (USHATE) “Acad. I. Penchev”, 

Medical University - Sofia and the Institute of Information and Communication Technologies 

(IICT), Bulgarian Academy of Sciences (BAS). The project extension was focused on one major 

direction: validation and assessment of PSIP ADE rules, of suitable subset of the PSIP Clinical 

Decision Support System (CDSS) modules and in general, of the PSIP’s contextualisation and 

interoperability potential. To support the compilation of a relatively large repository of patientrelated 

data in USHATE, IICT-BAS has implemented complex modules for automatic extraction 

of diagnoses, drugs and lab data from the free text of hospital patient records in Bulgarian 

language. USHATE has integrated a PSIP-compliant repository, has extended its hospital 

information system in a suitable manner and has performed the PSIP validation. 

This workshop presents mostly the results, performed in the extended PSIP project and focused 

on the PSIP validation in USHATE. At the same time the workshop presents all components 

from the main PSIP that are needed to understand the tasks performed in the project extension. 

Special emphasis is put on the validation exercise and the human factors relevant to the practical 

applications of IT solutions in clinical environments. 

The PSIP validation in USHATE shows that the Bulgarian medical experts have positive attitude 

towards innovative ICT solutions in the everyday clinical practice. The workshop attendees have 

also asked numerous questions and made valuable comments, which shows the need of further 

developing the eHealth environment in Bulgaria. We hope that the lessons learnt in PSIP will 

have long term impact on the IT applications for processing medical data in Bulgaria and that the 

linguistic resources in Bulgarian language, which have been collected for the PSIP project, will 

be further expanded and reused in the clinical practice. 

We would like to thank the authors for contributing to the workshop proceedings and to the 

workshop presenters for the interesting lectures delivered at the oral presentations. We are 

grateful to the Bulgarian Academy of Sciences for hosting this workshop it its Headquarters 

building. Our special gratitude goes to the Foundation Europartners 2000 (Chairman Miroslav 

Nikolov) for the support in many technical tasks related to the workshop organisation. 

June 2011 

Régis Beuscart, Dimitar Tcharaktchiev and Galia Angelova

Workshop Programme 

23 June 2011, Headquarters of Bulgarian Academy of Sciences, Sofia, Str. "15 November" 1 

9 00 -9 15 Opening 

9 15 -9 45 Régis Beuscart (University Hospital Lille) 

Overview of the PSIP project for the detection and prevention of Adverse Drug Events 

9 45 -10 15 Grégoire Ficheur (University Hospital Lille) 

Interoperability of Medical Databases: Construction of Mapping Between Hospitals 

Laboratory Results Assisted by Automated Comparison of their Distributions 

10 15 45 

-10 

Nicolas Leroy (University Hospital Lille) 

Human Factors in the PSIP project 

10 45 -11 15 Coffee Break 

Vassilis Kilintzis (Aristotel University of Thessaloniki) 

Knowledge Engineering and Development of Clinical Decision Support Systems for 

11 15 -11 45 Adverse Drug Event Prevention 

11 45 -12 15 Suzanne Pereira (Vidal SA, France) 

Comparison of Two Methods for French Drug Names Extraction 

12 15 -12 30 Discussion 

12 30 -13 30 Lunch Break 

13 30 -13 50 Galia Angelova (Institute of Information and Communication Technologies, Bulgarian 

Academy of Sciences) 

Automatic extraction of patient-related entities from Bulgarian hospital records 

13 50 -14 20 Svetla Boytcheva (Institute of Information and Communication Technologies, Bulgarian 

Academy of Sciences) 

Demo: Automatic extraction of ICD-10 codes for diagnoses and ATC codes and dosages 

for drugs 

14 20 -14 40 Dimitar Tcharaktchiev (University Specialised Hospital for Active Treatment of 

Endocrinology /USHATE/ “Acad. I. Penchev”, Medical University – Sofia) 

Development of experimental repository for Bulgarian patients and a testbed for 

integration of PSIP CDSS modules with the USHATE hospital information system 

14 40 -15 00 Hristo Dimitrov (University Specialised Hospital for Active Treatment of 

Endocrinology /USHATE/ “Acad. I. Penchev”, Medical University – Sofia) 

Demo: Testbed for integration of CDSS modules and PSIP validation 

15 00 -15 30 Coffee Break 

15 30 -16 00 Elske Ammenwerth and Martin Jung (University for Health Sciences, Medical 

Informatics and Technology, Hall in Tirol, Austria) 

Expectations and barriers of physicians versus CPOE (Computerised Physician Order 

Entry) 

16 00 -16 20 Krassimira Nechkova-Atanassova (University Specialised Hospital for Active 

Treatment of Endocrinology /USHATE/ “Acad. I. Penchev”, Medical University – Sofia) 

Survey of the Attitudes toward Innovation of the Physicians at USHATE 

16 20 -17 00 Round table, discussion 

17 00 Closing

Table of Contents 

PSIP Overview: Detection and Prevention of Adverse Drug Events 

Régis BEUSCART, Emmanuel CHAZARD …………………………………………...……. 1 

Interoperability of Medical Databases: Construction of Mapping between Hospitals 

Laboratory Results Assisted by Automated Comparison of their Distributions 

Grégoire FICHEUR, Emmanuel CHAZARD, Aurélien SCHAFFAR, Régis BEUSCART … 4 

Human Factors in the PSIP Project 

Nicolas LEROY, Romaric MARCILLY, Marie-Catherine BEUSCART-ZEPHIR …………. 9 

Developing Decision Support Services for Patient Medication Safety: 

A Knowledge Engineering Perspective 

Vassilis KOUTKIAS, Vassilis KILINTZIS, Régis BEUSCART, Nicos MAGLAVERAS .. 13 

Comparison of Two Methods for French Drug Names Extraction 

Suzanne PEREIRA, Catherine LETORD, Sophie TESSIER, Solenne REIN, 

Stefan DARMONI, Elisabeth SERROT ……………………………………………..…….. 17 

Automatic extraction of patient-related entities from Bulgarian hospital records 

Galia ANGELOVA ……………………………………………………………………….… 21 

Automatic extraction of ICD-10 codes for diagnoses and ATC codes and dosages for drugs 

Svetla BOYTCHEVA …………………………………………………………………….… 25 



Dimitar TCHARAKTCHIEV ………….…………………………………………………… 35 

Testbed for integration of CDSS modules and PSIP validation in USHATE, Medical University – 

Sofia 

Hristo DIMITROV ………….…………………………………………………….………… 39 

Expectations and Barriers versus cxCDSS-CPOE: A European User Survey 

Elske AMMENWERTH, Martin JUNG ..…………………………………………...……… 49 

Survey of the attitudes of the physicians at the USHATE hospital towards innovations 

Krassimira NECHKOVA-ATANASSOVA ..…..………………………………...………… 53

PSIP Overview: Detection and Prevention of Adverse Drug Events 

Régis BEUSCART 1 , Emmanuel CHAZARD 2 

1 

Lille University Hospital, UDSL-EA2694-France, Regis.BEUSCART@CHRU-LILLE.FR 

2 

Lille University Hospital, UDSL-EA2694-France, emmanuelchazard@yahoo.fr 

Abstract 

The development of Clinical Information Systems (CIS), 

Laboratory Management Systems (LMS) and Computer 

Provider Order Entry Systems (CPOE) for the 

computerization of hospital settings make available a large 

amount of data, that are now available for statistical 

analysis and epidemiological studies. In this paper, we 

describe how we utilized the data issued from more than 

100,000 medical records for detecting Adverse Drug 

Events. First we performed a Knowledge Data Discovery 

phase for discovering detection rules. Second, we applied 

these rules on real medical data bases to automatically 

identify potential ADE cases. Third, we implemented the 

rules in Clinical Decision Systems, which are now 

available through the Hospital Information System (HIS) or 

through the web. The resulting applications are currently 

used in routine for the prevention of ADE. 

Keywords 

Clinical Information Systems, Data Mining, Knowledge 

data Discovery, Adverse Drug Events, Hospital 

Information Systems, Clinical Decision Support Systems. 

1. Introduction 

For the last thirty years, ICT (Information and 

Communication Technology) applications have been 

developed and implemented in hospital settings. 

Remarkable efforts have been made to incorporate in 

Hospital Information Systems (HIS), specific 

applications to support the medication ordering, 

dispensing and administration functions. These 

applications are usually referred as Computer 

Provider Order Entry (CPOE) systems. As a 

consequence, huge amounts of data are available in 

Hospital Information System (HIS), Lab Management 

Systems (LMS) and CPOE. More often, these data are 

only used for the management of patients, or for 

medico-economic purposes. 

These data can also be utilized for a better knowledge 

of the hospitalized patients, a better comprehension of 

the hospitalization characteristics, the assessment of 

the quality of care and epidemiologic studies. In this 

paper, we will show how they can be used for Patient 

Safety. 

Patient Safety and the identification of Adverse Drug 

Events (ADEs) in Healthcare have become a major 

public health issue. Every national survey so far 

confirms the trend shown in U.S.: in about 10% of the 

admissions, serious mistakes occur, and in 60% of the 

cases, medications are involved [1]. But Adverse 

Drug Events are not systematically declared by 

physicians of pharmacists to pharmaco-vigilance 

units. It is estimated that only 1-2% of the ADEs 

occurring in an hospital are reported. So, most of the 

usually cited numbers are estimates and nationwide 

extrapolations issued from quantitative or qualitative 

regional studies. Therefore, these numbers or 

estimates bear a relative uncertainty. Moreover, their 

reliability can be questioned. 

Comparing, on one hand the difficulty to obtain 

reliable epidemiological information on ADE [2], on 

the other hand the availability of a large number of 

data recorded in the Hospital Information Systems, it 

seems possible to identify healthcare situations where 

the patient safety is at risk. The objective is, by using 

Data Mining techniques, to get a better knowledge of 

the prevalence of Adverse Drug Events, and of their 

characteristics. 

As a consequence, this knowledge can be used for two 

objectives: (1) a retrospective analysis of collected 

medical records to estimate the prevalence and the 

incidence of Adverse Drug Events in different 

contexts within hospitals; (2) a prospective prevention 

of Adverse Drug Events by using a Clinical Decision 

System able to detect the risks associated with 

medications to a patient in a defined context. 

2. Methodology 

Knowledge Data Discovery. 

Workshop Patient Safety through Intelligent Procedures in Medication, Sofia, Bulgaria, 2011 

1

The study has been performed in more than 100,000 

anonymized medical records obtained after extraction 

from HIS according to a common data model. 

Data Mining was first performed on a first set of 

records (30,000) for Knowledge Data Discovery 

(KDD), inducing new rules of knowledge for 

detecting ADE. This has been done by means of nonsupervised 

and supervised data mining methods. 

Supervised methods like Decision trees (CART, 

CHAID), and Association Rules have been the more 

efficient for KDD. 

The knowledge about ADEs issued from the 

application of data mining methods has been 

summarized by means of ADE detection rules. An 

ADE detection rule is made of one or several Boolean 

conditions that lead to an outcome, with a given 

probability, such as: 

Cause1 & Cause2 & Cause3 Outcome 

That representation is widely used either for 

retrospective ADE detection [3]. In this work, we 

obtain a set of 236 rules involving 1 to 4 Causes to 

lead to an Outcome. The Causes can be: demographic 

characteristics of the patients, drug administrations or 

discontinuations, laboratory results, or diagnoses. The 

number and the kind of the conditions were not 

constrained by the methods but were optimized by the 

use of statistical procedures. These rules enable to 

identify 56 different types of outcomes organized in 3 

categories: Coagulation disorders, Ionic and Renal 

disorders, Miscellaneous. 

Validation of the Knowledge Base (PSIP KB) 

Ruleset 

236 rules describing the knowledge for ADE 

identification has been collected. A phase of 

validation, and refinement has been necessary to make 

the rules usable in clinical practice. This validation 

has been realized on the set of the remaining collected 

70,000 records and on some test-cases provided by 

clinicians. Several technical meetings are organized 

with experts: physicians, pharmacologists, 

pharmacists and statisticians. The rules are examined 

by the experts and validated against summaries of 

products characteristics and bibliography. That review 

used several drug-related web information portals 

(Pharmacorama 2009, BDAM 2009, Thériaque 2009), 

Pubmed referenced papers (Pubmed 2009), and 

French summaries of products characteristics 

provided by the Vidal Company. 

During this review, the experts may suggest different 

things: queries to check what are the pathological 

context of the cases and what are precisely the drugs 

involved, or tuning of the rules, to test different 

classes or subclasses of drugs. Sometimes, rules are 

manually enforced in agreement with the academic 

knowledge in order to test some hypothesis. 

The following part presents two tools implementing 

the PSIP rules set for ADE identification and 

prevention. 

3. Results 

ADE Scorecards 

A special software “ADE Scorecards”, can semiautomatically 

detect potential cases of ADEs in a 

large set of medical records has been developed [4]. 

The calculation and presentation of the statistical 

analysis of a set of clinical records to detect potential 

ADEs relies on two steps: (1) a computation step, 

consisting in applying the ADE detection rules to the 

hospital stays in order to detect the ADE cases and to 

compute statistics about ADEs (e.g. incidence and 

prevalence of the identified Adverse Drug Events per 

hospital and per medical unit), (2) the second step, a 

web-based display tool, consists in displaying the 

statistics and the ADE cases. Thus global 

comparisons are also possible between medical units, 

hospitals, regions, and even countries. 

This application is now routinely used by two 

experimental hospitals in a protocol to improve the 

patient safety through the prevention or the 

monitoring of the potential ADEs. 

Contextualized Decision Support System. 

The Knowledge Base (PSIP KB) has also been 

exploited to build Clinical Decision Systems. The 

ADE Detection Rules have been implemented in a 

PSIP Knowledge Base System (KBS) to provide 

physicians and Healthcare Professionals with 

information and alerts concerning the treatment of a 

patient, taking into account the geographical 

environment, the clinical characteristics and the 

current status of the patient (through the lab tests 

results), and obviously the current medications. This 

PSIP KBS encapsulates signals capable of 

2 

Workshop Patient Safety through Intelligent Procedures in Medication, Sofia, Bulgaria, 2011

automatically detecting potential ADEs susceptible of 

occurring for a hospitalized patient. It has been 

implemented in existing medical application (e.g. a 

French Electronic Health Record, a Danish CPOE). A 

web prototype (independent of any medical 

applications) has been also developed. These CDSS 

are available on the PSIP website (http://psip.univlille2.fr/prototypes/public/). 

4. Discussion 

Many clinical data are now available from Clinical 

Information Systems including Electronic Health 

records, Lab Management Systems and CPOEs. By 

aggregating these data, and analyzing them with 

appropriate statistical tools, it is possible to identify 

retrospectively the existence, occurrence, and 

frequency of various events (nosocomial infections, 

Adverse Drug reactions, adverse drug events, 

medication errors, falls, surgical performances, etc.). 

In our study we performed Data Mining to identify 

cases where the common occurrence of certain 

clinical features, lab results and drug prescription 

make the probability of an Adverse drug Event 

higher. The performance of this method, estimated 

after an expert review, is about 60%. Moreover, the 

experts review shows that the world is complex and 

not black/white. In many cases, contraindicated drugs 

are prescribed in presence of another drug or despite a 

diagnosis. But experts explain that in complex 

medical situations, involving multiple diseases, this 

medical attitude is comprehensible; 

Given the data at hand in hospital databases, our 

studies demonstrate that it is possible, with a 

reasonable confidence, to identify Adverse Drug 

Events, and their frequency. 

It becomes also possible to provide physicians and 

nurses with alerts, information and recommendations 

for a better usage of medications during the 

prescription or the administration phases. 

By mining large sets of data, it is possible to gain 

automatically estimations of events with a better 

precision than the current manual methods. 

Clinical Information systems represent a vast source 

of data and knowledge that only starts to be exploited 

but must be more extensively explored through 

adequate statistical methods. 

The results of the PSIP Project have been presented in 

2 international workshops, the minutes of which are 

presented in 2 books [5, 6]. 

6. Acknowledgements 

The research leading to these results has received 

funding from the European Community’s Seventh 

Framework Program (FP7/2007-2013), under Grant 

Agreement n° 216130 – the PSIP project. 

7. References 

[1] Sauer F, Patient and medication safety, EJHP-P 11 2005-4. 

[2] Murff H. J., Patel V. L., Hripcsak G., Bates D. W. Detecting 

adverse events for patient safety research: a review of 

current methodologies, J Biomed Inform; 36 (1-2) 2003, pp. 

131-43. 

[3] Koutkias V. et al. Construting clinical decision systems fro 

adverse drug events prevention: a knowledge-based 

approach, Proc. AMIA Annu Symp. pp. 63-74, 2010. 

[4] Chazard E., Baceanu A., Ferret L, Ficheur G. The ADE 

Scorecards: a tool for ADE detection in Electronic health 

records, In Koutkias V. et al. [Eds] Patient Safety 

Informatics, Amsterdam, IOS Press, 2011. 

[5] Beuscart R, Hackl W, Nøhr C, Detection and prevention of 

Adverse Drug Events -information technologies and human 

factors. Preface, Stud Health Technol, IOS Press, 148, 2009. 

[6] Koutkias V., Niès J., Jensen S., Maglaveras N., Beuscart R., 

Patient Safety informatics, Stud Health Technol, IOS Press, 

166, 2011. 

5. Conclusion 

With the continuous increase in the collection of data 

in Clinical Information Systems, it is now possible to 

use these data for performing advanced statistical 

studies Through Knowledge Data Discovery, new 

knowledge or contextualization of existing knowledge 

can be obtained for helping the decision process in 

medicine. 


3

Interoperability of medical databases: 

Construction of mapping between hospitals laboratory 

results assisted by automated comparison of their 

distributions 

Grégoire Ficheur, MD 1 , Emmanuel Chazard, MD, PhD 1 , Aurélien Schaffar 1 , Régis Beuscart, MD, PhD 1 

1 

Department of Medical Information and Archives, CHU Lille, 

UDSL EA 2694; Univ Lille Nord de France; F-59000 Lille, France, 

gregoire.ficheur@gmail.com 

Abstract 

In hospital information systems, laboratory results are stored 

using specific terminologies which differ between hospitals. The 

objective is to create a tool helping to build a mapping between a 

target terminology (reference dataset) and a new one. Using a 

training sample consisting of correct and incorrect correspondence 

between parameters of different hospitals, a match probability 

score is built. This model allows also to determine the theoretical 

conversion factor between parameters. This score is evaluated on 

a test sample of a new hospital: For each reference parameter, best 

candidates are returned and sorted in decreasing order using the 

probability given by the model. The correct correspondent of 9 of 

15 reference parameters are ranked in the top three among more 

than 70. All conversion factors are correct. A mapping webtool is 

built to present the essential information for best candidates. 

Using this tool, an expert has found all the correct pairs. 

Keywords 

Interoperability, medical databases, data-mining, 

laboratory results, statistic distribution 


The construction of inter-hospital databases offers 

perspective to analyze larger databases and thus gain 

more power in analysis. The European PSIP project 

(1), in which this work is included, is a recent 

example of construction of such database. The aim of 

this project is to detect and prevent the occurrence of 

adverse drug reactions by analyzing large datasets 

using the methods of data-mining (2). 

This kind of analysis requires syntactic 

interoperability (ie between databases) and several 

standards have emerged in this context including 

Health Level 7 (3). Semantic interoperability (ie 

between terminologies) is the second aspect of this 

convergence and requires the use of common 

terminologies. Thus, one can cite the International 

Classification of Diseases (4) (ICD) for diagnoses and 

also Anatomical Therapeutic Chemical Classification 

(5) (ATC) for drugs. However, the encoding of 

laboratory results continues to pose a specific problem 

in this context and semantic interoperability of 

laboratory results is not a reality yet. 

The structured description of laboratory results in 

databases is based on different terminologies. Logical 

Observation Identifiers Names and Codes (LOINC®) 

(6-8), SNOMED-CT (9) or International Union of 

Pure and Applied Chemistry (IUPAC) (10,11) can be 

retained, but mostly a local terminology is used with 

low accuracy level (ie less detailed) and only adapted 

for clinical use. So a same parameter measured in one 

system with one method is named differently between 

2 different hospitals and it requires a difficult manual 

time-consuming work to build a mapping between all 

used terminologies and a unique reference. The 

difficulties to map a local to an international 

terminology are detailed by Lin and al. (12) analyzing 

the correctness of the LOINC mappings of 3 large 

institutions. 

2. Related Work 

Tools can be used to build mappings between these 

terminologies, they are based on character string 

recognition in the label of parameter or in reports. 

“The Map to LOINC” presented by A.N. Khan (13) 

uses an automatic mapping tool to map local 

laboratory labels to LOINC. Initially, two experts 

manually assigned LOINC codes to the tests in a 

master merging laboratory tests names and synonyms. 

Then an automated mapping tool was developed to 

map local laboratory test names to LOINC using the 

4 


mappings specified in the master file. Evaluation goes 

on 4,967 laboratory active tests, 67% have been 

mapped automatically, 19% manually and 14% have 

been considering as uncodable tests. Local naming 

choice was the principal cause of failure (including 

variant due to a different facility naming’s 

convention). 

K.N. Lee and al. (14,15) have built a tool based on the 

6 attributes of the LOINC terminology. Each LOINC 

code is based on an unique combination of 6 attributes 

so each code can be thought as having a unique set of 

6 relationships, one to each attribute. All this sets are 

stored in a knowledge base with also synonyms and 

rules. For a new laboratory result, a set of 

relationships is created based on information given by 

laboratory (like unit) and a tool compares the set of 

relationships of the new entrant to all the sets of 

relationships in the knowledge base. In this way, 

exact LOINC code can be found. This method needs 

manually work to create lists of synonyms and rules. 

Two more projects to map local radiology terms to 

LOINC (16) and to map clinical narrative to LOINC 

(17) can be cited: They don’t concern laboratory 

results specifically. 

Labels of laboratory results are not explicit in many 

cases: For example, for “total bilirubinemia”, the label 

“BT” is found in a first database, “Bilirubins” in 

another one, and “Bilirubine totale” in a third one. 

Most of labels are not explicit enough and, for this 

reason, the units, the bounds of normality and the 

distribution of parameters are used to find the correct 

correspondence during expert review. These steps are 

very useful for expert validation, so it might be useful 

to automatically compare the distributions of 

parameters to develop a new kind of mapping tool. 

In this context, Zollo and al. (18) defined each 

parameter using “name, frequency, unit, code, cooccurences” 

and also “mean and standard deviation”, 

these 2 last elements approaching the description of 

distribution of the parameter. This work doesn’t seem 

to detail how to find automatically the conversion 

coefficient factor for a same kind of laboratory result 

with different units. 

The objective is to create a tool helping to construct a 

mapping between a reference parameter (with a target 

terminology) and a new parameter from an incoming 

data using a different terminology. The target 

terminology has to be simple for clinical use and 

ready to use for data-mining. Two steps are planned: 

1. Construct a model giving a probability of 

correspondence of two parameters. The 

approach is not to use the labels but rather to 

compare the statistical distributions of 

parameters. This model must also identify the 

unit. 

2. Present the results and the discriminant 

information via a web tool that can help an 

expert to build this mapping. 

3. Material 

Here are presented the datasets used in this project. 

Table 1 contains more details about these datasets. 

The reference dataset with reference distributions of 

laboratory results is taken from Denain’s (FR) 

hospital center. From this reference data, a simple 

target terminology is manually created. This target 

terminology could be replaced by an existing 

international terminology. Among the 233 found 

parameters, 15 most used parameters are selected for 

this study, these 15 parameters are the target to match 

with the other datasets. 

Hospital 

Center 

Denain 

(FR) 

Rouen 

(FR) 

Copenhagen 1 

(DK) 

Copenhagen 2 

(DK) 

Use 

Name 

Reference set 

Target 

Learning set 

Newcomer 1a 

Learning set 

Newcomer 1b 

Validation set 

Newcomer 2 

Selected 

parameters 

15 

(to match) 

Terminology 

Target 

terminology 

74 Local 

terminology 

71 IUPAC 

74 IUPAC 

Table 1 - Details about datasets used in the project 

The two datasets used in addition to the dataset of 

reference for learning phase contain different 

terminologies. Only the parameters having a number 

of values above 1‰ of the total number of laboratory 

results are retained in the datasets of Rouen hospital 

center and Copenhagen hospital 1. This limit is 

chosen in order to have a good representation of the 

distribution and to remove rare parameters. The Table 

2 illustrates the terminologies used to represent the 

same laboratory results (sodium ion and glycemia) 

among 3 different datasets. 


5

Hospital 

center 

Label of 

parameter 

Kind 

Unit 

Denain NA1 Sodium ion meq/l 

Denain GLY1 Glycemia g/l 

Copenhagen 

1 & 2 

Copenhagen 

1 & 2 

NPU03429 Sodium ion mmol/l 

DNK35842 Glycemia mmol/l 

Rouen Sodium Sodium ion mmol/l 

Rouen Glycémie Glycemia mmol/l 

Table 2 - Same laboratory results in different datasets 

The evaluation is made with a dataset from a second 

hospital of Copenhagen (DK). These results are 

encoded according to IUPAC classification, but this 

setting is not used (of course) to make the 

correspondence between pairs of parameters, it is used 

only to check the quality of the found correspondence. 

4. Method 

The comparison of distributions is strongly 

intertwined with the finding of theoretical conversion 

coefficient between two parameters. 

For each reference parameters, theoretical conversion 

coefficients possible are known. For example, 

international unit for creatininium is “μmol/l”, but a 

lot of hospitals use another unit “mg/l”. The 

theoretical conversion coefficients are “1” if the two 

parameters have the same unit, “8.85” (from μmol/l to 

mg/l) and “1/8.85” (from mg/l to μmol/l). Figure 1 

illustrates this point: On the left is presented 

creatinemia’s distribution from two different data with 

two different units and on the right creatinemia from 

the second dataset is multiplied by the nearest 

theoretical conversion coefficient and distributions 

overlap. For the parameters counting a number of 

elements (for example the number of red blood cells), 

all the powers of 10 are possible coefficients. It is the 

same with some concentrations as hemoglobin that 

may be in g/l or g/dl. 

Figure 1 - Effect on distribution of conversion coefficient 

To determine the nearest theoretical conversion 

coefficient, all theoretical coefficients (known for a 

reference) are compared with the ratio of medians 

between the reference and the candidate. This nearest 

theoretical conversion coefficient is used to multiply 

the values of a new entrant parameter and thus the two 

parameters have the same scale. Next variables 

describing the distribution are evaluated after this 

conversion. 

5 robust and discriminating variables are retained to 

describe the distribution of a laboratory result: 

Kolmogorov test’s statistic 

10th, 65th and 85th quantiles ratio of the 2 

compared parameters 

R squared of the linear regression (with 

quantile new variable as dependent variable 

and quantile reference variable as 

independent variable). 

The 5 variables presented above are the explanatory 

variables. In the table containing training data, each 

line corresponds to the values obtained for a pair of 

parameters (target vs. newcomer 1a & 1b). This table 

also contains a column for the binary dependent 

variable “couple” that indicates whether the 

correspondence of the couple (seen on the line) is true 

or false. The variable “couple” is the variable to 

explain. A logistic regression is performed with these 

variables and a model is obtained. The model thus 

built directly returns a score describing the probability 

of match for a pair of tested parameters. 

Validity and robustness of the resulting model are 

studied. All analyses are performed with R software 

version 2.11.1 (19). 

This score is then evaluated on a test sample of a new 

hospital (newcomer 2). For each reference parameter, 

best candidates are returned and sorted in decreasing 

order using the probability given by the logistic 

model. All results are saved using a XML format. 

6 


A mapping web tool (using directly XML results 

files) is built to present the essential information of 

best candidates. This tool is designed to help an 

expert to take a decision. 

The web interface shows, for each reference 

parameter: 

A classification of best corresponding candidates 

sorted in decreasing order using the probability given 

by the logistic regression. Only the corresponding 

candidates with a probability above a limit score are 

retained. 

Their graphic distribution: Used Graphics combine 

the graphic distribution of the reference parameter and 

the graphic distribution of the new parameter 

multiplied by the nearest theoretical conversion 

coefficient. 

Their names and a few describing attributes 

aggregating the main information required to validate 

the visual impression: Unit, number of values, nearest 

theoretical conversion coefficient and bounds of 

normality. 

5. Results 

For each reference, all correct parameters are returned 

among the short list candidates and the median 

number of candidates is 6. The main result is the rank 

given by the model to the correct couple (a target 

parameter and its correspondent from newcomer 2). 

For the correct candidate, the first rank is obtained for 

7 of them and 9 are in the top three. The worst rank is 

8, obtained for 1 couple. All conversion factors found 

for these parameters are correct. These results are 

presented in Table 3. 

6. Conclusion 

A logistic model based on the comparison of 

distributions of laboratory results is built. It is then 

used to select a short list of compatible parameters for 

each of the target parameters. All relevant parameters 

are among the top 8 candidates and all the preselected 

conversion factors are correct. A short list of 

candidates is then presented via a web tool allowing 

to an independent expert to find all the correct pairs in 

a short time. 

Reference data compared 

to newcomer 2 

Results given by the model for 

the correct candidate 

Parameter Rank (/74) Conversion 

coefficient 

Sodium Ion 1 1 

Kalemia 1 1 

Creatininium 1 0.113 

Haemoglobin 1 1.667 

Leukocytes 1 1000 

C Reactive Protein 1 1 

Carbamide 1 16.6 

Calcium Ion 2 40 

Creatin Kinase 2 1 

Aspartate transaminase 2 1 

Glycemia 4 0.18 

Erythrocytes 4 1 

Alanine transaminase 4 1 

International Normalized 

Ratio 

5 1 

Bilirubins 8 0.585 

Table 3 - Rank and unit returned by the model for the correct 

correspondent parameters 

7. Discussion 

The number of reference parameters is reduced and 

related only to common parameters so well described 

in the dataset. For this reason, only the parameters 

representing more than 1‰ of the lines of the 

database are analyzed. It is possible that this model 

has lower performance on rare parameters like 

troponin or thyroid hormones. 

Some identical parameters (with same unit) have 

sometimes very different distribution and are thus 

difficult to compare with this method. For example, 

INR is a systematic rendering of one of our partner 

hospitals (when a prothrombin time is done) while in 

other places, it is reserved for patients with vitamin K 

antagonist. Thus, the distributions are very different 

between these places. Another example concerns the 

C reactive protein which shows very different 

distributions between hospitals: A first hypothesis is 

that the practices are different; another hypothesis is 

that it comes from the measurement tool. 

Finally, this method doesn’t take into account the 

textual results like some results of bacteriology. 


7

This analysis has focused on the comparison of 

distributions, but doesn’t try to model the distribution 

of each parameter individually. This complementary 

approach is perhaps a way to progress. 

Moreover, it seems possible to use a meta-rule that 

would organize the allocation of the same parameter 

between many references. For example, the parameter 

A is ranked first among candidates for a reference 1 

and only the fifth candidate for reference 2 so it seems 

reasonable to exclude the parameter A from the list of 

candidates for reference 2 and to keep it only in the 

list of candidates for reference 1. 

The fact to report on a web tool the short list of 

candidates (given by the model for one reference 

parameter) gives a real help to map a new 

terminology to a target terminology, the final decision 

being left to an expert. One of the interests of the 

mapping tool is to be used with any local or 

international terminology. 


The research tasks leading to these results have received 


Framework Programme (FP7/2007-2013) under grant 

agreement n° 216130 PSIP (Patient Safety through 

Intelligent Procedures in Medication). 

9. References 

[1] www.psip-project.eu [Internet]. [cit. 2011 Feb 

23];Available from: http://www.psip-project.eu 

[2] Chazard E, Preda C, Merlin B, Ficheur G, 

Beuscart R. Data-mining-based detection of 

adverse drug events. Stud Health Technol Inform. 

2009;150:552-6. 

[3] HL7. Health Level 7 [Internet]. Available from: 

http://www.hl7.org 

[4] WHO. International Classification of Diseases 

[Internet]. 2009;Available from: 

http://www.who.int/classifications/icd/en 

[5] WHO. Anatomical and Therapeutical 

Classification. 2009;Available from: 

http://www.whocc.no/atcddd 

[6] Logical Observation Identifiers Names and Codes 

(LOINC®) — LOINC [Internet]. [cit. 2011 Mar 

17];Available from: http://loinc.org/ 

[7] Forrey AW, McDonald CJ, DeMoor G, Huff SM, 

Leavelle D, Leland D, et al. Logical observation 

identifier names and codes (LOINC) database: a 

public use set of codes and names for electronic 

reporting of clinical laboratory test results. Clin. 

Chem. 1996 Jan;42(1):81-90. 

[8] McDonald CJ, Huff SM, Suico JG, Hill G, 

Leavelle D, Aller R, et al. LOINC, a universal 

standard for identifying laboratory observations: a 

5-year update. Clin. Chem. 2003 Avr;49(4):624- 

633. 

[9] IHTSDO. SNOMED-CT [Internet]. 

2009;Available from: http://www.ihtsdo.org 

[10] Pontet F, Magdal Petersen U, Fuentes-Arderiu X, 

Nordin G, Bruunshuus I, Ihalainen J, et al. 

Clinical laboratory sciences data transmission: the 

NPU coding system. Stud Health Technol Inform. 

2009;150:265-269. 

[11] IUPAC. C-NPU. 2009;Available from: 

http://www.iupac.org 

[12] Lin MC, Vreeman DJ, McDonald CJ, Huff SM. A 

Characterization of Local LOINC Mapping for 

Laboratory Tests in Three Large Institutions. 

Methods Inf Med [Internet]. 2010 Aoû 20 [cit. 

2011 Feb 1];49(5). Available from: 

http://www.ncbi.nlm.nih.gov/pubmed/20725694 

[13] Khan AN, Griffith SP, Moore C, Russell D, 

Rosario AC, Bertolli J. Standardizing laboratory 

data by mapping to LOINC. J Am Med Inform 

Assoc. 2006 Juin;13(3):353-355. 

[14] Lee KN, Yoon J, Min WK, Lim HS, Song J, Chae 

SL, et al. Standardization of terminology in 

laboratory medicine II. J. Korean Med. Sci. 2008 

Aoû;23(4):711-713. 

[15] Lau LM, Banning PD, Monson K, Knight E, 

Wilson PS, Shakib SC. Mapping Department of 

Defense laboratory results to Logical Observation 

Identifiers Names and Codes (LOINC). AMIA 

Annu Symp Proc. 2005;:430-434. 

[16] Vreeman DJ, McDonald CJ. A comparison of 

Intelligent Mapper and document similarity scores 

for mapping local radiology terms to LOINC. 

AMIA Annu Symp Proc. 2006;:809-813. 

[17] Fiszman M, Shin D, Sneiderman CA, Jin H, 

Rindflesch TC. A Knowledge Intensive Approach 

to Mapping Clinical Narrative to LOINC. AMIA 

Annu Symp Proc. 2010;2010:227-231. 

[18] Zollo KA, Huff SM. Automated mapping of 

observation codes using extensional definitions. J 

Am Med Inform Assoc. 2000 Déc;7(6):586-592. 

[19] R_Development_Core_Team. R: A Language and 

Environment for Statistical Computing [Internet]. 

2009;Available from: http://www.R-project.org 

8 


Human Factors in the PSIP project 

Nicolas LEROY 1 , Romaric MARCILLY 2 , Marie-Catherine BEUSCART-ZEPHIR 3 

1 2 3 

INSERM CIC-IT, Lille ; Univ Lille Nord de France ; CHU Lille ; UDSL EA 2694 ; F-59000 Lille, France 

1 nicolas.leroy-2@univ-lille2.fr 2 romaric.marcilly@univ-lille2.fr 3 mcbeuscart@univ-lille2.fr 

Abstract 

The goal of the ISO 13407 Human Centered Design 

Process is to ensure that the development and use of 

interactive system take into account the user’s needs. 

We applied this approach to the PSIP project and 

analyzed the user’s tasks connected to the medication 

use process. As a result some recommendations for 

the design of the system were produced. An iterative 

evaluation process of the usability of the PSIP tools 

was performed to ensure that the systems are easy to 

use and meet the user needs. 

Keywords 

Patient safety, Usability, User Centered Design 

Process, Adverse Drug Event, Human Factor 


The first objective of the PSIP project is to improve 

the safety of the medication use process. Besides, the 

efficiency of the organization and the well-being and 

satisfaction of users are also addressed. These two 

aspects are linked to the acceptability of the system by 

the organization and the final users. 

To reach these goals, we have some specific Human 

Factors objectives: the satisfaction of user’s needs, a 

good usability of the tools, and the consistency of 

organizations, policies and procedures. For the 

satisfaction of these objectives, specific human factors 

methodologies were used in a user-centered design 

process (figure 1). 

Figure 1 : Human Factors objectives and methodology 

2. Methods 

2.1 User-centered design process 

The ISO 13407 user centered design process is a 

multidisciplinary process divided into four parts 

(figure 2). 

The first part refers to the specification of the context 

of use. At this stage, the main objectives are to 

understand the real activity of the users, the second 

part refers to the specification of the user’s needs and 

the third part refers to recommendations for the design 

of the products and the last part refers to the 

evaluation of the design. This is an iterative process: 

if the results of the evaluation are not good enough, 

another loop of specifications and evaluations is 

performed. 


9

Figure 2 : ISO 13407 User Centered Design Process 

2.2 Analysis of the context of use 

The analysis of the work situations in the medication 

use process is mandatory to support the user-centred 

design of the system and therefore enhance the 

acceptance of this system and ultimately its 

efficiency. The methods used for this work analysis 

must allow the identification and description of the 

relationships between (i) the obligations inherent to 

the professionals’ tasks and activities, (ii) the 

constraints imposed by the work environment and (iii) 

the resources that can be mobilized by the 

professionals. A number of well-known techniques 

are available to help the analyst collect, organize, 

formalize and analyze relevant data. To structure the 

techniques and methods in the PSIP project, we rely 

on the general framework from Jens Rasmussen & al., 

the Cognitive Work Analysis (CWA) [1], which is the 

internationally recognized reference for the study of 

complex sociotechnical environments. 

The interview is one of the most commonly used 

approaches for information gathering [2]. It has to be 

completed with observational techniques such as 

direct visual observation, video-recording, participant 

observation, time-lapsed photography, etc. The 

triangulation of these techniques allows uncovering 

information on the work situation which cannot be 

acquired in any other way [3]. 

Figure 3: illustration of the analysis of the work situation 

(pharmacist assistant preparing the drug pills) 

Figure 4: illustration of the analysis of the work situation 

(Physicians during the design sessions) 

Three observers trained in naturalistic observation in 

complex settings conducted observations and 

interviews. To generalize the results to different 

cultures, countries and modes of functioning, an 

exploratory phase was undertaken to gather general 

information on the work situations and their 

organization. It was performed in Danish and French 

(CHU de Lille and CH of Denain) hospitals. 

Observations were supported by video recording 

and/or handwritten time-stamped, detailed field notes. 

Preliminary interviews were carried out with the 

target professional to gather general information about 

the work situation (global organization, technical 

system, habits of work, etc.). In the Denmark 

Hospital, the observations were supported by videorecording. 

In the French hospitals, as the HF team has 

been working on a daily basis in the CHU de Lille and 

involved in a great number of IT projects for over 7 

years, always using the same methods, a number of 

10 


observation notes from other observations were added 

to the main collection of data. 

Then, the observations and interviews were 

systematized according to the determinants of the 

work situations [4]. 

2.3 Specifications and design requirements 

This step of the requirements determination process 

allowed obtaining a set of common supports which 

could be exchanged with the different stakeholders of 

the PSIP project (users, analysts/designers, managers, 

ergonomists). Several techniques have been used to 

describe and structure collected data: 

1. Some techniques issued from human factors 

domain as quantitative data, SHEL model, and 

taxonomy. 

2. Some techniques issued from software 

engineering and Human Computer Interaction 

domain as UML models 

3. Textual description and mock-ups 

2.4 Evaluation of the design 

Methods for usability evaluation and re-engineering 

have been applied to six prototypes. Given the wide 

variety of prototypes it has not been adequate to apply 

the same evaluation method to all the prototypes. 

Different methods from the partner’s main toolboxes 

have been careful selected and adapted to each 

particular prototype to meet the unique demand of 

each situation. The usability evaluations have been 

performed in different locations and situations ranging 

from simple laboratory tests involving no real users to 

full field studies were the prototype is used in routine 

clinical work [5]. An overview of the relevant method 

for each test approach and the type of users involved 

are shown in Table 1. 

Laboratory 

Field 

Usability 

inspection 

In lab 

No users 

Focus 

groups 

and 

interviews 

End users 

Usability 

test 

In lab 

Simulatio 

n (tasks) 

End users 

Full 

scale 

simulati 

on 

Simulat 

ed 

environ 

ment 

End 

users 

Field 

Clinical study 

(on site 

usability 

study) 

Clinical field 

End users 

Real patients 

cases 

Table 1: Usability evaluation methods used in PSIP 

3. Results 

3.1 Description of work situation 

The descriptions of the work situations allowed 

identifying a total of twelve noticeable “moments” of 

the work situation. Each “moment” described refers to 

either the medication use process or the gathering of 

information about the patient’s state or to 

communications between the professionals: 

1. Ward round 

2. Drugs’ prescription 

3. Control of new prescriptions by the pharmacist 

4. Infusions’ preparation 

5. Medication organizer’s preparation 

6. Infusions’ administration 

7. Drugs’ administration rounds 

8. Laboratory ordering and reporting cycle 

9. Physiological parameters’ measurement 

10. Nurses’ shift handover 

11. Briefing physician / nurse 

12. Transfer of a patient 

The actual safety procedures currently performed by 

the Health care professionals for the prevention of 

ADEs were also analyzed. At a macroscopic level, 

four main types of safety procedures are 

distinguished: 

1. The control of the decision of the physician and of 

its implementation: is the medical decision 

adapted to the patient’s case and how, through the 

prescription, will it be carried out? 

2. The control of the validity of the order according 

to the evolution of the patient’s process: is the 

prescription still adapted to the patient’s condition 

in spite of the patient’s state evolutions? 

3. The control of the correspondence between what 

the patient is getting and what he should get: is 

what the patient is getting similar to what has 

been prescribed for him? 

4. The compliance of the patient about his 

medications’ taking and its control: is the patient 

taking what he is supposed to take? 

3.2 Specifications and requirements 

The initial prescribing step is of critical importance: 

The system should be a clinician partner and support 

the decision making of the physician. Moreover, the 

system should not interrupt the physician during the 


11

prescription and the content and display of the alert 

must be adapted to the clinical context [6]. 

For example, if a rule with a risk of hyponatremia is 

activated, we may have different types of clinical 

contexts. 

Example 1: If the physician prescribed a monitoring 

of the natremia, and the current results of the natremia 

are good, the context is “Risk of hyponatremia, but 

for the moment it’s OK” 

Example 2: if the physician didn’t prescribe the 

monitoring of the natremia, the patient may be in 

danger. The context is “Risk of hyponatremia, and the 

effect is not monitored, BE CARREFUL” 

Figure 5: example of clinical context for a PSIP alert 

The activity is also distributed for the control of the 

medication process. The system will have to support 

the existing collective continuous control and 

monitoring of the drug effects. As a consequence, the 

system should also be a team player and support the 

cooperation between the physician, nurses, 

pharmacists and patients. The system should also 

raise the collective awareness on ADEs occurring in 

the clinical setting. 

3.3 Evaluation of the design 

As the usability evaluations has been performed on all 

the six prototypes in the PSIP project almost the entire 

PSIP team (more than 45 persons) has been involved 

in various degrees in this process. 

Example of the web prototype: It is a web based tool 

that provides ADE information independently of the 

availability of CPOE systems. Through this 

application, healthcare professionals as physicians and 

pharmacists can display patient’s hospitalization data 

and check the medication prescriptions to verify 

whether they triggers a PSIP alert. 

The evaluation started with a usability inspection 

carried out with a heuristic evaluation method. The 

list of problems identified and rated for their severity 

was then presented to the designers / developers of the 

prototype, and several meetings allowed finding 

technical solutions to fix most of the problems. At the 

end of the re-engineering, the ergonomists 

documented the number of problems solved and the 

list of remaining problems (if any). Once the 

prototype considered usable enough (all major 

problems fixed) the re-engineering phase was 

validated, and usability tests were carried out with 

end-users. 

4. Discussion 

We applied a user centred design lifecycle to the PSIP 

project. The focus was on the analysis of the current 

work system and the design of the expected situation 

The expected HF benefits for the PSIP project are: (1) 

tools adapted to the user’s needs (2) a good 

acceptability by the final users (3) products easy to 

learn and to understand (4) The knowledge of PSIP 

really use by the clinicians. 


The research tasks leading to these results have 

received funding from the European Community’s 

Seventh Framework Programme (FP7/2007-2013) 

under grant agreement n° 216130 PSIP (Patient Safety 

through Intelligent Procedures in Medication). 

6. References 

[1] Rasmussen J, Pejtersen AM, Goodstein LP. Cognitive 

systems engineering. New York: Wiley; 1994. 

[2] Meister D. Behavioural Analysis and Measurement 

Methods. New York: Wiley and Sons; 1985. 

[3] Kirwan B, Ainsworth LK. A guide to task analysis. 

Philadelphia: Taylor & Francis; 2001. 

[4] Riccioli C. Leroy N. Pelayo S., The PSIP approach to 

account for human factors in adverse drug events: 

preliminary field studies, Stud Health Technol Inform 

148, 197–205, 2009. 

[5] Kanstrup AM. Christiansen MB. Nøhr C. Four 

principles for user interface design of computerised 

clinical decision support systems. Stud Health Technol 

Inform. 166:65-73, 2011. 

[6] Marcilly R. Leroy N. Luyckx M. Pelayo S. Riccioli C. 

Beuscart-Zéphir MC. Medication Related 

Computerized Decision Support System (CDSS): 

Make it a Clinicians' Partner! Stud Health Technol 

Inform. 166:84-94, 2011. 

12 


Developing Decision Support Services for Patient Medication Safety: 

A Knowledge Engineering Perspective 

Vassilis KOUTKIAS 1 , Vassilis KILINTZIS 1 , Régis BEUSCART 2 , Nicos MAGLAVERAS 1 

1 

Lab of Medical Informatics, Medical School, Aristotle University, Thessaloniki, GREECE, 

email: {bikout, billyk, nicmag}@med.auth.gr 

2 

Lille University Hospital, EA2694, FRANCE, email: regis.beuscart@univ-lille2.fr 

Abstract 

Adverse drug events (ADEs) constitute a major public 

health issue, endangering patients’ safety and 

introducing significant additional healthcare costs. 

The EU-funded project PSIP (Patient Safety through 

Intelligent Procedures in medication) aims to develop 

medication-related Clinical Decision Support Systems 

(CDSS) for ADE prevention. In order to identify the 

origin of preventable ADEs, PSIP employs novel data 

and semantic mining techniques as well as human 

factor analyses, all applied on patient records and 

actual hospital settings available from several clinical 

environments across Europe. This new knowledge 

combined with existing evidence, i.e. drug 

interactions and already identified ADE cases 

reported in the literature, is used to populate a 

Knowledge Base (KB) that constitutes the basis for 

developing contextualized CDSS modules for ADE 

prevention at the point of care. In this paper, we focus 

on the knowledge engineering approach employed 

towards the construction of a flexible Knowledgebased 

System (KBS) that is able to encompass 

updates of the source knowledge and support the 

deployment of medication-related CDSS modules. 

Keywords 

Adverse Drug Event (ADE), ADE Prevention, Clinical 

Decision Support System (CDSS), Medication-related 

CDSS, Knowledge Engineering, Knowledge-based System, 

Patient Medication Safety. 


Adverse Drug Events (ADEs) due to medication 

errors and human factors constitute a major public 

health issue, endangering patients’ safety and causing 

significant healthcare costs [1]. Information 

Technology (IT) is envisioned to play an important 

role towards the reduction of preventable ADEs by 

offering relevant knowledge and decision support 

tools to healthcare professionals along the entire 

prescription - dispensation - administration - 

compliance (PDAC) medication cycle [2]. Although 

there have been several efforts towards this direction, 

their efficiency was impeded by the lack of reliable 

knowledge about ADEs, and the poor ability of such 

solutions to deliver contextualized knowledge focused 

on the problem. The European project PSIP (Patient 

Safety through Intelligent Procedures in medication) 

aims at preventing medication errors and ADEs in 

particular by: (1) facilitating the systematic 

production of epidemiological knowledge on ADEs, 

and (2) improving the PDAC cycle in hospitals by 

introducing sophisticated medication-related Clinical 

Decision Support System (CDSS) modules at the 

point of care based on the above knowledge. 

In particular, the first sub-objective involves 

generating new knowledge on ADEs, in order to 

identify as exactly as possible (per hospital) their 

number, type, consequences and causes. Data and 

semantic mining techniques applied on structured 

hospital databases and data collections of free-texts 

(letters, reports, etc.) respectively, have been 

employed to identify observed ADEs along with their 

frequencies and probabilities, thus giving a better 

understanding of potential risks. The second subobjective 

involves the development of a knowledgebased 

framework integrating the knowledge 

discovered with existing evidence, such as drug 

interactions and already known ADE cases reported in 

the literature. The aim of this framework is the 

delivery of contextualized knowledge fitting the local 

risk parameters in the form of alerts and decision 

support functions to healthcare professionals. 


13

The proposed knowledge framework constitutes the 

backbone of the PSIP platform that is independent of 

existing Clinical Information Systems, such as CPOE 

(Computerized Physician Order Entry) and EHR 

(Electronic Health Record) systems. It is important to 

note that the platform provides the appropriate 

connectivity mechanisms enabling such systems to 

access, exploit and integrate the incorporated 

knowledge in their local context. In this paper, we 

briefly present the overall rationale of PSIP and focus 

on the knowledge engineering approach employed 

towards the construction of a flexible Knowledgebased 

System (KBS) and framework constituting the 

core part of the CDSS modules for ADE prevention. 

2. ADE Prevention: A Systematic, 

Knowledge-based Approach 

The research and development activities of PSIP have 

been organized in the following three stages: (1) 

improvement of knowledge on ADEs, (2) 

development of CDSS modules, and (3) 

contextualization of CDSS modules and integration in 

existing IT solutions and usage. Initially, the available 

patient data have been identified residing at the 

Hospital Information Systems (HIS) of the 

participating healthcare organizations. In this context, 

a common data model has been developed [3], in 

order to exhaustively define the nature, type, and 

possible values of parameters that are potentially 

relevant to ADEs, including drug-related information, 

diagnostic information, lab examination results, 

description of procedures, demographic data, etc. 

The subsequent knowledge discovery activities that 

were applied focused primarily on the identification 

of clinical cases that are potential ADEs. In particular, 

the analysis was performed in more than 115,000 

patient records exported from RegionH hospitals 

(Copenhagen, Denmark), the General Hospital of 

Denain (Denain, France), the General Hospital of 

Rouen (Rouen, France), the University and Research 

Hospital of Lille (Lille, France) and the USHATE 

hospital (Sofia, Bulgaria). Among the techniques 

employed for knowledge discovery on ADEs were 

multivariate correspondence analysis, hierarchical 

classifications, decision trees and association rules 

[4]. The outcome of these activities resulted in: (1) a 

set of production rules of the form “IF Condition-1 & 

Condition-2 & ... & Condition-N THEN Effect-X”, 

corresponding to potential ADE signals, and (2) 

characterization of hospital stays according to their 

probability of showing (or being associated) with an 

ADE. 

The outcomes of the knowledge discovery phase have 

been filtered and validated, in order to eliminate 

artifacts or clinically irrelevant results [5]. More 

specifically, the above rules were confronted with the 

existing clinical pharmacological knowledge available 

in the scientific literature and/or in specialized 

information repositories such as the VIDAL® 

knowledge source (expertise available in the PSIP 

Consortium). In addition, specialist physicians or 

clinical pharmacologists extensively reviewed these 

data, in order to determine whether the corresponding 

patient stays present or not potential ADEs. As most 

of the abnormal stays were also attached to one or 

several rules, their review by human experts also 

allowed for assessing the capacity of the clinicians to 

understand the discovered knowledge, and the 

contextual clinical relevance of the rule(s) attached to 

each reviewed stay. 

Following the knowledge discovery and validation 

phase, a knowledge engineering framework has been 

established to articulate and electronically encode this 

knowledge, so that it may be efficiently exploited and 

incorporated in CDSS modules. 

3. Knowledge Engineering 

Knowledge employed in PSIP can be considered as 

belonging in three categories [6]: a) domain 

knowledge, in terms of types and facts, which is 

generally static and structured via concepts, relations 

– associations, attributes, and rule types (expressions); 

b) task knowledge, in terms of functional 

decomposition, and control; in this regard, knowledge 

is elaborated with respect to combination of tasks to 

reach a goal/workflow, or oppositely, decomposition 

of complex tasks into separate processes; c) inference 

knowledge, in terms of basic reasoning steps that can 

be made in the domain and are applied by tasks. 

The elaborated knowledge model architecture is 

depicted in Fig. 1. Roughly, its components are 

discriminated into the following categories: 

a) Drugs: Defines all possible drugs, containing also 

their categories and subcategories, based on the ATC 

(Anatomical Therapeutic Chemical) standard 

classification. 

b) Diagnosis: Defines medical conditions to be used 

as input parameters for identifying possible ADEs 

14 


ased on the ICD-10 (International Classification of 

Diseases) standard classification. 

c) Lab results: Defines the terminology for 

expressing lab results in the “Conditions” part of 

ADE rules based on the C-NPU (Committee on 

Nomenclature, Properties and Units) standard 

classification of IUPAC (International Union of Pure 

and Applied Chemistry). 

d) Effects: Ontology-based representation of the 

effects as entities based on attributes containing the 

recommendation, the level of severity, type of risk, 

etc. 

e) Patient parameters: PSIP-specific ontology-based 

representation defining the terminology for expressing 

conditions of the ADE rules. 

f) Context parameters: Set of context-related 

parameters defined to allow future contextualization 

for the CDSS modules. 

g) Procedural Logic – Clinical Protocols and 

Guidelines: Description of clinical procedures, 

protocols and guidelines related to medication that 

aim to enable expressing knowledge related to human 

factors and complex ADE rules. 

h) ADE rules: The core component encapsulating 

knowledge about potential ADEs in the form of rules 

associating a number of conditions to an effect. 

Figure 1. Components of the knowledge model. 

According to the above, the PSIP knowledge model is 

defined as a set of ontology-based structures, either 

specific to the problem domain or standard 

classifications to be used as terminology. In addition, 

a rule-based component is included that is defined via 

a set of classes and populated with rules. The 

ontology-based structures and the rule-based 

component constitute the fundamental elements to 

define complex procedural logic in terms of protocols 

and guidelines, following an electronic formalism, 

i.e., the guideline modeling component. The 

conceptual description of the knowledge model is 

thoroughly presented in [6]. 

4. Implementation and Results 

Figure 2 depicts the overall knowledge engineering 

and CDSS development procedures followed. Taking 

into account the requirements posed by the knowledge 

sources considered in the project and a state-of-the-art 

analysis, GASTON (http://www.medecs.nl/) was 

selected as the KBS/CDSS platform. The core of 

GASTON consists of a guideline representation 

formalism relying on a combination of knowledge 

representation approaches and concepts, i.e., 

primitives, problem-solving methods (PSMs), and 

ontologies. This formalism uses ontologies as an 

underlying mechanism to represent guidelines in 

terms of PSMs and primitives in a consistent way. 

All the rules generated from the knowledge discovery 

activities have been implemented in the respective 

Knowledge Base (KB), in subsequent development 

iterations, while standard and problem-specific 

terminologies have been also incorporated and 

defined, respectively. Aiming to avoid potential 

ambiguities in the description of terms/concepts 

contained in the discovered knowledge, an XML 

schema has been defined and agreed upon for 

knowledge exchange among the knowledge discovery 

team and the knowledge authors participating in PSIP. 

In addition, appropriate inference mechanisms were 

developed, in order to filter rules and eliminate over 

alerting by applying contextual criteria. Aiming to 

provide a flexible KBS that can constitute the basis 

for deploying CDSS modules, various knowledge 

management tools were also developed. In particular, 

the construction of the KB and the generation of the 

corresponding CDSS modules are performed via fully 

automatic procedures using the source XML files. The 

procedure includes syntactic verification of the source 

files against the defined terminology, the population 

of the KB with instances (data, meta-data, rules, etc.) 

and the generation of CDSS modules capable of 

providing up to date knowledge on ADE prevention 

that corresponds to the local context (language, 

statistical significance of rules, etc). 


15

Figure 2. Knowledge engineering and CDSS development procedures followed. 

5. Conclusion 

Identification of ADEs constitutes a challenging 

problem, considering the complexity of medical 

information that has to be processed for ADE signals 

discovery, as well as the validation of this new 

knowledge with respect to its medical/clinical gravity. 

PSIP employed a systematic approach towards the 

identification, validation, and efficient representation 

of ADE-related knowledge. The ultimate goal of our 

research involved the development of flexible, 

contextualized CDSS modules for preventing ADEs 

in the entire PDAC medication chain at the point of 

care. To this end, the decision support services 

deployed are available for exploitation to clinical IT 

systems via a highly interoperable, scalable, and 

robust platform that provides the appropriate 

connectivity mechanisms enabling systems such as 

CPOE to access and integrate the relevant knowledge 

and functionality in their local context. Major 

challenges for future development include the 

incorporation of more complex knowledge sources in 

the KB, i.e., knowledge related to human factors and 

tacit knowledge, as well as exhaustive computational 

testing of the CDSS modules in real and demanding 

environments. 


The research leading to these results has received 


Framework Programme (FP7/2007-2013) under grant 

agreement n° 216130 PSIP (Patient Safety through 

Intelligent Procedures in medication), and has been 

performed by the PSIP Consortium. 

7. References 

[1] L.T. Kohn, J. Corrigan, and M. S. Donaldson, “To 

err is human: Building a safer health system”, 

National Academy Press, 2000. 

[2] D.W. Bates et al., “Detecting adverse events using 

Information Technology”, J Am Med Inform 

Assoc., vol. 10, no. 2, 2003, pp. 115–128. 

[3] E. Chazard et al., “Detection of adverse drug 

events: Proposal of a data model”, Stud Health 

Technol Inform., vol. 148, pp. 63–74, 2009. 

[4] E. Chazard et al., “Detection of adverse drug 

events: Data aggregation and data mining”, Stud 

Health Technol Inform., vol. 148, pp. 75–84, 

2009. 

[5] N. Leroy et al., “Human factors methods to 

support the experts’ review of automatically 

detected adverse drug events”, Stud Health 

Technol Inform., vol. 150, pp. 542–546, 2009. 

[6] V. Koutkias et al., Constructing clinical decision 

support systems for adverse drug event 

prevention: a knowledge-based approach, AMIA 

Annu. Symp. Proc. (2010) 402–406. 

16 


Comparison of two methods for French Drug names extraction 

Suzanne Pereira 1 , Catherine Letord 1 , Sophie Tessier 1 , Solenne Rein 1 , Stefan Darmoni 2 , Elisabeth Serrot 1 

1 VIDAL, Issy-Les-Moulineaux, suzanne.pereira@vidal.fr & elisabeth.serrot@vidal.fr 

2 

CISMeF, Rouen, stefan.darmoni@chu-rouen.fr 

Abstract 

The paper describes two methods for drug names extraction 

for French medical records. We compared two approaches 

integrated in the FMTI (French multi-terminological and 

multilingual indexer) tool. The article summarizes the 

evaluations performed. 

Keywords 

Abstracting and Indexing/methods; algorithms; Information 

Storage and Retrieval/methods; Evaluation Study, France; 

Natural Language processing; Medical records; Drug 

therapy. 


Representing medical record data using standardized 

terminologies facilitates medical practice, information 

retrieval and costs management [1, 2]. 

Structured drug data is needed for drug adverse events 

automatic detection. 

In some hospitals, information on drug prescriptions 

are available in the computerized physician order 

entry (CPOE). Unfortunately, all hospitals are not 

equipped (e.g. CHRU Lille and CHU Rouen) in this 

case the documents written in free text contain most 

of the drug prescriptions: hospitalization reports, 

letters, test results, etc. 

An extractor can extract this information based on the 

text (see figure 1). We have tested this idea during the 

PSIP project (Patient Safety through Intelligent 

Procedures in medication) [3, 4], using FMTI, an 

automatic extracting tool enable to extract, from 

documents, terms of several terminologies. 

The paper is structured as follows: Section 1 we begin 

with the study of existing similar works. Section 2 

describes our system and the evaluation of its 

performances for drug data extraction. Finally we 

discuss the results and conclude. 


Extraction of drug treatments in the hospital records 

and discharge summaries is a complex task. Indeed, 

drug treatments can be described in the reports using 

the brand names, the generic names, the substances, 

and abbreviations in any section of the document 

(Background, Evolution, Conclusion, Treatment, 

etc.).. Moreover, unlike most of the names of 

diseases, drug names or substances don’t follow the 

French syntax. The task is made even more difficult 

for these names hard to spell which cause the 

appearance of numerous misspellings in the 

documents. The natural language processing must be 

very specific here. 

Few computational approaches have been exploited 

until now: 

• Approaches based on lexicons have been explored. 

For example, Levin et al. [5] and Sirohi et al. [6], 

Evans et al. [7] developed a system based on 

lexicons (drug names and abbreviations). 

• Approaches based on regular expressions (names 

of generic or brand names) [7]. 

• Approaches based on phonemization algorithms to 

handle spelling errors [5]. 

Figure1. Example of a French discharge summary 


17

and the corresponding extraction of the drug names 

3. FMTI 

3.1.1 FMTI background 

F-MTI© 1 integrates 26 terminologies (ICD10, 

SNOMED3.5, ICD9, CCAM…) and the mappings 

between them, in 6 languages (French, English, 

Danish, Portuguese, Spanish, Deutch). 

For the PSIP project, VIDAL integrated four new 

terminologies and their mappings devoted to drugs 

into F-MTI’s knowledge sources: the Anatomical, 

therapeutical and chemical Classification (ATC) 

(N=5 514), the substances INN (N=2 942), the brand 

names (N=11,922) including all the abbreviations and 

generical names provided by the VIDAL Company 

and the MeSH Supplementary Concepts for chemical 

substances and pharmacological action terms 

translated into French by the CISMeF team 

(N=6 505). 

Called via a Web API, FMTI enables to extract 

automatically, from documents, terms of several 

terminologies. The indexing is performed in less than 

one second for a discharge summary. 

This tool was already evaluated for Web document 

indexing using MeSH [8], discharge summaries 

indexing using ICD10 [9] and SNOMED3.5 [10]. For 

drugs, the system was evaluated for therapeutical data 

extraction in SPC [11]. 

In this article, we present FMTI performances for 

drug names extraction from discharge summaries in 

2009 and 2010. The first evaluation was published in 

2009 [12]. 

3.1.2 FMTI system description 

The basic method of Bag of Words algorithm was 

using only stemming for normalization purpose [8]: 

For each discharge letters, the document is first 

broken into sentences. Then each sentence is 

normalized (accents are removed, all words are 

switched to lower case and stemmed…) and stop 

words are removed to form a bag of words containing 

all the content words. The “bag” thus obtained is 

matched independently of the order of the words 

against all the different terminologies terms that have 

been processed in the same way. All terms formed 

1 

FMTI is the proprietary of the VIDAL Company. 

with at least one word of the sentence are retrieved. 

VIDAL team worked on a phonemization algorithm 

to replace stemming, improve the system and solve 

some mistakes found in the previous study. Indeed, F- 

MTI encountered difficulties to recognize brand 

names due to incorrect spelling of the names in the 

discharge summaries. Some brand names are written 

improperly with dash ("-") or underscore ("_") or with 

an incorrect space " " (e.g. di-antalvic, diffu k, di 

hydan, cacit D, calcidose vit D, co renitec). On the 

contrary, some brand names were written without 

dash ("-") or underscore ("_") or space (" "), as 

normally they should have to (e.g. chibroproscar 

instead of chibro-proscar; bipreterax instead of 

bipreterax). 

Brand names are particular as they don’t follow the 

French syntax. Band Names can have English or Latin 

origins and in the spelling they don’t mean anything 

at all. That’s why using stemming (removing suffixes 

from the word) is logical for terms like those from 

ICD10 but not for Brand names. Another alternative 

of using the exact word is to use a phonemization 

algorithm. Phonemization [13] is based of the 

pronunciation of a word dependant of the language. 

The word is transformed into the corresponding list of 

phonemes. This method enables to take into account 

some incorrect spellings like “Eupanthyl” and 

“Eupantil”. 

The algorithm was significantly changed to address 

the specific purpose of brand names recognition. 

Specific English, Latin and Roman pronunciations 

were taken into account (“free” equal “fri”, “chol” 

equal “kol”, “I” equal “un”, etc.). Some abbreviations 

were added (“vit” equal “vitamin”), single letters 

(“PH”) were kept. The problems of dash, underscore 

and space were solved via alternative spellings. Some 

common words were removed (the brand name 

“PAR” equal to “par”). And finally some items of the 

discharge summary were not taken into account: “way 

of life” and “exams” that can produce some noise in 

the indexing. 

4. Evaluation 

In this study, the objective is to determine (a) the 

performances of FMTI using a stemming algorithm 

and (b) the gain of the use of a phonemization 

algorithm for extracting drug names from documents 

narrative. 

18 


For this purpose, 50 discharge summaries were 

extracted from the corpus of 4,000 discharge 

summaries and letters used in the PSIP project to 

evaluate this task. 

Three key metrics were used to show the 

performances of the automatic extraction produced by 

FMTI compared to the reference, the manual 

identification: Precision, Recall and F-measure. 

5. Evaluation Results and Discussion 

5.1 Drug names extraction using the stemming 

algorithm: Agreement between FMTI and experts 

FMTI, using the stemming algorithm, enabled to 

index 50 French discharge summaries. Automatic 

indexing was manually analyzed by a CISMeF expert 

pharmacist in Rouen [13]. 

• Overall precision is P = 0.84 

• Overall recall is R = 0.93 

• F-measure is F = 0.88 

5.2 Drug names extraction using the phonemization 

algorithm: Agreement between FMTI and experts 

FMTI, using the phonemization algorithm, enabled to 

index 50 French discharge summaries. Automatic 

indexing was manually analyzed by a VIDAL expert 

pharmacist in Issy-Les-Moulineaux. 

• Overall precision is P = 0.97 

• Overall recall is R = 0.95 

• F-measure is F = 0.96 

5.3 Discussion 

The results showed that the F-MTI tool obtained 

better results when using the customized 

phonemization algorithm than when using the 

stemming algorithm: plus 0.13 points for precision 

and plus 0.02 points for recall. 

The phonemization algorithm enabled to solve some 

problems linked with incorrect spelling of the brand 

names in the discharge summaries: dash ("-"), 

underscore ("_") and incorrect space " " (e.g. diantalvic, 

diffu k, di hydan, co renitec, chibro-proscar; 

bipreterax instead of bipreterax etc.). But we could 

not resolve the problem of words written with a dash 

in the discharge summary and written with no dash in 

the terminology. 

It also enabled to solve some misspellings or 

mistyping problems (e.g. ketoderme instead of 

ketoderm and dextropropoxifene instead of 

dextropropoxyfene). Some other misspellings quite 

frequent (e.g. triapridal instead of tiapridal, 

genopevaryl instead of gynopevaril, piperacetam 

instead of piracetam) could not be solved by this 

method. VIDAL team is working on similarity 

measures to improve the system. 

Some terms are ambiguous (e.g. albumin, which is 

both a laboratory result and a drug name). We will 

add a disambiguation algorithm to avoid the resulting 

noise. 

Finally, the phonemization algorithm is dependant of 

the language used. A specific algorithm is needed for 

each language used in FMTI. 

6. Conclusion 

We developed a multi-terminological and a multi 

lingual extractor useful for extracting drug names 

from free text. The evaluation showed that FMTI 

using a lexicon and a robust phonemization algorithm 

obtained very good results: 97% of precision, 95% of 

recall and 96% of F-measure. This is encouraging for 

our project PSIP, for which we have shown that in the 

absence of computerized prescribing system, FMTI 

could help extract drug information from the 

electronic patient record. After some improvements 

we hope to see FMTI integrated in an electronic 

health record system. 







8. References 

[1] Campbell JR, Carpenter P, Sneiderman C, Cohn S, 

Chute C, Warren J. Phase II Evaluation of Clinical 

Coding Schemes : Completeness, Taxonomy, 

Mapping, Definitions and clarity. JAMIA , 1997, pp. 

238-251. 

[2] Wasserman H, Wang J. An applied evaluation of 

SNOMED CT as a clinical vocabulary for the 

computerized diagnosis and problem list. AMIA 


19

Annu Symp Proc 2003; 699–703. 

[3] Patient Safety by Intelligent Procedures in 

medication. See : http://www.psip-project.eu [october 

2010]. 

[4] Beuscart R, McNair P, Brender J. Patient safety 

through intelligent procedures in medication: the 

PSIP project. Stud Health Technol Inform. 2009 ; 

148 : 6–13. 

[5] Levin MA, Krol M, Doshi AM, and Reich DL. 

Extraction and mapping of drug names from free text 

to a standardized nomenclature. AMIA Annu Symp 

Proc 2007; 438–42. 

[6] Sirohi E, and Peissig P. Study of effect of drug 

lexicons on medication extraction from electronic 

medical records. Pac Symp Biocomput 2005; 308–18. 

[7] Evans DA, Brownlow ND, Hersh WR and Campbell 

EM. Automating concept identification in the 

electronic medical record: an experiment in 

extracting dosage information. Proc AMIA Annu Fall 

Symp 1996; 388–92. 

[8] Pereira S, Sakji S, Névéol A, Kergoulay I, Kerdelhué 

G, Serrot E, Joubert M, Darmoni SJ. Abstract multiterminology 

indexing for the assignment of MeSH 

descriptors. AMIA symp. 2009; 521–525. 

[9] Pereira S, Névéol A, Massari P, Joubert M, Darmoni 

SJ. Construction of a semi-automated ICD-10 coding 

help system to optimize medical and economic coding. 

Stud Health Technol Inform 2006; 124: 845–850. 

[10] Pereira S, Massari P, Buemi A, Dahamna B, Serrot E, 

Joubert M, Darmoni SJ. F-MTI : outil d'indexation 

multi-terminologique : application à l'indexation 

automatique de la SNOMED. Risques, technologies 

de l'information pour les pratiques médicales : 

comptes rendus des treizièmes journées francophones 

d'informatique médicale (JFIM), Informatique et 

santé 2009; 17: 57–67. 

[11] Pereira S, Plaisantin B, Korchia M, Rozanes N, Serrot 

E, Joubert M, Darmoni SJ. Automatic construction of 

dictionnaries, application to product characteristics 

indexing. MIE2009, the XXII International conference 

of the european deferation for medical informatics 

2009:150;512-516. 

[12] Merlin B, Chazard E, Pereira S, Serrot E, Sakji S, 

Beuscart R, Darmoni SJ. Can F-MTI semantic-mined 

drug codes be used for Adverse Drug Events detection 

when no CPOE is available? Proc. MEDINFO 2010; 

1025–1029. 

[13] Brouard F. L’art des Soundex. See: 

http://sqlpro.developpez.com/cours/soundex/ 

20 


Automatic Extraction of Entities from Hospital Patient Records 

in Bulgarian Language 

Galia ANGELOVA 

Institute of Information and Communication Technologies (IICT), Bulgarian Academy of Sciences (BAS) 

Block 25A, Acad. Georgi Bontchev Str., Sofia 1113, Bulgaria 

e-mail: galia@lml.bas.bg 

Abstract 

This paper overviews the tasks performed by IICT-BAS in 

the PSIP project. It considers the three experimental 

extractors implemented by the IICT team: for automatic 

mining of (i) ICD-10 diagnoses, (ii) drugs and dosages and 

(iii) values of clinical tests and lab data. The evaluation of 

these components is presented as well. The article also 

discusses issues related to the contextualisation and 

integration of all entities which have been automatically 

extracted from the texts of discharge letters. 

Keywords 

Automatic extraction from free text, contextualisation, 

integration 


The project Patient Safety through Intelligent 

Procedures in Medication (PSIP) investigates novel 

ICT scenarios for reducing the risk of preventable 

Adverse Drug Events (ADEs) in hospital settings. 

PSIP analyses large volumes of patient-related data, 

integrated within a decision-making framework, to 

provide accurate alerting, signalling of risks and 

supporting healthcare professionals in their decision 

making. The availability of large data bases with 

patient information is a must for the successful transferability 

of PSIP to a new hospital. Therefore the 

PSIP validation in USHATE (University Specialised 

Hospital for Active Treatment of Endocrinology 

“Acad. Iv. Penchev”, Medical University – Sofia) 

required to prepare an experimental repository of 

patient data, where ADEs can be discovered. 

It is well known, however, that a variety of important 

medical findings are traditionally stored as free text 

descriptions in the patient-related documentation. 

Many entities are structured in the Hospital 

Information Systems (HIS) – for instance, the drugs 

prescribed to the patients are maintained via the 

Hospital Pharmacy and the values of lab tests are 

automatically entered in predefined fields whenever 

the tests are made in the hospital. Nevertheless the 

textual paragraphs in the hospital Patient Records 

(PRs) contain essential information that needs to be 

mined, extracted and structured. The task of IICT- 

BAS in PSIP was focused on the extraction challenge 

in order to compliment the entities available in the 

HIS of USHATE. The team has developed software 

modules which mine the free texts of PR discharge 

letters to identify entities important for PSIP: 

• a drug extraction component, which recognises 

drug names and dosages and juxtaposes the 

corresponding ATC code to each drug name; 

• a component for extraction of ICD-10 codes, 

which assigns ICD-10 codes to accompanying 

diseases enumerated in the PR text (the principal 

diagnose is inserted in the HIS), and 

• a component for extraction of values of clinical 

tests and lab data which delivers information 

about the tests made outside USHATE (these 

values are typed in the PR text paragraphs). 

This article briefly describes the extractors, presents 

the extraction accuracy and discusses the integration 

and contextualisation aspects. 

2. Automatic Extraction of Entities from 

the Free Texts of Discharge Letters 

The performance of Information Extraction (IE) from 

clinical narratives gradually improves and currently 

exceeds 90% sensitivity, especially for English texts 

[1]. There are numerous potential IE applications with 

far-reaching effects but the present prototypes are 

mostly used in research settings because of scalability 

and generalisability issues. Nevertheless automatic 

text analysis is viewed as the only means to cope with 

the enormously large amounts of medical information 

available in textual form [2]. Projects for automatic 

text processing are running for dozens of natural 


21

languages and they all deliver slowly but gradually 

improving components for automatic recognition of 

various kinds of medical entities. Semantic miners for 

French language have been developed in the PSIP 

project too; they mark the state-of-the-art results for 

languages other than English. The French Multi- 

Terminology Indexer F-MTI is applied for the 

automatic detection of ADEs in discharge letters. It 

extracts ATC codes from the free text of French 

discharge letters with accuracy 88% when compared 

to the manual extraction [3, 4]. 

Working for Bulgarian language and performing 

some tasks for the first time (e.g. drug extraction), we 

took advantage of the relatively structured paragraphs 

in the hospital discharge letters. These letters have 

mandatory content which is fixed in the Official State 

Gazette as Article 190(3) of the legal Agreement 

between the National Health Insurance Fund and the 

Bulgarian Medical and Dental Associations [5]. It 

seems that the (somewhat conservative) practice to 

segment the PRs into sections exists in Bulgaria since 

many decades and results from the centralised health 

requirements that were imposed to all state hospitals 

during the last 60-70 years. Table 1 shows the 

percentage of PRs with available standard sections in 

a corpus of 1,300 PRs. 

Section title PRs containing the section 

Diagnoses 100% 

Anamnesis 100% 

Past diseases 88,52% 

Allergies risk factors 43,56% 

Family Medical History 52,22% 

Patient Status 100% 

Lab data, clinical tests 100% 

Examiners’ comments 59,95% 

Debate 100% 

Treatment 26,70% 

Table 1: Percentage of PRs including standard sections 

(which can be automatically split with accuracy 99,99%) 

In this way, when our ICD-10 code extractor analyses 

the discharge letter texts in order to find diagnoses of 

accompanying diseases, it mines the section 

Diagnoses only – i.e. paragraphs of nominal phrases 

which enumerate only names of illnesses or special 

states (e.g. 'deficiency of Vitamin D'). Similarly, the 

drug extractor mines the Anamnesis for accompanying 

drugs; the Anamneses often contain subtitles like e.g. 

“therapy at the admission” or other phrases which 

signal current therapy with high probability. The 

extraction of lab data values is performed over the 

text of the sections “Lab data, clinical tests” which 

contain only this kind of values. Thus the extraction 

algorithms operate on input texts with relatively 

predictable structure and content, they are more 

precise and the system performance for Bulgarian 

texts is very good. The extensive evaluation of these 

extractors was performed on 6200 anonymised PRs of 

USHATE patients, diagnosed with diabetes and other 

endocrine diseases. The results show very high 

extraction accuracy for the three extractors [6, 7]: 

• The drug extraction component identifies 1,537 

drug names in 6200 PRs with accuracy 98,42% 

and dosage with accuracy 93,85%, 

• The component for extraction of ICD-10 codes 

assigns correctly ICD-10 codes to 84,5% of the 

disease names enumerated in the PR section 

'Diagnoses', and 

• The component for extraction of values of 

clinical tests and lab data works with precision of 

98,2% for the corpus of 6200 anonymised PRs. 

The in-depth overview of related work shows that the 

performance of these miners for Bulgarian language is 

comparable to the state-of-the-art achievements in 

biomedical Information Extraction. 

A major difficulty in the implementation was due to 

the lack of electronic linguistic resources for 

Bulgarian language. For instance, the dictionary of 

drugs names in Bulgarian was compiled semiautomatically 

with final manual assignment of 

relevant ATC codes to Bulgarian drug names. The 

USHATE Hospital Pharmacy operates with 1,182 

drugs but occurrences of another 355 drug names 

were found in the discharge letter texts. Thus the 

present version of the drug extractor uses a dictionary 

of 1537 drug names. Similarly, the dictionary of 

measurement units and Bulgarian names of clinical 

tests was compiled from the raw texts in the PSIP 

corpus especially for this project. 

Another complication is related to the established 

medical practice to use terminology in both Bulgarian 

and Latin, mixing Cyrillic and Latin symbols. This 

requires to develop multilingual dictionaries and to 

support long lists of transliteration rules. It turned out 

that considerable amount the efforts, needed for 

transferring PSIP to a new language environment, 

have to be invested in the construction of machinereadable 

vocabulary in the respective language. 

22 


3. Contextualisation and Integration 

Our extractors deliver three types of entities identified 

in the PR texts and we need a strategy how to 

integrate them into coherent data sets with the values, 

available in the USHATE HIS for each patient. The 

diagnoses of accompanying diseases are clearly 

relevant at the hospitalisation moment; similarly, all 

clinical tests made outside USHATE for all patients 

are anchored to Day 0 of the respective hospitalisation 

as these tests are entered as valid ones in the 

corresponding PR sections. Thus the main issue is to 

invent an algorithm for automatic recognition of 

'current drug treatment'. This is a nontrivial task 

which can be tackled only by heuristics collected on a 

representative PR corpus. 

Drug names as tokens participate in all PR sections, 

including Diagnoses (e.g. 'Amiodaron-induced 

hypothyroidism'). We have found 355 drugs in the 

experimental corpus of 6200 PRs which might be 

taken by the patients for accompanying and chronic 

diseases (in addition to the 1182 drugs that are in use 

in USHATE during the period relevant for our 

experiment). Table 2 shows the frequency of 

occurrence of these 'external' 355 drugs in the PR 

sections. So we need to know which drugs, brought to 

the hospital by the patient, are taken at the moment of 

hospitalisation and during the hospital stay. Table 3 

shows the accuracy of recognition of these 355 drug 

occurrences in the PR texts. Some percentage of 

hypothetical or future treatment discussions for these 

drugs is wrongly considered as actual one (6% overgeneration). 

PR sections 

Percentage 

In the Anamnesis under header 

“Accompanying treatment” 

6% 

As prescription by External medical 

examiner (e.g. gynecologist) 

26% 

In the Debate and Treatment 68% 

Table 2: Distribution of occurrences of 355 'external' drugs 

in the PR sections 

Evaluated feature 

Percentage 

Precision 88% 

Sensitivity 92,45% 

Over-generation 6% 

Table 3: Accuracy of extraction of medication events 

related to mentioning 355 'external' drugs in the PR texts 

Heuristic observations of the local context explicate 

the typical phrases signaling descriptions of drug 

treatments. Most often the text expression, signalling 

the treatment at Day 0, is the phrase “at the 

admission” (при постъпването). This phrase occurs 

with slight variations in 2122 PRs (34%). On average 

25% of all drugs in the Anamnesis are recognised as 

“Day 0 medication”. The second preferred text 

expression is “at the moment” (в момента... ). It 

occurs in 908 PRs from the test set of 6,200 PRs; in 

703 PRs (77% of all occurrences) the phrase refers to 

explicitly specified drugs in the local context. The 

heuristic observations are based on automatically 

prepared concordances of local contexts but the final 

inspection is manual, e.g. the abovementioned 703 

cases were encountered after reading and marking of 

the list of all 908 occurrences. Please note that these 

typical expressions can be used in other phrases as 

well, e.g. “therapy at the admission: none”, “at the 

moment without complains” hence the training phase 

of our algorithms includes learning of positive and 

negative examples. 

The three extractors, implemented by IICT-BAS, 

were run on ,200 anonymised PRs. They have 

delivered numerous data items to the PSIP repository 

which has been constructed in USHATE (see Table 

4). Many values of clinical tests and lab data were 

available in the USHATE HIS, so the automatic 

extraction contributed mostly values of hormonal tests 

(which are often made outside the hospital). These 

entities were integrated with the HIS data to constitute 

the PSIP repository; and the repository was used for 

discovery of USHATE-specific ADEs. In all cases of 

overlapping descriptions the HIS data are preferred as 

more exact and reliable. Actually the extractors, 

developed for the Bulgarian PR texts, provided 

interoperability between the USHATE PRs and the 

PSIP data formats: once structured information is 

extracted from the free texts, it can be recorded in 

various databases using ATC and ICD codes. Further 

details are available in [8, 9]. 

Number 

Entity type of Entities Precision Sensitivity 

extracted from 

the PR texts 

Diagnoses 26,826 97.30% 74.69% 

Drugs 160,892 97.28% 99.59% 

Values of 

lab tests 

114,441 99.99% 99.04% 

Table 4: Entities extracted automatically from free PR texts 


23

The excellent performance of the drug extractor 

enabled its on-line integration in the USHATE 

validation testbed. When storing an Anamnesis in the 

hospital information system, the drug extractor 

automatically displays the list of drugs taken at the 

moment of hospitalisation (see Figure 5 in [10]). The 

validating doctors were quite positive about the 

experimental integration of this extractor as an on-line 

analyser because it provides structured data in a 

convenient format that can be further used for 

prescriptions. However, we note that the extractor 

uses a pre-fixed dictionary of drug names; if a quite 

new drug name appears it will not be recognised. 

Thus the integration of IE modules requires constant 

updates of the system dictionaries. 

4. Conclusion 

Here we have presented a research effort in automatic 

extraction of structured information from hospital 

PRs, performed in order to integrate a data base for 

experimental discovery of ADEs. The repository for 

PSIP validation in USHATE is relatively small but 

sophisticated as it comprises the HIS data as well as 

the entities delivered by the automatic extractors for 

each patient. Our prototypes demonstrate very high 

extraction accuracy which is partly implied by the 

established structure of the discharge letters in 

Bulgarian hospitals. The false or negative results 

(including overgeneration) are an inevitable aspect of 

the IE performance. In our case the small percentage 

of false positive entities is statistically insignificant 

and practically negligible because PSIP has its own 

thresholds for discovery of ADEs and issuing alerts 

(we note that manual human processing of clinical 

texts would also include some percentage of errors). 

In this way our prototypes support the statement that 

the automatic information extraction is a valuable 

component in eHealth projects dealing with focused 

analysis of patient data. 







6. References 

[1] Meystre, S., G. Savova, K. Kipper-Schuler, and J.F. 

Hurdle. Extracting Information from Textual 

Documents in the Electronic Health Record: A Review 

of Recent Research. IMIA Yearbook of Medical 

Informatics, 2008, pp. 138–154. 

[2] Demner-Fushman, D., W. Chapman and C. McDonald. 

What can natural language processing do for clinical 

decision support? Journal of Biomedical Informatics, 

Volume 42, Issue 5, October 2009, pp. 760-772. 

[3] Merlin B., E. Chazard, S. Pereira, E. Serrot, S. Sakji, R. 

Beuscart, and S. Darmoni. Can F-MTI semantic-mined 

drug codes be used for Adverse Drug Events detection 

when no CPOE is available? Stud. Health Technol. 

Inform. 2010, 160 (pt 1), pp. 1025-1029. 

[4] Pereira, S., C. Letord, S. Tessier, S. Rein, S. Darmoni, 

and E. Serrot. Comparison of two methods for French 

Drug names extraction. In R. Beuscart, D. 

Tcharaktchiev, G. Angelova (Eds) Proc. Int. Workshop: 

Patient Safety through Intelligent Procedures in 

Medication, INCOMA, 2011, pp. 17-20. 

[5] National Framework Contract between the National 

Health Insurance Fund, the Bulgarian Medical 

Association and the Bulgarian Dental Association, 

Official State Gazette №106/30.12.2005, updates 

№68/22.08.2006 and №101/15.12.2006, Sofia, 

http://dv.parliament.bg/. 

[6] Tcharaktchiev, D., G. Angelova, S. Boytcheva, Z. 

Angelov, and S. Zacharieva. Completion of Structured 

Patient Descriptions by Semantic Mining. In Koutkias 

V. et al. (Eds), Patient Safety Informatics, Stud. Health 

Technol. Inform. 2011 Vol. 166, IOS Press, 2011, pp. 

260-269. 

[7] Boytcheva, S. Shallow Medication Extraction from 

Hospital Patient Records. In Koutkias V. et al. (Eds), 

Patient Safety Informatics, Stud. Health Techn. Inform. 

2011 Vol. 166, IOS Press, 2011, pp. 119-128. 

[8] Boytcheva, S., D. Tcharaktchiev and G. Angelova. 

Contenxtualisation in Automatic Extraction of Drugs 

from Hospital Patient Records. To appear in Proc. of 

MIE-2011, the 23th Int. Conf. of EFMI, Norway, 28-31 

August 2011, published by IOS Press. 

[9] Boytcheva, S., G. Angelova, Z. Angelov, D. 

Tcharaktchiev, and H. Dimitrov. Integrating Patient- 

Related Facts using Hospital Information System Data 

and Automatic Analysis of Free Text. To appear in the 

Proc. of SALUS (Special track on electronic 

healthcare), International Conference MURPBES: 

Multidisciplinary Research and Practice for Business, 

Enterprise and Health Information Systems, Vienna, 

Austria, 22-26 August 2011, Springer LNCS. 

[10] Dimitrov, H. Testbed for integration of CDSS modules 

and PSIP validation in USHATE MU Sofia. In R. 

Beuscart, D. Tcharaktchiev, G. Angelova (Eds) Proc. 

Int. Workshop: Patient Safety through Intelligent 

Procedures in Medication, INCOMA, 2011, pp. 39-48. 

24 


Automatic extraction of 

CD Codes for Diagnoses and 

ATC Codes and Dosages for Drugs 

Svetla BOYTCHEVA 

Institute of Information and Communication Technologies (IICT), 

Bulgarian Academy of Sciences and 

State University of Library Studies and Information Technologies, 


svetla.boytcheva@gmail.com 

Abstract 

This paper presents approach for automatic 

extraction of diagnoses and medication information 

from discharge letters. The proposed algorithms are 

designed for processing free text document in 

Bulgarian language. Two algorithms were designed 

and implemented: an algorithm for mapping of 

International Classification of Diseases 10 th 

revision (ICD10) to diagnoses extracted from 

patient records (PRs) and an algorithm for 

automatic extraction of drug events and association 

to them of codes from Anatomical Therapeutic 

Chemical (ATC) Classification System. The system 

presented in here was developed and applied in the 

PSIP project for the preparation of an experimental 

repository for PSIP validation from University 

Specialized Hospital for Active Treatment of 

Endocrinology “Acad. I. Penchev” (USHATE) at 

Medical University – Sofia. 

Keywords 

Medical Information Extraction, Semantic mining 


We deal with hospital Patient records (PRs) which are 

anonymized by the hospital information system of the 

University Specialized Hospital for Active Treatment 

of Endocrinology “Acad. I. Penchev” (USHATE) at 

Medical University – Sofia. 

The system presented in here was developed and 

applied in the PSIP project for the preparation of an 

experimental USHATE’s repository for PSIP 

validation [1]. 

PRs in all Bulgarian hospitals have mandatory 

structure, which is published in the Official State 

Gazette within the legal Agreement between the 

Bulgarian Medical Association and the National 

Health Insurance Fund [2]. 

In the hospital PRs, medical terminology is 

recorded in both Bulgarian and/or Latin language. 

There is no preferred language for the terminology so 

the two forms are used like synonyms. 

2. Drug Events Information Extraction 

We have developed automatic procedures [3] for 

analysis of free texts in hospital patient records in 

order to extract information about: drug names; 

dosages; modes; frequency and treatment duration and 

to assign the corresponding ATC code [4] to each 

medication event. 

Actually the system enables extraction of drug 

related information about drugs which are mentioned 

in the PR texts as accompanying medications but are 

not prescribed by the Hospital Pharmacy. 

The list of registered drugs in Bulgaria is provided 

by the Bulgarian Drug Agency [5] and it contains 

about 4,000 drug names and their ATC codes. 

Currently the Hospital Pharmacy operates with 1,537 

medications because USHATE is specialized mostly 

for treatment of diabetic patients. 

Our aim in the PSIP project is to extract 

information about drugs, taken by the patient, which 

are not prescribed via the Hospital Pharmacy. 

There are two versions of the implemented 

algorithms for information extraction of drug events 

from discharge letters in Bulgarian language: 

Retrospective – This version was used in the 

initial preprocessing stage, where major task was 


25

information collection from full text discharge letters 

included in USHATE archive (total 6200 PRs ). The 

implemented system usually uses automatic model for 

analysis of all PRs in experimental repository (Fig. 

10). For this task were extracted drug events from all 

sections of text personal records. In order to 

determine which drugs were used for treatment during 

hospitalization we maintained information from: 

Anamnesis – including history of disease 

and information for previous and current 

treatment; 

Medical Examiners comments – including 

information for treatment of accompanying 

diseases; 

Debate – including information for the 

provided treatment during hospitalization. 

The extracted information concerning drug events 

from these three sections is processed as follows: 

If some drug is mentioned in the Anamnesis 

section, but not discussed further in Debate 

section, this means that treatment with it was 

stopped and we are not concerning this drug as 

“current treatment”. 

If some drug is mentioned in the Anamnesis 

section and discussed further in debate section, 

this means this drug should be marked with 

“current treatment” marker. For such drugs we use 

last mentioned information about their dosage, 

because it can happened at hospital admission 

daily scheme to be changed. In such cases this new 

information is placed in Debate section. If the 

current treatment continues with the same scheme 

 

as admission we use information about dosage 

from Anamnesis section (if any) 

All drugs mentioned in Medical Examiners 

comments section are marked with “current 

treatment” marker, because they are recognized by 

the system with high confidence as present events 

(and not future prescriptions), except cases with 

'stop' and 'replace' phrases. 

Prospective This is the integrated version 

with Hospital information system at USHATE. There 

is also available manual mode of the system mainly 

used for testing and experiments (Fig. 1). For this 

task the algorithm provides runtime information 

extracted from Anamnesis section only, because the 

whole discharge letter is not available during the 

hospital admission. If anamnesis contains section 

“Current treatment”, system extracts information 

available only in this section. In other cases we 

process temporal information concerning history of 

disease and treatment in Anamnesis. All text is 

separated by temporal events markers and segments 

between them are concerned as episodes describing 

diagnoses, treatment, symptoms and complain. The 

negation events are taken in consideration as well. 

Drugs in Anamnesis, only if they are listed under the 

headers 'medication at the moment of hospitalization' 

or 'accompanying treatment'. Some phrases like 

'started treatment with' can be also interpreted as hints 

for 'current medication' but only if they are not 

followed by phrases including 'replaced by' which 

signal past events. 

 

Fig. 1 Manual mode of the system 

Allows processing of a single PR stored as 

text file. The text of PR is opened in section 

(1). After opening the text file, PR is 

automatically separated on sections and the 

text from each section is presented in 

separated tab (2): (i) personal data; (ii) 

diagnoses Diag; (iii) anamnesis; (iv) patient 

statusStatus; (v) lab dataabs; (vi) medical 

examiners comments Consult; (vii) 

discussion Debate; (viii) treatment; and 

(ix) recommendations. The first section 

“personal data” is skipped, because we 

process anonymized PRs. The third section 

“anamnesis” is spited on two tabs – Current 

Treatment (if any) and Anamnesis. The 

information from the last two sections is 

merged and presented in the Treatment tab. 

26 


Fig. 2 Drug events manual analysis 

PR section can be processed separately by 

the system. For the main task “identification 

of current treatment at hospital admission” 

we process information from “Current 

Treatment” section, if available or in other 

case from “Anamnesis”. After choosing 

“Analyze” function from menu bar the 

selected section (1) is processed and 

automatically is generated list with 

recognized drug names within the text. In 

this example from “Current Treatment” 

section are recognized three drugs 

“достинекс” (Dostinex), „лтиоксин” ( 

Thyroxin) and „дилтиазем” (Diltiazem). In 

this case all events are marked as present. 

 

 

 

 

 

 

Fig. 3 Drug events Current Treatment 

Drugs included in the generated list can be 

processed separately by using “Find” button 

and selecting drug name from the list or as 

bunch using “Find All” button. 

In the current example is selected 

“достинекс” (Dostinex) from the list (2). 

The system identifies in the PR text (1) the 

scope of the corresponding drug event and 

presents it in section (3). Then using rules 

and regular expressions is identified 

information about drug name, dosages, 

modes, frequency and treatment duration 

(4). The system assigns the corresponding 

ATC code to this medication event (5) and 

stores information in CSV format (6). 

 

 

 

 

Fig. 4 Drug events disambiguation 

In this example in check box list (5) are 

presented four possible drug packs for „л 

тиоксин” (Thyroxin) with 0.5 mg, 1 mg 

and 50 mg. Because in the PR text (1) the 

recognized dosage is 50 micrograms the 

closest dosage of 0.5 milligrams is selected 

from the list (5) and the ATC code of 

“лтиоксин 0.5 м” (Thyroxin 0.5 mg) 

is associated to this drug event. 

 

 


27

Fig. 5 Drug events daily dosage 

In this example in check box list (5) is 

presented only one possible drug packs for 

„дилтиазем” (Diltiazem). Because there is 

no other options its ATC code is 

automatically is associated to this drug 

event. In the PR text (1) the recognized 

dosage is 2x90 mg per day and in section (4) 

is calculated the daily dosage 180 mg. 

 

 

Fig. Drug events negation arers 

In some cases even there is available 

“Current treatment” section in PR it 

contains negation marker. 

In this example “няма” (No) causes 

association of empty drug events list for 

current treatment at hospital admission. 

Fig. Drug events nanesis 

In this case “Current treatment” section is 

not available and the system processes 

temporal events in “Anamnesis” section. In 

this example as current treatment is 

recognized the sentence “по повод на 

стенокардна симптоматика е хоспитализиран 

в кардиологична клиника, където е започнато 

лечение с оликард и изоптин, което 

продължава и до момента.” (....due to anginal 

symptoms was hospitalized in the cardiology 

clinic, where started treatment with with 

Olicard (Olicard 40 retard) and Isoptin, 

and which continues to date ...) Because for 

Olicard and Isoptin is not presented 

information for dosage we use N/A marker 

and further use DDD (default daily dosage). 

28 


In this case “Current treatment” section is 

not available and the system processes 

temporal events in “Anamnesis” section. In 

this example as current treatment is 

recognized information in the sentence “от 

началото на заболяването на 

интензифицирана схема с новорапид и 

лантус, дозировки при постъпването 

новорапид 6+6+6E, лантус 18E в 22ч.” 

(Since the beginning of the decease with 

intensified regimen of treatment with Lantus and 

NovoRapid, dosages at hospital admission 

NovoRapid 6+6+6E , Lantus 18e at 10 pm.) 

For the scheme 6+6+6E for NovoRapid is 

calculated daily dosage 18E. 

 

From example in Fig.8 for Lantus is 

recognized daily dosage 18E. If some drug name 

is mentioned more than one time in the text, than 

using rules for temporal events analyses we 

choose the dosage for the closest to present time 

marker. 

 

In automatic mode can be processed all PRs 

stored in the selected folder. The result file 

can be generated in CSV format, XML 

format or Excel format. Extracted 

information contains PR ID, ATC, drug 

name and drug pack, daily dosage, mode 

and scope of the recognized drug event from 

the narrative text. 


29

3. Diagnoses Information Extraction 

The Bulgarian hospitals are reimbursed by the 

National Insurance Fund via the “clinical pathways” 

scheme. When a patient is hospitalized, they often 

select from the Hospital Information System (HIS) 

menu one diagnosis which is sufficient for the 

association of the desired clinical pathway to the 

respective patient. Thus most of complementary 

diseases diagnosed by the USHATE medical experts 

are recorded in the personal history as free text. They 

are entered in the Discharge letter section Diagnoses 

as free text and some of them are only mentioned in 

the section Discussion of the discharge letter. 

This developed an approach for automatic 

mapping of International Classification of Diseases 

1th revision (ICD1) [6] to diagnoses extracted 

from discharge letters. The proposed algorithms are 

designed for processing free text document in 

Bulgarian language. 

Diseases are often described in the medical patient 

records as free text using terminology, phrases and 

paraphrases which differ significantly from the ICD 

disease description. than those used in ICD1 

classification. In this way the task of diseases 

recognition (which practically means e.g. assigning 

standardized ICD codes to diseases’ names) is an 

important natural language processing (NLP) 

challenge [8]. 

To solve this task we are using the following 

resources provided in Bulgarian language [6]: 

ICD1 in Excel format 

Index of diseases and pathological states and 

their modifications (Fig. 11). 

Fig. 11 Index of diagnoses, pathological states and their 

modifications 

The other obstacle is mixture of Latin and 

Bulgarian terminology used in free text diagnoses 

presentation. For some of Latin terms is used 

transliteration in Cyrillic. 

This component works in three steps [1,6]: (i) 

shallow text analysis by regular expressions and 

patterns matching, (ii) searching disease names in the 

terminology resource bank medical terminology 

dictionary, list of abbreviations rules and Latin – 

Cyrillic transliteration rules, and (iii) application of 

terminology binding rules manually added by experts. 

The regular expressions, applied at step (i) for shallow 

syntactic analysis, encode grammatical patterns of 

text phrases which describe medication events in the 

particular training corpus (of endocrinology patients 

treated at USHATE). These expressions are extracted 

semiautomatically from the training texts by machine 

learning techniques. 

 

 

Fig. 12 Diagnoses manual analysis 

Allows processing of a single PR stored 

as text file. The text of PR is opened in 

section (1). After opening the text file, PR 

is automatically separated on sections and 

the text from diagnoses section is 

displayed in section (2). After choosing 

“Analyze” function from menu bar the 

extracted text in section (2) is processed 

and automatically is generated list (2) 

with recognized diagnoses within the text. 

 

30 


Fig. 1 1 aignn 

Diagnoses included in the generated list 

(2) can be processed separately by using 

“Find” button and selecting diagnose from 

the list or as bunch using “Find All” 

button. After selection of diagnose from 

list (2) to be processed its name is 

automatically excluded from list (2) and 

displayed in section (3). In the current 

example the selected diagnose “захарен 

диабет тип 1” (Diabetes mellitus type 1) 

is displayed in sections (3). The system 

identifies possible ICD10 codes 

assignments and displays them in list (4) – 

E10 Инсулинозависим захарен диабет 

(Insulindependent diabetes mellitus) 

 

 

 

 

 

Fig. 1 1 aignn 

The data for processed diagnoses from list 

(2) are displayed in list (5) for further 

storage in CVS format text file. 

It is possible the system to identify more 

than one possible codes for assignment, in 

this case different options are displayed in 

list (4) in decreasing order of ranking. The 

most appropriate association is ranked 

first. There are two options – automatic 

assignment of ICD10 code and manual 

assignment. If there are more than one 

options listed in (4), user can choose by 

checkbox the more appropriate code. In 

automatic mode the first ICD10 code 

with highest rank is associated. 

 

 

 

 

 

Fig. 15 Ranking 

In diagnose “овариална поликистоза” 

(Polycystic ovarian syndrome) for Latin 

term “овариална” (ovarian) (яйчници – 

in ulgarian) in 3 chars categories ICD10 

are recognized N83, Q50, C56, D27, E28. 

We exclude C56 and D27 branches, 

because none of “neoplasm”, “carcinoma 

in situ” and “melanoma” terms were 

presented. For further specification we 

search in 4 characters categories for the 

second term “поликистоза” (Polycystic) 

and we set highest rank to E28.2, which 

contains it, for the other codes we 

recognize Q50.3, N83.2 and E28.9 that 

refer to unspecified disorders. 


31

Fig. 16 Latin terminology processing 

In this example the diagnose “струма 

нодоза гр 1б/еутироидес”( Struma Nodosa 

Euthyroides in Latin) (Euthyroid nodular 

goiter in English) is presented using latin 

terminology with transliteration. The first 

term “струма” corresponds to “гуша” 

(goiter) in Bulgarian language. In ICD10 

3 characters categories it corresponds to 

the cluster E00E0. 

 

 

 

 

 

 

 

Fig. 17 Latin terminology processing 

In this example the diagnose 

“феохромицитома” (pheochromocytoma) 

is presented using latin terminology with 

transliteration. This term corresponds to 

“Доброкачествено новообразувание на 

надбъбречна жлеза” (neoplasm of 

Adrenal gland) in Bulgarian language. In 

ICD10 chars categories it corresponds 

to D35.0. The next diagnose “киста 

оварии декстра” (киста на яйчника – in 

Bulgarian, cyst of ovary – in English) () is 

processed similarly to example on Fig. 15 

with assigned code N83.0 Фоликуларна 

киста на яйчника (N83.0 Follicular cyst 

of ovary). “декстра” in Latin means (дясна 

– in Bulgarian, Right – in English) is not 

considered in classification in this case 

Fig. 18 Diagnoses automatic analyses 

In automatic mode can be processed all 

PRs stored in the selected folder. The 

result file can be generated in CSV 

format, XML format or Excel format. 

Extracted information contains PR ID, 

diagnose name from the narrative text in 

PR, ICD10 code, and diagnose name 

according to ICD10 classification. 

32 



In NLP the performance accuracy of text extraction 

procedures usually is measured by the precision 

(percentage of correctly extracted entities as a subset 

of all extracted entities), recall (percentage correctly 

extracted entities as a subset of all entities available in 

the corpus) and their harmonic mean 

easre: F=2*Precision*Recall/(Precision+Recall). 

The experiments were made with a training corpus 

containing 1,300 PRs and the evaluation results are 

obtained using a test corpus, containing 6,200 PRs. In 

the test corpus there are 5,859 PRs with prescribed 

drugs during the hospitalization. The remaining 341 

PRs concern patients hospitalized for clinical 

examinations only; these 341 PRs are excluded from 

the evaluation. 

Evaluation results (Table 1 and Table 2) shows 

high percentage of success in drug name, drug dosage 

and diagnoses recognition in PRs texts. 

Table 1 Extraction sensitivity according to the IE 

performance measures 

Precision Recall core 

Drug ame 97.28% 99.59% 98.42% 

Dose 92.25% 95.51% 93.85% 

Diagnoses 97.3 % 74.68* 84.5% 

Table 2 Extraction sensitivity according to the IE 

performance measures 

Drug ame 

Extracted entities from the PRs text 

16 82 

Diagnoses 26 826 

The major reasons for incorrect recognition of 

medications are: misspelling errors, unrecognized 

drug events for allergies, incorrect detection of 

negation scope, drug events occurrence in other 

context and etc. 

The incorrect assignment of ICD10 codes for 

diagnoses is mainly due to: misspelling errors, 

unrecognized abbreviations, incorrect transliteration 

of Latin terminology and description of specific 

pathological states which is hard to classify according 

to ICD10 even for humans. 

Comparing with systems performing similar task 

in the PSIP project like French MultiTerminology 

Indexer (FMTI) [9], which indexes documentation in 

several health terminologies, we note that despite all 

complications in processing USHATE discharge 

letters the performance of our system is satisfactory. 

FMTI is applied for automatic detection of Adverse 

Drug Events in discharge letters. The extraction of 

ATC codes from the free text of French discharge 

letters is performed with fmeasure 88% when 

compared to the manual extraction; however, 

compared to the CPE content, the fmeasure is 49%. 

The main reason for better performance of our 

systems is that the discharge letters in French seem to 

have no predefined structure, which is available in 

Bulgaria that significantly helps to recognize events. 

During the Third i2b2 Shared Task and Workshop 

“Challenges in Natural Language Processing for 

Clinical Data: Medication Extraction Challenge” [10] 

several semi and unsupervised systems for medical 

information extraction were presented. They report 

result for machine learning base algorithms [11] high 

fmeasure (91.40% for medication and 94.91% for 

dosage). Statistical hybrid methods that combine 

machine learning and rulebased modules [12] report 

fmeasure for medication 89.9% and for dosage 

93.6%. The other approaches for medication 

information recognition are mainly based on 

techniues like information extraction [13], rulebased 

[14], event driven [15] and semantic mining [16] and 

report similar performance. 

For the second task assignment of ICD10 codes 

to diagnoses the results are comparable with recent 

systems as MIDAS (Medical Diagnosis Assistant) [17], 

SynDiATe [18] with about 76% Fmeasure based on 

combination between text parsing and semantic 

information derivation from a Bayesian network, the 

system reported in [19] uses a hybrid approach 

combining examplebased classification and a simple 

but robust classification algorithm (naive Bayes) with 

high performance over 22 millions PRs: fmeasure 

98,2%; for about 48% of the medical records at Mayo 

clinic, another 34% of the records are classified with 

fmeasure 93,1%, and the remaining 18% of the 

records are classified with fmeasure of 58,5%. The 

article [20] compares three machine learning methods 

on radiological reports and points out that the best f 

measure is 77%. 


33

5. Conclusion and Further Work 

The paper presents software modules for PSIP+ 

project which supports the automatic extraction of 

medication information and diagnoses from PR texts 

The Semantic mining modules are strictly oriented 

to Bulgarian language. The plans for their further 

development and application are connected primarily 

to Bulgarian local context. Future enhancements are 

planned for extension of the name and dosage 

recognition rules, to cope with certain specific 

exceptions and section filtering rules. The preliminary 

correction of spell errors and other kinds of typos will 

also increase the IE accuracy. 

For diagnoses recognition task we plan 

improvement of rules for more precise code 

assignments. 




Seventh Framework Programme (FP7/20072013) 



7. References 

[1] Boytcheva, S., D. Tcharaktchiev and G. Angelova. 

Contenxtualisation in Automatic Extraction of Drugs from Hospital 

Patient Records. To appear in the Proc. of MIE2011, the 23th Int. 

Conf. of the European Federation for Medical Informatics, Norway, 

2831 August 2011, published by IOS Press. 

[2] National Framework Contract between the National Health 

Insurance Fund, the Bulgarian Medical Association and the 

Bulgarian Dental Association, Official State Gazette 

№106/30.12.2005, updates №68/22.08.2006 and №101/15.12.2006, 

Sofia, Bulgaria, http://dv.parliament.bg/. 

[3] Boytcheva, S. Shallow Medication Extraction from Hospital Patient 

Records. Stud. Health Technol. Inform. 2011, 166: pp. 260269: pp. 

119128. 

[4] ATC drugs classification, http://atc.thedrugsinfo.com/, last visited 

on 01/06/2011 

[5] Bulgarian Drug Agency, Drug List at 

http://www.bda.bg/images/stories/documents/register/Mp.htm 

[6] National Center of Health Information, 

http://www.nchi.government.bg/download.html 

[7] Boytcheva, S. Assignment of ICD10 Codes to Diagnoses in 

Hospital Patient Records in Bulgarian. In: Alfred, R., G. Angelova 

and H. Pfeiffer (Eds.). Proc. of the Int. Workshop “Extraction of 

Structured Information from Texts in the Biomedical Domain” 

(ESITBioMed 2010), ICCS2010, Malaysia, Published by MIMOS 

BERHAD, ISBN 9789834137137, July 2010, pp. 5666. 

[8] DemnerFushman, D., W. Chapman and C. McDonald. What can 

natural language processing do for clinical decision support? Journal 

of Biomedical Informatics, Volume 42, Issue 5, October 2009, 

(2009), 760772. 

[9] Merlin B., E. Chazard, S. Pereira, E. Serrot, S. Sakji, R. Beuscart, 

and S. Darmoni. Can FMTI semanticmined drug codes be used for 

Adverse Drug Events detection when no CPOE is available? In 

Studies in Health Technology and Informatics, Proceedings of the 

13th World Congress on Medical Informatics, Cape Town, South 

Africa, olume 160, Number pt 1, (2010), 10251029. 

[10] Third i2b2 SharedTask and Workshop Challenges in Natural 

Language Processing for Clinical Data: Medication Extraction 

Challenge”, https://www.i2b2.org/NLP/Medication/. 

[11] Patrick, J. and M. Li. A Cascade Approach to Extracting Medication 

Events. In: Proc. Australian Language Technology Workshop 

(ALTA), (2009), 99103. 

[12] Halgrim, S., F. Xia, I. Solti, E. Cadag, and Ö. Uzuner, Extracting 

medication information from discharge summaries, In Louhi '10 

Proceedings of the AACL HLT 2010 Second Louhi Workshop on 

Text and Data Mining of Health Documents, (2010), 6167. 

[13] Meystre, S. M., G. K. Savova, K. C. KipperSchuler, and J. F. 

Hurdle, Extracting Information from Textual Documents in the 

Electronic Health Record: A Review of Recent Research, IMIA 

Yearbook 2008: Access to Health Information, ol. 1 (2008), 128 

144. 

[14] Evans, D. A., N. D. Brownlowt, W. R. Hersh, and E. M. Campbell. 

Automating Concept Identification in the Electronic Medical 

Record: An Experiment in Extracting Dosage Information. AMIA 

1996 Symposium Proceedings, (1996), 388392. 

[15] Miura, Y., E. Aramaki, T. Ohkuma, M. Tonoike, D. Sugihara, H. 

Masuichi, and K. Ohe, AdverseEffect Relations Extraction from 

Massive Clinical Records, Proceedings of the Second Workshop on 

LP Challenges in the Information Explosion Era (LPIX 2010), 

Coling 2010 Beijing, 2010, 7583. 

[16] Chazard E., C. Preda et al., Delierable D2.3 Results of data 

semantic mining, PSIP Project, URL: https://www.psip 

project.eu/2010 

[17] SotelsekMargalef, A. and J. illenaRomn. MIDAS: An 

InformationExtraction Approach to Medical Text Classification 

(MIDAS: Un enfoque de extracción de información para la 

clasificación de texto médico), Procesamiento del lenguaje Natural 

n. 41, 2008, pp. 97104. 

[18] Hahn, U., K. Schnattinger and K. Markó. Wissensbasiertes Text 

Mining mit SynDiKATe (Knowledge based text mining with 

SynDiKATe). Künstliche Intelligenz, vol. 2, 2002. 

[19] Pakhomov, S., J. Buntrock and C. G. Chute. Automating the 

assignment of diagnosis codes to patient encounters, Journal of 

American Medical Informatics Association, 13, 2006, pp. 51652. 

[20] Coffman, A. and N. Wharton. Clinical Natural Language Processing: 

AutoAssigning ICD 9 Codes. Overview of the Computational 

Medicine Center’s 2007 Medical Natural Language Processing 

Challenge. Available online at 

http://courses.ischool.berkeley.edu/i256/f09/Final%20Projects%20w 

riteups/coffmanwhartonprojectfinal.pdf 

34 




Dimitar TCHARAKTCHIEV 

University Specialised Hospital for Active Treatment of Endocrinology (USHATE) “Acad. Ivan 

Pentchev”, Medical Univeristy – Sofia, Bulgaria, dimitardt@gmail.com 

Abstract 

This paper presents the set up of an experimental repository 

for patients treated in the University Specialised Hospital 

for Active Treatment of Endocrinology (USHATE). The 

Bulgarian repository is aligned with the structure of the 

Global repository developed in the PSIP project. The 

testbed for integration of PSIP Clinical Decision Support 

System (CDSS) modules with the research version of 

USHATE’s Hospital Information System is also discussed. 

Keywords: CDSS integration, HIS, EHR, EMR, CPOE. 


Huge amounts of data are available in Hospital 

Information Systems (HIS), Laboratory Information 

Management Systems (LIMS) and Computer Provider 

Order Entry or Computerized Physician Order Entry 

(CPOE) systems [1]. The Hospital Information 

System of USHATE designed in 1987 includes 

several modules: registration of demographic and 

medico-administrative data (diagnoses, procedures, 

clinical pathways, Diagnosis-Related Groups (DRGs), 

data concerning the admission, transfer and 

discharge), structured vital signs and clinical findings, 

management of the laboratory results ordering and 

reporting, radiology, pharmacy, dietology, clinical 

documentation management etc. The discharge 

summaries are also recorded in the HIS. The patient is 

the centre of the integration of all clinical and 

administrative data. Documentation and messages 

conforming to the rules of UN/EDIFACT are created. 

The standard EN/ISO EN13606 on Electronic 

Healthcare Record (EHR) communication and 

archetype paradigm are implemented [2]. Roger’s 

definition for Minimum Basic Data Set (MBDS) – the 

core of information with most commonly available set 

of items and most extensive range of usages – is 

adopted [2]. 

Our laboratory modules provide different quality 

assurance functions during the data entry – the results 

being audited as they are being placed into the 

databases. The user can be alerted to any unusual 

results and take the proper corrective action before the 

result leave the bench. The results provided on-line by 

the laboratory analyzers undergo the same checks as 

manually entered data. The quality assurance include 

several futures: zones of analyses of lab data (normal, 

pathologic, intermediate), life threatening value 

checks (produce signals, alerts and questions 

concerning the acceptance of the result), analyses of 

all laboratory data in accordance with age, sex, 

method, patient condition, influence of some drugs or 

tests, routine statistical analysis of lab data and 

control results. This complex structure of the 

USHATE HIS provides a solid base for its inclusion 

in the PSIP project. 

2. Development of an experimental 

repository for Bulgarian patients 

The experimental repository for patients treated in the 

University Specialised Hospital for Active Treatment 

of Endocrinology is aligned with the structure of the 

Global PSIP repository, where are stored more than 

90 000 complete Electronic Medical Records (EMR), 

extracted from all 6 participating hospitals (from 

France, Denmark and Bulgaria) [1]. 

Simplified representation of the PSIP data schema is 

presented at the figure 1. 

The extracted information includes demographic data 

(age and sex), data describing the hospital stay – dates 

of the admission, transfer and discharge, the length of 

stay, diagnoses coded using ICD-10, Acts (ICD-9 CM 

codes), Diagnosis Related Groups (DRGs), Major 

Diagnostic Categories (MDCs), patient medication 

codes using ATC classification, laboratory tests, and 

text files of anamnesis, patient status and discharge 

letters. All the data are anonymised before the transfer 

to the scientific databases. 

Most of the values are automatically transferred from 

the USHATE Hospital Information System (HIS) to 

the scientific repository. A significant amount of data 


35

Figure 1. The PSIP data schema of the experimental repository 

is added by three extractors processing a free text of 

discharge letters and providing more information for 

diagnoses, lab tests and medication [3]. The 

importance of the automatic text analysis for accurate 

filling of experimental repository can be demonstrated 

by the following examples: the discharge letter texts 

contain on average 5,26 drugs administrated per 

patient, while in the Hospitalization Patient Record 

are registered only 1,9 drugs prescribed via the 

hospital pharmacy; for 6 200 hospital patient records 

including discharge letters the extractor of ICD-10 

diagnoses finds in the texts 22 667 diagnoses, while 

there are only 9321 diagnoses registered in the 

USHATE Hospital Information system. 

The stored data in the scientific repository in 

USHATE are integrated in the Global PSIP 

Repository through middleware services. 

3. Testbed for CDSS integration 

The Testbed for integration of PSIP Clinical Decision 

Support System (CDSS) modules with the research 

version of USHATE Hospital Information System 

gives the possibility for real-time presentation of the 

generated alerts to the medical practitioners. It alerts 

the medical practitioners about ADE’s during the 

prescription process and whenever a patient file is 

open. Manual ADE’s checks are also possible [4]. On 

figure 2 are presented the relations between some 

modules of the USHATE HIS (research version), 

Drug Information Extraction module (developed by 

the IICT – BAS) and PSIP ADEs Web service. The 

connection of Bulgarian prototype to the CDSS 

Knowledge base is realised using PSIP Connectivity 

Platform and is shown on figure 3. The applied 

messages are based on Simple Object Access Protocol 

and eXtensible Markup Language over Hypertext 

Transfer Protocol (SOAP and XML over HTTP). 

The process of integration of alerts in the prescription 

process is presented on figure 4. 

4. Results and Discussion 

The complex HIS structure, including all the data 

defined in the PSIP “Common Data Model” [1], gives 

the possibility of USHATE to provide to the Global 

PSIP repository 6800 anonymised Electronic Medical 

Records. The natural language processing (NLP) 

considerably enrich recorded in the experimental 

repository medical data. This enables PSIP partners to 

36 


Figure 2. UML representation of the relations between some modules of the USHATE HIS (research version), 

Drug Information Extraction Module (developed by the IICT – BAS) and PSIP ADEs Web service 

Figure 3. The connection of Bulgarian prototype to 

the CDSS Knowledge base 

adequately calculate the frequency of ADEs in 

USHATE and to generate contextualised rules for 

issuing alerts to the medical practitioners. 

The physicians expressed their positive opinion 

concerning the alerts generated by the PSIP CDSS 

Knowledge Base and presented by the Bulgarian 

prototype [5]. No additional input is needed as the 

system operates on data which are automatically 

collected. The doctors support the further 

development of Bulgarian prototype. 

5. Conclusion 

The repository compliant with PSIP model is 

developed and set up in USHATE and includes 

clinical data about Bulgarian patients. The developed 

testbed enables to validate and assess the PSIP 

approach in Bulgarian clinical setting. 

The evaluation results show that the extraction 

accuracy of the applied NLP techniques is similar to 

the state-of-the-art achievements of leading research 

groups in the field [3]. The functionalities of 

Bulgarian prototype are comparable with other 

integrated test environments developed by innovative 

high-tech companies as Medasys and IBM [6]. 








37

Figure 4. The integration of generated alerts in the prescription process 

The USHATE scientists are grateful to all PSIP experts 

who contributed for the smooth integration of the Bulgarian 

achievements into the elaborated PSIP framework and 

especially to the teams of Régis Beuscart, Nicos 

Maglaveras, Jean-François Forget, Elske Ammenwerth and 

Galia Angelova. 

7. References 

[1] Beuscart R. PSIP: An Overview of the Results and 

Clinical Implications. In Koutkias V. et al. (Eds), 

Patient Safety Informatics, Stud. Health Technol. 

Inform. 2011 Vol. 166, IOS Press, 2011, pp. 3-12. 

[2] Tcharaktchev D., A. Dimitrov, G. Dimov et al. 

MEDICA – 9 years of development and use of a 

Clinical Information System in the University Hospital 

of Endocrinology and Gerontology – Sofia. In J. 

Brender et al. (Eds), Medical Informatics Europe ’96, 

IOS Press, 1996, pp. 458–462. 

[3] Tcharaktchiev, D., G. Angelova, S. Boytcheva, Z. 

Angelov, and S. Zacharieva. Completion of Structured 

Patient Descriptions by Semantic Mining. In Koutkias 

V. et al. (Eds), Patient Safety Informatics, Stud. Health 

Technol. Inform. 2011 Vol. 166, 2011, pp. 260-269. 

[4] Dimitrov H. Testbed for integration of CDSS modules 

and PSIP validation in University Specialized Hospital 

for Active Treatment of Endocrinology, Medical 

University – Sofia. In R. Beuscart, D. Tcharaktchiev, 

G. Angelova (Eds) Proc. Int. Workshop: Patient Safety 

through Intelligent Procedures in Medication, 

INCOMA, 2011, pp. 39-48. 

[5] Nechkova-Atanassova, K. Survey of the attitudes of the 

physicians at USHATE hospital towards innovations. 

In R. Beuscart, D. Tcharaktchiev, G. Angelova (Eds) 

Proc. Int. Workshop: Patient Safety through Intelligent 

Procedures in Medication, INCOMA, 2011, pp. 53-56. 

[6] Bernonville S., J. Nies, H. Pedersen et al. Three 

Different Cases of Exploiting Decision Support 

Services for Adverse Dreg Event Prevention. In 

Koutkias V. et al. (Eds), Patient Safety Informatics, 

Stud. Health Technol. Inform. 2011 Vol. 166, IOS 

Press, 2011, pp. 180-188. 

38 


Testbed for integration of CDSS modules and PSIP validation in University 

Specialized Hospital for Active Treatment of Endocrinology, Medical University - 

Sofia 

Hristo DIMITROV 

University Specialised Hospital for Active Treatment of Endocrinology (USHATE) “Acad. Iv. Pentchev” 

Medical Univeristy – Sofia, Bulgaria 

h_dimitrov@yahoo.com 

Abstract 

The paper presents the development of a testbed for 

integration of Clinical Decision Support System 

(CDSS) modules with the research version of 

USHATE Hospital Information System (HIS). These 

CDSS modules enable the validation of the PSIP 

approach in USHATE. The interface which introduces 

the CDSS alerts to the physicians in the validation 

process is described. 

Keywords 

CDSS integration, PSIP validation 


The HIS of the USHATE is organized in modules. 

The first modules, operational in the end of 1987, 

represent the core of necessary functions and make 

possible registration of administrative data, clinical 

data, management of the laboratory results ordering 

and reporting. Later pharmacy module was set up [1]. 

The new version of the system from 2007 is aligned 

with the CEN/ISO EN13606 EHR standard and 

includes the application of published archetypes [2]. 

The system presented here was developed and applied 

in the PSIP project for the preparation of a testbed in 

USHATE to support PSIP results validation. 

Our aim is to demonstrate the possibility to integrate 

PSIP CDSS module in the process of drug 

prescription with Computerized Physician Order 

Entry (CPOE) subsystem, part of the USHATE HIS. 

In this way we can obtain an improvement of the 

quality of clinical activity and patient safety. 

2. CDSS modules integration 

Initially the USHATE experts have considered three 

different ways for integration of PSIP CDSS modules: 

• Real time Adverse Drug Events (ADE’s) 

checks during the prescription process - the 

major benefit of this method of integration is that 

it allows the physicians to be aware of the events 

that may occur even before the actual 

prescription. 

• Automatic and Manual ADE’s checks - 

automatic checks are performed in the USHATE 

HIS whenever a patient file is opened. The 

background check allows the physicians to 

continue their work without having to wait for 

the response of the PSIP CDSS module. The 

automatic check is done only once when a 

patient file is opened. After that additional 

checks may be performed but they are manual. 

• Backend process executing ADE’s checks - the 

idea of this integration is to embed ADE’s 

checks into USHATE drug prescription module. 

Different events will trigger ADE’s checks and 

they will be done in background, independent 

from the user interaction with the system. This 

approach requires availability of additional 

information in the USHATE drug prescription 

module including diagnoses, acts, laboratory test 

results, etc. In this way the CDSS ADE’s alerts 

can be saved into the system and only updated 

when new events occurs (but this required major 

changes in the CPOE - HIS architecture, to 


39

provide the necessary additional information to 

the drug prescription module). 

USHATE implemented two of the presented models 

of integration – during the drugs prescription process 

and in the moment of patient file opening. Manual 

check is also possible. Requests are done in parallel 

with normal execution so they do not disturb the user 

interaction with the system. 

All the examples below illustrate the USHATE testbed environment. No real patients are shown. The 

presented cases are simulated and are only used for the purpose of PSIP integration and testbed validation. 

The full demonstration video can be watched at: http://www.medicalnet-bg.org/psip/ 

After we have successfully logged in into USHATE HIS, we can see the list of hospitalized patients which are 

currently in the hospital using the button marked as “4” on Fig. 1. The window that appears reflects the structure 

of the USHATE clinics with their rooms and beds. The line (rectangle) marked as “1” shows the “Pituitary 

clinic” that we are browsing at the moment and the rectangle marked as “2” shows the room “01” and its beds. 

All this information is represented as a tree view. On Fig. 1 we have selected the patient marked as “3” which is 

currently situated on bed 3 in room 01 in clinic 003 (Pituitary pathology). After double clicking the patient we 

can open his file and see the details of the hospitalization. 

Figure 1. Presentation of hospitalized patients in the clinics of USHATE 

40 


Fig. 2 shows patient clinical details during the current stay. When the window appears it automatically 

triggers an ADE’s check based on the available information till the current moment. If some alerts are found, 

yellow exclamation mark appears in the upper right corner of the window. We can see it marked as “6” on Fig. 2. 

The medical professional can click on the icon and review the alerts. Since the window on Fig. 2 is the main 

interface for patient data manipulation, it contains other vital information as well: 

• Patient names, gender and age (in Rectangle “1”); 

• Principal diagnosis for the current stay coded in ICD 10 (in Rectangle “2”); 

• Diagnostic and treatment plan which is used to enter different diagnoses (preliminary, complications, 

comorbidities and differential) during the stay (in Rectangle “3”); 

• Laboratory tests and their results (in Rectangle “4”); 

• Performed acts coded in ICD 9 (in Rectangle “5”); 

• All the paper documents required during the stay like: status, anamnesis, discussion, discharge letter, 

etc. (in Rectangle “7”); 

• A dropdown menu where the PSIP components (scorecards, web prototypes) are integrated and the 

drugs prescription is performed (in Rectangle “8”). 

Figure 2. Patient details during the current stay in the hospital, with links to the integrated PSIP modules 


41

If we click on the icon marked as “6” on Fig. 2, the window shown on Fig. 3 appears. The screen contains 

found alerts and their descriptions. Rectangle “1” there points to the current medication including drug ATC 

codes, markers denoting the first day of application (0, 0, 2) and drug names in Bulgarian. Rectangle “2” 

represents a short description of the alerts which is oriented to the physician. Rectangle “3” presents the full 

information about the alerts including: 

• the short description for the physician, 

• drug that causes the event, 

• a long description for the physician, 

• confidence value, 

• source of the alert and 

This information is passed from the PSIP CDSS web service to the testbed in USHATE. The received 

information is formatted in HTML, thus it can be easily represented in different forms. For every alert a row 

appears in the zone marked as “2” and full information concerning the alert is presented in the zone marked as 

“3”. 

The ADE found in the present example states: “Кортикостероидите могат да увеличат панкресните 

ензими (Corticosteroids can increase the enzymes of pancreas). This event is triggered by Дехидрокортизон 

(Dehydrocortison) drug”. 

Figure 3. Presentation of automatically triggered ADE’s checks and found alerts 

42 


Fig. 4 shows the integration of PSIP services in the USHATE hospital information system. Four different 

services are included in the dropdown menu described below: 

• An interface providing manual check for ADE’s based on the available information regarding the 

patient till the current moment (Rectangle “1” on Fig. 4); 

• The PSIP Scorecards link directly points to the PSIP web site including the contextualized 

information for participated hospitals and their departments (Rectangle “2”); 

• The PSIP web prototype link in Bulgarian language directly points to the web site for drug check for 

professional usage (Rectangle “3”); 

• The PSIP Prototypes portal link directly points to the web site presenting the PSIP major prototypes 

including the PSIP Patient Component (Rectangle “4”). 

Figure 4. Integration of PSIP service in the USHATE Hospital Information System 


43

To fully describe the patient drug treatment, Fig. 5 presents the process of automatic extraction of drugs from 

the free text of the anamnesis. Rectangle “1” points to the part of the anamnesis that states the current treatment: 

Терапия при постъпването: Дехидрокортизон 5 мг – 1 т/дневно; Л-Тироксин – 75 мкг/дневно; 

(Therapy in the moment of hospitalization: Dehydrocortison 5 mg – 1 tablet per day; L-Thyroxin – 75 mg per 

day). 

Drug extraction is performed automatically when the anamnesis is modified and saved or manually when the 

menu marked by “5” is clicked. When the medication extraction [3] is activated, it opens a new window if it finds 

drugs that have been already prescribed in ambulatory care. This window contains a table with rows for each 

found drug. The information that is presented within the table on Fig. 5 is: drug ATC code, drug name in 

Bulgarian, dosage of the application, units and the context in which the drug was found in the text. In our example 

two drugs were successfully extracted and marked with rectangle “2”. The first column of the table contains a 

checkbox which can be used to discard the drug and not to apply it further during the hospitalization. Button “3” 

is used to validate and apply the selected drugs; button “4” is used to discard the whole operation. 

Figure 5. Automatic extraction of drug therapy at the moment of hospitalization from the free text of the 

anamnesis 

44 


It is also possible to perform ADE check when new drugs are prescribed. This is shown on Fig. 6. The 

process starts with clicking on button “1”: Изписване на медикамент (New drug prescription). After that a new 

window appears where the drug for the prescription has to be selected. In our example Ацетизал (Acetysal) is 

chosen. When the user successfully selects a drug, an automatic ADE check is performed. If some alerts are found 

the same yellow exclamation mark (rectangle “6”) appears in the upper right corner. Again the user can click on 

that icon to review the alerts. Other fields marked as “2”, “3”, “4”, “5”, “7”, and “8” are used to complete the 

prescription process: 

• Start date and hour of the application (Rectangle “2”); 

• Form of the application based on predefined schemas (in this case 2 times daily, Rectangle “3”); 

• Textual description of the admission schema parameters (in this case one tablet 2 times daily in 08:00 

and 20:00, Rectangle “4”); 

• Number of days and last hour of the reception (the number of the days is defined in the schema and 

only visualized on Fig. 6 as a reference, Rectangle “5”); 

• Button “Apply the prescription” (Rectangle “7”); 

• Button “Discard the prescription” (Rectangle “8”). 

Figure 6. ADE check during the drugs prescription process 


45

Fig. 7 shows the alerts found during the prescription shown on Fig. 6. As we can see in our example 

(Rectangle “1”) new alerts have appeared due to the prescription of Ацетизал (Acetysal). Rectangle “2” shows 

the short descriptions of the new ADE alerts and the full description is in the HTML component below. 

Figure 7. Detected ADEs during the prescription of Acetysal 

46 


Fig. 8 shows the alerts generated by the PSIP Web Prototype in Bulgarian when the same clinical and prescription data 

as of a patient presented on fig. 5 and fig. 6 are used. 

Figure 8. Detected ADEs during the prescription of Acetysal presented in PSIP Web Prototype in Bulgarian 


47


The Testbed for integration of CDSS modules 

implemented in USHATE gives the possibility to alert 

the medical experts about ADE’s during the 

prescription process and whenever a patient file is 

open. Manual ADE’s checks are also possible. These 

functionalities are comparable with other integrated 

test environments (Medasys Prototype, IBM 

Prototype and PSIP Web based application) [4]. 

In USHATE, like as other participating hospitals in 

the PSIP Project, the medical professionals are in 

favor of automatic alerts, part of a CPOE system [5]. 

It is important to mention that the physicians working 

at the USHATE are open to innovations concerning 

the patient safety [6]. 

4. Conclusion and Further Work 

Successful integration was performed and CDSS 

module was introduced to be used by the physicians 

in USHATE. 

In some cases physicians continue the drug therapy 

regarding the CDSS alerts. Because they cannot 

discard some of them (stop showing the alerts), alerts 

triggered by new drug prescription can become 

indistinguishable from the old ones. Eventually this 

will lead to over alertness and physicians will 

decrease their attention to the subsequent warning 

messages. To avoid this USHATE drug prescription 

module should locally store CDSS alerts and allow 

the physicians hide some of them (stop showing 

them). This approach requires CDSS module 

integration as a backend process as discussed in 

section 2 of the current report. 





under grant agreement n° 216130 - The PSIP project 

(Patient Safety through Intelligent Procedures in 

Medication). 

6. References 

[1] Tcharaktchev D., A. Dimitrov, G. Dimov et al. 

MEDICA – 9 years of development and use of a 

Clinical Information System in the University Hospital 

of Endocrinology and Gerontology – Sofia. In J. 

Brender et al. (Eds), Medical Informatics Europe ’96, 

IOS Press, 1996, pp. 458–462. 

[2] Tcharaktchev D. Development of experimental 

repository for Bulgarian patients and a testbed for 

integration of PSIP CDSS modules within the USHATE 

hospital information system. In R. Beuscart, D. 

Tcharaktchiev, G. Angelova (Eds). Proc. International 

Workshop: Patient Safety through Intelligent 

Procedures in Medication (PSIP), INCOMA, 

Shoumen, 2011, pp. 35-38. 

[3] Boytcheva S. Shallow Medication Extraction from 

Hospital Patient Records. In Koutkias V. et al. (Eds) 

Patient Safety Informatics, IOS Press, 2011, pp. 119- 

128. 

[4] Bernonville S., J. Nies, H. Pedersen et al. Three 

Different Cases of Exploiting Decision Support 

Services for Adverse Dreg Event Prevention. In 

Koutkias V. et al. (Eds) Patient Safety Informatics, 

IOS Press, 2011, pp. 180-188. 

[5] Ammenwerth E. Expectations and Barriers versus 

cxCDSS-CPOE: A European User Survey. 

International. In R. Beuscart, D. Tcharaktchiev, G. 

Angelova (Eds) Proc. International Workshop: Patient 

Safety through Intelligent Procedures in Medication 

(PSIP), INCOMA, Shoumen, 2011, pp. 49-52. 

[6] Nechkova-Atanassova K. Survey of the attitudes of the 

physicians at USHATE hospital towards innovations. 

In R. Beuscart, D. Tcharaktchiev, G. Angelova (Eds) 

Proc. International Workshop: Patient Safety through 

Intelligent Procedures in Medication (PSIP), 

INCOMA, Shoumen, 2011, pp. 53-56. 

48 


Expectations and Barriers versus cxCDSS-CPOE: 

A European User Survey 

Elske AMMENWERTH 1 , Martin JUNG 2 

1 

University for Health Sciences, Medical Informatics and Technology, 

EWZ 1, 6060 Hall in Tirol, Austria, elske.ammenwerth@umit.at 

2 

University for Health Sciences, Medical Informatics and Technology, 

EWZ 1, 6060 Hall in Tirol, Austria, martin.jung@umit.at 

Abstract 

User acceptance can be crucial for the success of a health 

IT project. We analysed the attitudes of physicians in 

various European hospitals with regard to automatic 

alerting as part of CPOE systems. Overall, 275 physicians 

from five countries participated. Most surveyed doctors are 

in favour of automatic alerts as part of a CPOE system, but 

fear alert overload. Consequently, they prefer an adaption 

of alert presentation with regard to the clinical context. The 

level of CPOE implementation in the hospitals seems to 

play a minor role in the attitude of the physicians. 

Keywords 

CPOE, automatic alerting, user survey, attitudes, 

quantitative study 


The introduction of CPOE systems, and especially the 

introduction of automatic alerts within CPOE 

systems, is a huge organizational change, with impact 

on the clinical workflow and patient safety [1-2]. User 

acceptance is often crucial for the adoption of CPOE 

systems and for the success of the overall 

implementation project [3-4]. 

The objective of this study is to determine the attitude 

of physicians towards decision-support within CPOE 

systems in the PSIP partner hospitals. 


Patient safety is a serious public health issue [5-6]. 

The Council of Europe defines patient safety as “the 

identification, analysis and management of patientrelated 

risks and incidents, in order to make patient 

care safer and minimize harm to patients” [7] (p.8). 

Estimates from the World Health Organization 

(WHO) show that one out of ten patients is harmed 

while receiving hospital care [6]. 

Medication-related events are among the most 

common adverse events [8]. These Adverse Drug 

Events (ADE) are defined as “any injury occurring 

during the patient’s drug therapy and resulting either 

from appropriate care or from unsuitable or 

suboptimal care” [7, p. 1]. ADEs resulting from a 

medication error are considered to be preventable 

ADEs [7, 9]. A review article by von Laue [10] in 

2003 found ADE rates to be ranging from 0.7% to 

6.5% per admission of hospitalized patients. 

The use of computerized physician order entry 

(CPOE) systems can reduce medication errors and 

ADEs [11-12]. CPOE systems can be equipped with 

different levels of clinical decision support [13]. 

However, drug safety alerts generated by CPOE 

systems often show low specificity due to too many 

false-positive warnings [14]. 

Overriding drug safety alerts in CPOE systems is very 

common and occurs in 49–96% of cases [15]. 

Constant over-alerting may cause alert fatigue. Alert 

fatigue is described “as the mental state that is the 

result of too many alerts consuming time and mental 

energy, which can cause important alerts to be 

ignored along with clinically unimportant ones” 

[15, p. 139]. 

Several studies have provided insight into the 

attitudes of prescribers towards drug safety alerts in 

CPOE systems [4, 16-17], but nothing is known about 

the acceptance of CPOE and automatic alerting in the 

PSIP partner hosiptals. 


49

3. Methods 

In 2010, a one-group cross-sectional quantitative 

questionnaire survey was performed in the following 

hospitals: 

• Three Hospitals of Region Hovedstaden, 

Danemark 

• Hospital of Denain, France 

• University Hospital of Novara, Italy 

• University Hospital of Rouen, France 

• University Specialized Hospital of Active 

Treatment of Endocrinology Sofia, Bulgaria 

• Academic Medical Centre of Amsterdam, the 

Netherlands 

All participating hospitals, with the exception of 

Rouen, already had a CPOE system in use, each with 

a different level of decision-support. 

The standardized questionnaire was developed based 

on the published questionnaires by Magnus [17], Hor 

[4] and Ko [16], using a four-point Likert scale. In 

addition, two open questions were added: What would 

you expect as the biggest possible benefits of 

automatic alerting? And what as the biggest possible 

problems? 

The questionnaires were analyzed with descriptive 

statistics. Furthermore, acceptance scores were 

calculated based on a factor analysis. The open 

questions were analyzed by means of a quantitative 

content analysis according to Mayring [18] . 

4. Results 

Overall, 275 physicians completed the questionnaire. 

Table 1 shows the distribution of returned 

questionnaires. 

Hospital 

Distributed 

questionnaires 

Returned 

valid 

questionnaires 

Return 

rate 

Amsterdam 998 78 7.8% 

Copenhagen 207 94 45.4% 

Denain 60 26 43.3% 

Novara 8 5 62.5% 

Rouen 100 41 41% 

Sofia 53 31 58.5% 

Overall 1,426 275 19.3% 

Table 1. Participation and return rate of the survey. 

The overall acceptance of automatic alerts was 

medium to high in all hospitals. The median 

acceptance scores, calculated by the sum of 8 

questions, was between around 24 and around 26 in 

all hospitals (the acceptance score could range from 8 

= minimum to 32 = maximum acceptance). 

Fig. 1 – Fig. 4. present selected results of individual 

questions. They show that most physicians support 

the idea of automatic alerting, and do not feel limited 

in their prescribing freedom. However, over half of 

the participants feel that there may be too many alerts. 

Consequently, the majority says that alerts should be 

filtered according to the clinical context. 

In the free text comments, the problem of alert 

overload was also mentioned by nearly half of the 

participants. 

Figure 1. Answers to the question: “I find automatic 

alerts a useful tool in electronic prescribing” ( “agree” 

and “partly agree” aggregated). 

Figure 2. Answers to the question: “I believe that 

CPOE systems limit my freedom in prescribing” ( 

“agree” and “partly agree” aggregated). 

50 


Figure 3. Answers to the question: “These CPOE 

systems generate too many alerts that are irrelevant 

for the patient” ( “agree” and “partly agree” 

aggregated). 

Figure 4. Answers to the question: “Alerts should be 

filtered according to the context of the clinical 

situation” ( “agree” and “partly agree” aggregated). 

5. Discussion 

The majority of physicians from all study hospitals 

was in favour of automatic alerting as part of CPOE 

systems. Interestingly, the overall opinions did not 

differ much between the hospitals, disregarding the 

differences in organaization, used technology and 

clinical workflow. This can be seen as an indication 

that the basic attitudes are quite independent from the 

organizational and technical context. 

The survey was carried out in six hospitals within the 

European Union; therefore the results reflect an 

international focus. We included hospitals with 

different availablity of CPOE and automatic alerting. 

The translation of the questionnaire was done by nonprofessional 

translators that were familiar with the 

field. 

6. Conclusion 

Most surveyed doctors are in favour of automatic 

alerts as part of a CPOE system, but fear alert 

overload. Consequently, they prefer a differentiation 

according to the importance of the alerts and an 

adaption to the clinical context. The level of CPOE 

implementation in the hospitals seems to play a minor 

role in the attitude of the physicians 

In all hospitals, high acceptance scores were found. 

This supports the idea of introducing further ways of 

decision-support as part of the prescription process. 

No major difference in the acceptances scores 

regarding countries or level of CPOE implementation 

could be spotted, which we see as an indication for a 

good transferability of the PSIP solutions to all 

participating hospitals. However, the possible 

problem of alert overload is a serious issue that should 

be addressed. 







We thank all colleagues in the participating hospitals 

that supported this survey, namely: Jens Barholdy, 

Carlo Cacciabue, Mauro Campanini, Stefan Darmoni, 

Jean Doucet, Laurie Ferret, Giuseppina Ferrotti, 

Monique Jaspers, Maurice Langemeijer, Kitta 

Lawton, Philippe Lecocq, Michel Luycks, Philippe 

Massari, Krassimira Neshkova, Flemming Steen 

Nielsen, Lars Nygård, Costanza Riccioli, Dimitar 

Tcharaktchiev, Jesper Vilandt 

8. References 

[1] Ash J.S., Sittig D.F., Seshadri V., Dykstra R.H., Carpenter 

J.D., Stavri P.Z. Adding insight: a qualitative cross-site study of 

physician order entry. Int J Med Inform. 2005 Aug;74(7-8):623-8. 

[2] Aarts J., Van Der Sijs H. CPOE, Alerts and Workflow: 

Taking Stock of Ten Years Research at Erasmus MC. Stud Health 

Technol Inform. 2009;148:165-9. 

[3] Campbell E.M., Sittig D.F., Ash J.S., Guappone K.P., 

Dykstra R.H. Types of unintended consequences related to 

computerized provider order entry. J Am Med Inform Assoc. 

2006 Sep-Oct;13(5):547-56. 

[4] Hor C.P., O'Donnell J.M., Murphy A.W., O'Brien T., 

Kropmans T.J. General practitioners' attitudes and preparedness 

towards Clinical Decision Support in e-Prescribing (CDS-eP) 


51

adoption in the West of Ireland: a cross sectional study. BMC 

Med Inform Decis Mak. 2010;10:2. 

[5] Byers J., White S., editors. Patient safety: principles and 

practice. New York: Springer; 2004. 

[6] WHO. 10 facts on patient safety. Geneva: World Health 

Organization;; 2010; Available from: 

http://www.who.int/features/factfiles/patient_safety/patient_safety 

_facts/en/index.html. 

[7] Council of Europe. Committee of Experts on Management of 

Safety and Quality in Health Care (SP-SQS) - Expert Group on 

Safe Medication Practices: Glossary of terms related to patient 

and medication safety. . . 2005; Available from: 

http://www.who.int/patientsafety/highlights/COE_patient_and_me 

dication_safety_gl.pdf. Last accessed: April 1st, 2008. 

[8] EC. Communication from the Commission to the European 

Parliament and the Council on patient safety, including the 

prevention and control of healthcare-associated infections. 

Brussels: European Commission; 2008; Available from: 

http://ec.europa.eu/health/ph_systems/docs/patient_com2008_en.p 

df. 

[9] Institute of Medicine, editor. Committee on Identifying and 

Preventing Medication Errors. Aspden P, Wolcott JA, Bootman 

JL, Cronenwett LR, editors. Preventing medication errors. 

Washington, DC: National Academies Press; 2007. 

[10] von Laue N.C., Schwappach D.L., Koeck C.M. The 

epidemiology of preventable adverse drug events: a review of the 

literature. Wien Klin Wochenschr. 2003 Jul 15;115(12):407-15. 

[11] Ammenwerth E., Schnell-Inderst P., Machan C., Siebert U. 

The Effect of Electronic Prescribing on Medication Errors and 

Adverse Drug Events: A Systematic Review J Am Med Inform 

Assoc. 2008;15(5):585-600. 

[12] Hug B.L., Witkowski D.J., Sox C.M., Keohane C.A., Seger 

D.L., Yoon C., et al. Adverse drug event rates in six community 

hospitals and the potential impact of computerized physician 

order entry for prevention. J Gen Intern Med. 2010 Jan;25(1):31- 

8. 

[13] Kuperman G., Bobb A., Payne T., Avery A., Gandhi T., 

Burns G., et al. Medication-related clinical decision support in 

computerized provider order entry systems: a review. J Am Med 

Inform Assoc 2007;14(1):29-40. 

[14] Khajouei R., Jaspers M.W. The impact of CPOE medication 

systems' design aspects on usability, workflow and medication 

orders: a systematic review. Methods Inf Med. 2010;49(1):3-19. 

[15] van der Sijs H., Aarts J., Vulto A., Berg M. Overriding of 

drug safety alerts in computerized physician order entry. J Am 

Med Inform Assoc. 2006 Mar-Apr;13(2):138-47. 

[16] Ko Y., Abarca J., Malone D.C., Dare D.C., Geraets D., 

Houranieh A., et al. Practitioners' views on computerized drugdrug 

interaction alerts in the VA system. J Am Med Inform Assoc. 

2007 Jan-Feb;14(1):56-64. 

[17] Magnus D., Rodgers S., Avery A.J. GPs' views on 

computerized drug interaction alerts: questionnaire survey. J Clin 

Pharm Ther. 2002 Oct;27(5):377-82. 

[18] Mayring M. Qualitative Inhaltsanalyse: Grundlagen und 

Techniken: Utb; 2007. 

52 


Survey of the attitudes of the physicians at the USHATE hospital 

towards innovations 

Krassimira NECHKOVA-ATANASSOVA 1 , 2 

1 

University Specialized Hospital for Active Treatment of Endocrinology “Acad. Ivan Pentchev” (USHATE), 

Medical University Sofia, 2 Zdrave Str., Sofia 1431, Bulgaria, 

2 

Institute for Population and Human Studies, Department of Psychology, Bulgarian Academy of Sciences, 

Block 6, Acad. Georgi Bontchev Str., Sofia 1113, Bulgaria 

e-mail: krassimiran@yahoo.es 

Abstract 

This paper presents a survey of the physicians’ attitudes at 

the University Specialized Hospital for Active Treatment of 

Endocrinology (USHATE), Medical University - Sofia 

towards the introduction of automatic medication safety 

alerts. The great majority of the participants was in favor of 

the PSIP Web prototype in Bulgarian, detecting several 

Adverse Drug Events (ADEs) concerning the prescribed 

medicaments. Integrating the Clinical Decision Support 

System (CDSS) alerts in the process of drug prescription 

using the Computerized Physician Order Entry (CPOE) 

subsystem, part of the USHATE HIS, the extra time for 

data registering is not needed and almost all medical 

professionals expressed their satisfaction and support. 

Keywords 

Innovations, attitudes, automatic medication safety alerts 


One of the goals of the project “Patient Safety through 

Intelligent Procedures in Medication” /PSIP/ is to 

introduce a CDSS for automatic alerts concerning the 

adverse drug events, based on the observation of large 

electronic patient information archives. Because this 

is an innovation, one of the tasks of our team was to 

investigate the attitudes of the medical staff towards 

innovations. 

Attitudes are a major factor in introducing and 

accepting innovations. The attitude is a relatively 

stable habitual internal stance, or the person’s 

predisposition to react in a particular way as a 

precondition of action and experience. In general, it is 

a mediator between the stimulus and the reaction. It is 

the way of being in a particular situation. 

The Concise Oxford Dictionary of Current English [1] 

defines innovation as “introducing novelty” and 

“making changes”. In the literature there are three 

main paradigms of interpreting innovations as defined 

by Linn and Zaltman in their concise review of the 

existing theoretical approaches towards defining 

innovations [2]: 

1. The term “innovation” is used as a synonym 

of invention, of constructing something new 

through a creative process; 

2. Innovation is a process by which an already 

existing creation is accepted by the person 

and becomes a part of her/his knowledge and 

behavior; 

3. Innovation is viewed as a description of 

something material or ideal which has been 

invented and is considered new independent 

from its acceptance or non-acceptance and its 

dissemination. 

One of the most influential theories for studying 

innovations is the theory of cognitive styles of M. 

Kirton [3, 4]. It focuses on the strategies preferred by 

individuals in situations of change. This theory is a 

basis for a well-known empirical method. According 

to Kirton innovative processes start at an individual 

level but often turn to changes on the level of the 

social system. Innovation as a personal characteristic 

should be considered on a continuum between 

adaptation and innovation; accordingly, individuals 

can be regarded as adaptors and innovators. Between 

the two poles – the highly adaptive and the highly 

innovative are situated the different cognitive styles in 

posing and solving problems and the creativity styles. 

Adaptors prefer the already established methods of 

problem solving, and the innovators – the radically 

new ones. 

According to empirical studies described in the 

literature there are representatives of the both 

personality types in every organization. This means 


53

that one part of the group readily and with a great 

interest accepts innovations and starts using them as 

soon as possible, whereas the other part is slow in 

accepting changes and needs more time to adapt to 

innovations. 

The goal of our study was to investigate the attitudes 

and the perceptions of the medical staff at the 

USHATE before and after the introduction of the new 

system for automatic alerts in prescribing. 

2. Method of Study 

We used two questionnaires provided by the PSIP 

partners E. Ammenswerth and her colleagues from 

UMIT (University of Health Sciences, Medical 

Informatics and Technology in Hall, Tirol, Austria) 

[5]. Part of the survey was conducted by Martin Jung 

and Werner Hackl from UMIT. The survey conducted 

at the USHATE was divided into two phases. The 

first phase began in January and ended in March, and 

the second phase began in May and ended in the 

beginning of June. The goal of the phase one was to 

study the physicians’ attitudes towards the automatic 

medication safety alerts and their opinions towards 

systems facilitating the decision making process. The 

second part was conducted when the physicians 

already had some experience with the new system. 

The preparation of the surveys in USHATE included 

the following activities: 

• translation and adaptation of the questionnaires, 

• approbation of the questionnaires, 

• defining the target groups, 

• conducting the research, and 

• processing and analyzing the results. 

We decided to include in the target groups all the 

physicians working at the hospital clinics and 

laboratories. The entire number of the USHATE 

personnel is 169, including 53 physicians. All the 

physicians were invited to fill in the questionnaires. 

So in the first survey the target group consisted of 53 

physicians. Some 33 responded; one part of the 

remaining physicians was on leave, and another part 

refused to collaborate due to a heavy workload. From 

the 33 questionnaires received 31 were filled out 

properly and could be processed; the remaining two 

were invalid. The second target group consisted of 19 

physicians who already had worked with the system. 

To obtain our data we applied two questionnaires. The 

administration of the second questionnaire was 

accompanied by an interview with the participant. 

The survey proceeded in the following way: the 

psychologist handed the questionnaire to each of the 

participants in person, the participant gave answers to 

questions arising, and then the questionnaires were 

collected. 

For analysis of the results descriptive statistic 

methods were used. 

3. Results and Discussion 

On Figure 1 the results of the first item are shown. It 

was the statement “I find automatic alerts a useful 

tool in prescribing”. Some 67,7% of the participants 

responded that they agreed with the implementation 

of an automatic medication safety system; 29% were 

hesitating but rather agree, and 3,2 % disagree. This 

results correspond to previous research showing on 

innovators (those who would accept the innovations 

immediately) as well as adaptors (who prefer the 

traditional way of working) (see Section 1, also [4]). 

Figure 1. Assessment of the usefulness of the automatic 

alerts 

The regression analysis was applied to find out to 

what degree the preferences about the implementation 

of the automatic medication safety alert system were 

influenced by gender. The results are shown at the bar 

chart on Figure 2. 

95,2 % of the women are innovation oriented – 76,2% 

agree that the automatic alerts are a useful tool for 

prescribing, 19,0% partly agree; however 4,8% were 

against. 100 % of the males are innovation oriented - 

50% of the males agree, and exactly the same 

percentage 50% partly agrees. 

54 


90 

80 

70 

60 

50 

40 

30 

20 

10 

0 

I find automatic alerts a useful tool in prescribing 

partly disagree partly agree agree 

Figure 2. Gender influence on preferences 

Male 

Female 

The descriptive statistics (see bar chart on Figure 3) 

shows the impact of age on answering the question 

about the usefulness of the ADE-scorecards as a tool 

to learn about ADEs occurring at the respective 

department. The “ADE Scorecards” is the application 

designed to describe the statistical results and to 

improve the awareness of the healthcare professionals 

on the ADEs occurring in their medical unit or their 

hospital [6]. 

column 3 are presented the partly agreeing medical 

practitioners and in column 2 – the partly disagreeing 

practitioners. 

Altogether the preliminary expectations and the 

attitudes of the physicians at the USHATE hospital 

towards introducing new automatic safety alert 

systems are highly positive. The percentage of 

innovators is significantly higher then the percentage 

of adaptors. 

Some 94,7% of the participants agree with the 

statement „I would recommend using ADE-scorecards 

to my colleagues.”, and only 5,3% agree only partially 

and show reservations towards the system. This 

shows that also the physicians who have reservations 

are in favor of the new system (see Figure 4). 

Figure 4. Positive attitudes towards novel solutions 

Figure 3. Impact of age on the preferences 

It is obvious that the physicians with a longer 

professional experience and age tend to value the 

introduction of the ADE-scorecards higher than their 

younger colleagues. In the column 4 are presented the 

medical professionals agreeing that the ADEscorecards 

are a useful tool to learn about ADEs, in 

This questionnaire contains also two open questions. 

The first question: “What are the biggest benefits of 

ADE scorecards” was answered by 57,9 % of the 

participants. The biggest benefits are the easy access 

to information and reducing the side effects, followed 

by better safety and security for the patients and the 

improvement of clinical work. 42,1% of the 

participants have not answered this question (Figure 

5). 

The second open question was “What are the biggest 

problems of ADE scorecards?” The problem indicated 

most often is the “limited scope”, i.e. the limited 

number of cases at hand to gather the necessary 

information. 


55

Figure 5. Assessment of score cards benefits 

Concerning the use of PSIP Web prototype some 

doctors recommended to reduce “the amount of time 

necessary to put in the information”. 

These results are shown in Table 1. 

Valid 

Missing 

The limited 

scope 

The 

amount of 

time 

necessary 

to put in 

the data 

Frequ- 

ency 

Valid 

Percent 

Percent 

Cumulative 

% 

2 10,5 50,0 50,0 

2 10,5 50,0 100,0 

Total 4 21,1 100,0 

15 78,9 

Total 19 100,0 

Table 1: Problems mentioned by the medical practitioners. 

4. Conclusions 

The results obtained from this survey testify in a 

convincing way that the physicians at the USHATE 

hospital find the automatic safety alert system useful 

and would use it in their daily work. 

The benefits expected from implementing the system 

are: 

• more information concerning the adverse drug 

events of the prescribed therapy, 

• reducing the errors due to inadvertence, 

• quicker access to information, and 

• improved patient safety. 

The physicians who already use the system have made 

the following recommendations: first, to increase the 

number of cases for analysis, and secondly, to reduce 

the amount of time necessary to upload the data in the 

PSIP Web prototype. Integrating the CDSS alerts in 

the process of drug prescription using the CPOE 

system, the extra time for data handling is not needed 

and the medical professionals were satisfied. 

The results show that the physicians working at the 

USHATE hospital are open to innovations concerning 

the patient safety. In our view this is a great advantage 

that could be used for the benefit of the patients. 





under grant agreement n°216130 PSIP project (Patient 

Safety through Intelligent Procedures in Medication). 

We thank all medical professionals from the 

USHATE, participating in this survey, and Martin 

Jung and Werner Hackl from UMIT conducted part of 

the interviews, and all the team of E. Ammenwerth 

(UMIT) providing the standardized questionnaire. 

6. References 

[1] The Concise Oxford Dictionary of Current English, 

1965. Fifth Edition. 

[2] Lin N. and Zaltman G., 1973. Dimensions of 

innovations. – In: Processes and Phenomena of 

SocialChange. New York: Wiley&Sons. 

[3] Kirton M., 1976. Adaptors and Innovators: A 

description and Measure. Journal of Applied 

Psychology, Vol. 61, 5, pp. 622-629. 

[4] Kirton M., 1978. Have adaptors and innovators equal 

levels of creativity? Psychological Reports, 42, pp. 

695-698. 

[5] Ammenwerth, E. and M. Jung. Expectations and 

Barriers versus cxCDSS-CPOE: A European User 

Survey. In R. Beuscart, D. Tcharaktchiev, G. Angelova 

(Eds) Proc. International Workshop: Patient Safety 

through Intelligent Procedures in Medication (PSIP), 

INCOMA, Shoumen, 2011, pp. 49-52. 

[6] Beuscart R. PSIP: An Overview of the Results and 

Clinical Implications. In Koutkias V. et al. (Eds) 

Patient Safety Informatics, Amsterdam, IOS Press, 

2011, pp. 3-12. 

56

INTERNATIONAL WORKSHOP Patient Safety - MedicalNet-BG

Create successful ePaper yourself

Delete template?

Save as template?