pdf 820Kb - INSEAD CALT

The ONTO-LOGGING Consortium consists of (alphabetical order): 

ARCHETYPON,DELTATEC,FZI, INDRA, INSEAD, META4 

Status: Confidentiality: 

[ 

[ 

[ 

[ 

Corporate Ontology Modelling and Management 

System 

X 

] 

] 

] 

] 

Draft 

To be reviewed 

Proposal 

Final / Released to CEC 

Project ID: IST-2000-28293 

Deliverable ID: D8b –version 1.0 

Workpackage No: WP8 

[ 

[ 

[ 

X 

] Public 

] Confidential 

] Restricted 

Title: 

Evaluation report of the use of 

Onto-Logging platform in the 

user site 

Version: 1.0 

Date: 27 january 2004 

Pages: 110 

Responsible Author: INSEAD 

Co-Author(s): INDRA 

- for public use 

- for Onto-logging consortium and Commission 

services 

- restricted to a group specified by the Onto-logging 

consortium and including Commission services 

Title: Evaluation report of the use of Onto-Logging platform in the user site 

Summary / Contents: 

The main goal of this document is to propose a framework for the evaluation, and its 

application for the evaluation of the Ontologging System.

Evaluation report of the use of Onto-Logging 

platform in the user site 

Deliverable ID: D8b 

Table of Contents 

Page : 2 of 110 

Version: 1.0 


Status: Final 

Confid.: Public 

1 Introduction ....................................................................................................................................6 

2 The Principles of the Evaluation.....................................................................................................7 

2.1 Evaluation: Objective & Challenges .........................................................................................7 

2.2 Evaluating What? ....................................................................................................................8 

2.2.1 Levels of evaluation ............................................................................................................8 

2.2.2 The phases of the evaluation: formative, summative and “substantive value”.........................10 

2.3 The Evaluation Methods & Tools ...........................................................................................12 

2.3.1 Evaluation: an overview .....................................................................................................12 

2.3.2 Observational methods ......................................................................................................13 

2.3.3 User’s Opinions methods ...................................................................................................13 

2.3.4 Interpretative methods .......................................................................................................14 

2.3.5 Predictive methods ............................................................................................................14 

2.3.6 Test, Experiment and Usability Engineering methods ..........................................................15 

2.4 Defining the evaluation process ..............................................................................................16 

2.4.1 The definition of the research questions ..............................................................................16 

2.4.2 Selecting the evaluation methods (defining the instrument) ..................................................18 

2.4.3 Executing the instrument and gathering the data ..................................................................19 

2.4.4 Analysing the data, theory confirmation & building .............................................................19 

3 Evaluating the Ontologging System ..............................................................................................21 

3.1 Overview ..............................................................................................................................21 

3.2 Goals and scope of the evaluation of Ontologging ...................................................................22 

3.2.1 Knowledge Management & Ontologies, some clarification ..................................................22 

3.2.2 The goals of Ontologging: the design of a Knowledge Management System that better 

support some knowledge processes ..................................................................................................24 

3.3 Selecting the elements to be evaluated (evaluation criteria) ......................................................25 

3.3.1 Identifying the elements associated to each of the goals .......................................................25 

3.3.2 Levels of evaluation for Ontologging. What could be evaluated............................................27 

3.3.3 The different phases of the evaluation .................................................................................30 

3.3.4 A pre-selection of the elements to be evaluated ...................................................................31 

4 Gathering the Data.......................................................................................................................36 

4.1 Previous work.......................................................................................................................36 

4.1.1 The IR (Information Retrieval) approach ............................................................................36 

4.1.2 The Ontology-based approaches .........................................................................................37 

4.2 The approach used for evaluating Ontologging........................................................................39 

4.3 The definition of the instruments and the gathering of the data .................................................40 

4.3.1 The context of the evaluation .............................................................................................40 

4.3.2 The Questionnaires............................................................................................................41 

4.3.3 Focus groups.....................................................................................................................44 

4.3.4 Interviews.........................................................................................................................45 

4.3.5 Experiments, via Scenarios ................................................................................................46 

4.4 Some directions towards a more quantitative (and rigorous?) evaluation...................................46 

4.4.1 The reason of the qualitative evaluation ..............................................................................46




Page : 3 of 110 

Version: 1.0 


Status: Final 


4.4.2 Some direction for a quantitative evaluation........................................................................46 

4.4.3 Some prospective operationalization of “SMART” quantitative evaluation............................47 

4.4.4 Concluding remarks on the quantitative evaluation ..............................................................49 

5 Results & Analysis........................................................................................................................50 

5.1 Phase 0: State of the situation (formative evaluation) ...............................................................50 

5.1.1 Results..............................................................................................................................50 

5.1.2 Analysis............................................................................................................................53 

5.2 Phase 1: Ontology Building (and some content population) ......................................................54 

5.2.1 Ontology building (at INDRA) ...........................................................................................54 

5.2.2 The reengineering of the Ontology .....................................................................................57 

5.2.3 Some lessons learned. ........................................................................................................60 

5.3 Phases 3: Content population .................................................................................................60 

5.4 Phases 4: Evaluating Ontologging “knowledge retrieval” .........................................................61 

5.4.1 Evaluating of the basic knowledge retrieval.........................................................................62 

5.4.2 Evaluating of the user-centred usages .................................................................................65 

5.5 A Comparison with a more traditional knowledge management system (KnowNet) ...................66 

5.5.1 Description of the evaluation..............................................................................................67 

5.5.2 Comparing the two systems................................................................................................67 

5.5.3 Lessons learned from this comparison.................................................................................67 

5.6 Final words ...........................................................................................................................68 

6 Discussion and Conclusions ..........................................................................................................69 

7 References.....................................................................................................................................72 

8 Annex...........................................................................................................................................75 

8.1 Annex 1: Ontologging Goals & focus. The consortium perspective. ..........................................75 

8.2 Annex 2: Description of INDRA (the main user group)............................................................76 

8.2.1 An overview .....................................................................................................................76 

8.2.2 Activities of Indra’s Competence Centres ...........................................................................77 

8.2.3 The tendering process at INDRA........................................................................................80 

8.2.4 References ........................................................................................................................81 

8.3 Annex 3: Ten Usability Heuristics (Nielsen J. and Molich P.)...................................................81 

8.4 Annex 4: Glossary of terms....................................................................................................82 

8.5 Annex 5: Pre-questionnaire for the participants in the Ontologging usability test .......................83 

8.6 Annex 6: Questionnaire Ontology-based approach for the structuring of knowledge ..................86 

8.7 Annex 7 Spanish questionnaires .............................................................................................89 

8.8 Annex 8 User modelling tools and knowledge distribution agents questionnaires .......................95 

8.9 Annex 9 Ontologging project questionnaire........................................................................... 101




Change Log 

Page : 4 of 110 

Version: 1.0 


Status: Final 


Vers. Date Author Description 

8-April-03 Thierry Nabeth Very First draft version of deliverable (methodology) 

12-Nov-03 Liana Razmerita Works on chapter 4 and integrating preliminary evaluation results 

23-Dec-03 Liana Razmerita Integrating evaluation results and lesson learned 

27-Jan-04 Thierry Nabeth Finalisation and final conclusion 

Acronym/ 

abbreviation 

List of Acronyms and Abbreviations 

Resolution

Executive Summary 




Page : 5 of 110 

Version: 1.0 


Status: Final 


The main goal of this document is to propose a framework for the evaluation, and its 

application for the evaluation of the Ontologging System – a next generation ontology-based 

knowledge management platform - that has been designed as part of the Ontologging IST 

project. 

This document is structured in several sections. 

The first section presents the methodological aspects and principles of the evaluation, and in 

particular describes what are the objective of the evaluation, what are the different phases for 

the evaluation (formative or summative), and what are the different categories of methods 

(observational, users’ opinion, interpretative, etc.) that can be used. This section also 

proposes to use of an adapted version of the Donald Kirkpatrick model, in order to evaluate 

the different levels of efficiency and effectiveness achieved by the solution. 

The second section presents how these principles are to be applied to the concrete evaluation 

of the different components of the Ontologging platform. In particular, this section identifies 

the activities that are going to be evaluated (structuring knowledge, capitalizing knowledge, 

searching knowledge, and sharing knowledge), selects the different methods to be used and 

establishes an action plan. 

The third section (chapter 4) is related to the execution of the action plan of the evaluation. It 

describes the setting of the evaluation (what is the context of the evaluation at Indra, what is 

the profile of the participants of the evaluation). It also presents on the evaluation methods 

used and the elaboration of the evaluation instruments, and collection of the data. 

Then, the fourth section comprises an analysis of the data, and some preliminary results of 

the evaluation. 

Finally, the last section draws the main conclusion of this evaluation, and defines some future 

work in this domain to be conducted.

1 Introduction 




Page : 6 of 110 

Version: 1.0 


Status: Final 


The main goal of this document is to propose and to apply a framework for the evaluation of 

the Ontologging System – a next generation ontology-based knowledge management 

platform– that has been designed as part of the Ontologging IST project. 

This document tries to provide a complete view of the evaluation: 

The rational for the evaluation, what are the methods available, what methods have been 

selected for evaluating the Ontologging system, how these methods are operationalized, 

executed, and how is the result analysed. 

This document is structured in several sections. 

The first section presents the methodological aspects and principles of the evaluation: what is 

the evaluation for? What can be evaluated? What should be evaluated? What are the different 

methods available to conduce an evaluation? Etc. 

The second section presents how these principles are to be applied to the concrete evaluation 

of the different components of the Ontologging platform (a next generation, ontology-based 

knowledge management platform). In particular, this section establishes an action plan for 

evaluating the system, and in particular defines the methods that have been chosen to capture 

the data. 

The next section (chapter 4) is related to the execution of the action plan of the evaluation. It 

presents concretely the setting of the evaluation (what is the context of the evaluation, who 

are the participant), the elaboration of the instrument and the collection of the data. 

Then, the following sectio n is concerned with the analysis of the data, and of the result of the 

evaluation (this section may be empty at an initial stage of this document). This section also 

presents how the different results are taken into account in order to improve the system. 

Finally, the last section draws the main conclusion of this evaluation, and defines some future 

work in this domain to be conducted.



2 The Principles of the Evaluation 


Page : 7 of 110 

Version: 1.0 


Status: Final 


2.1 Evaluation: Objective & Challenges 

The evaluation of a research project always raises many questions and challenges. 

A first set of questions is related to the objective of the evaluation. Is the evaluation 

conducted to guaranty that the resources have been properly utilized for what they were 

intended for (in particular when it concerns the spending of public money), or is the objective 

of the evaluation to provide the participants an assessment and some feedback that will help 

them to better pilot the project and in particular maximize the generation of value by this 

project? 

A second set of questions is related to the scope of the evaluation. Are we interested in 

assessing the process of advancement of the project or in evaluating the quality of the results 

that are generated by this project? Are we interested by evaluating the technical system (the 

demonstrator) that is being designed, or by the approach that this system is expected to 

validate? 

Another set of questions concerns the dynamics of the execution of the project. What are the 

consequences of the evaluation on the dynamics of advancement of a project? Indeed the 

evaluation is rarely neutral and brings several secondary effects such as: an increase 

transparency and a magnification of the importance of the elements that are being monitored; 

a reduction of the flexibility and an increase risk-avoidance by the people whose activity is 

being monitored. 

An additional set of questions has to do we the operationalization of the evaluation: What is 

the amount of resources that should to be dedicated to the evaluation of the project? How can 

we evaluate the effort, and in particular decide how the evaluation resources have to be 

allocated? How should we direct the effort (prioritization)? How do we deal with all the risks 

associated with the evaluation, and in particular the resistance of people and organizations to 

participate in an activity that consume their time, and may threaten their position? 

Finally, a last set of question is related to the analysis of the result of the evaluation and the 

use of the evaluation. How do we proceed to extract the maximum of this evaluation, identify 

the most significant results and learn from them? 

The answer to all these questions is difficult, and is well is beyond the scope of this 

document. Indeed, if the main focus of a research project should be the maximization of the 

effectiveness of the evaluation effort in the perspective of the value of the generated 

knowledge (value for the end user; innovativeness of the solution; capability to exploit this 

knowledge), a project very rarely provides the time to evaluate all the potential impact on the 

society of the knowledge that has been created. 

Besides, and as indicated, many factors can make the operationalization of the evaluation 

delicate and difficult to achieve such as: cost and time factors (the evaluation do not come for 

free, and usually takes time); or some less tangible factors such as the willingness of people 

and organization to participate in the evaluation that they can consider as a thread, and the




Page : 8 of 110 

Version: 1.0 


Status: Final 


effect of the evaluation on the dynamic of exec ution of a project (with the bureaucratic 

syndrome). 

The approach we have decided to adopt in this document is to concentrate as much as 

possible on the dimension of the evaluation that is related to the substantive value 

(effectiveness) of the knowledge that is generated (in order work the effectiveness of the 

approach), and to support the other dimensions when this first dimension can not be 

adequately covered. 

2.2 Evaluating What? 

2.2.1 Levels of evaluation 

Donald Kirkpatrick, who started his work on learning evaluation in 1959, proposes four 

levels of evaluation of training programmes (Donald Kirkpatrick, 1996). Level 1, measures if 

the Learner liked the training (Did Learners like it?); Level 2 measures if the learning 

objectives have been met (Did Learners learn?); Level 3 measures the transfer of skills back 

to the job (Are they using it?); Level 4 measures the impact on the business (Did it matter?). 

We believe that this model of evaluation, which addresses both the efficiency and the 

effectiveness aspects of training programmes, can easily be transposed to the evaluation of 

systems in general, and in particular in systems that include a strong technical component 

(such as it is the case for the Ontologging system). 

We will therefore rely for the evaluation, to the four Kirkpatrick levels, plus two additional 

levels (technicalities & economics) that appear to be useful for evaluating technical systems: 

• Level 0: Technicalities. Do the approach/system perform technically well? 

• Level 1: Users’ acceptance: do they like the approach/system? 

• Level 2: Assessment: Do the management of knowledge approach/system function? 

• Level 3: Transfer: Is the approach/system used? 

• Level 4: Impact: Measures the impact in supporting the organizational processes 

(Does it deliver substantive value to the Organization?) 

• Level 5: Economics. Do the approach/system performs economically well? 

Level 0: Do the system performs technically well? This level reflects the performance of 

the system according to a technical perspective. It covers elements such as speed, scalability, 

reliability, (technical) flexibility, simplicity (capability to evolve) and openness (ability to 

interoperate with other systems). 

Level 1: Users’ acceptance (do they like it)? This level reflects the perception of the system 

by the user and is sometimes called the “smile sheet”. It is important to note that a good 

perception by the user do not guaranty that the system is useful (for instance, very nice 

looking graphics or gizmos will please the user, but will not contribute to his performance).




Page : 9 of 110 

Version: 1.0 


Status: Final 


However, the importance of user’s perception should not be underestimated, since, it 

contributes to the motivation of the user, which can be considered very important for the 

adoption any radically new system. 

The elements that can be evaluated at this level include the user -friendliness of the system, 

and the perceived value and limitation that the user gets from the system, or level of 

interactivity. The table 1 summarize the different levels of evaluation. 

Level 2: Assessment: Do the approach/system function? This level reflects the richness & 

the completeness of the functionalities of the system. In particular, it covers elements such as 

the ability to conduct a full session of the system, the number of functions available, and how 

deep the system support the user’s activities. 

Level 3: Transfer: Is the approach/system used? This level reflects the adoption of the 

system in an operational context (beyond the testing). In particular, it covers elements such as 

the adoption of the approach/system by an organization, and also clarifies the context of use. 

Indicators may include the number of installation effectively used in the organization, the 

number of activities supported for the organization, the features adopted by the use, and the 

rate of growth of new installation. 

Level 4: Impact: Measures the impact on the organization. This level reflects the 

substantive value of the approach/system for the organization, and for the people that use it. 

Indicators of impact include: improved work productivity, better work quality, faster result, 

and cognitive / behavioural transformation of the users leading to an organization better 

adapted to its environment. 

Level 5: Do the approach/system performs economically well? This level is concerned 

with the evaluation of the system according to a market perspective, and in particular its 

capability to be exploited in a competitive environment. Indicators include the cost of 

acquisition and the cost of ownership of the solution. 

It is important to remind that users organizations are ultimately driven by results & therefore 

should mainly be interested by level 4 (Impact on the Organization), and in particular to the 

answer to the following questions: (1) Is the proposed approach & system delivers 

substantive value to the organization that is translated into improved flexibility and better 

adequation to the company needs? (2) What are the key elements of this approach & system 

that contribute the most to the impact, and what is the nature of their contribution? However, 

Level 4 cannot easily be measured, and the answer to these questions can mainly be observed 

in the long run. 

The evaluation of Level 2 (assessment) provides a useful perspective: It reflects the level of 

functionalities delivered by the approach & system. However, this level can definitively not 

be considered sufficient since it only provide a partial picture that do not take into account the 

meaningfulness & usefulness of the approach and of the system. For instance, we can very 

well imagine systems (and actually, this applies to most approaches & systems which design




Page : 10 of 110 

Version: 1.0 


Status: Final 


is only technology driven) that works perfectly and provides very powerful functionalities, 

but that are considered as totally useless by the final user! 

Evaluation level 

Level 0: 

Technical 

performance 

Level 1: 

User’s 

acceptance 

Level 2: 

Functionalities 

Level 3: 

Transfer 

Level 4: 

Impact 

Level 5: 

Economic 

performance 

Question addressed 

(elements assessed) 

Do the system performs 

technically well? 

(technical performance) 

Do the users like it? 

(perceived value) 

Do the approach/system function? 

(functionalities) 

Is the approach/system used? 

(level of adoption) 

What is the impact on the 

organization? 

(substantive value) 

Do the approach/system performs 

economically well? 

(economic value) 

Description 

Measures & Indicators 

Performance of the tools supporting this 

process. 

Ergonomics of the tools supporting this 

process. 

Level of support of the users’ activities. 

Richness or limitations of the functions. 

Features adopted by the users. 

Impact of the improvement of this 

process on the performance of the 

organization. 

Cost / benefit of the proposed solution. 

Table 1: Evaluating the support of domain processes 

2.2.2 The phases of the evaluation: formative, summative and “substantive value” 

The evaluation can be done before any system has been implemented (for instance using 

paper based scenario). In this case, the objective of the evaluation (referred as the formative 

evaluation) is to provide an assessment that will be used to provide guidance for the design 

of the system. 

A system can also be evaluated after its implementation. In this case, this category of 

evaluation (referred as the summative evaluation) aims at testing the proper functio ning of 

the system. 

Finally, a system can be evaluated according to its more substantive value in a competitive 

perspective (taking also into account the unique characteristics that this system brings when




Page : 11 of 110 

Version: 1.0 


Status: Final 


compare to other systems). We will refer to this category of evaluation as the “substantive 

value” evaluation. Note: contrary to the formative and summative evaluations, which are 

terms well recognized and accepted by evaluation practitioners and experts, the term 

“substantive” evaluation has only been introduced in this document for emphasizing the 

importance of assessing the effectiveness dimension of systems. 

The elements that are evaluated at the formative phase: 

• The needs. This assessment determines who needs the system, how great the need is, 

and what might work to meet the need. 

• The evaluability. The evaluability assessment determines whether an evaluation is 

feasible and how stakeholders can help shape its usefulness. 

• The feasibility. The feasibility evaluation helps to determine and identity the risks 

associated to the implementation of the solution both from a technical and nontechnical 

(socio-psycho-eco) perspective. 

• The effectiveness. The evaluation of the effectiveness helps stakeholders to better 

understand the substantive value of the solution, and in particular, the impact it can 

have in the organization. 

The elements that are evaluated at the summative phase: 

• The outcome . The outcome evaluations investigate whether the system or technology 

caused demonstrable effects on specifically defined target outcomes 

The elements that are evaluated at the “substantive value” phase: 

• The impact. The impact evaluation is broader and assesses the overall or net effects - 

- intended or unintended -- of the system or technology as a whole 

• The cost. The cost-effectiveness and cost-benefit analysis address questions of 

efficiency by standardizing outcomes in terms of their dollar costs and values. 

• The capability to develop. This analysis re-examines all the elements in order to 

understand the real potentia l of the solution, and it capability to evolve and deliver 

more value in the future. 

• The overall value. The meta -analysis is done to integrate the outcome estimates from 

multiple studies to arrive at an overall or summary judgement on an evaluation 

question. 

These phases of evaluations formative, summative and “substantive value” can be considered 

as presenting another perspective of the evaluation level presented in the previous chapter.

2.3 The Evaluation Methods & Tools 

2.3.1 Evaluation: an overview 




Page : 12 of 110 

Version: 1.0 


Status: Final 


Evaluation is based on some observations, and of a model of interpretation. In quantitative 

research, these observation usually correspond to quantitative data that can be captured, and 

the model of interpretation correspond to a set of measures that consist in a me ans of 

associating numerical values to these data that provide some scale of comparison. In 

qualitative research (McBride and Schostak, 1995), these observations may take the form of 

the capture of events (such as “the user has done a specific action”) or the result of an 

interview with a user (.i.e. some more or less structured text or a video that record the content 

of the interview). The model of interpretation corresponds to method of analysis (the next 

chapter will present the main methods available). 

The quality of the sample used (characteristics and the size of the data) is also an element to 

take into account in the evaluation, since it has some important implications in the validity of 

the evaluation (for instance if the size of the observation is too small, or if the sample use for 

the observation is biased). Many works on this subject exist in quantitative research (and 

actually constitutes a major dimension of statistics) and to a less extent in qualitative research 

methodology. In the case of quantitative research, probability theories will be used to 

calculate a size of the sample that correspond to a given error margin. In qualitative 

evaluation, the size of the sample may be chosen more empirically in a way that is considered 

reasonable and significant enough to provide useful insight. 

Note: The size and on the characteristic of the sample used for the observation is a variable 

that should be manipulated with precaution when defining an evaluation since it is directly 

correlated to the ef fort needed to run the evaluation. Besides, the choice of the size and of the 

composition of the sample may be limited by real world constraints (for instance finding a 

large sample of motivated test users can represent an important difficulty). 

Many evaluation methods can be used to capture and analyse the data. Some methods 

(considered to be more objective) try to observe and evaluate the system in real operations, 

while other methods use more indirect proxy such as the collection of users’ perception and 

opinions. Each method has its advantages and disadvantages (related to the reliability, the 

level of effort required, the time, the complexity, etc.), and different methods should be used 

conjointly for the evaluation of a system. 

The next paragraph will present different categories of methods available.




Page : 13 of 110 

Version: 1.0 


Status: Final 


2.3.2 Observational methods 

Observational methods involve real people using working systems. They are based on 

observing users interacting with the actual system and can range from being almost entirely 

informal to highly structured. Observational methods should try to be as invisible as possible 

to the user. 

Direct observation. Users are observed directly doing special or ordinary task in their work, 

while the observer makes notes about the their behavio ur and performance that she consider 

important to record. The evaluators do not have direct contact with the user. 

Indirect observation and software logging. Videoing a user interaction with a system 

allows recording several aspects of the user activity. Combined with some forms of automatic 

keystroke or interaction logging enable to collect vast amount of data. In some cases, 

software-logging tools can be used to record time-stamped key presses and real time 

recording of the interaction between users and technology. Besides, in many cases, 

information systems are usually able to track and monitor the activities of the users and to log 

them (for instance, that a user has posted a document, or has launched a search). 

Think aloud protocol. The data collected by think aloud protocol contains users´-spoken 

observation by addressing the cognitive activity and provide a subjective feeling about the 

activity they are performing. Users are asked to think aloud to describe what they are doing at 

each stage, and why. The evaluators record the users actions by using tape recordings, video, 

or pencil and paper. 

Note: Of course, the think aloud protocol may not be considered as totally neutral on the 

activity of the user since it introduce a cognitive load that can dist ract this user. 

2.3.3 User’s Opinions methods 

The objective of the users’ opinion method is to elicitate the (subjective) perception of the 

system by the different categories of users. 

Structure interview. Interviews are particularly useful in eliciting information about user 

attitudes, impressions, and preferences. Interviews can be used in conjunction with thinkingaloud 

protocols for retrospective analysis of user responses and events. 

Questionnaires. Questionnaires have been used as instruments to assess different aspects of 

usability of the human-computer interface. Collecting users' subjective opinions about a 

system can remove unpopular and unusable parts early in the design or after delivery. 

Focus groups. Focus Groups are effective Research Tool and are very commonly used in 

software development. They are similar to interviews, but because we have a group of 

participants the group dynamics generate more data then one to one interviews. Focus groups 

involve gathering 8 - 12 members of a target audience for an open-ended discussion of




Page : 14 of 110 

Version: 1.0 


Status: Final 


particular topic of interest. Each group session typically lasts 1 to 1 ½ hours and should be 

moderated by a professionally trained facilitator. Groups should always be audiotaped or 

videotaped. 

2.3.4 Interpretative methods 

The interpretative evaluation consists in deriving, from the observation of the users in natural 

settings (working on at least partly controlled tasks), the structure and thinking behind the 

visible behaviours. 

Cooperative evaluation. Cooperative evaluation uses the think aloud protocol, which allows 

the user to ask questions, comments and suggest appropriate alternatives of the evaluators, 

and the evaluators to prompt the user. Here, the user is encouraged to act as a collaborator in 

the evaluation to identify usability problems and their solutions. 

Ethnography. Ethnography is an approach developed by anthropologist, which is used in 

HCI to inform system design. This method is used "in order to look at aspects of social 

interaction and work practices in natura l work environments. By doing so, they attempt to 

reveal the detailed and systematic interaction between members in an organization and 

therefore explicate the underlying work practices". 

Scenario -based design. The phrase “scenario-based design” represents a diverse field of 

design practice (Carroll, 1995). The various techniques are connected by the use of scenarios 

to ground the design process in the situated tasks of users. These scenarios describe 

sequences of actions taken by a user with a specific goal in mind. These methods are meant to 

make explicit those assumptions and activities of the design process which are usually 

implicit. Dialogue can for instance be used as to conduct the inquiry (Erskine, Carter-Tod, 

and Burton, 1997). Practically, the user participants are asked in a dialogue to provide a 

context (who they are), a specific goal (for instance they may want to capitalize the 

knowledge of a just finished project), and an action (a detail description of how they would 

proceed to achieve this goal using the system that is being tested). 

Contextual inquiry. Contextual Inquiry (Beyer and Holtzblatt, 1998) is a field study method 

where usability specialists observe people performing their usual job tasks in the context of 

their actual work situations. Contextual inquiry is basically a structured interviewing method 

based on a few core principles (context, partner, focus): (1) the users are observed in their 

own working situation; (2) the researcher and the users are equal partners in investigating and 

understanding the usage of a system; (3) a focus is defined, which can be seen as a number of 

assumptions and beliefs concerning what needs to be accomplished and how to accomplish it. 

2.3.5 Predictive methods 

Experts separately review a system and categorize and justify problems. Predictive evaluation 

can be based on a specific theory (for instance in cognitive psychology) or on a short set of 

heuristics (rules of thumb).




Page : 15 of 110 

Version: 1.0 


Status: Final 


Cognitive walkthrough. Cognitive walkthrough is an evaluation technique that considers 

psychology theory into the informal and subjective walkthrough technique, proposed by 

(Polson et al., 1992). It aims to evaluate the design in terms how well its supports the user as 

he learns how to perform the required task. The designer or an expert in cognitive psychology 

performs the walkthrough. The expert works through the design for a particular task, step by 

step, identifying potential problems against psychological criteria. The analysis is focused on 

the users goals and knowledge. The cognitive walkthrough must show if, and how the 

interface will guide the user to generate the correct goal to perform the required task, and to 

select the necessary action to fulfil a goal. 

Heuristic evaluation. Heuristic evaluation (proposed by Nielsen and Molic h) involves 

experts assessing the design. In this approach a set of usability criteria or heuristic is 

identified to guide design decision. Evaluators independently run through the performance of 

the task set with the design and assesses its conformance to the criteria at each stage. Nielsen 

suggests that around five evaluators found about 75% of potential usability evaluation. 

2.3.6 Test, Experiment and Usability Engineering methods 

Tests, experiment and assessments evaluation methods consist in setting up a more formal 

setting for the evaluation, and in particular the organization of evaluation sessions with clear 

evaluation objectives. 

Tests and assessments can be useful tools in evaluation to measure the impact of the system 

on the participants’ processes. The tests are conducted successively without and with the use 

of the system, and the results are compared. 

Self-reports participants write reports on how much the system has contributed to improve 

their work process (more than an opinion, users are asked for an explanation). 

Usability engineering. Usability engineering is an approach to system design where the 

usability issues of a system are specified quantitatively in advance. The development of the 

system is based in these metrics, which can be used as cr iteria for testing the usability of a 

system or prototype. Examples of such metrics are 'percentage of errors', and 'number of 

commands used'. 

Experiments. Experiments are the most formal methods of usability testing that requires 

preparation and knowledge to evaluate the design of a system or prototype by quantifying 

user performance such as 'time on task', 'number of errors', etc.

2.4 Defining the evaluation process 




Page : 16 of 110 

Version: 1.0 


Status: Final 


For the different reasons we have already mentioned (cost, time, availability to find 

participants, etc.), evaluating everything is not possible, and to some certain extend it is not 

really desirable (lack of focus, introduction of rigidity, etc.). 

An evaluation process aiming at delimiting the research and conducting it realistically can be 

used. 

The different stages of this research process in social sciences are the following: 

• Stage 1: The definition of a research question. This involves contributing to 

building a body of knowledge and developing theory. 

• Stage 2: The development of an instrument. Having defined the research question, 

the research investigator needs to develop measurement instruments to capture the 

data for future analysis and to select the context (the site, the users, etc.) in which the 

measurement will take place. 

• Stage 3: The data gathering. This stage is related to the execution of the chosen 

instruments in the selected context in order to collect the data. This stage may include 

the execution of some preliminary actions, such as for instance trust building in the 

case that the data has to be collected from people. 

• Stage 4: The analysis of the data. This stage consists in the analysis of the data in 

the perspective of the research questions that have been previously identified, and the 

determination of the learning of the research. 

• Stage 5: The dissemination. This stage consists in the selection and the 

dissemination of the most important (originality, impact) research finding generated 

by this research. 

These different stages can be slightly relaxed in the Case Study Research method (Eisenhardt 

1998; Yin 1994; Meredith 1998), a method favoured for conducting research in a field of 

research that is still largely unexplored, and which characterised by a paucity of theories, and 

the complexity and lack of well supported definitions and metrics (Stuart et al., 2002). For 

example, Eisenhardt (1989), while acknowledging the role of good research questions and 

theoretical constructs, argues that propositions can be developed (and tested) during data 

collection, rather than prior to it. Because the aim is to obtain a rich understanding of the 

cases in all their complexity, insights gained during data collection can be used to inform the 

theory. 

2.4.1 The definition of the research questions 

An initial definition of the research question should be made in order not to become 

overwhelmed by the volume of the data. A priori specification of constructs can also help to 

shape the initial design of theory-building research, although this type of specification is not




Page : 17 of 110 

Version: 1.0 


Status: Final 


common in such studies. It is important to recognize that the early definitions of the research 

question and the possible constructs are only tentative. 

2.4.1.1 Identifying the goals and the scope 

The identification of the most important points and objectives is very specific to the syste m & 

approach that is subject to the evaluation. 

Some of the critical points to identify are the main processes and dimensions that the system 

is expected to support. For instance, in context of learning systems, the processes and 

dimension may include “individual learning”, “experiential learning” or “collaborative 

learning” (via learning network). In the context of knowledge management, the selected 

processes and dimension are to be taken from the different knowledge processes used in 

organization such as “knowledge creation” (creativity), “knowledge acquisition”, “knowledge 

structuring”, “knowledge capitalization”, “knowledge searching”, knowledge sharing (and in 

particular the support of the social aspects), etc. 

These objectives and points should include both short-term objectives & long-term 

objectives, distinguish the points that are very general or very narrow and also be used to 

provide insights both for the design process itself (summative evaluation), for the functioning 

of the result (formative evaluation), and its effectiveness (substantive value of the solution). 

More horizontal & global characteristics may also be evaluated such as: 

• “Usability”: A system is usable if desired task quality can be achieved with acceptable 

costs to users. 

• “Flexibility”: A system is flexible if it is able to adapt with a reasonable level of 

effort, to changes that were not initially planned. 

• “Social acceptability”: A system is socially acceptable, if its introduction does not 

generate a too important level of resistance from the people due to a too radical 

change in their role. 

• Etc. 

Again, the choice of these characteristics is very dependant of the target characteristic of the 

system to be evaluated. For instance, “flexibility” may be considered of very little relevance 

in the application to the design of a satellite system for which the possibilities of intervention 

are very limited once the satellite has been launched. 

2.4.1.2 Defining elements (criteria and indicators) for evaluating the goals. 

The evaluation of each goa l will rely on the definition a set of elements (criteria and 

indicator) that reflect the best how the different goals are fulfilled by the approach / system 

that is to be evaluated. 

The selection of the elements to be evaluated will take into account the relevance of these 

elements related to the importance of the goals and points that have been identified, as well as 

the level of detail desired or possible (affordable).




Page : 18 of 110 

Version: 1.0 


Status: Final 


First, these elements should include the most important mechanisms that the system is 

providing to support in the domain addressed. For instance “the search function” will be 

considered as a critical function to be evaluated related to the Goal “providing support to 

knowledge identification”. 

Second (and perhaps even more importantly), these elements should include the features that 

are considered as unique for this system. For instance, if automatic clustering represents a 

unique feature of this system, it will have to be evaluated in dept, since it will represent the 

competitive advantage of this particular system when addressing the goal of “supporting 

knowledge identification”. 

Another element not to underestimate is that the choice of the priority also has consequences 

on the dynamics of the project. The evaluation if not neutral, and we can believe that the 

elements that be evaluated will receive more attention in the execution of the project. 

Besides, the evaluation may trigger some resistance both from people participating to this 

evaluation (they may see it as an additional effort, and consider the associated monitoring 

threatening their position) and from organizations (who may be reluctant to put some light on 

some of their current practices and systems they are currently using). 

2.4.2 Selecting the evaluation methods (defining the instrument) 

The concept of population is crucial (what kind of data we are interested to collect). It defines 

the set of entities from which the research sample is to be drawn, controls variation and helps 

define the limits of generalizing the findings. Theory-building research relies on theoretical 

sampling Crafting Instruments and Protocols. Theory-building researchers typically combine 

multiple data collection methods. 

The choice of the evaluation methods to use and the elaboration of the action plan will 

depend of different factors such as: 

• The elements and indicators that have been previously identified. 

• The stage of system development 

• The resources available 

• The time available 

• The availability of potential users 

• Acceptability of intrusion on user 

• The type of output required 

• The precision / reliability desired 

• Etc.




The action plan will consist in different phases: 

Page : 19 of 110 

Version: 1.0 


Status: Final 


• The formative phase. Conducted mainly using interviews, mock-ups and early 

prototypes, the objective of the formative evaluation is to pr ovide an assessment that 

will be used to provide guidance for the design of the system. 

• The summative phase. This phase will focus more on assessing the functionalities of 

the system being designed (this phase will focus on the testing of the prototype, and 

will collect data via interviews, and observational methods). 

• The “substantive value”. This phase will focus on evaluating the value of the system. 

It will involve the users testing a consolidated version of the prototype, as well as 

experts and business development manager assessing the real value of the system 

(what are the unique features, how it compare with other systems) and its 

“economical” potential (ROI, business plan, etc.). 

2.4.3 Executing the instrument and gathering the data 

For each phase will have to be defined the following steps: 

• Identification of the source of data for the evaluation (and in particular identifying the 

users’ groups participating in the evaluation and the experts). 

• Any preparation that would be necessary before the evaluation (for instance some 

evangelisation work towards the users, and their recruitment). 

• The setting of dates for the tests to be conducted negotiated with the users’ groups 

(date of sessions, starting date and duration in the case of continuous evaluation, etc.). 

• The setting of date for the analysis of the data (milestones). 

• The definition of how the results of the analysis are going to be fed-back to the 

system. 

Note: The evaluation is not necessary as linear as it may appear, but consist in a series of 

iterations: test, analysis, and integration of the findings. 

2.4.4 Analysing the data, theory confirmation & building 

Analysing the data. This is the least codified part of the process. The overall idea is to 

become intimately familiar with the data collected. This process allows the unique patterns of 

each case to emerge before investigators push to generalize patterns across cases. 

Searching for Patterns. There are various tactics for doing that. The idea behind these 

tactics is to force investigators to go beyond initial impressions, especially through the use of 

structured and diverse lenses on the data. 

Shaping Hypotheses. Researchers constantly compare theory and data - iterating toward a 

theory, which closely fits the data. This stage includes sharpening the constructs by refining




Page : 20 of 110 

Version: 1.0 


Status: Final 


their definition as well as building evidence, which measure them in each case. Another step 

is verifying that the emergent relationships between constructs fit with the evidence in each 

case. 

Enfolding Literature. Linking results to the literature is crucial in theory building research 

because the findings often rest on a very limited number of cases. 

Reaching Closure. Two issues are important in reaching closure: when to stop adding cases 

and when to stop iterating between theory and data. In both cases, researchers should stop 

when a theoretical saturation is reached. That is when the incremental improvement to the 

theory is minimal.




3 Evaluating the Ontologging System 

Page : 21 of 110 

Version: 1.0 


Status: Final 


This section of the document will present now how the principles that have been presented 

previously are being applied for the evaluation of the Ontologging knowledge management 

system. 

3.1 Overview 

The evaluation of the Ontologging project will consist in the use of a combination of different 

techniques described in the previous section and in particular: (1) predictive methods aiming 

at exploiting the expertise and know-how present in the consortium related to knowledge 

management (both from a technical and an organizational perspective) and Ontology; (2) user 

opinion methods aiming at integrating the user’s needs (and not the user’s want) into the 

proposed system, as well as validating the substantive value of the approach proposed to the 

users; (3) observational methods and test and experimental methods aiming at assessing the 

capability of the designed system to fulfil the objective of the system, as well as its adoption, 

both from a technical and organizational perspective (overcoming resistance to change) by 

the users. 

The result of the evaluation will be used for the following purpose: 

• To help the design process. Indeed, Ontologging uses an iterative process (spiral of 

analysis of needs, specification, testing of result, adjustment) in which the system is 

progressively detailed (versus the top-down V design process). The feedback from the 

evaluation will help the process of definition and adjustment of the system. 

• To assess the substantive value of the system that is being generated, and to guaranty 

that the focus of the project is not lost in favour of other sub-goals (such as technical 

goals) and remain consistent with the original high level objectives. 

As indicated in the previous section of this document, different phase will be used for the 

evaluation of Ontologging: 

• The formative evaluation phase. The objective of this phase is to provide a first 

feedback that will flow into the design of the system. This phase will mainly be done 

via the collection of users’ and expert opinion (via questionnaire, interview), as well 

as the initial testing of the mock-up / early version of the prototype for validating the 

main ideas (versus testing the functionalities). 

• The summative evaluation phase. The objective of this phase is to assess the 

functioning of the system. This phase will start with the availability of the first 

version of the prototype (and not the mock-up), and will aim at assessing the different 

functions of the system, both according to its performance, the richness of the 

functionality, and their ease of use. The methods used will mainly be based on users’ 

opinion (related to ergonomics), as well as on the organization of specific test for 

evaluating the different functions of the system (such as the authoring of ontologies, 

the population of content, the search for content and of collaborative capabilities)




Page : 22 of 110 

Version: 1.0 


Status: Final 


• The “substantive value”. The objective of this phase is to assess and reflect the 

substantive value (from a usability and economic perspective) of the Ontologging 

system in addressing the high level objective of the project (how ontology concept 

can help to better support the knowledge management processes). This phase will 

mainly make use of the consolidated version of the prototype (the tester will be able 

to focus their attention on the possibilities of using the system to support their 

working practices). Practically, the method used will rely on users’ opinion methods 

(related to their perceived usefulness to contribute to their work and their willingness 

to adopt the system) and on observational methods (in order to identify what is the 

real usage of the system, and in particular what are the functions that are effectively 

used). 

3.2 Goals and scope of the evaluation of Ontologging 

One of the main objectives of Ontologging is to investigate how ontologies technologies 

approaches can be used to design knowledge management systems that better support to 

knowledge management processes in organizations. 

We can indeed expect, that ontology-based knowledge management systems will be able to 

benefit of the number of advantage of Ontology approaches such as: powerful knowledge 

representation, easy evolution, interoperation, and minimum ambiguity. 

3.2.1 Knowledge Management & Ontologies, some clarification 

Before going into the exercise of defining the goals and the scope, let’s first provide some 

clarification of the different concepts such as knowledge management, knowledge 

management systems and Ontology. 

What is knowledge management and what is a knowledge management system? 

Knowledge management refers to the set of activities and processes which aims at explicitly 

supporting the way the knowledge is managed and in particular: 

• How the knowledge is acquired (business intelligence, training, etc.)? 

• How the knowledge is created (innovation)? 

• How the knowledge is assessed? 

• How the knowledge processes are structured? 

• How t he knowledge is structured and transformed? 

• How the knowledge, stored, and more generally capitalized? 

• How the knowledge retrieved and searched? 

• How the knowledge is shared amongst people (CoP, benchmarking)? 

• How the knowledge is applied (adapted, helping to the decision, etc.)? 

• Etc.




Page : 23 of 110 

Version: 1.0 


Status: Final 


Knowledge management systems represent technical infrastructures and methods that can be 

used to support a set of this knowledge activities and processes. 

Note: 

Our definition of knowledge is very broad, and refers to any form of information which 

embed some context and some level of actionability. Knowledge may refer to some form of 

intellectual assets that people can activate in other to generate new objects, but also to 

processes (knowledge process, organizational knowledge) that in embedded in the 

functioning of the organization. 

What is an Ontology? 

Ontology represent a proposed approach for representing knowledge. 

"An ontology is an explicit specification of a conceptualization.” (Gruber, 1993) 

“An ontology provides an explicit representation for structuring all the entities and 

relationships that are to be talked about in some domain” (Bateman, 2004). 

More concretely, ontology propose a conceptual framework that provide the capability: 

• To model deeply the knowledge (semantic web) making the search more powerful 

(possibility to define more precise requests) and less ambiguous (less noise). 

• To evolve relatively easily. Ontology technologies support the change in the structure 

of the knowledge without impacting (or at least in a very localized way) the whole 

system. 

• To interoperate: Ontology models provide very clear and rigorous interfaces, which 

facilitate the interoperability between different systems. Besides, mapping 

mechanisms allowing ontologies translation from one ontology to another can help 

the connection of systems that implement similar knowledge & data models but that 

were defined separately. 

• To help to structure the knowledge processes in organization by contributing to the 

establishment of a common language. Ontology provides the means to define 

unambiguously the different terms of a domain. 

Ontology technologies / systems (such as Ontology editors) are just some tools that have 

been design to support this approach. 

Practically, Ontology systems & technolo gies propose mechanisms that help: 

• The definition of ontologies formalizing a target domain (the semantic of this domain 

is explicitly defined). 

• The use of these ontologies to formalize explicitly the different knowledge objects 

(documents, practices, people, etc.) of the domain addressed. 

• Semantic search of the different knowledge objects (the queries express the semantic, 

and not only the existence of keywords).




Page : 24 of 110 

Version: 1.0 


Status: Final 


• People communication, and organizational interoperation (Ontologies provide nonambiguous 

languages, facilitating the interoperation between people and 

organization). 

3.2.2 The goals of Ontologging: the design of a Knowledge Management System that better 

support some knowledge processes 

The goal of the Ontologging project is to design a system that take advantage of the Ontology 

approach to better support some of the knowledge management process of the organization. 

This general goal can be refined in the following sub-goals: 

• Goal 1: The Ontologging system will help organizations to better structure knowledge 

and elicitate their knowledge processes. 

• Goal 2: The Ontologging system will help organizations to better capitalize their 

knowledge assets. 

• Goal 3: The Ontologging system help organizations to better retrieve (locate / search) 

their knowledge. 

• Goal 4: The Ontologging system helps people in organizations to better share their 

knowledge (assets & processes). 

3.2.2.1 Goal 1: Ontologging will help the structuring of the organization 

Ontologging provides a framework (tools and theories) for representing unambiguously how 

the knowledge is structured. Practically, a domain is modelled via the definition of concepts 

and properties associated to these concepts, and the relationships between these concepts. 

Ontologging will in particular help the organizations to capture (elicitate) the complexity of 

the knowledge and the processes used in the organization, via the definition of a common and 

semantically well specified language (ontologies) for describing the domains in which the 

organization operate and the structure of the different processes. 

3.2.2.2 Goal 2: Ontologging will help the capitalization of the knowledge of the 

organization 

Ontologging provides the means (based on the previous structure definition) to represent the 

knowledge (knowledge asset) of the organization. Practically, this representation consists in 

categorizing the “chunks” of knowledge (documents, people description, project description, 

etc.) according to the shared ontologies of the organization, and the creation of semantic 

networks that rely on these ontologies.




3.2.2.3 Goal 3: Ontologging will facilitate knowledge retrieval 

Page : 25 of 110 

Version: 1.0 


Status: Final 


By contributing to the explicit definition of the semantic of the knowledge, Ontologging 

approaches makes the operations browsing and searching this knowledge more powerful and 

higher level. In particular browsing and searching can be done not only at the lexical level, 

through keywords, but also exploit the high level concepts that have been defined in the 

Ontology and also navigate the semantic networks. 

3.2.2.4 Goal 4: Ontologging will facilitate knowledge sharing and exchange 

Ontologging by providing shared vocabulary (the ontologies), and making the knowledge 

more visible, will contribute to improve the shared understanding of the values of the 

organization, and facilitates the communication and the inter-operation between the different 

parts of the organizations. 

3.3 Selecting the elements to be evaluated (evaluation criteria) 

The support of all the knowledge processes / sub-goals that have been identified previously 

(structuring role, capitalizing knowledge, searching and browsing knowledge, sharing 

knowledge) will have to be evaluated according to a formative, summative and substantive 

dimension. 

As indicated however, a complete and detailed evaluation of each goal according to the levels 

of evaluation introduced in the first section of this document would require a too big effort 

and would take too much time. Therefore, only a partial evaluation will be considered, and 

will consist in the selection of a subset of the most important and representative elements that 

will be evaluated. 

This section will first try to identify for each of the goals the different elements that could be 

considered candidate for the evaluation. Then, base on their importance, it will select a few of 

these elements that will be subject to the effective evaluation of the system in the evaluation 

plan. 

3.3.1 Identifying the elements associated to each of the goals 

Ontologging proposes different means, tools and approaches for fulfilling the goals that have 

been previous ly identified. 

3.3.1.1 Elements associated to a structuring role (Goal 1) 

Ontologging proposes different components / elements to define a rich vocabulary of 

concepts, with a well defined (unambiguous) semantic, and agreed across the whole 

organization:




Page : 26 of 110 

Version: 1.0 


Status: Final 


• The Ontology Editor for the creation of the ontologies for modelling a domain of 

application (domain ontologies, events ontologies, users ontology) in term of concepts 

and properties. 

• The Ontology & Knowledge Evaluators (OKE) provides the means to evaluate (and 

therefore to readjust) the authored ontologies. 

Other less technical elements will need also to be considered such as: 

• The effort and the expertise required to author complex ontologies. 

• The difficulty to make these ontologies evolve. 

• How to address the eventual conflicts that may occur when defining ontologies at the 

companies level (what happen if two departments have some diverging definition of a 

term?). 

3.3.1.2 Elements associated to knowledge capitalization (Goal 2) 

Ontologging provides several tools that can be used for representing the knowledge of the 

organization: 

• The DUI (Distributed User Interface) is a tool that allows users (authors) to describe 

(using the Ontology “share vocabularies”) and enter new knowledge into the 

Knowledge Base. 

• The Office Connector allows users (authors) to enter knowledge directly from inside 

Microsoft office. The Office Connector is a visual basic extension of Microsoft 

Office, which can be used to annotate document being edited with the domain 

Ontology and enter the result in the Knowledge Base. 

• The user profile editor allows the users to define their profile (based on the user 

ontology that has been previously defined using the ontology editor). The User profile 

editor contributes to support the management of the tacit knowledge of the 

organization, and in particular the access of the knowledge that in only present in the 

people head. 

• The OKE provides the means to evaluate the knowledge stored in the Ontologging 

server, and therefore provide some indication useful for its reorganization. 

Non- technical elements will need also to be considered such as: 

• What is the overhead effort in the knowledge capitalization process? 

• Will formal roles (such as authors) have to be defined in the organization? 

• What are the incentive measures that will encourage people in the organization to 

capitalize knowledge, or to update their profile? 

• How to overcome resistance in adopting new knowledge capitalization practices? 

3.3.1.3 Elements associated to knowledge searching and browsing (Goal 3)




Page : 27 of 110 

Version: 1.0 


Status: Final 


Ontologging provides several tools that can be used for searching and navigating the 

knowledge: 

• The Ontologging DUI provides the means for browsing and searching the knowledge 

that was stored in the Ontologging server. The underlying search engine is aware of 

the semantic of the knowledge, and in particular goes beyond the single keyword 

search. Besides, the inference engine (embedded in the DUI) allows querying of 

ontologies to discover information not explicitly present in the ontology. 

• The user profile editor provides a specialized tool to enter and maintain user 

information. This tool helps in particular to navigate the network of expertise (people) 

of the organization. 


• What is the expertise necessary to exploit properly the technology? 

• Will training be necessary? 

• Will the value of Ontology techniques be visible enough (versus more traditional 

methods)? 

3.3.1.4 Elements associated to knowledge sharing and exchange (Goal 4) 

Different tools in Ontologging that contributes to the facilitation of knowledge sharing in the 

organization, and knowledge exchange between the departments: 

• The Ontology Editor for the creation of the shared ontologies. Besides, Ontologging 

even provide some mechanisms for the collaborative authoring of ontologies (using 

agents for notification of conflicts). 

• The Ontology mapping tool, provides the means to reconcile ontologies that have 

been defined separately by different parts of the organization (different departments), 

allowing them to interoperate. 

• The user profile editor, by making more explicit people profile, can help the 

circulation of tacit knowledge (present in the people head) of the organization. 


• Will the availability of tools be enough, and in particular strong signal from the top 

management will also be important in better inter-department communication? 

• What will be the incentive for people for sharing their knowledge, in particular for the 

sales people (what is it for me)? 

3.3.2 Levels of evaluation for Ontologging. What could be evaluated 

Let’s provide for information a perspective of the evaluation according to the levels as it was 

described in the first part of the document.




3.3.2.1 Level 0: Do the system performs technically well? 

Page : 28 of 110 

Version: 1.0 


Status: Final 


This level addresses the evaluation of the technical characteristic of the Ontologging platform 

& components. In particular, if the different tools (DUI, Ontology editor, User profile editor, 

the reconciliatory agent, etc.) are easy to set-up, are reliable, are fast, are interoperable, etc. 

3.3.2.2 Level 1: Users’ acceptance (do they like it)? 

This level is related to the ergonomic of the Ontologging component such as the perceived 

speed, the aesthetics, the readability of the fonts, the navigability, and the interactivity (drag 

& drop capabilities, etc.). 

3.3.2.3 Level 2: Assessment: Do the approach/system function? 

This level addresses the functionality of the different components of Ontologging and for 

instance: can we create any ontology or what are the limitations? What are the functions 

currently implemented related to knowledge searching? What are the different means for an 

agent to intervene to support the collaborative authoring process? What can be evaluated, etc. 

3.3.2.4 Level 3: Transfer: Is the approach/system used? 

This level addresses the use of the Ontologging systems in the tester companies (and in 

particular Indra) for supporting their knowledge management processes. This level also 

assesses the condition of use of Ontologging, in particular related to other knowledge 

management approaches (such as the Meta4 product Knownet). For instance, it evaluates the 

use of the system in better supporting the work processes (for instance the tender process) 

and the collaborative dimension (for instance, are the agent mechanisms effectively used to 

support collaborative ontology authoring?). Other questions: are the advanced searching 

capabilities (inference) used? How often the different Ontologies are updated? Do user use 

knowledge evaluation features? 

3.3.2.5 Level 4: Impact: Measures the impact on the organization 

This level addresses the effectiveness of the Ontologging system in making the employee to 

better manage the knowledge of the company and making it operate better (for instance 

improving the relationships with its customers, increasing its flexibility and its reactiveness). 

More concretely, it may provide some assessment in how ontologies contribute to provide a 

better management of the knowledge (because it enables flexibility and also the support for 

the structuring of the knowledge-oriented activities). Finally, this level will also try to assess 

the cognitive transformation of the user, and the changes in his work practices: are the beliefs 

and the behaviours of the users different been radically transformed by the system (or has the 

user radically learned)?




3.3.2.6 Level 5: Do the approach/system performs economically well? 

Page : 29 of 110 

Version: 1.0 


Status: Final 


This level is concerned with the evaluation of Ontologging in the perspective of exploitation 

in a competitive environment. In particular this level will analyse what are its unique features, 

as well as its weaknesses. 

The following table present some illustrations of the evaluation of knowledge processes along 

the different levels.

Evaluation level 

Level 0: 

Technical 

performance 

Level 1: 

User’s 

acceptance 

Level 2: 

Functionalities 

Level 3: 

Transfer 

Level 4: 

Impact 

Level 5: 

Economic 

performance 



Structuring role 

Performance of the 

ontology authoring 

tools. 

Easiness to author 

an ontology (create 

new concepts & 

properties) 

Features of the 

ontology editor. 

Level of adoption of 

Ontologies. 

Do the structuring 

of the knowledge 

(via ontology) 

brings a better 

performance for the 

organization. 

What is the effort / 

benefits to structure 

the knowledge 


Knowledge processes & Goals 

Capitalise 

knowledge 

Performance of 

the DUI 

(publisher mode) 

and of the MS 

Word extensions. 

Ergonomics of 

entering and 

annotating new 

documents in the 

DUI, and the MS 

Word extensions. 

Features & 

limitation of the 

DUI-publishing 

function, and of 

the MS Word 

extensions. 

level of 

capitalization of 

knowledge 

Do the 

capitalisation of 

the knowledge 

assets provide 

benefit to the 

organization. 

What is the cost 

of capitalizing the 

knowledge? 

Retrieve 

knowledge 


DUI (user mode). 

General ergonomics 

of the DUI. 

Capability to display 

and manipulate 

complex 

information. 

level of reuse of 

existing knowledge 

Is the knowledge use 

in the organization 

less reinvented, 

conducing to a more 

efficient 

organization. 


benefits of searching 

the knowledge 

(training, support, 

etc.). 

Table 1: Examples of level evaluation according to the knowledge processes 

Page : 30 of 110 

Version: 1.0 


Status: Final 


Share knowledge 

Scala bility of the 

multi-ontology 

mapping. 

3.3.3 The different phases of the evaluation 

As indicated in the first section, the evaluation will fulfil three objectives: 

• Provide some first feedback to the design process (formative evaluation). 


user profile editor 

Ergonomics of the 

user profile editor. 

Scalability of the 

multi-ontology 

mapping. 

Evolutivity of the 

User Ontology in 

the user profile 

editor 

level of sharing 

knowledge amongst 

people 

Is the acceleration 

of the diffusion of 

knowledge inside 

the organization 

translated to 

tangible benefit for 

the organization 

(reactivity, 

flexibility, etc.)? 


benefits to put to 

facilitate the 

knowledge sharing. 

(facilitator, 

incentives, etc.)




Page : 31 of 110 

Version: 1.0 


Status: Final 


• Assess the functionalities of what has been designed (summative). 

• Assess the substantive value of what has been designed (substantive value). 

In the first case, formative evaluation will help to readjust the plans and the priority of the 

Ontologging project, to better understand the users’ needs, and to refine the definition of the 

different Ontologging components, putting more emphasis on what is considered to be the 

most important. 

This phase will first be based on setting-up at the user place an early version of the prototype, 

and asking them for their feedback. This phase will also consist in evaluating how the 

different components interoperate with one anot her, and in identifying the functionalities that 

may be missing, or that would need more attention. 

In the second case, summative evaluation will help to access what Ontologging has produced, 

and what are the functionalities available. 

This phase will consist in the evaluation of the main Ontologging components at the users’ 

site, and will be achieved with a functioning version of the prototype. 

In the last cases, the substantive value evaluation will help to understand what are the areas of 

knowledge management in which the system can contribute the most, and where is the real 

innovation (incremental or radical) in the Ontologging system. 

This phase will consist in the analysis of the different components of Ontologging according 

to a usability perspective (what is really usable for the organization). This phase will be 

achieved using the result of a consolidated version of the prototype 

3.3.4 A pre-selection of the elements to be evaluated 

Let’s now define, for each of the sub-goals that we want Ontologging to achieve, some 

elements and criteria to evaluate. 

3.3.4.1 Goal 1: Ontologging for helping to structure the work in the organization 

The structuring role of Ontologging originates from its capability to support the definition of 

the shared Ontologies that specify the semantic of the different concept and that are used 

across the whole organization to communicate. 

Several elements can be taken into account when evaluating the capability of Ontologging to 

support this structuring role. 

Technical view 

The first element is the technical and instrumental, and is related to the capability of the 

Ontology authoring tools that are provided, to support the definition of the different 

Ontologies that need to be represented. The evaluation of this element mainly consists in the 

evaluation of the technical components that are used to author the ontologies (these




Page : 32 of 110 

Version: 1.0 


Status: Final 


components include the Ontology editor, and the Ontology Knowledge Evaluator for its 

diagnostics functions). The evaluation of technical tools represents a relatively well-known 

territory, which consist in assessing several dimensions such as: the reliability of the system 

and its scalability, the richness of the functionalities, the ergonomics, etc. 

Usability view 

A second element to be considered is related to the ability the ontology approach to 

effectively model the concept of domain in a complete way, and to take into account 

evolution. In order words, are the theories of knowledge representation rich and powerful 

enough to represent the concepts that need to be represented? A subsidiary question is related 

to the difficulty to effectively model these concepts: what level of competency is required 

from an author to model and maintain an ontology (does he need to be a rocket scientist, or is 

just common sense enough)? Answering these questions may consist in setting up some 

experiment in modelling the semantic of a particular domain (for instance the tendering 

process), and measuring the effort required to arrive the definition of an Ontology that satisfy 

the modelling need of this domain. 

A final question is related to the amount of effort that is required to create and then, to 

maintain these ontologies. The experience of using ontologies will help to answer this 

question and in particular the key question: will the ontologies stabilize rapidly enough to 

shared ontologies that will require very minimum maintenance, or on the contrary, will we 

the ontologies will have to enter into a process of constant evolution, in which the concepts of 

the ontologies will have to be modified, and new concept will have to be introduced to take 

into account the need of the organization. 

Organizational view 

The last element is organizational and relates to role and responsibility: which structure in the 

organization will be responsible and entitled (has the resources) to author these ontologies? 

How to guaranty that these ontologies will effectively be adopted by all the parts of the 

organization? How should conflict on the definition of the semantic be managed? The 

answer to these questions will have to be addressed as part of the deployment process (related 

to the non technical dimensions of this deployment). 

3.3.4.2 Goal 2: Ontologging for supporting knowledge capitalisation 

The knowledge capitalization role of Ontologging originates relates from to the formalization 

of the knowledge of the organization. It has to be reminded that this knowledge can include 

both very concrete knowledge elements such as documents, but also less tangible elements 

such as processes knowledge (present in cases, experiences shared in a bulletin board), an 

expertise (via the explicit representation of people competency). 

Technical view




Page : 33 of 110 

Version: 1.0 


Status: Final 


The technical elements to be considered here include all the software components that can be 

used to formalize the knowledge. These elements have already been identified and include: 

The DUI (Distributed User Interface) and the Office Connector used for representing the 

most concrete knowledge elements; the user profile editor that is used by people to publish 

their competency (and other personal information); the OKE for providing a diagnostic of the 

knowledge enter, and helping the restructuring of this knowledge. 

As previously indicated, the evaluation of these software components consists in the usual 

assessment, with a more im portance focussed on productivity and ergonomics (since the 

capitalisation of knowledge should not become a too important burden for the contributors). 


The second element to consider is related to effort necessary for capitalizing the knowledge 

(annotating knowledge takes time), and its difficulty (bad knowledge annotation may result in 

a system that generate inaccurate results and noise). Another question has to do with how 

broadness of this capitalization: will this capitalization concern only the most critical 

knowledge element of the organization, or will it concern the all knowledge of the 

organization? 

The answer to this question will also depend on the productivity of the knowledge 

capitalization tool. 


The last ele ment is organizational and relates to role and responsibility: which structure in the 

organization will be responsible and entitled (has the resources) to capitalize the knowledge? 

Will all the people using the system have an authoring role as part of their activity, or will 

this knowledge capitalisation role will be dedicated to a limited number of people. In the 

former case, what will an incentive structure will have to be put in place to encourage this 

knowledge capitalization, or can we expect that people will spontaneously integrate this task 

as part of their activity? 

3.3.4.3 Goal 3: Ontologging for supporting knowledge retrieval (searching and browsing) 

The knowledge searching & browsing view relates to the effective retrieval of the knowledge 

by the people of the organization to better do their work. 


The technical elements to be considered here include all the software components that can be 

used to browse, search and retrieve the knowledge. These elements have already been 

identified and include: The DUI (Distributed User Interface) for its role searching and 

browsing functionalities, and the user profile editor for viewing the different people profile. 

Since these tools represent the main user interface of the system by the users (both expe rts 

and non-experts) and will finally determine the perceived and substantive value delivered by 

the system to the organization, a particular emphasis will be the ergonomics of the system, 

and its powerfulness (expressiveness of the search engine, accurate level of the results).




Page : 34 of 110 

Version: 1.0 


Status: Final 



The second element to consider is related to the effective use of the system to support the 

work of the knowledge worker. 

Will Ontologging system become a central tool supporting the work of the knowledge worker 

(use d intensively in many of its activity), or will it only be peripheral, and be used in some 

limited and not very critical case? 

The answer to this question will obviously depend a lot on the ergonomic of the tools and of 

their powerfulness. However, it also depends on more profound consideration related to the 

effort to put to knowledge capitalization, and knowledge obsolescence. For instance, it might 

be desirable not to try to capitalize every knowledge of the organization (knowledge that can 

become too ra pidly obsolete to justify the effort), more only the more critical one (important 

document, process knowledge, etc.), and put more effort on the mechanisms (such as the 

description of the people expertise) that facilitate the circulation of the tacit knowledge of the 

organization. 


The organizational view in this case relates to the importance the INDRA organization will 

see in using the system across the organization. The use of the main browsing & searching 

tools is intuitive enough to be used very easily by any user of the organization, once perhaps 

solved the questions of confidentiality (which as to be decided at the organization level, and 

can be very critical for INDRA because of its involvement in the defence sector). Other uses 

could also be considered, such as the possibility to open the system (searching and browsing 

only) too some close partners via an Intranet. 

Technical limitation (the DUI is a special client that need to be installed and not a web 

application) may also play a role in giving the INDRA organization the flexibility to deploy 

the Ontologging application. 

3.3.4.4 Goal 4: Ontologging for facilitating knowledge sharing and exchange 


The technical elements to be considered here include all the software components that 

facilitate the exchange of knowledge in the organization as well as the sharing of the 

knowledge among people. These elements include: the Ontology Editor (including its agent 

coordination component) for its contribution to the creation of a common language used 

across the organization, the Ontology mapping tool, for its contribution the better 

interoperability of the communication between departments (for connecting Ontologies that 

have been designed separately), and the user profile editor for its support to the management 

of the tacit knowledge (facilitating people to people communication via the contribution to 

people expertise and trust).




Page : 35 of 110 

Version: 1.0 


Status: Final 



The second element to consider is related to the effectiveness of Ontologging for supporting 

knowledge exchange between departments, and knowledge sharing amongst people. To 

which extent a more explicit specification of the semantic of the concepts manipulated by the 

organization will really contribute to the communication process? Will it reduce the risk of 

ambiguity? Will it increase trust (because understand better each other)? Will it reduced the 

coordination time (because people share the same understanding of the concepts, and do not 

need to invest as much time to know how to communicate with each other)? More concretely, 

what will be the issues related to the mapping of different Ontologies in term of cost and 

difficulty, and will it generate a really useable solution (for instance if the mapping conduct 

to inconsistencies)? 

Organizatio nal view 

The organizational view relates to the importance the INDRA organization is attaching to the 

Ontologging system to contributing to the communication and interoperation between the 

different departments, and the development of a shared culture. Are there any role, resources, 

and responsibility to be assigned or dedicated to support this role? On the knowledge sharing 

sides, will the availability of tools guaranty the willingness of people to share their 

knowledge (what’s in for me? in particular concerning the sales forces)? Will there be some 

incentive system to be set? Will the effective working of the approach need the commitment 

of the top management? 

This pre-selection of the elements to be evaluated will be refined in the next chapter in the 

elaboration of the evaluation plan, and in particular take into account the efforts that can 

effectively accomplished for the evaluation (the evaluation of all the pre-selected elements 

might not be possible).

4 Gathering the Data 




Page : 36 of 110 

Version: 1.0 


Status: Final 


In this section, we are going to present the orientation chosen for this evaluation (qualitative 

rather than quantitative), the selection and elaboration of the instruments for capturing the 

data, as well as a description of the process of gathering of the data that was conducted. 

4.1 Previous work 

We do not pretend that the use of Ontology in the context of knowledge management is 

totally new, and that nothing has been done in the field of evaluating the performance of 

these systems. However, we believe that the previous work does not address (or at least only 

partially cover) some of the critical dimension of the contribution of ontology to the design of 

the next generation knowledge management systems, in particular related to the new usages. 

In the next chapters, we are going to examine different approaches that have been used to 

evaluate ontology-based knowledge management systems, and that we could consider using 

for our evaluation. 

4.1.1 The IR (Information Retrieval) approach 

Many of the previous researches work on evaluation originate from a database conceptual 

background, and focus in evaluating the performance of information retrieval (IR) from a 

search perspective. In this context, one of the principal methods used for the evaluation is the 

“precision and recall” method (Ragha van, Jung, and Bollmann, 1989), which consists in 

measuring the noise associated to the result of a search (percentage of relevant documents 

among the retrieved ones) and the coverage of this search (percentage of retrieved relevant 

documents among all existing relevant documents). 

This method has been applied in several cases to the evaluation of Ontology-based system 

(Ehrig, 2002), sometime to demonstrate how Ontology can contribute to the improvement of 

the search process (Aitken and Reid, 2000; Clark, et al, 2000). 

Investigating how an ontology-based knowledge management system can compare to a 

traditional document-centred (IR focussed) present however little interest. Indeed documentcentred 

knowledge management systems have mainly failed to get widely adopted by the 

companies (with the exception of some niches such as competitive intelligence), probably 

because the functions they propose and usages they support do not really fulfil enough the 

companies’ and people’s needs. Besides, trying to make a innovation only to improve the 

support of an existing practice is doomed to fail. Innovations succeed because of their 

capacity to invent and to support new practices. The example of the Object oriented databases 

technologies that have tried in the past to mimic the approach of the older relational database




Page : 37 of 110 

Version: 1.0 


Status: Final 


technology (defining a schema that appeared not to be flexible, inventing a similar –even if 

more powerful- query language) is here to testify that this is not a good strategy. 

4.1.2 The Ontology-based approaches 

Ontology-based approaches clearly follows another direction grounded to the theory of the 

semantic networks, which we believe is more conform to the “grand vision” of the semantic 

web (Berners-Lee, Hendler, and Lassila, 2001). According to this perspective, the underlying 

idea consists in the explicit definition of the concepts of a domain, the explicit structuring of 

the knowledge and information via semantic networks, and the navigation through these 

networks. In other words, the identification of know ledge happen principally not by searching 

terms that have been previously clearly defined (the user knows very well what he/she is 

looking for), but by the serendipitous discovery of this knowledge (the user do not have a 

precise idea of what he/she is looking for, and the navigation process helps him/her to 

concretise it). 

Note: we can considerer that these two modes of searching to be complementary, the former 

one being more appropriate in a context of relatively well specified knowledge processes, 

whereas serendipitous discovery being more adapted to support more fuzzy and creative 

knowledge processes (increasingly important in the learning organization). 

Actually, ontology-based systems include aspects that go much beyond knowledge retrieval, 

and in particular include elements such as ontology building and evolution, content 

population and maintenance, and new usages (peer-to-peer knowledge exchange in 

communities, knowledge evolution and usage monitoring, etc.). 

Two different perspectives can be distinguished for the evaluation of a semantic web 

approach: 

(1) An approach aiming at evaluating the performance according to an operational 

perspective (Angele and Sure, 2002) and the capacity of the tools to support the 

different phase of the design and the operation of a system; 

(2) Another approach aiming at evaluating the usage and business value of ontologybased 

systems (OntoWeb, 2002). 

4.1.2.1 Evaluation according to the performance perspective 

The first approach addresses the efficiency dimension (covering the leve l 0 and 2 of our 

adapted Kirkpatrick model) and consists in the evaluation of the usability of ontology-based 

tools for supporting the different phases intervening in a knowledge management system that 

have been mentioned previously (ontology building, content population, knowledge retrieval, 

etc.).




Page : 38 of 110 

Version: 1.0 


Status: Final 


For instance (Missikoff, Navigli and Velardi, 2002) reports some experiences in the design of 

domain ontology building, and for instance the use of some tools (OntoLearn) for speedingup 

the elaboration of such ontology and its content population. These tools propose for 

instance to provide support to the automatic extraction of concepts, to the validation of the 

extracted concepts and to ontology content population. The assessment of these tools has 

used as criteria some proxy for the quality of the ontology (coverage, consensus and 

accessibility) and for the quality of the content (measured via precision and recall). 

In a similar way (Guarino and Welty, 2002) proposes, with OntoClean, a methodology for 

designing and evaluating “good” ontology, in a more systematic manner than what is the 

current practice. 

Whilst these approaches are useful, and important to really asset the quality of a technical 

system, they do not guaranty the value of this system according to an organizational 

perspective. In particular, they fail to recognize (or do not justify) the profound factors that 

are important to the real adoption and success of a system in an operational environment. 

Besides they can, in particular in an early stage of a research (pursuing radical innovation), 

distract the attention from the factors that are really matters. We refer here to the metaphor of 

“the person looking for his lost keys under the street lamp” (although the keys were most 

likely lost far from the street light, the person laments that he has little chance of finding the 

keys in the dark so why waste time looking there). Indeed, the availability of instruments 

introduces some cognitive bias, and in particular tend to reinforce research in areas that have 

already been explored in disfavour of more peripheral areas that can potentially lead more to 

radical innovation, and for which the instruments has not been invented yet. For instance, in 

the case of an Ontology-based system, such an evaluation may acknowledge the efficiency of 

a search mechanism but fail to recognise that value of this mechanism is limited (people use 

mainly navigation). In a similar way, this evaluation may acknowledge the capability of this 

system to manage an important quantity of documents (given a sophisticated model to 

capitalize documents) but fail to recognise that in a real environment, it may be undesirable to 

formalize extensively documents (because of problems of obsolescence, and because of 

problems of diminishing (or even negative) return related of the generation of noise). For 

instance, (Buckingham Shum 1997) advocates to avoid premature formalization and that only 

stable and sanctioned knowledge should be formalised. This confirms the personal experience 

of the author of this Ontologging evaluation report that documents represent generally chunk 

of knowledge that outdate very rapidly, and therefore should not be systematically 

formalized. This is to compare to “knowledge items” such as projects, people and topics on 

the contrary have a much longer lifespan (experts usually do not easily radically change their 

domain of expertise, projects have generally stronger identity than documents and can be 

recalled a long time after their completion, topics are not really changin g but are mainly 

getting richer) and that are better candidate for extensive capitalization. 

To conclude, we believe that this category of evaluation is particularly meaningful in a 

second stage once the ideas underlying an approach have been clarified and validated. Used 

too early, these methods bring the risk of distracting the attention to the investigation of the 

radically new knowledge management practiced that Ontology-based systems support. To be




Page : 39 of 110 

Version: 1.0 


Status: Final 


also frank, these methods also usually require a cons iderable amount of effort and resources 

that may better be allocated in an initial stage, to more qualitative mode of evaluation. 

4.1.2.2 Evaluating according to the usage, perception and business value perspective 

The second approach is more oriented towards usage, cognitive perception and business 

value of the systems and the tools, and addresses the effectiveness of the approach (covering 

the level 3, 4 and 5 of our adapted Kirkpatrick model). The objective of the evaluation 

consists in this case in assessing to which level the new approach support new practices, that 

are more effective at the individual level (for instance by supporting higher level cognitive 

processes) and at an organizational level (for instance, by enabling the organisation to be 

more flexible, more continuously learning, more able to exploit previous experiences, and 

more creative). 

Practically, the nature of the evaluation is more qualitative, and relies on instruments such as 

questionnaire or experiments via scenarios (Giboin et. al., 2002) trying to elicitate needs, 

perception, and more effective usage patterns that could be better supported by a technical 

infrastructure, and offered to companies. 

4.2 The approach used for evaluating Ontologging 

The Ontologging project possesses the following characteristics: 

• A domain relatively new, for which little previous work (theories, experience, etc.) is 

available. 

• The relatively immaturity of the technologies (the system is still far from an industrial 

product). 

• The limited size of the “population” conducting the test (it is difficult indeed to get a large 

sample of users for a system that is only in a prototypical stage). 

• The desire to evaluate the high-level usage value of the knowledge that is generated (and 

in particular, radically new usage patterns). 

Besides, it was decided to orient the focus of the evaluation towards the effectiveness of the 

approach (what is the substantive value delivered to the users?), rather than the efficiency of 

the technical infrastructure (are the tools “functioning” well?) that was being developed. 

As a consequence, it was decided to use a more qualitative form of evaluation. 

Questionnaires were therefore elaborated and their result analysed, focussed groups and 

experiments (cognitive walkthroughs) were conducted. 

Note:




Page : 40 of 110 

Version: 1.0 


Status: Final 


The use of more quantitative and “rigorous” methods was actually considered at the 

beginning of the project, but they were finally discarded, because they would have distracted 

the attention of evaluating the new usages (the core of the innovation), and would have 

necessitated too much resource that would have to be taken from the qualitative evaluation. 

A next chapter of this document will however indicate some directions of how to conduce a 

more quantitative evaluation that could be envisaged in the future (as an initial stage of a 

productarisation effort). 

4.3 The definition of the instruments and the gathering of the data 

The Ontologging system has been installed in the three competency centres, and has been 

tested by a number of employees belonging to theses centres. 

Questionnaires were the instruments that were the most extensively employed for capturing 

the data, since this method is relatively simple to set-up, and is particularly well adapted to 

accomplish a qualitative evaluation. But other methods were also employed to complement 

this capture such as: focus groups , interviews and experiments. 

4.3.1 The context of the evaluation 

Let’s now enter more in the details of this evaluation by presenting the context in which this 

evaluation took place. 

The evaluation of how the Ontologging system is able to support knowledge processes has 

mainly taken place at INDRA, an important (6000 employees) Information Technology 

service Spanish company, and to a lesser extent at Meta 4, a smaller IT company designing 

enterprise software products (such as accounting, people management, knowledge 

management software). 

In the following part of this document, we will principally focus our attention on the 

evaluation that has been conducted at INDRA. 

The evaluation of the Ontologging system at INDRA has consisted in testing the systems for 

supporting three activities: 

• Tendering. 

This activity consists in the elaboration proposals answering to call for tenders for new IT 

contracts. 

• Development 

Business development activities (looking for new projects, etc). 

• Technology watch. 

This activity consists in the elaboration of reports on topics/disciplines that are considered 

as important for the INDRA organization.




Page : 41 of 110 

Version: 1.0 


Status: Final 


These activities are fulfilled by a horizontal structure of the IN DRA organization (the 

competency centres), which role is to provide more long-term services to the operational 

centres (the production centres and the sales lines) of the INDRA organization. 

Competence centres mission is to lead the innovation of Indra’s services and solutions. These 

centres provide support to the operational centres (the production centres), and in particular 

play the role of interface with the business lines (the commercial forces in contact with the 

clients), lead the innovation and capitalize expertise for INDRA. See the Annex 2 for a detail 

description of the role of competence centres. Annex 2 also presents an overview of the 

tendering process at INDRA. (INDRA, 2002) 

The profiles of the employees participating to the test are diverse (consultants, software 

analysts, system architects), and the employees originate from different domains (security, 

ERP, etc.). 

Besides, different roles can be distinguished such as: 

• The end-users (focussed on content) 

o The occasional users. (Main activity: browse and search the content) 

o The traditional users. (Main activity: browse, search & edit) 

o The “power” users. (Main role & activity: browse, search, edit & organize) 

• The Knowledge Managers (focussed on structure) 

o The technical knowledge manager 

Main role: set-up and maintain the IT/knowledge infrastructure; activity: set-up, 

clean, merge, restructure. 

o The knowledge management consultant / manager. 

Main role: help to structure the work processes in the organization; Activity: 

defines ontologies. 

The size of the group of the users participating in the evaluation has been decided in with the 

objective of: (1) collecting a reasonable amount of data that will make the result of the 

evaluation useful and significant; (2) making the evaluation doable (realistic) in the time 

frame, and with a good level of quality (the running of session can be considered as a 

relatively “heavy” operation). 

4.3.2 The Questionnaires 

A series of questionnaires has been elaborated to collect the data: 

• An early questionnaire (given in Spanish), which aim was to make a very early 

assessment of the situation and of the users’ need. 

• A pre-questionnaire, which was to get a good understanding of the users participating to 

the test. 

• A questionnaire for evaluating the Ontologging knowledge structuring and content 

population process.




Page : 42 of 110 

Version: 1.0 


Status: Final 


• A questionnaire for evaluating the usage of the document -centred tools of Ontologging 

system by the final end-users. 

• A questionnaire for evaluating the more user -centred tools of the Ontologging system by 

the final end-users. 

These questionnaires have been distributed both to INDRA and to Meta4 (for which not all 

the questionnaires have been distributed though). 

The process of the elaboration of the questionnaires was relatively straightforward, and 

included both close questions (the user has to select a choice) and open questions (the user is 

asked to use free text to answer). The main difficulty was to address the risks of 

misunderstanding of some of the terms used and of the blank answers (questions considered 

as irrelevant to some users). The experience showed that these problems were not totally 

avoided since a few misunderstanding were identified (such as the definition of the semantic 

of a knowledge management tool), and some of the questions requiring some effort from the 

users (such as the description of knowledge management usage scenario) were left answered 

by some users. 

An example of an answer indicating that the definition of knowledge tools was not totally 

clear is given bellow: 

“Google may not be KM in pure sense as I understand it, but it definitely solves most of my 

problems most of the time”. 

4.3.2.1 The early questionnaire (in spanish) 

The objective of this early (spanish) questionnaire “Cuestionario de Opinión sobre 

Características para Seleccionar Ontologías” was to get the initial situation and needs of the 

organization INDRA. This questionnaire was given at an initial stage of the project, well 

before the prototype was designed, and can be considered as participating to the formative 

evaluation of the Ontologging system. The result was used as an input for the user need to 

help to define and drive the design of the system. 

The content of this questionnaire was relatively high level, and was principally addressing 

aspects related to knowledge representation via Ontology. The content of this questionnaire is 

available in Annex 7. 

4.3.2.2 The pre-questionnaire 

In a first stage when the prototype was still not fully functional, a pre-questionnaire has been 

used to collect data related to the end-users knowledge and practices of knowledge 

management. The objective of this pre -questionnaire was to get a better understanding of the 

people participating to the test and also to help to adjust the design of the prototype before its 

initial release.




Page : 43 of 110 

Version: 1.0 


Status: Final 


This pre-questionnaire surveyed knowledge processes of the knowledge workers, what are 

the knowledge management tools used, what are the perceived needs of the users for 

knowledge oriented processes and what are the suggested improvements related to current 

KM tools. 

Practically, this questionnaire has consisted in a series of question related to: 

• The profile of the participant: Title, position and role in the organization (programming, 

managing, consulting, marketing, sales, etc), knowledge activities in which he/she is 

involved (write reports, write tenders, programme, prospect clients). 

• The perception of this participant of the discipline of knowledge management and its 

current situation (opinion). 

• His/her experience and practices in this domain (what were the knowledge process used 

to perform the work). 

• His / her expectation in this domain (what are the support he/would need / like to have). 

This questionnaire was distributed to the group of users a little time before the system was 

deployed, and 14 questionnaires have bee n collected. 

An extract of the answers to the questionnaire to some of the open questions related to the 

KM definitions and suggested problems related to KM are given bellow: 

“For me a KM tool is the one that allows me to access the information I need to perform the 

tasks associated to my job in a fast and efficient manner”. 

“KM should help people benefit from experience not to have to redo things again”. 

“Part of the problem is to have a system in which people are willing to contribute.” 

KM tools don’t improve work. “That’s because currently it’s necessary to spend to much time 

looking for material.” 

The content of this questionnaire is available in Annex 5. “Annex 5: Pre -questionnaire for 

the participants in the Ontologging usability test”. 

4.3.2.3 Questionnaire 1: Ontology-based approach for the structuring of knowledge 

The objective of this questionnaire was to evaluate the Ontologging system (tools and 

approach) capacity to help the structuring of the knowledge (and in particular the elaboration 

of the Ontology that was to be used to organize the knowledge) and the population of the 

content. Practically, this questionnaire was trying to identify the main limitations and 

problems that occurred during the process of design of the main ontology, both from a 

conceptual point of view (methodology used), and from a technical point of view (how did 

the tools helped to support this process). 

The content of this questionnaire is available in Annex 6.




4.3.2.4 Questionnaire 2: Ontologging Project Questionnaire 

Page : 44 of 110 

Version: 1.0 


Status: Final 


The objective of this questionnaire was to evaluate the main document-centred tools and 

approach of the Ontologging system, and in particular the DUI (Distributed User-Interface). 

The final users were asks: 

• To indicate some usage scenarios about how they had used the tools. 

• To provide feedback on the efficiency of the main Ontologging tool (DUI). Were the 

functionalities complete enough, did it have a good ergonomic, etc.? 

• To provide feedback on the effectiveness of this tool. Was the tool supporting well 

knowledge management processes? 

• To indicate more in detailed knowledge management processes that were supported. 

• To provide a longer-term (3 years) perspective about possible and desire evolution. 

The content of this questionnaire is available in Annex 9. 

4.3.2.5 Questionnaire 3: Evaluation of user modelling processes and the evaluation of the 

knowledge distribution agents 

The objective of this questionnaire was to evaluate the Ontologging usage according to its 

more people-centred dimension, and in particular to assess the aspects related to the 

modelling of the users, the sharing of knowledge amongst the users, and the knowledge 

distribution via agent mechanisms. More specifically, this questionnaire has collected data 

concerning the user profile editor tool and the knowledge distribution agent mechanism. This 

questionnaire has also addressed the question of privacy and knowledge sharing, as well as 

the personalisation issues. 

The content of this questionnaire is available in Annex 8. 

4.3.3 Focus groups 

A series of meetings were organized with the user participants at different periods of the 

advancement project. 

First, several meeting were organised between INDRA and the development teams in order to 

introduce the concepts of the system to the users, and raise their interest. These meetings 

were also the opportunity to assess the initial situation and to collect the user needs and 

therefore contributed to the formative evaluation of the project. Some of the focus of these




Page : 45 of 110 

Version: 1.0 


Status: Final 


meeting were the identification of the current practices, the nature and the amount of 

knowledge manipulated by the different competency centres of INDRA participating to the 

evaluation, and some investigation of the issues related to the complexity of the main 

ontology to be used for structuring the knowledge. 

For instance, for the tender use case (one of the activity that will be used to test the 

Ontologging system), the original information system was organized in a similar way as a file 

system, and the documents could be searched by year project/customer. Concerning the order 

of magnitude, the use case could generate around 250 documents per year, person instances 

could add up to around 150, project instances around 50. 

Second, after he Ontologging system had been deployed, the consortium took the opportunity 

of different consortium meetings in the INDRA settings, to organize meetings involving 

some (and sometime all) the partners and the different users. The objective and focus of these 

discussions was to collect the user feedback, and in particular to identify and understand the 

difficulties and problems faced by the user as well as the different practices that had been 

adopted by these users. For instance, these discussions have helped to spot a number of 

elements that had not been really envisaged at the beginning of the project such as: (1) a 

difficulty of the design process of the ontology that had been largely underestimated, 

resulting in an initial low quality domain ontology that made the content population difficult 

and the result not very convincing, and finally to the redesign of the Ontology and the 

migration of the initial content; (2) a usage pattern that was clearly oriented towards the 

navigation via a variety of objects (projects, people, topics, etc.) connected to one another, 

rather than document search using the (ontology-based) search engine provided in the system. 

These reasons of these results, which should not come to a surprise today given the better 

understanding that we have today of the semantic web concepts, originate from the fact that 

the perspective of several of the designers of the Ontologging system was biased towards 

document centred systems (for which they had most of their experience) or towards a very 

technical and tools orientation, rather than an knowledge engineering orientation that would 

be more appropriate when designing system including an important knowledge (content) 

dimension. 

4.3.4 Interviews 

As for the focus groups, several interviews were conducted at different period of the project 

principally via phone calls. 

Interviews were particularly used during the phase of the reengineering of the INDRA 

domain ontology, and of the content migration. This phase (and the lessons learned) was 

indeed particularly rich of insights related to some of the most critical issues of ontologybased 

systems such as: (1) the quality of the Ontology: the reengineering had to address the 

problem of the initial ontology such as a set of concepts that were incomplete and not 

consistent with the “implicit” ontology used by the INDRA organization; (2) the content 

population and migration: the value of some of the knowledge engineering tools such as ORA 

(Ontology Reconciliation Agents) could be acknowledged or in some case questioned such as




Page : 46 of 110 

Version: 1.0 


Status: Final 


in the case of the agent mechanism for Collaborative Ontology Authoring or the Ontology 

Knowledge Evaluator (OKE) that still appear more than ever as potentially valuable, but for 

which easier way of operationalising them would be desirable; (3) the usage patterns and 

expectation of an ontology system: some usage feedback was available to which kind of 

knowledge should be present in the system (users were expecting a rich variety of high 

quality content but that would be limited in quantity). 

4.3.5 Experiments, via Scenarios 

Finally, a set of usage scenarios was identified (from current practices), further elaborated 

and tested to assess the capability of the Ontologging system to effectively support and 

improve some of the knowledge processes, and also to understand the usage patterns of a next 

generation of an Ontology-based knowledge management system. 

4.4 Some directions towards a more quantitative (and rigorous?) evaluation 

4.4.1 The reason of the qualitative evaluation 

As indicated previously, the Ontologging project has favoured the choice of a more 

qualitative evaluation versus a more quantitative evolution. This decision was actually not 

taken initially (a quantitative evaluation was even considered), but appeared later after 

observing the usage patterns of the end users. Indeed, contrary to the what was foreseen at the 

beginning of the project, the users did not see the Ontologging management system as a more 

powerful document management system that would allow to search more effectively an 

important amount of documents, but rather as a system able to capture the complex 

relationship between a relatively small numbers of a variety of objects (project, people, 

topics, and documents) intervening in knowledge management processes, and giving the 

possibility to the user to navigate this network of semantic relations. In this perspective, 

quantitative evaluation aiming at assessing the capability (quality) and the performance of the 

system at managing (manipulating, searching, retrieving, etc.) huge amount of data presented 

little interest since it was only covering usages (Information retrieval, and document 

orientation) that were not central for Ontologging. 

Does it mean that qualitative evaluation is meaningless? 

The answer is definitively no: as indicated previously, more quantitative evaluation should be 

useful at a later stage, once the different knowledge processes have been better understood, 

and a more precise evaluation is needed to assess how much the system is able to support 

these processes. 

4.4.2 Some direction for a quantitative evaluation




Page : 47 of 110 

Version: 1.0 


Status: Final 


Concerning the quantitative evaluation of aspects that would be more semantic web oriented 

(semantic domain definition, semantic knowledge capture, and semantic navigation), much 

less previous experience is available (than evaluation accordin g to an Information Retrieval 

perspective). 

We can first indicate some work in this direction related to semantic disambiguation that was 

done by (Missikoff, Navigli and Velardi, 2002), and which measured the performance of 

different disambiguating heuristics, and more generally all the work on Ontology building, 

and ontology learning and population that has developed over recent years. However, the 

perspective of these approaches remains very technical, and is quite far from a more semantic 

web and cognitive vision that were adopted by the ontologging users and that we would like 

to promote in this project. 

A probably better perspective to explore is the domain Organizational Memory research 

(Abecker et al. 2003), which is more in line with higher-level knowledge management 

concepts, but for which evaluation still would need more elaboration (see for instance 

(Weinberger, Teeni and Frank 2003) for some work in this direction, for instance in 

evaluating the completeness of a manual knowledge population). 

Fina lly, an even more promising approach (but also far fetched) is the approach of knowledge 

management emphasizing cognitive and social factors (Thomas, Kellogg, and Erickson, 

2001). Along this line is all the work related to social translucence, which rely on tools 

monitoring quantitatively the knowledge activity of a whole community. Obviously, the 

monitoring of the activity via the usage log represents a source of quantitative data that could 

easily be exploited (actually it is exploited by the Ontologging OKE) by quantitative 

evaluation, and in particular related to the (not only usage retrieval) usage of the system. 

4.4.3 Some prospective operationalization of “SMART” quantitative evaluation 

Let’s now indicate some direction of how we could build some quantitative evaluation for the 

Ontologging project. In the illustration that will be given later will try to follow the SMART 

principle and be Specific, Measurable, Attainable, Relevant and Time-bound. 

4.4.3.1 Ontology building 

The first quantitative evaluation could relate to Ontology building, and could assess the 

capacity of the Ontology approach and tools to support the elaboration of the domain 

Ontology. 

Different quantitative indicator could be used here: 

• The time to design or redesign the Ontology. How many hours, days, or weeks would be 

necessary to design the main Ontology, or some sub-ontology? 

• The complexity of the designed ontology. What is the level of complexity of the 

Ontology being elaborated (number of concepts, number of properties associated to this 

concept, level of nestledness / how need is the inheritance hierarchy). 

• Quality of the resulting Ontology (redundancy, ambiguity, etc.).




Page : 48 of 110 

Version: 1.0 


Status: Final 


• Completeness. For instance, when trying to capitalize a given set of relevant documents, 

or other knowledge objects, what is the percentage of time that a concept or a relationship 

appears to be missing? 

Example of a scenario of a quantitative evaluation: 

Evaluate (with the criteria that have been indicated) the design of the sub-Ontology for the 

military sector at Indra. This Ontology has to be able to be used to elaborate proposals 

ranging from 500000 Euros to 5 million Euros. The size of the existing data to be captured in 

this sector is in the order of magnitude of: 250 documents, 400 people, 80 projects, and 120 

topics. The first version of this Ontology will have to be available in a maximum 2 months 

and a half; the definitive version will have to be available in 8 months. 

4.4.3.2 Ontology content population 

The second quantitative evaluation could relate to Ontology content population, and could 

assess the effort necessary to populate the content of an Ontology with the Ontologging tools. 


• The time to populate the content of the Ontology. How many hours, days, or weeks would 

be necessary to design the main Ontology, or some sub-ontology? 

• The richness of this ontology population. Number of instances, average number of 

relationships between the instances and more generally complexity and nature of the 

resulting network (according to graph theories). 

• Quality of the resulting Ontology population (noise, navigability, etc.). 

• Completeness. Percentage of knowledge considered as relevant that has been captured. 

• Level of knowledge sharing in the organization. For instance average num ber and nature 

of the distribution, of the sharing of documents and other knowledge item by the 

population of the knowledge workers. 


Evaluate (with the criteria that have been indicated) the population of the sub-Ontology for 

the military sector at Indra that has been described in the previous phase. 

Note: some shorter evaluation experience tests could also be considered here such as: 

Time necessary to capitalize a set of 10 documents. Time necessary to model 5 people 

instance. Time necessary to model 3 projects. 

4.4.3.3 Usage




Page : 49 of 110 

Version: 1.0 


Status: Final 


The usage quantitative evaluation objective is to assess the capability of the Ontologging 

system to support knowledge related activities, and in particular to improve the productivity 

of the knowledge worker (time to complete a task, quality of the work). 


• The time to identify the knowledge items (project, people, documents, etc.) in the initial 

phase of a knowledge management task (for instance, the elaboration of a new tender). 

• Level of quality of the retrieve information (precision and recall). 

• Level of cognitive load (could be connected to the precision). 

• Level of use of the different knowledge items (statistical distribution in term of level of 

access to the different items). 


Evaluate (with the criteria that have been indicated) the elaboration of the answer to a set of 

tenders in the military sector. An idea could also consist in the comparison of the effort to 

answer to a proposal with and without the Ontologging tools. This category of experience 

may however appear not to be Attainable (or realistic) at Indra for human and deontological 

reasons. 

4.4.4 Concluding remarks on the quantitative evaluation 

As we have seen, some level of quantitative evaluation would have been possible in this 

project. However, we can observe that it is more difficult with this category of evaluation to 

capture aspects that are difficult to formalize, such as for instance qualitative practices (such 

as the use of navigation instead of search) or level of knowledge sharing that however appear 

to be key elements in this project. If we had done so, consequence would probably have been 

an over-emphasis and some bias on knowledge retrieval methods, and the overlook of the 

elements that we can consider as the most important in this projects (in particular a semantic 

web orientation). 

If in the longer term, and as a further step, we can however consider that a more quantitative 

evaluation would be very desirable (and actually almost mandatory in order to validate 

“scientifically” the knowledge generated), in particular if we manage to define some 

quantitative indicators able to measure realistically the most important elements of the 

semantic web “vision”.

5 Results & Analysis 




Page : 50 of 110 

Version: 1.0 


Status: Final 


In this analysis, we are going to distinguish and evaluate the different aspects and phases that 

correspond to the total cycle of the setting-up and the use of an ontology-based knowledge 

management solution. 

These aspects and phases include: 

• The evaluation of the initial situation (to intervene in the formative evaluation) 

• The elaboration of the structure (ontology building) 

• The population of the content 

• The usage 

• The overall result 

5.1 Phase 0: State of the situation (formative evaluation) 

An analysis of the initial questionnaire and of the pre-questionnaire has helped to make a 

state of the initial situation that was used at the level of the formative evaluation to provide 

some input for driving and adjusting the design (via a prioritisation of what appeared to be 

the most critical elements from a user perspective). 

5.1.1 Results 

Different elements were collected from these questionnaires: 

• The perceived needs of the end-users. 

• The practices and tools currently used. 

• The users’ expectation (longer term). 

5.1.1.1 Perceived needs of the end-users 

The answers to the pre-test questionnaires revealed the perceived needs and expectations of 

the end-users. In particular, users expect Knowledge Management systems to help them to 

access to information, to reuse past experience, to save time, and to locate experts. 

The following paragraph quotes some answers of the pre-questionnaire: 

(Good Access to information) 

“To facilitate access to information/knowledge relevant to the current work tasks in order to 

optimize work processes and improve productivity”;




Page : 51 of 110 

Version: 1.0 


Status: Final 


“The most important function is to facilitate access (on time) to useful information in order to 

make my work easier and best.“ 

“Searching the information that I need” 

“Discover and consume the knowledge generated by the company that could help me in my 

everyday tasks.” 

“To get briefings of news” Julian 

(Better re-use of past experience) 

“Re-use past experience”. 

“ Reuse Indra’s past experience: documents, experien ces, etc.” 

“Not loosing time studying problems which have been solved before” 

(Saving time) 

“Saving time when I am searching for a solution”, Jose. 

“When I must do a tender, I need a lot, and diverse information about products, prices, 

references, news, etc. There is, normally, scarce time to do it and to have information on time 

if it is necessary.” 

“A major advantage is saving time when I’m searching for a solution” 

“The most important functionality of this system is that it save me time in searching any kind 

of information that I need” 

(To find the right person, and get help) 

“Look for experts in order to ask for a tacit knowledge” 

“Find the right people to solve concrete problems” 

“Also some KM tool should provide information to know what people knows about something 

or has a previous experience with some technologies and products”. 

“Direct chat with experts”. 

5.1.1.2 Current practices (and KM tools used) 

In order to accomplish various work tasks, the Competence Centres employ a variety of 

(structured and unstructured) knowledge tools supporting a large set of knowledge processes. 

These tools include: 

• An Enterprise portal (Indr@web): integrates and manages a wide variety of corporate 

information channels and services. 

• A Knowledge and information repository infrastructure: databases and electronic 

document management systems 

• Knowledge maps and guides to available knowledge resources: thesaurus, taxonomy and 

ontology generation 

• Search and delivery information services to access analysis and strategy external sources 

of information




Page : 52 of 110 

Version: 1.0 


Status: Final 


• Collaboration services 

o Directory, calendar, agenda services 

o Threaded discussions (forums) 

o Asynchronous (e-mail) 

o Shared spaces (document sharing, white -boarding). 

• E-learning and human resource management portal (Employee Application Portal). This 

portal is still however in a prototypical stage. 

All these tools are not integrated under the common framework of a knowledge management 

system, but appear mainly as a set of tools independent from one another. The level of usage 

is also different and can be ranked according to the following order: email, search tools (such 

as Google), databases, communication tools, and the portal. 

It has to be noted that since the core business of INDRA is centred on information 

technologies (they propose and integrate IT solutions to their clients), the company’s culture 

is very favourable to the use of technology, and people are very keen to adopt new 

technologies. 

5.1.1.3 Suggestions for improvements of the actual KM tools 

The end-users do not seem to be satisfied by the solution currently offered by the knowledge 

management tools and ask (for instance in the pre-questionnaire) for many improvement and 

advanced features. 

Examples of desires include: 

• To better organize the content of the knowledge management tools and the ”quality of 

content” is perceived as a very important issue. “Content is not correctly organized, not 

updated or duplicated.” 

• To make experience of people more visible in organization: “to know what people know 

and to make their experience with technology and products accessible”. 

• Personalization and adaptive features “to include mechanisms in order to acquire 

knowledge about user profile and filter information and noise”, “adapt the tools to each 

company or secto r” 

• To integrate collaborative tools 

• To improve the interface and to integrate the functionality under a common framework; 

• KMSs are still “document management systems” enabling to access documents in an 

organized way. “A KM should be more powerful, should be able to relate in an active 

way people and knowledge, should be more dynamic and make know people profile 

evolve.” 

• To address Information overload. “Yes, this tools need major improvement to allow users 

use Knowledge tools in an easy way, spending few time and don’t lose among hundreds 

of document” 

• More intelligent search of documents (“not only by descriptors”)

• List of connected users to the system 

• Direct chat with experts 

5.1.2 Analysis 




Page : 53 of 110 

Version: 1.0 


Status: Final 


The analysis of the different initial questionnaire, of the pre -questionnaire, and of the 

different discussions have helped to assess the situation, to identify some (perceived) needs 

and has revealed a set of important features that need to be enhanced in actual KM tools. 

An organization already equipped but with disparate tools 

As it can be observed, INDRA competence centres are already relatively well equipped with 

information systems providing support to knowledge intensive work of the knowledge 

workers. Besides, the level of acceptance of these technologies is good, tha nks to a 

company’s culture very favourable to the use of technology. 

However, these tools are not integrated and the support of the knowledge processes remains 

shadow: most of these tools appear to be closer to information tools than knowledge tools. 

The need to better organize the content 

The organization of the content and the quality of content are some of the most important 

issues. Actual knowledge management tools facilitate access to knowledge, but the 

knowledge workers complain that “content is not correctly organized, not updated or 

duplicated”. 

Enhanced support for searching and filtering knowledge; 

The knowledge workers want to “save time when I am searching for a solution” and “not 

loosing time studying problems which have been solved before”. The enhanced support for 

filtering and searching knowledge is related to “saving time” in achieving the various work 

related tasks. 

Adaptive features and personalization 

Personalization issues are related to the issue of filtering information but also a need of easy 

to be used expressed as: “Yes, these tools need major improvement to allow users use 

Knowledge tools in an easy way, spending few time and don’t lose among hundreds of 

document.”; but also a need “to include mechanisms in order to acquire knowledge about 

user profile and filter information and noise”. Adaptation features are required in order to 

“adapt the tools to each company or sector”. 

Support for expertise finding or a need “to know what people know”




Page : 54 of 110 

Version: 1.0 


Status: Final 


Being allocated to different projects, writing projects proposals for various clients and 

proposing IT solutions are current tasks for Indra’s knowledge workers. The access to the 

right information and collaboration with experts help their efficiency and help them taking 

better decisions. Therefore it is necessary to make experience of people more visible in 

organization. The need “to know what people know, to make their experience with 

technology and products accessible” and “to find the right people” is useful for solving 

everyday tasks. 

These needs of the users validate the main research challenges of the project namely: 

• Research on mechanisms to better structuring and reusing knowledge (e.g. ontologies), 

• Research on more powerful mechanisms for searching and retrieving knowledge (e.g. 

advanced query mechanisms and the use of semantics annotations for retrieving 

knowledge), 

• Personalized information spaces for avoiding a condition of information overload and 

easiness of the use of the system (e.g. taking into account preferences of the users, the 

users’ needs, etc) 

• Better management of the tacit knowledge expressed as: “to know what people know and 

to make their experience with technology and products accessible”. 

5.2 Phase 1: Ontology Building (and some content population) 

Ontologies aim to structure and represent domain knowledge in a generic way which may be 

reused and shared across applications and groups. 

“Disparate backgrounds, languages, tools and techniques are a major barrier to effective 

communication among people, organizations and/or software systems (…) the implementation 

of an explicit account of a shared understanding (i.e. an “ontology”) in a given subject area, 

can improve such communication, which in turn can give rise to greater reuse and sharing, 

interoperability and more reliable software.” (Uschold and Gruninger, 1996). 

5.2.1 Ontology building (at INDRA) 

5.2.1.1 Description of the building process 

The process of knowledge domain modelling at INDRA, namely the tendering process 

modelling, went through an iterative phase in order to get to a shared understanding of 

concepts, to get to a complete ontology and to reach consensus towards the definition of 

concepts. The process of building the domain ontology was used in the process of the 

evaluation of the ontology modelling processes and tools. The design and implementation of 

the domain ontology at INDRA brought some interesting insights, which were captured 

through a specialized questionnaire. The questionnaire is attached in Annex 6.




Page : 55 of 110 

Version: 1.0 


Status: Final 


In a first phase, the ontology modelling process was assessed at INDRA during may-june. A 

Spanish questionnaire was designed with this purpose. (See Annex 7) Some conclusions 

drawn by the ontology engineers are briefly summarized bellow: 

• Related to ontology’s concepts: 

It is very important that the concepts match the domain’s needs. 

Concept relevance is more important that completeness. 

The number of concepts defined is not important. 

• Related to ontology’s relations: 

It is very important that relations match the domain’s reality. 

• Related to the visual clarity of the ontology: 

It is very important that the ontology interface displays in a clear manner the information 

captured in the ontology. 

From this experience of building and refining the domain ontology we have extracted some 

thoughts of the ontology engineers. The ontology engineers from Indra and Meta4 were asked 

to share their experience related to the ontology modelling process. The questionnaire was 

designed with the purpose to capture their experience, problems and it was complemented 

with some discussions via telephone. Indra and Meta4 faced different problems. 

The first domain ontology used at Indra faced a series of problems at the usage stage. For 

example the terminology used in the everyday tasks by the knowledge workers was not the 

same with the conceptualization of the domain ontology. A set of too generic concepts were 

confusing for the end-users so they were misused or used for everything. Implicitly too 

specific concepts were never used. A set of missing concepts has been also identified at the 

usage. 

5.2.1.2 Findings and analysis 

The ontology reengineering process emphasized the fact that getting to a shared 

conceptualization is not a straightforward process . The design of good domain ontology 

is an iterative process. Amongst the main reasons that determined the ontology reengineering 

process are: 

1. The initial ontology wasn’t offering enough support for the given scenario (tender, 

development, technology) 

2. The initial ontology was not complete 

3. The terminology used was not consistent with the general usage 

4. It is difficult to reach a consensus towards a definition of concepts




Some other reasons were suggested bellow by the ontology engineer of Indra: 

Page : 56 of 110 

Version: 1.0 


Status: Final 


One important problem of the ontology was located in some concepts. These concepts 

weren’t too clear to users, so they misused it, causing some chaos in the information. This 

problem was related with the terminology (proposed reason 3), because some of the 

problematic concepts weren’t used in daily work, and also related with consensus about 

concepts (proposed reason 4), because if a concept is not specific, each user will apply his or 

her own idea of the concept. 

For example: 

• Term and Topic were fuzzy concepts, so people tend to use it for everything they did not 

know where to put. Therefore the instances for these two concepts were totally 

heterogeneous because the name of the concept was too general, so each user had are own 

meaning. 

• Technology and its sub-concepts: some of these concepts refer to similar ideas (software, 

operating systems, commercial products), some of them were useless (requirements), or 

some were too general (services). The consequence of these problems was that nobody 

used these concepts and their instances homogeneously. 

In case of other concepts, the problem was with the nomenclature or formats: the format for 

dates, for numbers. For example, in the attribute duration of a project, the unit was not 

specified, so each user had its own idea about the format: using months, years, days, the 

initial and final date of the project. 

Therefore, a first conclusion may be that it is very important to use specific concepts, in order 

to avoid misunderstandings. Although the DUI includes the possibility of adding some kind 

of help to users (specifying the format and a brief tool-tip for each concept), it would be very 

interesting that it would be possible attaching a description to each concept of the ontology, 

making this description or help available in the DUI. This way, the users can always review 

the help for a concept in order to clarify their doubts without receiving a previous course 

about the ontology. For example: “The organizations related with an operation using the 

relationship -implies- can be clients, partners, providers…” 

Regarding the reasons 1, these were clearly present in the Documents concepts definition. 

This definition was very scarce, so it was difficult to distinguish the different types of 

documents involved in a proposal or in a project. Also the original ontology didn’t include 

the option to relate a Proposa l to the corresponding project (if the proposal is awarded). 

The ontology was also uncompleted in people definition, because there was only a concept 

for all the people. In the new ontology this has been addressed including three sub-concepts 

of the “People” concept.




Page : 57 of 110 

Version: 1.0 


Status: Final 


Therefore, the original ontology was too general, especially in some concepts, which are key 

for some of the proposed scenarios (especially Tendering). On the contrary, it was too 

specific in some of its branches, so a more general approach ha s been used in the new 

ontology. 

The usage of the domain ontology (queries, submitted documents) helps the ontology 

engineer to decide what concepts are missing, which concepts have not been used 

appropriately and which ones cause problems. The queries of the users can also help to 

identify new relationships between concepts. 

In addition to the suggestions from the other users some limitations of the domain ontology 

were identified when the ontology engineer used the tool as a normal user (e.g. the missing 

concepts or relationships can be detected when entering or querying data). Therefore, a first 

step to take before refining the ontology is review users’ work and suggestions, and also 

experiment the DUI as a normal user, trying to reproduce the more frequent use cases. Once 

compiled all this notes, the ontology engineer begins to make the corresponding changes. 

5.2.2 The reengineering of the Ontology 

As the level of quality of the first version of the Ontology (used for the first stage of the 

evaluation) was found not adequate (not complete, confusing, noisy, etc.), a process of major 

redesign was initiated. 

5.2.2.1 Description of the ontology re-reengineering process 

This section describes the process accomplished to update the original version of Indra’s 

domain ont ology, according to the users’ comments, and to the work made for the domain 

modellers in order to solve the detected problems and to adapt the ontology to the proposed 

use scenarios. 

A detailed description of the process of refining and migrating the domain ontology (by the 

ontology engineer of Indra) is presented bellow. 

Step 1 - ‘In paper’ redesign 

The first step carried out when redesigning the ontology was to analyse the original ontology 

to detect which ‘branches’ of the ontology store homogeneous and appropriated data and 

which ones not. This allows the managers to distinguish those concepts that are well defined 

from those one that are confused and cause misunderstandings or problems. 

It is also important that the manager use the DUI as other users, because the best way of 

viewing what the user may need is being in the user’s place. Therefore, it is important to take 

into account the use scenarios, in order to detect which may be the most frequent queries that




Page : 58 of 110 

Version: 1.0 


Status: Final 


the users would need, or which concepts may be interesting for them. These allow us to add 

those concept missed in the original ontology. 

Once a first draft of the new ontology has been sketched, it is better to work directly with the 

Ontologging system tools, which provide a richer environment to redesign ontologies, even if 

the new ontology is not totally defined. 

Step 2 – From paper to Ontologging: ‘rough’ migration 

The ‘sketched’ ontology was transferred to the system using KAON. The original ontology 

was used as a reference, since some of the concepts were the same. 

Once the first version was introduced in the system with KAON, the next task was defining 

the mappings or translations required to move the already existing data from the original 

ontology to the new one. This process was ma de using the ORA (Ontologging Reconciliation 

Agent), programming in XML the required mappings. 

This tool allowed avoiding the loss of a huge amount of data, already introduced in the 

original ontology. However, this migration process does not allow to map all the instances to 

their corresponding concept, because when a concept from the original ontology is mapped to 

different concepts in the new ontology, it may be difficult to establish a way to distinguish 

the to which concept the instances must be assigned. Therefore, it will be necessary an 

additional work to separate this instances in a later stage. 

Step 3 – Polishing the ontology – ‘sharp’ migration 

Although ORA allowed moving an important amount of data, the result still required some 

additional work of adjustment and tuning. The DUI was used at this stage for completing the 

work. The DUI was used: 

• To check if its design is correct from the point of view of the final users 

• To complete the data population with some instances that were not present in the original 

Ontology. Indeed the new Ontology had also introduced new concepts and new relations, 

for which associated instances and relationships had to be created. 

At this final stage, the work was an iterative cycle between KAON (the Ontology editor) the 

DUI, and sometime the ORA (with was used to move some data not included in the ‘gross’ 

migration). 

Following is the iteration cycle:




Page : 59 of 110 

Version: 1.0 


Status: Final 


• The DUI was used to review a part of the ontology and to see which instances were 

required to complete the attributes or relationships that are important and have not been 

included in the new ontology. 

o We had to check if those required instances or values where present in the original 

ontology and their links had been destroyed in the migration process. For this 

purpose , the KAON was used to navigate the original ontology to check the old 

values of these properties or attributes. This procedure allowed us to locate those 

instances that in the migration process were moved to a wrong concept, so we can 

take note of them to change the parent concept later with the KAON. 

• We created a lot of instances to complete the empty attributes or relationships, using the 

DUI. These instances were new, and were not present in the original ontology. 

• We modified the instances already existing in the original ontology but moved to a wrong 

concept in the new one. Using the KAON, we changed their parent concept to the correct 

one. Later, again with the DUI, we relate appropriately these instances to others. 

• Sometimes, it was easier to re-create the instance from scratch, just using the instance in 

the original ontology as a reference to copy its attributes and to re -link its relationships. 

This is so because sometimes we want also to change the tag that identifies the instance, 

perhaps because the original was wrong according to the nomenclature established. 

Therefore, in the iteration we kept going over the instances of a concept, checking if all the 

key relations were established, filling the pending values. This process implied the creation of 

new instance, which again meant reviewing all their properties and perhaps creating new 

instances. 

This process was repeated over the instances of other concept. 

Once all the data was complete, the ontology was again ready to be used for all the users. 

5.2.2.2 Analysis of the redesign process 

The Ontology resulting from the re-engineering was considered of an adequate level of 

quality, and in particular made the final system usable by the end users. 

This re-engineering represented a major effort, and took several weeks to be completed. 

This redesign was very useful in that it provided the setting for investigating deeply and 

concretely the issues related to Ontology design, one of the most critical element in the 

design of an ontology-based system. 

Indeed, we can consider that reengineering represents a facet inherent to the Ontology design 

process, and not only an operation that we had to carry out in this project because we would 

been to hasty in design of the initial ontology. Ontologies are in any case subject to 

evolutions (this flexibility is actually one of the main reason of using ontology-based system) 

during the lifecycle of a knowledge management system. Ontology building is also inherently




Page : 60 of 110 

Version: 1.0 


Status: Final 


iterative, since it is difficult to have it right the first time (actually ontology building is 

sometime considered today more as an art than as science). 

This redesign presented the opportunity to evaluate the capability of the Ontologging system 

to support the work of more experienced designers (the designers had indeed acquired some 

experience since the beginning of the project), which provided a better Outlook of how the 

Ontology design process would happen in the future. 

5.2.3 Some lessons learned. 

It is clear that the difficulty of the process of Ontology building was under-evaluated in this 

project, even if the more user -oriented participants of this project (the main user partner Indra 

and the partner in charge of the evaluation) had raised some concerns early after the 

beginning of this project. However, addressing this issue was not an easy task, and would 

have required a lot of attention that was perhaps beyond the scope of this project. 

Yet, this issue was actually not totally ignored, and was to some limited extend undertook by 

the work of (OKE) Ontology Knowledge Evaluation, which role was to provide some tools to 

do some diagnostics of the quality of an Ontology and about ontology population, and that 

should definitively help the ontology design and population processes. However, this 

category tool relies a lot on the definition of some heuristics that were not really available at 

the beginning of the project (the theoretical heuristics appeared to be very artificial and of 

little value), and are also difficult to make easy to use. Another category of work in this 

project is related to the mechanisms (agent notifications) supporting the collaborative 

authoring of the Ontologies. As indicated, this mechanism was however not really tested 

(excepted technically) for the design of the Indra Ontology given the relatively limited size of 

the ontology design team. Finally, worth to be mentioned is the sophisticated and very 

powerful rollback mechanisms present in the KAON system, which allow the Ontology 

designer to test in advance and without risks the consequence s of a design decisions. 

The use of an Ontology building methodology, such as OntoClean (Guarino and Welty, 

2002)), the benchmarking with projects having to confront the same difficulties (see for 

instance (Missikoff, Navigli and Velardi, 2002)), and a be tter identification of tools and 

environments supporting Ontology building may have probably helped to identify earlier the 

problems and provided some directions for overcoming these problems. 

5.3 Phases 3: Content population 

Two tentative of ontology content population can be distinguished: An initial content 

population that happened with the initial domain Ontology, and a final content population 

that happened with the new re-engineered Ontology. 

The initial tentative of ontology content population was done by different users with 

moderate level of expertise, that have tried to capitalized into the system using the DUI, with 

documents and information that they though would be useful. As already indicated, they




Page : 61 of 110 

Version: 1.0 


Status: Final 


found this capitalization process difficult and painful, first because the quality of the 

Ontology was bad (not the right concepts) and second because they did not have some 

guideline or style to follow. The result was not surprisingly of low quality: not homogeneous, 

and noisy (the end-users complaine d that the knowledge had “a lot of crap”), and almost 

unusable. 

The second (successful) tentative of ontology content population was accomplished only by 

the limited team (two-three) of people that elaborated (or reengineered) the main Ontology. 

This manner of proceeding had the advantage of better controlling the quality of this 

population, since experts were used for this purpose. It had also the advantage of making the 

processes of ontology elaboration and population happen concurrently and in an inte rrelated 

way with the design of the main Ontology itself, facilitating therefore the elaboration of this 

Ontology (populating the content helps in the identification the “good” concepts to be present 

in the ontology). 

Whilst the quality of the resulting Ontology and Ontology population was considered as good 

(the designer dedicated a lot of effort for this), this situation cannot be considered as 

satisfactory. Indeed, every person in the organization should be able to share his/her 

knowledge with others and contribute to “feed“ the electronic memory of the organization. 

We can however imagine (this can even become a research question to investigate in the 

future) that this process of collective content population would be easier once a minimum of 

Ontology population has been accomplished by the team of expert (users have good examples 

and styles to follow), in particular if it is followed by a set of accompanying measures 

(coaching, training, knowledge population review process, etc.) guarantying that the quality 

is preserved. 

What is the conclusion and lesson learned of this population? That as for Ontology-building, 

Ontology content population has appeared to be more difficult than what was expected. 

Besides, this problem do not seem to be related to the tools for Ontology population that have 

been designed in Ontologging, but more profoundly connected to Ontology theories and 

usage methodologies. Indeed, the quality of the content in Ontology-based system is more 

critical than in the case of traditional information systems, and the capitalization process 

inherently more difficult (because for instance of ambiguity (Garigue, 2003)). 

The answers to these questions would require some further investigations. The directions that 

have already been indicated for Ontology building appear also applicable and consist in more 

methodological work, as well as some work on tools and environments helping to support 

these methodologies. 

5.4 Phases 4: Evaluating Ontologging “knowledge retrieval” 

The evaluation of the system from an end user perspective (knowledge retrieval) has 

distinguished two dimensions that will be presented in the next chapter: first, a basic




Page : 62 of 110 

Version: 1.0 


Status: Final 


knowledge retrieval dimension aiming at evaluating the capability of the system to support 

the more basic knowledge retrieval processes (navigating and searching knowledge); second 

a more advanced dimension more centred on the users and groups of people in the 

organization, and the advanced means to support these users and groups (via personalization 

mechanisms, support for knowledge sharing, etc.). 

5.4.1 Evaluating of the basic knowledge retrieval 

Questionnaire 2 (Ontologging Project Questionnaire) was used to collect the “user” feedback, 

related to the main usage of the Ontology system. In particular, this questionnaire has helped 

to collect information related to the use of the central Ontologging tool: the DUI (Distributed 

User Interface). It has to be reminder that this tool allows the final user to visualize 

knowledge, to navigate into the knowledge, to search knowledge, and also to add new 

knowledge items (knowledge capitalization). 

Some additional feedbacks were collected from the different focus groups and interviews that 

were organized. Finally, experiments (via scenarios) were used to validate different usages 

and in particular to identify the difficulties, and to elicitate (cognitive walkthrough) the 

internal cognitive process followed by the end users. 

5.4.1.1 Evaluating the main tool (DUI) 

The main tool (the DUI) was well perceived and considered as adequate, but some 

improvement would be well appreciated. 

Bellow, are some opinions extracted from the questionnaire: 

The system is all right … 

“The word enjoy is not correct (but) the DUI is ok and the experience is satisfactory”, “(liked) the 

flexibility of the tool to navigate the taxonomy, enabling and disabling the desired relationships, 

choosing which concept to see and which one not”; “(liked) to have a global and comprehensive view 

of the elements, entities and items involved in the tendering process”, “(liked) the possibility to search 

information through link in natural language”, “(liked) using the interface for navigation”. 

… but not perfect. 

“The way of showing all the information related with one instance is not very useful, is much better 

viewing everything navigating the ontology”, “Could be more attractive” , “(problem with) 

Navigation when there is too much documents stored”, “Sometimes it’s difficult to know the best way 

to perform the searches” , “(disliked) The hierarchical representation of the information. I prefer a 

more graphical layout.” 

5.4.1.2 Usage scenarios 

More interesting are the usage scenarios of the tools that help to understand how the end user 

perceived and appropriated the system.




Page : 63 of 110 

Version: 1.0 


Status: Final 


The scenarios bellows represent use cases that the users were asked to elaborate in order to 

later experiment the system: 

Tendering scenario 1: Finding ‘experts’ in a specific knowledge area. 

Approach: Use of a query template called ‘Experts in Knowledge area’. 

Examples of selections to enter in the query are: OCR, ‘Automatización de procesos 

(Workflow)’, ‘Gestion de redes’ 

Tendering scenario 2: Querying all the activity related with a specific customer 

Approach: Use of a browse template called ‘Customer’ and so you can browse all the 

operations related with a customer, and all the attributes and relations of these operations. 

Examples of selections: 

o AMENA: to see the status of the different operations (activating the two Current Status 

properties) 

o Comision Europea (IST): To see involved companies (activating the involves relation) 

o AENOR: To see that Indra has a strong position in EDM technology in this client 

(activating refers-to relations) 

Tendering scenario 3: Viewing the operations categorised by the market. 

Approach: Use a browsing template (Operations by Market) 

Examples of selections: 

o ‘Administración Pública y Sanidad’: many operations 

o Energía, Operadores y Media: médium number of operations 

o Espacio: No operations at all 

Project development scenario 1: Finding the people who worked on a proposal 

Approach: Just navigating the ontology, going to operations->proposals and selecting some 

specific as ‘Implantación de un Proyecto Piloto SIG’ (activating has-participant and refers-to) 

to see that Juan Antonio Alamillos is responsible, and then we can navigate through the 

Knowledge Area to see other similar projects. 

Project development scenario 2: Searching documents and contact people related to a third 

party product. 

Approach: Just browsing the ontology and going to products, activating (~related with 

(products)) to see related documentations, and activating ‘manufactured-by’ to get the 

manufacturer, then the produced-by and then the ‘~works-in’ to see the contact people for 

this product. 

Examples of selections: ‘Eyes & Hands FORMS’, ‘BEA Weblogic’




Page : 64 of 110 

Version: 1.0 


Status: Final 


Project development scenario 3: View original proposal and project management issues 

corresponding to a project. 

Approach: Use of a query template called ‘All management documents for a project’. Then 

you can view the type of each document activating the Parent Concepts property to see the 

concept for each document. 

Examples of selections: Smartgov, SINEA 

Examples extracted from the Questionnaire 2 are also given bellow: 

“I need to write an specific type of document, and I search documents of the same type in order to 

have an index, some examples, ideas about how to structure the information… Even I can find other 

documents of the same type and directed to the same customer, so that I can see if they use an specific 

format, and if usually they ask for some specific information”. 

“I have to add Indra’s references in a tender, showing in which similar project the company has 

worked. Using the DUI, I can search project or proposal related with the same knowledge area, or 

with customer in the same market, for example… or even previous experience with the same 

customer.” 

“I am working as developer in a project and the user is asking me for some requirements that are new 

for me. I can search my project, the related proposal, and locate if that requirements were originally 

included in the contract or in our proposal, or the customer is trying to get more developments for 

free. I also can locate the person who made the deal, so I can talk with him about the relation with the 

customer, how flexible we have to be…” 

“Tenderi ng process 

Purpose: to get effective co-ordination of tendering process between parties involved” 

“Project Development. Designing of the system and programming (developing). I used the tool for 

searching documents related with my project in the designing and developing stages.” 

etc… 

5.4.1.3 Analysis 

One of the main finding of this study is that people have adopted a view of knowledge 

management system radically different than the document-oriented and search driven 

approaches of the traditional knowledge management systems. Practically, the users have 

reasoned in a way that is much more conform to the vision of the semantic web, and which 

consists in visualizing and navigating a web of a variety of knowledge elements. 

This approach should not come to a surprise today when we know about some the recent 

orientation of the work conducted in the Ontology-based systems, and in particular the 

Ontology based portals that have flourished in different places (see for example the OntoWeb




Page : 65 of 110 

Version: 1.0 


Status: Final 


project web site at http://www.ontoweb.org/). However, it was relatively unexpected to some 

of the designers of the Ontologging system which experience in knowledge management 

system was very document centred. 

5.4.2 Evaluating of the user-centred usages 

Questionnaire 3 (Evaluation of user modelling processes and the evaluation of the knowledge 

distribution agents) was the main questionnaire used to evaluate usage-centred usage of the 

Ontologging system. Questionnaire 2, also was useful to collect information related to 

knowledge sharing and motivational aspects. 

The problem of user modelling in KMSs relates to the last two issues mentioned above 

namely: the information overload issue and the need to better manage the tacit knowledge. 

The need for enhance d user support for filtering and retrieving the knowledge available in the 

system expressed as “to not get lost” amongst hundreds of documents and to filter 

“information and noise” relates to research on personalization and adaptive hypermedia. A 

series of user modelling techniques for personalized interaction enables to build systems that 

adapt to the user’s characteristics. The evaluation of the user modelling tools has been done 

combining the questionnaire with other empirical evaluation methods such as: focus group discussion 

and semi-structured interviews. 

The main issues addressed by the evaluation of the advanced user centred usages were: 

• Employees view on sharing personal information and user modelling processes 

• The perceived need of personalization of KM tools and the use of knowledge distribution 

agents 

• Knowledge sharing incentives 

What is the end-user’s view on sharing personal information and user modelling? 

Personalized systems require users to submit user data (personal information). The disclosure 

of user data opens a series of problems like privacy and security but it also opens up new 

forms of personalization, communication, collaboration and social interactions. Some of the 

user data can be acquired explicitly by filling in a form or implicitly by various usermodelling 

techniques. 

In our case the UPE (User Profile Editor) enables the users to enter and update personal 

information (instantiate the user ontology). The user is in control of his “user profile” data. 

The UPE also enables to visualize the other’s profile in order to support collaboration and 

communication between the employees. 

User modelling techniques enable to capture certain characteristics of the users interacting 

with a KMS, a so-called behaviour of the users in the system. But the integration of the user 

models and user modelling in KMSs is a sensitive issue. The questionnaires and semi-




Page : 66 of 110 

Version: 1.0 


Status: Final 


structured interviews with the end-user emphasized the fact that certain users are concerned 

with privacy and trust issues. This categor y of users seems to be reluctant related to the use of 

their data by the organization. Therefore according to the user opinion the user profiles 

should be made partial available to the other end-users and fully available to human 

resources. 

What would be the main motivation for knowledge sharing and creation for Indra end-users 

(money, virtual rewards, recognition versus other incentives)? 

The behaviour of the users in the system can be associated with incentives provided to the 

users to share their knowledge and be active in the system. Of course the issue of sharing 

knowledge and contribute in the system is complex and it shouldn’t be limited to simple 

incentives. It might imply changes of the current work practices and it can be associated with 

other managerial interventions. We have surveyed different types of incentives a company 

might use to stimulate knowledge sharing and knowledge creation. According to the user’s 

opinion from Indra, reputation and promotion in organization would be the right incentives to 

stimulate a knowledge sharing culture in the organization. However a bonus associated with 

the salary seems to be also a right incentive for experts to spend extra time-sharing their 

knowledge. Some expert knowledge workers have expressed their concern in being 

recognized as experts and having to do extra work. 

Conclusions on the evaluation of the advanced user -centred mechanisms 

o Personalization of KMSs is important due to two important factors: “information 

overload” problem and the heterogeneity of the users. Ontologging uses very simple 

personalization mechanisms such as: the adaptation of content using templates and 

notifications of users through knowledge distribution agents. Both mechanisms are 

perceived as very useful by the knowledge workers. 

o Expertise modelling is important because it facilitates collaboration between the peer 

knowledge workers and implicitly facilitates taking decisions and work processes. At the 

same expertise modelling facilitates a better management of the tacit know ledge of the 

organization. 

o Some users are concerned with privacy and trust issues and therefore user’s profile should 

be only partial available for all the users of a KMS 

o Recognition seems to be key incentive for knowledge sharing; 

5.5 A Comparison with a more traditional knowledge management system (KnowNet) 

To conclude our work, we have also compared the Ontologging system with a more 

traditional knowledge management product (KnowNet) that is currently developed by Meta4, 

one of the main partner of this pr oject. The aim of this comparison is to better evaluate the 

aspects of Ontologging that are directly connected to the use of an ontology approach rather 

than the aspects related to knowledge management in general.

5.5.1 Description of the evaluation 




Page : 67 of 110 

Version: 1.0 


Status: Final 


Meta4 has considered two alternative scenarios for Onto-KnowNet The evaluation focused 

on the approach, (i.e. what are the benefit of introducing ontologies in the considered 

scenarios.) Focus on some of the aspects (browse and search, knowledge edition), in 

particular to allow a comparison with their existing knowledge management product 

(KnowNet). 

Two different ontologies, oriented to different type of users: 

• First modelled scenario described the knowledge base of customer support cases, and 

product bugs. This was intended for developers that were currently using a proprietary 

application based on relational database. 

• Second modelled scenario described competences, knowledge areas and knowledge units. 

This scenario was presented to some managers to compare with our existing KM and HR 

solutions. 

5.5.2 Comparing the two systems 

Meta4 has compared and assessed the added value of an ontology approach comparing it with 

KnowNet. The comparison was done not only at product level but also taking into account 

usage experience, acceptance, etc. Product managers wanted to evaluate whether the 

approach of ontologies/semantic techniques could be a strategy for some future products. 

The main enhancements compared to KnowNet are: 

• Providing navigational capabilities: allow browsing fr om one object to other related 

objects, finding important knowledge by visual means. Ontology navigation complements 

the directed search, similarly to browsing the Internet. 

• Easiness of the knowledge creation process, the integration with the Office environment 

• Enhanced search capabilities: allow searching for documents related to other entities 

(projects, etc.) 

• Permitting easy extension/modification of the data model (domain ontology) 

• Integrating logical inferences 

5.5.3 Lessons learned from this comparison 

• Use of ontologies seems to be an important enhancement to knowledge-oriented 

applications, and it is critical to support a flexible evolution of the domain. 

• Ontology navigation complements the directed search, similarly to browsing the Internet.




Page : 68 of 110 

Version: 1.0 


Status: Final 


• Technically, the approach was adequate to allow a possible integration with other 

products (Web services, import/export tools). However, introducing ontologies is not by 

itself the solution nor is “magic”. 

• Therefore, Ontologging must be considered as a platform, a set of tools that should aim to 

integrate with – and not replace – existing applications. 

5.6 Final words 

The ontology engineers from Indra and Meta4 have been questioned related to the advantage and 

limitations of the use of ontologies, and it appears useful to report this more high level perspective. 

What are the difficulties and what are the advantages of using ontologies? 

The key advantage is in the power of the relationships, which enables users to navigate from 

one concept (and its instances) to another (and its instances) very easily. The main difficulty 

is that the users are not used to work with this kind of map or representation. Some of the 

Indra knowledge workers faced problems “understanding what an ontology really is.” People 

are used with taxonomies, trees for structuring knowledge and they faced problems to think 

in terms of concepts, relationships, etc. 

However, a taxonomic view, a well-known interface similar to folders in Windows explorer 

made easier understanding the structure beneath the application to the end-users. 

From the Meta4 perspective, the advantage is the flexibility: it is very easy to change the 

model maintaining consistency. No major difficulties were found, if performed as an 

individual process. Collaborative modelling should be given more support. 

Is an ontology based management information meta -model useful for better structuring the 

domain model? Why? 

Yes, because it enables users to see all the domain elements and its relationships, providing 

easy navigation. There is no sim ilar system with the same power and possibilities.

6 Discussion and Conclusions 




Page : 69 of 110 

Version: 1.0 


Status: Final 


Concluding the work and outcome of the Ontologging project, which the evaluation has 

helped to determine, appears to be difficult: 

On one hand, the technical objectives of this project have been fulfilled: a whole set of tools 

relying on ontology based technologies that provide support to many of the very important 

dimensions of the knowledge processes have been elaborated. These tools provides the means 

to define “knowledge schema” that will be used to structure the organization of the 

knowledge in a KM system, to populate this system with content, to retrieve this knowledge, 

and finally to evaluate this knowledge as well as the different knowledge processes being 

conducted in the system. Even if some room for improvement still remains, the design of 

ontology-based knowledge management systems does not appear an unachievable overtaking. 

From this perspective, moving to a productisation phase would not to raise any major 

difficulties, given in particular the fact that most of the technological building blocks for 

designing these systems are now available. 

On the other hand, this project has “kind of ” revealed a real “Pandora box”: Ontology 

oriented knowledge management systems are radically different systems than the traditional 

document centric knowledge management system of today, and raises many more nontechnical 

issues that are not particular trivial to solve. 

For instance the design of the structure (ontology building) is difficult, and require time. In 

the case of the Ontologging project, it took several months and a major redesign to obtain a 

correct domain Ontology. (Missikoff, Navigli and Velardi, 2002) indicates that it took one 

year for the project Harmonize project to release their first domain Ontology (in the domain 

of tourisms) that comprises about 300 concepts. Even if we can obviously imagine that in the 

later case the participants were not dedicating all their time on this design (they also 

mentioned some techniques that have helped to reduced considerably this time), ontology 

design is a complex operation that requires time and expertise. 

In a similar way, the ontology content population requires also an important amount of effort 

and rigor, and cannot be improvised. Hence, in the Ontologging project, the first 

(unsupervised) population resulted in a result that was barely usable. Indeed, Ontology 

systems appear also to be less tolerant to low quality and noise than more traditional 

information-centred systems. The solution that was finally adopted to have the content 

population be accomplished by a reduced and specialised team, was working all right, but 

seems to go against the general idea that all the employees should participate to the 

knowledge capitalisation process and contribute to enrich the repository of knowledge of the 

company. Even if this solution is only temporary, some investigation needs to be 

accomplished to make possible the collaborative capitalization process. 

Finally, the retrieval of this knowledge is also more complex (but also richer) and prone to 

dispersion, since it goes much beyond the use of search mechanisms but also include the 

navigation in a maze of node of information elements connected to one another, and the more




Page : 70 of 110 

Version: 1.0 


Status: Final 


closely support highly cognitive knowledge management processes (in one case, the 

elaboration of a tender). 

One of the most interesting finding of this project is that a totally new vision has emerged 

related to perception of the usage of an ontology-based knowledge management system than 

what was originally envisaged. Ontology-based knowledge management systems do not 

represent only the (incremental) evolution of the traditional document -centred knowledge 

management system, but a radically different concept: the document in no longer the unique 

element capturing the all knowledge of the organization (other elements also intervene such 

as knowledge object representing people or projects, but also all the relations connecting 

these different objects) and the knowledge retrie val do not happen via the global search of 

information fulfilling some criteria (typically a set of key words), but via the navigation 

thorough a semantic network and progressive discovery of this knowledge. Ontology-based 

systems also appear to provide a much better support to the management of many of the more 

complex and difficult to formalize knowledge processes. First as indicated, they provide a 

much higher and cognitive level of abstraction, closer to the ones that are used by the 

knowledge worker in his/her reasoning. Second, by giving to possibility to deeply capture 

people information (via user modelling) they are able to provide some support to the 

management of the tacit knowledge of the organisation. They are able to support the 

implementation of much more sophisticated mechanisms such as personalization or active 

mechanisms such as agents. Finally, they also offer some way to integrate seamlessly the 

different information of an organization into a single system by being the wrapper of all the 

information systems of the organization (one of the idea that appeared to be the most 

appreciated by the users). 

These finding should be considered as globally positive. Traditional knowledge management 

have mostly failed to be adopted for a numerous number of deep reasons. These reasons 

include the lack of flexibility and poor “alignment” with the company processes (Malhotra, 

2002), little or no support for the management of the tacit knowledge and of complex 

knowledge management processes, ignorance of the human and social factors; a whole set of 

reasons to which Ontology-based systems seem to offer some answers and some new ideas. 

And trying to make the traditional knowledge management system just evolve was hopeless. 

Will Ontology-based knowledge management system be more successful than their more 

traditional counterpart? There is much hope and belief that it will be the case, and that 

Ontology will represent a major innovation in the next generation information systems 

(Pisanelli, Gangemi and Steve, 2002). Perhaps they will also enable the design of the 

knowledge management “killer app” that we have been expected (for instance with systems 

able to better support the knowledge networking (Smith. and McKeen, 2003))? 

Still a lot of issues and problems remain open, and we have the impression in this project 

only to have scratched the surface, and that the whole field of knowledge management field 

should be revisited from an knowledge representation, engineering, and cognitive perspective 

in order to address questions of design, population, evolution, and usage. 

Besides many additional questions that are deeply rooted to the Ontology concepts and global 

vision of knowledge management that would be worth to be investigated such as: (1)




Page : 71 of 110 

Version: 1.0 


Status: Final 


effectiveness of serendipitous knowledge discovery versus simple search retrieval; (2) 

cultural implication of the introduction of ontology in an organization (influence on the 

knowledge sharing, contribution to a shared set of value); (3) cultural and personal style 

adequation of an Ontology based (semantic web) representation. For instance concerning the 

later case, we can mention the recent work that tends to demonstrate a cognitive orientation 

difference between East Asian and Westerners (typically Americans) in the way of perceiving 

object and networks (Nisbett et al., 2001; Chiu, 1972), and the possibility that some culture 

(Asians) be more able to adopt a semantic web point of view.

7 References 




Page : 72 of 110 

Version: 1.0 


Status: Final 


Abecker Andreas, Bernardi Ansgar, van Elst Ludger, and Klein Bertin (2003); Organizational 

Memory Information Systems for Global Organizations - Design Principles and Research 

Directions. Submitted for: Hamid R. Nemati & Riad Ajami (eds.): Global Knowledge 

Management - Challenges and Opportunities. Ivy League Publishing. To appear 2003. 

ACM SIGCHI (1992). Curricula for Human -Computer Interaction . ACM Special Interest 

Group on Computer-Human Interaction Curricula Development Group, New York 

Aitken, S. and Reid, S. (2000); Evaluation of an Ontology-Based Information Retrieval Tool 

Workshop on the Applications of Ontologies and Problem-Solving Methods, (eds) Gómez- 

Pérez, A., Benjamins, V.R., Guarino, N., and Uschold, M. European Conference on 

Artificial Intelligence 2000, Berlin 

Andre, E., Klesen, M., Gebhard, O., Rist, T. (2000); ‘Exploiting Models of Personality and 

Emotions to Control the Behavior of Animated Interactive Agents’, Fourth International 

Conference on Autonomous Agents, pp. 3-7, Barcelona, 2000 

Angehrn A, J. Atherton (1999), A Conceptual Framework for Assessing Development 

Programmes for Change Agents, ECIS '99, COPENHAGEN, 1999. 

Angehrn A., J.-F. Manzoni (1997), Understanding Organisational Implications of Change 

Processes: A Multimedia Simulation Approach, 30th Annual Hawaii International 

Conference on Systems Sciences, Vol. II, IEEE Computer Society Press, 1997, pp. 655-664. 

Angele, J., Sure, Y. (2002); Whitepaper: Evaluation of Ontology-based Tools. Excerpt from 

the IST-2001-29243 Report, OntoWeb. D1.3. Tools. (2001). Available at: 

http://www.aifb.uni-karlsruhe.de/WBS/ysu/publications/eon2002_whitepaper.pdf 

Berners-Lee Tim, Hendler James, and Lassila Ora (2001); The Semantic Web; Scientific 

American, 284(5):34-43, May 2001 

Bateman John (2004), Ontology Portal, 

http://www.fb10.uni-bremen.de/anglistik/langpro/webspace/jb/infopages/ontology/ontology-root.htm 

Beyer, H. & Holtzblatt, K. (1998); Contextual Design: Defining Customer-Centered Systems. 

San Francisco: Morgan Kaufmann Publishers, 1998. 

Brusilovsky P. (2001); Adaptive Hypermedia, User Modeling and User-Adapted Interaction, 

Kluwer, Academic Publishers, 2001, Printed in the Netherlands, pp. 87-110 

Buckingham Shum, S. (1997); Balancing Formality with Informality: User-Centred 

Requirements for Knowledge Management Technologies. AAAI Spring Symposium on 

Artificial Intelligence in Knowledge Management (1997), Stanford Univer sity, Palo Alto, 

CA. AAAI Press. Available at: 

http://kmi.open.ac.uk/people/sbs/org-knowledge/aikm97/sbs -paper1.html 

CALT Team, 2000, Advanced Learning Approaches & Technologies: The CALT 

Perspective, Working paper, INSEAD CALT, October 2000. 

Chiu, L.-H. (1972). A cross-cultural comparison of cognitive styles in Chinese and American 

children. International Journal of Psychology, 7, 235-242.




Page : 73 of 110 

Version: 1.0 


Status: Final 


Clark, P., John Thompson, Heather Holmback, and Lisbeth Duncan. (2000). Exploiting a 

Thesaurus-Based Semantic Net for Knowledge-Based Search. Proceedings of IAAI-2000. 

Pp. 988-995. AAAI Press. 

Dix A., Finlay J., Abowd G. and Beale R. (1993): Human-Computer Interaction, Prentice 

Hall, New York. 

Eisenhardt, Kathleen M. (1989): "Building Theories from Case Study Research", Academy of 

Management Review , Oct89, Vol. 14 Issue 4, p532 

Ehrig Marc (2002), Ontology-Focused Crawling of Documents and Relational Metadata; 

Master’s Thesis University of Karlsruhe, (MatrNr. 0926607), January 31, 2002 

Erskine, L. E., Carter -Tod D. R. N., and Burton, J. K. (1997). Dialogical techniques for the 

design of web sites. International Journal of Human Computer Studies, 47, 169-195. 

Fink, J., Kobsa, A. (2000);A Review and Analysis of Commercial User Modeling Servers for 

Personalization on the World Wide Web, in User Modeling and User Adapted Interaction, 

Special Issue on Deployed User Modeling, 10, p.204-209, 2000 

Garigue Robert (2003); Managing Ontological Ambiguity: Extending the Knowledge 

Management Framework; 2nd Annual Knowledge Summit 2003 Doctoral Consortium, 

Queen's KBE - Centre for Knowledge -Based Enterprises 

Giboin A., Gandon F., Corby O., and Dieng R. (2002); Assessment of Ontology-based Tools: 

Systemizing the Scenario Approach; Proceedings of EON2002: Evaluation of Ontologybased 

Tools Workshop at the 13th International Conference on Knowledge Engineering and 

Knowledge Management EKAW 2002, Siguenza (Spain), 30th September 2002. Available 

at: http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS//Vol-62/ 

Gruber T. R. (1993); A translation approach to portable ontologies. Knowledge Acquisition, 

5(2):199-220, 1993 

Guarino Nicola and Chris Welty (2002). Evaluating Ontological Decisions with OntoClean. 

Communications of the ACM. 45(2):61-65. New York: ACM 

Jirotka M. (1992) Ethnomethodology and Requirements Engineering. Technical Report. 

Oxford University. 

Kay, J. (2001); Scrutability for personalised interfaces, ERCIM NEWS, Special Theme Issue 

on Human Computer Interaction, 46, July, 49-50, 2001. 

Kirkpatrick, D.L. (1994), Evaluating Training Programs: The Four Levels, Berrett-Koehler 

Publishers, San Francisco, 1994-1996. 

Kobsa, A., Koenemann, J. and Pohl, W. (2000); Personalized hypermedia presentation 

techniques for improving online customer relationships, The Knowledge Engineering 

Review 16, p111-155 

Malhotra, Y. (2002), Why Knowledge Management Systems Fail? Enable rs and Constraints 

of Knowledge Management in Human Enterprises. In Holsapple, C.W. (Ed.), Handbook on 

Knowledge Management 1: Knowledge Matters, Springer-Verlag, Heidelberg, Germany, 

577-599, 2002 http://www.brint.org/WhyKMSFail.htm 

McBride Rob and Schostak John (1995), An Introduction to Qualitative Research. 

http://www.uea.ac.uk/care/elu/Issues/Research/Res1Cont.html 

Menon Tanya, Jeffrey Pfeffer (2003); “Valuing Internal vs. External Knowledge: Explaining 

the Preference for Outsiders”, Management Science, Volume: 4,, Number: 4, April 2003.




Page : 74 of 110 

Version: 1.0 


Status: Final 


Meredith, Jack (1998) "Building operations management theory through case and field 

research", Journal of Operations Management, 16(4): 441-454. 

Missikoff M., R. Navigli, and P. Velardi (2002); The Usable Ontology: An Environment for 

Building and Assessing a Domain Ontology. Proceedings of the International Semantic 

Web Conference 2002, Springer, 2002, pp. 39-53. 

Nielsen J. and Molich P. (1992). Heuristic evaluation of user interfaces. In Empowering 

people - CHI'90 conference proceeding. ACM Press, New York 

Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems of thought: 

Holistic vs. analytic cognition. Psychological Review, 108, 291-310. 

OntoWeb (2002); Deliverable 2.1: Successful scenarios for ontology-based applications; 

OntoWeb Consortium, 2002. 

Oppermanna Reinhard and Harald Reitererb (1997). Software Evaluation using the 9241 

Evaluator, Paper published in: Behaviour Information Technology. 16 (1997), 4/5, 232 

Pisanelli D.M., Gangemi A., Steve G. (2002); Ontologies and Information Systems: the 

Marriage of the Century?; Proceedings of Lyee Workshop, Paris, 2002. 

Polson P., Lewis C., Rieman J. and Wharton C. (1992) Cognitive walkthroughs: A methods 

for theory-based evaluation of use interfaces. International Journal of Man - machine 

Studies, 36, 741-73 

Preece J., Rogers Y., Sharp H., Benyon D., Holland S. and Carey T. (1994). Human- 

Computer Interaction. Wokingham: Addison Wesley. 

Raghavan V. V., Jung G. S., and Bollmann P. (1989); A Critical Investigation of Recall and 

Precision as Measures of Retrieval System Performance; ACM Transactions on Office 

Information Systems , pages 205--229, July 1989. 

Smith Heather A. and McKeen James D. (2003); Network: Knowledge Management’s ‘Killer 

App’?; working paper 03-06, Queens University Centre for Knowledge-Based Enterprises, 

2003. http://business.queensu.ca/kbe/papers/abstract_03_06.htm 

Stephanidis, C. (2001); Adaptive Techniques for Universal Access, User Modeling and User- 

Adapted Interaction, 2001, 11: 159-179, Kluwer Academic Publishers 

Stuart, I., McCutcheon, D., Handfield, R., McLachlin, R. Samson, D. 2002. Effective case 

research in operations management: a process perspective. Journal of Operations 

Management, 20: 419-433 

Thomas J. C., W. A. Kellogg, and T. Erickson (2001); The knowledge management puzzle: 

Human and social factors in knowledge management; IBM Systems Journal, Volume 40, 

Number 4, 2001 

Uschold, M. and Gruninger, M. (1996); “Ontologies: principles, methods, and applications”, 

KnowledgeEngineering Review, volume 11, number 2, pages 93–155, 1996. 

Weinberger H., Teeni D. and Frank A. (2003); Ontologies of Organizational Memory as a 

Basis for Evaluation; 11th ECIS'03 European Conference on Information Systems, Naples, 

Italy, June 2003. 

Yin, Robert K. (1994): “Case Study Research: Design and Methods”; Sage, Thousand Oaks, 

CA; first published in 1984.

8 Annex 




8.1 Annex 1: Ontologging Goals & focus. The consortium perspective. 

Page : 75 of 110 

Version: 1.0 


Status: Final 


The different partners of the consortium were asked to identify what they considered to be the 

goals of the project, as well as their individual goals in the projects. 

More concretely, each partner was asked to provide some answers to the following questions: 

• Application goals : what we expect the Ontologging application will perform. 

• End user goals: what our end-users (INDRA) expect to gain by using the 

Ontologging application. 

• Intangible goals : some of our goals are more intangible (hence more 

difficult/subjective to measure). 

• Individual partners goals: each partner may have different individual objectives. 

The answers were sent via email to the coordinator, who aggregated these answers. The result 

was furthermore consolidated at a consortium meeting. 

The next paragraphs present the result of this process: 

Application goals: 

• User-friendly browsing capabilities. 

• Improved search by means of ontology capabilities. 

• Combination of ontology search with document keyword search. 

• Integration/embedding with MS Office. 

• Personalized interest notifications. 

• Integration with KM (or other) solutions. 

• Reusability + future exploitability. 

• Sca lability. 

End-user goals: 

• Helping internal processes: reusing, classifying, searching documents, people. 

• Improving knowledge sharing, common understanding, use of a common language, 

lead to better communication. 

• Enrich document definition. 

• Introduce semantic-based knowledge organisation and retrieval (there are no semantic 

tools currently at INDRA). 

• Help the tendering process. 

• Ease of use and deployment. 

• No interference with the current system and practices. 

• Customisation + flexibility.




• Mail integration + single sign-on (not for the pilot). 

Page : 76 of 110 

Version: 1.0 


Status: Final 


Intangible goals: 

• Bringing new value into ontology management that has scientific merit. 

• Research on Ontology evolution-reuse-query answering. 

• Assess the impact of introducing ontologies in KM. 

• Assess the use of ont ologies versus relational database. 

• Ability to capture the complexity of corporate processes and knowledge. 

Individual goals: 

• Implement an open source ontology management system. 

• Added value for KnowNet: ontology navigation, data model flexibility, more 

powerful search + inference, ease creation process. 

• Modelling different aspects of the users in a deep way. 

• Experimenting with new technologies. 

The individual partners focus: 

• Ontology representation-querying-evolution(FZI). 

• User-friendly MSOffice -integrated interface (Deltatec). 

• To improve KnowNet by integrating ontologies (Meta4). 

• JADE integration, contribution to JADE (Archetypon). 

• Evaluating new Knowledge Management approach and impact on organizations 

(CALT). 

• To help day-to-day processes as tendering a nd engineering (Indra). 

8.2 Annex 2: Description of INDRA (the main user group) 

8.2.1 An overview 

Indra is the leading Spanish company in Information Technologies. Indra’s activities are 

distributed into three lines of business: Information Technologies (80%) and Simulation and 

Automatic Test Systems, and Electronic Defence Equipment (20%). 

In year 2000, Indra had over 5,000 employees. Over 76% of these 5,000 employees are 

specialised graduates. 

In regards to financial performance, Indra’s year 2000 revenues tota led 676.9 million Euros. 

About 40% of its revenues were derived in the international marketplace (outside of Spain). 

During 2001, Indra exceeded its growth objectives for financial year 2001 in terms of both 

revenues and profits, earning a net profit of 48 million Euros (€), which represents a 25% 

increase (compared to the 18% figure established as the company's initial objective). The 

company also achieved growth of 28% in revenues from its Information Technologies




Page : 77 of 110 

Version: 1.0 


Status: Final 


business area (not counting balloting projects) and 26% in its Simulation and Automatic 

Testing Systems (SIM/ATS) and Defense Electronics Equipment (DEE) businesses. 

Indra’s head office is located in Madrid. Indra has 15 other locations in Spain as well as 

offices in Argentina, Chile, Peru, USA , Germany, China, Portugal and Venezuela. 

Worldwide, Indra is present in more than 40 countries on five continents. 

8.2.2 Activities of Indra’s Competence Centres 

Competence Centres are one of the basic pillars supporting Indra’s strategy of growth and 

adaptation to clients’ needs. They come about as a result of market requirements. Clients 

which makeup the market that Indra serves demand a quick and effective capacity to react 

and respond. 

The Competence Centres’ mission is to lead the innovation of Indra’s services and solutions. 

Although the number and type of the Competence Centres vary depending on the market and 

the clients’ needs, the following lines of business are related to knowledge management and 

ontology building: 

1. Supply chain management (SCM): e-business services aimed at both private individuals 

(B2C, G2C, etc.) and companies (B2B). 

2. Customer relationship management (CRM): Business Intelligence systems and Contact 

Centres. 

3. Network and systems management: Internet/Intranet infrastructure (including access 

from mobile phones and network security) and knowledge management systems. 

The commercial operations which address customer demands and project execution are the 

two major activities carried out in Competence Centres. Thus, tight coordination is ne eded 

with other organisational units involved in the marketing, sale, and production of information 

systems. 

The following diagram depicts the relationships that exist between the Competence Centres 

and the Production Centres. The vertical axis depicts the sequential activities and the 

horizontal axis shows the participating units.

User 

requirements 

Activities 



Line of 

business 

Identification 

for marketing 

Commercial 

Proposal 


Competence 

Centers 

Technical 

Solution 

Execution 

Plan 

Production 

Centers 

Page : 78 of 110 

Version: 1.0 


Status: Final 


Organization 

Installation & 

Development 

Installation 

At the conclusion of these activities is “installation.” Installation may occur at either an Indra 

customer’s site or an Indra employee, or both. 

In the Competence Centres, the key work practices that are particularly important to focus on 

are: 

• Identify knowledge and skill gaps and manage the development of human resources 

competencies. 

• Provide expert, trained and skilled consultants and technicians to meet service 

requirements. 

• Plan and implement learning and training activities. 

• Select, test or prototype and certify new technology resources and providers as needed. 

• Provide and deliver the service to specific customers ensuring the maximum quality of 

service. 

The generic role of and the guiding principles of the Competence Centres are: 

- Provide engineering and commercial support to the remainder of Indra. 

- Set up a knowledge-sharing environment within Indra. 

- Mobilise the best practices in technology across the organisation. 

- Generate reusable components for problem resolution. 

- Enable Indra to maximise performance and innovation.




Page : 79 of 110 

Version: 1.0 


Status: Final 


- Reinforce the growth of both Indra and the people aim at enhancing creativity and 

effectiveness. 

In order to accomplish these key work practices and fulfil the guiding principles, the 

Competence Centres employ technologies that provide users with access to structured and 

unstructured information. Technologies also facilitate the ability to identify users with 

relevant skills and expertise. 

Here is a brief explanation of the Competence Centres’ tools. 

♦ Enterprise portal (Indr@web): integrates and manages a wide variety of corporate 

information channels and services. 

♦ Knowledge and information repository infrastructur e: databases and electronic document 

management systems 

♦ Knowledge maps and guides to available knowledge resources: thesaurus, taxonomy and 

ontology generation 

♦ Search and delivery information services to access analysis and strategy external sources 

of information 

♦ Workflow services for corporate process automation in the area of human resource 

management 

♦ Collaboration services 

♦ Directory, calendar, agenda services 

♦ Threaded discussions (forums) 

♦ Asynchronous (e-mail) 

♦ Real-time communication (chat, net-meeting, conferencing, audio, video). 

Scheduled, not implemented yet. 

♦ Shared spaces (document sharing, white-boarding). Scheduled, not 

implemented yet. 

♦ E-learning and human resource management portal (Employee Application Portal) 

Prototyped, but not implemented ye t. 

For 2002, the Competence Centres are focused on these goals: 

• Increase the commercial contacts with clients. 

• Increase our offering volume in products and services. 

• Serve as a reference for the remaining business lines. 

• Provide counselling for improving project’s problems and opportunities. 

• Reach the maturity phase in the emergent KM line at Indra.




Page : 80 of 110 

Version: 1.0 


Status: Final 


8.2.3 The tendering process at INDRA 

The tendering process at INDRA aims to select a supplier and a proposal for the considered 

services and systems, and to agree with the chosen supplier on a contract defining the 

deliveries and responsibilities of both parties. 

The tendering process includes the following sub-processes: 

• preparation of request for proposal 

• response preparation, 

• supplier selection 

• contract preparation. 

These sub-processes do not need to be executed in the sequence of Figure 2: they may 

overlap in time, and there may be iterations of groups of these sub-processes. 

Tendering 

Preparation of 

request for proposal 

Response 

preparation 

Supplier selection 

Contract preparation 

Contract monitoring 

Decision point 

execution 

Contract completion 

Administrative 

completion 

Procurement 1 

Acquisition Initiation 

Tendering 



Response 

preparation 





execution 



completion 

Procurement 2 

Tendering 



Response 

preparation 





execution 



completion 

Procurement n 

Figure 2: The acquisition process model




Page : 81 of 110 

Version: 1.0 


Status: Final 


See (INDRA, 2002) or a detailed description of the tendering process at INDRA. 

8.2.4 References 

Albert A. Angehrn and Larry Todd Wilson (2002), ‘Knowledge Management & Ontology 

Building at INDRA’, INSEAD Case 2002. 

INDRA (2002). ‘Indra Case Study: Using Ontologies in Tendering Process’, INDRA internal 

document. 

8.3 Annex 3: Ten Usability Heuristics (Nielsen J. and Molich P.) 

Visibility of system status 

The system should always keep users informed about what is going on, through 

appropriate feedback within reasonable time. 

Match between system and the real world 

The system should speak the users' language, with words, phrases and concepts 

familiar to the user, rather than system-oriented terms. Follow real-world conventions, 

making information appear in a natural and logical order. 

User control and freedom 

Users often choose system functions by mistake and will need a clearly marked 

"emergency exit" to leave the unwanted state without having to go through an 

extended dialogue. Support undo and redo. 

Consistency and standards 

Users should not have to wonder whether different words, situations, or actions mean 

the same thing. Follow platform conventions. 

Error prevention 

Even better than good error messages is a careful design which prevents a problem 

from occurring in the first place. 

Recognition rather than recall 

Make objects, actions, and options visible. The user should not have to remember 

information from one part of the dialogue to another. Instructions for use of the 

system should be visible or easily retrie vable whenever appropriate. 

Flexibility and efficiency of use 

Accelerators -- unseen by the novice user -- may often speed up the interaction for the 

expert user such that the system can cater to both inexperienced and experienced 

users. Allow users to tailor frequent actions. 

Aesthetic and minimalist design 

Dialogues should not contain information which is irrelevant or rarely needed. Every 

extra unit of information in a dialogue competes with the relevant units of information 

and diminishes their relative visibility. 

Help users recognize, diagnose, and recover from errors




Page : 82 of 110 

Version: 1.0 


Status: Final 


Error messages should be expressed in plain language (no codes), precisely indicate 

the problem, and constructively suggest a solution. 

Help and documentation 

Even though it is better if the system can be used without documentation, it may be 

necessary to provide help and documentation. Any such information should be easy to 

search, focused on the user's task, list concrete steps to be carried out, and not be too 

large. 

8.4 Annex 4: Glossary of terms 

Flexibility the multiplicity of ways the user and system exchange information. 

Learnability the ease with which new users can begin effective interaction and achieve 

maximal performance. 

Robustness the level of support provided to the user in determining successful achievement 

and assessment of goals. 

Usability a measure of the ease with which a system can be learned or used, its safety, 

effectiveness and efficiency, and the attitude of its user towards it. 

Evaluation method a procedure for collecting relevant data about the operation and usability 

of a computer system. 

Formative evaluation an evaluation that takes place before actual implementation and which 

influences the development of the product. 

Summative evaluation an evaluation that takes after implementation and has the aim of 

testing the proper functioning of a product.




Page : 83 of 110 

Version: 1.0 


Status: Final 


8.5 Annex 5: Pre -questionnaire for the participants in the Ontologging usability test 

The following pre-questionnaire what distributed to a group of users at Indra in October 

2003. 

Title: Pre -questionnaire for the participants in the Ontologging usability test 

This pre questionnaire is intended to get to know better the end-users/testers of Ontologging 

system and their real needs. It serves to the evaluation of Ontologging system; it is a first 

phase of evaluation. The evaluation of the Ontologging system might include a comparison of 

a classical approach (actual KM tools) with an ontology-based approach. 

Name (optional)………………………………… 

Function………………………………………… 

Gender………………………………………….. 

Years of experience……………………………… 

What are your main job tasks and responsibilities at INDRA? ( e.g. programming, managing, 

consulting, marketing, sales, etc.) 

Can you indicate some of your knowledge oriented activities? ( e.g. customer relationship 

management, work on various projects, write tenders, write reports, etc.)




Can you please indicate briefly what KM means for you? 

Page : 84 of 110 

Version: 1.0 


Status: Final 


Have you worked with other KM tools before? Which ones? (KM tools can be a Knowledge 

Management System (KMS) or various separate tools such as: Intranet portals, groupware 

tools, forums of discussions, databases, search engines, email systems) 

Are you using KM tools in your daily work processes? 

(If yes give 2 or 3 concrete examples) 

What is the main purpose of using this KM tools for you? (or What is the most important 

functionality of this system for your daily tasks?) 

Does the KM tool in use improve your work? If yes, why and what is the main advantage




of using the actual KM tool in your everyday work tasks? 

Page : 85 of 110 

Version: 1.0 


Status: Final 


From your perspective do the KM tools you use require further improvements? Can they be 

designed better? What improvement would you suggest? (e.g. quality of content, organization 

of content, missing functionality, collaborative tools, filtering of irrelevant data, notifications, 

etc.) 

How often have you used the Ontologging system? 

(never, once, more than once, several times) 

For any further clarifications please contact: Liana.Razmerita@insead.edu




Page : 86 of 110 

Version: 1.0 


Status: Final 


8.6 Annex 6: Questionnaire Ontology-based approach for the structuring of 

knowledge 

Title: Ontology-based approach for the structuring of knowledge 

• What were the major problems/limitations of the initial domain ontologies? 

Amongst possible reasons are: 

1. The ontology wasn’t offering enough support for the given scenario? (Tender, 


2. The ontology was not complete? 

3. The terminology was not consistent with the general usage? 

4. It is difficult to reach a consensus towards a definition of concepts? 

5. Other reasons 

• Can you describe a little this process of “refining” the domain ontology? 

How did you proceed? 

• Can you describe the process of conceptual modelling of the domain? By conceptual 

modelling we understand the process of defining concepts, sub-concepts and their 

relationships? (Add new concepts, Delete, Create new properties?)




Page : 87 of 110 

Version: 1.0 


Status: Final 


• What is the current complexity of the domain ontology? Number of concepts and 

subconcepts? Number of relations? Number of instances? 

• What are the difficulties and what are the advantages of using ontologies? 

• Is an ontology based management information meta-model useful for better structuring 

the domain model? Why? (e.g. The ontology offers good support for the business process 

in general) 

Could you tell us your views on the use of ontology to support the given scenario? 

Do you believe it is: 

1. Strongly beneficial 

2. Beneficial 

3. Not very beneficial 

4. Not at all beneficial 

5. Not sure

Ontology based tools : 




Page : 88 of 110 

Version: 1.0 


Status: Final 


• What are the limitations and advantages of using KAON ontology editor for building the 

domain ontology? 

• Any functionality is missing? 

• What are the features that you enjoyed the most? 

• KAON Ontology editor 

KAON Ontology editor 

Tool perspective 

Functionalities provided 

Interface design 

Ergonomics 

Friendliness 

Usability 

Help level 

Response time 

Effort to load data 

Effort required to use the tool 

Very 

poor 

Poor Adequate Good Very 

good




Page : 89 of 110 

Version: 1.0 


Status: Final 


8.7 Annex 7 Spanish questionnaires 

In order to facilitate the evaluation of the domain ontology, the following questionnaire has 

been prepared. Due to the users are Spanish speaking, the first version of this one is in 

Spanish. 

Cuestionario de Opin ión sobre Características para Seleccionar Ontologías 

En este anexo se muestra un cuestionario que propone preguntas sobre la importancia de 

características de ontologías. 

Mapa del cuestionario 

C- Sobre el contenido. 

C1) Sobre la coincidencia de los conceptos (clases). 

C2) Sobre las relaciones. 

C3) Sobre la taxonomía de los conceptos. 

C4) Sobre los axiomas o reglas. 

T- Sobre el editor de la ontología. 

T2) Sobre la visualización de los términos de la ontología. 

T3) Sobre la edición de los términos de la ontología. 

T8) Sobre integración de ontologías.




Page : 90 of 110 

Version: 1.0 


Status: Final 


CUESTIONARIO SOBRE CARACTERÍSTICAS PARA SELECCIONAR ONTOLOGÍAS 

Entrevistado:_______________________________________________________________ 

_____ 

El supuesto es construir un sistema y buscar una ontología existente para integrarla en él. ¿En 

qué debemos fijarnos?, ¿qué es lo importante?, ¿de qué va a depender elegir una ontología 

frente a otras? Por favor, contesta a este cuestionario evitando pensar en un proyecto 

concreto, sino considerando las características que a ti te parezcan importantes para 

seleccionar una ontología para cualquier tipo de sistemas. Vamos a suponer también que 

existe un gran número de ontologías que pueden ser reutilizadas. 

Valora el cuestionario sobre características de ontologías indicando qué importancia das a 

cada una de ellas. Para ello, marca una o varias casillas, e indica tu opinión sobre la 

valoración realizada. Si no estás seguro de lo que significa la pregunta, preferimos que no 

contestes. Si piensas que olvidamos o no es necesaria alguna característica te agradeceríamos 

que nos lo comentaras. 

SOBRE EL CONTENIDO 

Se valora aquí la importancia que tiene la coincidencia de los términos que están 

representados en la ontología, con los términos que se supone debe utilizar el nuevo sistema. 

Es decir, sobre la parte del dominio que la ontología cubre de las necesidades de nuestro 

nuevo sistema. 

En general, sobre el grado de coincidencia del contenido de la ontología con las 

necesidades del sistema: 

Es suficiente que la ontología tenga un dominio parecido al buscado. Servirá como base 

para reelaborar una ontología para el sistema. 

La ontología debe coincidir en parte con las necesidades del sistema. Se deberá hacer un 

proceso de adaptación. 

La ontología debe coincidir en un alto porcentaje con las necesidades del sistema. Las 

adaptaciones deberían ser mínimas. 

La ontología debe coincidir casi totalmente con las necesidades del sistema. Las 

adaptaciones deberían ser sólo puntuales. 

La ontología debe coincidir totalmente con lo buscado. No debería hacerse ninguna 

adaptación. 

No estoy seguro. 

Otra idea:




Page : 91 of 110 

Version: 1.0 


Status: Final 


SOBRE LA COINCIDENCIA DE LOS CONCEPTOS (CLASES) 

Se evalúa en este apartado la coincidencia de los conceptos existentes en la ontología con las 

necesidades del sistema que se va a desarrollar. 

C1 En general, di si te parece 

importante que coincidan los conceptos 

en la ontología evaluada con lo 

necesitado en el sistema. 

C11 Los conceptos que son esenciales 

para el sistema se encuentran en la 

ontología. 

C12 Los conceptos que son esenciales 

para el sistema se encuentran en los 

niveles superiores de la ontología. 

C13 Los conceptos se encuentran 

descritos convenientemente en lenguaje 

natural. 

C14 La especificación formal de los 

conceptos coincide con la descripción 

de los conceptos en lenguaje natural. 

C15 Los atributos de los conceptos los 

describen de forma precisa. 

C16 El número de conceptos que son 

representados en la ontología. 

Comentario sobre la valoración: 

No es 

importante 

No es 

fundamental 

Es 

importante 

Es muy 

importante 

Es 

fundamental 

SOBRE LAS RELACIONES 

Se evalúa en este apartado la coincidencia de las relaciones entre conceptos existentes en la 

ontología con las necesidades del sistema que se va a desarrollar. 


importante que coincidan las relaciones 

definidas en la ontología con lo 

necesitado en el sistema. 

C21 Las relaciones que son esenciales 

para el sistema se encuentren definidas 

en la ontología. 

C22 Los conceptos de la ontología se 

encuentran relacionados tal y como se 

necesitan para el sistema. 

No es 

importante 

No es 

fundamental 

Es 

importante 

Es muy 

importante 

Es 

fundamental

C23 La especificación formal de las 

relaciones coincide con la descripción 

de las relaciones en lenguaje natural. 

C25 Las relaciones tienen especificadas 

propiedades formales necesarias para el 

sistema: (reflexividad, irreflexividad, 

simetría, asimetría, antisimetría, 

transitividad, intransitividad). 

C26 El número de relaciones definidas 

en la ontología. 

Comentario sobre la valoración. 




Page : 92 of 110 

Version: 1.0 


Status: Final 


SOBRE LA TAXONOMÍA DE LOS CONCEPTOS 

Sobre cómo están clasificados los conceptos en la ontología y su coincidencia con la 

organización necesaria para el sistema. 


importante la coincidencia de las 

taxonomías de conceptos con las que 

necesita el sistema. 

C31 Los conceptos de la ontología se 

encuentran clasificados desde varias 

perspectivas. 

C32 Existen especificadas relaciones del 

tipo "Not-Subclass-Of' necesarias para 

el sistema. 

C35 La profundidad máxima a la que 

llega la jerarquía de conceptos. 

C36 El número medio de subclases que 

tienen las clases. 


No es 

importante 

No es 

fundamental 

Es 

importante 

Es muy 

importante 

Es 

fundamental 

SOBRE LOS AXIOMAS 

La existencia de axiomas en la ontología quizá sea necesaria en el nuevo sistema para hacer 

deducciones, mantener la consistencia o restringir el valor de las instancias. 


importante que existan axiomas o reglas 

C41 Los axiomas pueden utilizarse para 

hacer deducciones, utilizados para 

No es 

importante 

No es 

fundamental 

Es 

importante 

Es muy 

importante 

Es 

fundamental

esolver consultas. 

C42 Existen axiomas para completar los 

valores sobre los atributos de conceptos 

C43 Existen axiomas que se utilizan 

para verificar la consistencia de 

términos en la ontología. 

C44 Existen axiomas definidos como 

elementos independientes (no 

vinculados a conceptos). 

C45 El número de axiomas definidos en 

la ontología. 





Page : 93 of 110 

Version: 1.0 


Status: Final 


SOBRE LA VISUALIZACIÓN DE LOS TÉRMINOS DE LA ONTOLOGÍA 

Sobre cómo nos muestra los términos y su contenido las aplicaciones software de 

visualización de ontologías. 

T2 En general, di si te parece 

importante cómo el editor muestra el 

contenido de los términos. 

T21 Permiten visualizar toda la 

información que se encuentre 

espe cificada en la ontología. 

T22 Permiten ver la información con el 

nivel de detalle deseado. 

T23 Permiten ver, de alguna forma, la 

taxonomía de los conceptos 

T24 Permiten usar el modo gráfico para 

mostrar las relaciones ad-hoc. 

Comentario sobre la valoración 

No es 

importante 

No es 

fundamental 

Es 

importante 

Es muy 

importante 

Es 

fundamental 

SOBRE LA EDICIÓN DE LOS TÉRMINOS DE LA ONTOLOGÍA 

En el caso de tener que hacer modificaciones en los términos de la ontología para adaptarla al 

nuevo sistema, deben valorarse ciertas características de las aplicaciones que ayudarán a 

realizar estas tareas. Debe tenerse en cuenta que los entornos pueden estar limitados por las 

capacidades expresivas del lenguaje que soporta la ontología. 

T3 En general, di si te parece 

importante la forma en la que el editor 

permite modificar los términos. 

T31 Con el entorno podemos 

No es 

importante 

No es 

fundamental 

Es 

importante 

Es muy 

importante 

Es 

fundamental

implementar todo lo que podemos 

implementar directamente con el 

lenguaje de implementación. 

T32 Permite realizar modificaciones de 

los términos en cualquier momento; es 

decir, siempre podemos editar la 

ontología y añadir nuevos términos, 

modificar y borrar términos, sus 

propiedades, sus relaciones, etc 

T33 Permite realizar modificaciones en 

las relaciones taxonómicas de forma 

gráfica. 

T34 Permite trabajar de forma gráfica 

con relaciones ad-hoc. 





Page : 94 of 110 

Version: 1.0 


Status: Final 

Confid.: Public




Page : 95 of 110 

Version: 1.0 


Status: Final 


8.8 Annex 8 User modelling tools and knowledge distribution agents questionnaires 

GOAL 

Evaluation of user modeling processes and 

the evaluation of the knowledge distribution agents 

• The evaluation of the user modeling tools; 

• The evaluation of the knowledge distribution agents; 

Hypothesis/questions to test 

• Employees view on sharing personal information and user modeling processes; 

• The perceived need of personalization of KM tools and the use of knowledge distribution 

agents; 


Function………………………………………… 

Gender………………………………………….. 


Methodology: questionnaire 

After the end-users fill in user profiles using the User Profile Editor (UPE), they are asked to answer a 

set of questions: 

User Profile Editor 

Tool perspective 

Functionalities provided 

Interface design 

Ergonomics 

Friendliness 

Usability 

Help level 

Response time 

Effort to load data 

Effort required to use the tool 

Functionality of the system 

Very 

poor 

Poor Adequate Good Very 

good



What are the main shortcomes of the UPE? 


Page : 96 of 110 

Version: 1.0 


Status: Final 


• Was the terminology (concepts) used for describing user profiles clear? Any missing 

characteristic of the user? Was the user profile too detailed? 

• What did you like the most in the UPE? 

• What enhancements would you suggest?




Page : 97 of 110 

Version: 1.0 


Status: Final 


• Are you happy to disclose personal data in exchange of personalization, communication, and 

collaboration facilities? (e.g. for example being able to look for experts easily in different 

areas) 

• Would you prefer to enter your data explicitly using the user profile editor or to be modeled 

implicitly through user modeling techniques (based on your activity in the system, e.g. based 

on your queries, based on your browsing behavior)? 

• Through user modeling techniques the system can infer your “Behaviour” in the system. ( e.g 

type of activity (reader, writer, lurker) and level of activity –(active, inactive, passive) in the 

system. These inferred characteristics would enable the organization to provide incentives to 

share your knowledge and to be more active in the system (e.g. provide answers to the 

problems in the forum of discussions). 

What would be the appropriate incentives for you to be active in the system and to share your 

knowledge more (recognition, promotion versus other types of incentives e.g. bonus, money, 

virtual rewards)?

Privacy issues 




Page : 98 of 110 

Version: 1.0 


Status: Final 


• Are you concerned with any privacy issues related to your personal data being available in the 

system? If yes, what particular aspects are you concerned with? 

• Do you think that a “privacy policy” related to the user’s data would be required in Indra? 

Future enhancements of KM tools/ KMSs and personalization issues 

How is perceived the agent support for the distribution of knowledge? 

For example, suppose you are interested in Knowledge Management. 

Scenario of notification agents: 

* Raphael Smith from team unit X has started a new project in the area of Knowledge 

Management; 

* A new document related to optimizing the tendering process has been authored by Fernando 

Salinas;




Would these notifications be an added value in a KMS for you? Why? 

Page : 99 of 110 

Version: 1.0 


Status: Final 


What type of notification of news (e.g. new documents available in your domain of interest, new 

events at Indra) would you prefer? 

1. character-based notification 

2. e-mail notification 

3. pop-up windows 

• What kind of adaptations would you prefer to be included in a next generation of Knowledge 

Management Tools? (Check the feature that you would enjoy to be available, put a question 

mark if it’s unclear for you) 

Adaptation of content of the KMS system: 

Filtering of content according to my interests and expertise; 

Optional detailed information and automatic summarization of documents 

Notification agents as similar to the previous scenario 

Adaptation of the modality 

Different types of layouts, 

Different skins; 

Different colors; 

Adaptation of structure 

Personalized view of the KM system and adapted functionality according to my own 

work tasks and needs; 

Personalized view of the KM system and adapted functionality according to the 

specificity of the team I am working in;

Other suggestions: 




Page : 100 of 110 

Version: 1.0 


Status: Final 


Which type of adaptations amongst the previous three is the most important for you? If you 

have other ideas suggest other types of personalization which KMSs could include: 

8.9 Comments to: Liana.Razmerita@insead.edu




Annex 9 Ontologging project questionnaire 

This questionnaire is divided into 5 sections: 

Ontologging Project Questionnaire 

Page : 101 of 110 

Version: 1.0 


Status: Final 


A. Ontologging System –DUI, Usage Perspective ............................................................... 102 

B. Ontologging System –DUI Tool Perspective................................................................... 104 

C. Ontologging System – DUI Ontology Perspective .......................................................... 105 

D. Detailed KM Process View............................................................................................. 106 

E. Ontologging System -Future Perspective ....................................................................... 110 

Glossary 

Knowledge capitalisation-includes processes such as submitting valuable knowledge assets in 

the system (e.g. writing a tender and making it available in a database or KMS or other 

knowledge assets which you have not necessarily written.) Adding annotations is part of the 

knowledge capitalisation process. 

Knowledge sharing- the process of knowledge sharing can be facilitated by the knowledge 

management tools (shared folders, collaborative tools, forum of discussions) but also by other less 

informal means like coffee break conversations with peers. 

Knowledge retrieving- is facilitated by querying/searching and browsing m echanisms included in the 

KM tools, but also by a good conceptualization of the domain knowledge.





Function………………………………………… 

Gender………………………………………….. 


A. Ontologging System –DUI, Usage Perspective 

Describe three contexts / scenarios in which you used the DUI ? 

Scenario 1: 

Scenario 2: 

Scenario 3: 

What have you enjoyed the most in your experience using the DUI ? 

What have you disliked in the DUI? 

Page : 102 of 110 

Version: 1.0 


Status: Final 

Confid.: Public

1 The functionalities provided by 

DUI were useful 


DUI were complete 


DUI fulfilled my expectations 

4 DUI supports very well knowledge 

capitalisation process 

5 In general, the functionalities 

support very well navigation 

processes 


querying (searching/ process) 


sharing process 




Strongly 

disagree 

Disagree Slightly 

disagree 

Page : 103 of 110 

Version: 1.0 


Status: Final 


Slightly 

agree 

Agree Strongly 

agree




B. Ontologging System –DUI Tool Perspective 

Ontologging system 

DUI tool perspective 

1 Functionalities provided 

2 Interface design 

3 Ergonomics 

4 Friendliness 

5 Usability 

6 Help level 

7 Response time 

8 Effort to load data 

9 Effort required to use the tool 

Very 

poor 

Page : 104 of 110 

Version: 1.0 


Status: Final 


poor adequate good Very 

good 

What have you perceived as the main strengths and weaknesses of the different tools? 

Knowledge capitalisation 

Strengths 

Knowle dge capitalisation 

Weaknesses 

Knowledge searching 

Strengths 

Knowledge searching 

Weaknesses 

Knowledge sharing 

Strengths 

Knowledge sharing 

Weaknesses




C. Ontologging System – DUI Ontology Perspective 

Page : 105 of 110 

Version: 1.0 


Status: Final 


Have you noticed any advantage in using an ontology approach for defining the domain 

knowledge? 

DUI an ontology perspective Strongly 

disagree 

1 The ontology offers good support 

for the given scenario (tender, 


2 The ontology offers good support 

for the business process in 

general 

4 The ontology is complete 

(concepts, 

instances) 

relations and 

5 The terminology is consistent 

with general usage 

Disagree Slightly 

disagree 

Slightly 

agree 

Agree Strongly 

agree 

Have you annotated documents using DUI? From your perspective are these annotations 

useful?

D. Detailed KM Process View 

Capitalising knowledge 




1 Report a concrete and personal experience 

of knowledge capitalization in your 

organization. 

What knowledge did you intend to 

store/capture? Why? How? 

2 Assessment of the effort to load 

documents 

How many documents have you submitted 

in the system? 

How long did it take you? 

3 What were the difficulties and limitations 

you encountered? 

Was it simple? Was it time consuming? 

Quality of the result? 

Were you able to capture what you 

wanted? 

4 What is your general conclusion about this 

experience? 

Was it positive? Would you do it again? If 

you had to do it again, what would you 

change? 

Additional comments 

Retrieving knowledge 

1 Report a concrete and personal experience 

of searching for knowledge that has been 

previously stored in a database, file system 

or Ontology-based system. 

What kind of knowledge were you looking 

for? What was the context? 

2 How did you proceed? 

Page : 106 of 110 

Version: 1.0 


Status: Final 


Using the DUI system Using traditional KM tools 

(DB or file system) 

Using the DUI 

system 

Using traditional KM tools (DB or 

file system)



Keyword search, browsing, navigating? 


3 Effort spent on the search and 

effectiveness of the result. 

Was the proc ess of searching satisfactory 

(all right or too long, amount of noise)? 

What were the main limitations? 

4 What is your general conclusion about this 

experience? 

Was it positive? Would you do it again? 

What would you suggest as improvement? 

Additional comments 

Page : 107 of 110 

Version: 1.0 


Status: Final 

Confid.: Public

Sharing knowledge 




Page : 108 of 110 

Version: 1.0 


Status: Final 


What is the most important way to share knowledge for you? 

1. Through shared document repositories (databases, file systems, ontologies) 

2. Through informal ways to share knowledge (via your social network, via email, 

coffee-room) 

3. Through structured processes (for instance your department organize regular 

meetings) 

Efficiency of the process 

In your opinion, are these knowledge sharing processes adequate? 

What is working well or not working well in these knowledge-sharing processes? What are the actual 

barriers?




Page : 109 of 110 

Version: 1.0 


Status: Final 


General conclusion about knowledge sharing as it is now 

Does it work well? What are the really important knowledge sharing processes or the ones that do not 

work? How do you think they could be improved? 

What would be the appropriate incentives for you to be active in a KM system and to share your 

knowledge (a better recognition, promotion versus other types of incentives e.g. bonus, money, virtual 

rewards)? 

Can you assess the value of using ontology for groups or inter-groups communication? 

Ontology defines a shared vocabulary. Amongst the possible reasons are: 

1. It enables modelling more deeply the knowledge 

2. It enables connecting knowledge with people, etc. 

3. It helps people to gain visibility as it connects knowledge assets with the people who 

authored them or commented them. 

Other reasons:




E. Ontologging System -Future Perspective 

What would be your recommendations for further development of the system? 

Could you see yourself using a similar system in three years time? 

Page : 110 of 110 

Version: 1.0 


Status: Final 


If you had to focus some efforts on the design of the next generation ontology-based system, where 

would it be? 

Can you imagine another usage of Ontology to support the knowledge processes?

pdf 820Kb - INSEAD CALT

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?