pdf 820Kb - INSEAD CALT

More documents

Recommendations

Info

Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 60 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public iterative, since it is difficult to have it right the first time (actually ontology building is sometime considered today more as an art than as science). This redesign presented the opportunity to evaluate the capability of the Ontologging system to support the work of more experienced designers (the designers had indeed acquired some experience since the beginning of the project), which provided a better Outlook of how the Ontology design process would happen in the future. 5.2.3 Some lessons learned. It is clear that the difficulty of the process of Ontology building was under-evaluated in this project, even if the more user -oriented participants of this project (the main user partner Indra and the partner in charge of the evaluation) had raised some concerns early after the beginning of this project. However, addressing this issue was not an easy task, and would have required a lot of attention that was perhaps beyond the scope of this project. Yet, this issue was actually not totally ignored, and was to some limited extend undertook by the work of (OKE) Ontology Knowledge Evaluation, which role was to provide some tools to do some diagnostics of the quality of an Ontology and about ontology population, and that should definitively help the ontology design and population processes. However, this category tool relies a lot on the definition of some heuristics that were not really available at the beginning of the project (the theoretical heuristics appeared to be very artificial and of little value), and are also difficult to make easy to use. Another category of work in this project is related to the mechanisms (agent notifications) supporting the collaborative authoring of the Ontologies. As indicated, this mechanism was however not really tested (excepted technically) for the design of the Indra Ontology given the relatively limited size of the ontology design team. Finally, worth to be mentioned is the sophisticated and very powerful rollback mechanisms present in the KAON system, which allow the Ontology designer to test in advance and without risks the consequence s of a design decisions. The use of an Ontology building methodology, such as OntoClean (Guarino and Welty, 2002)), the benchmarking with projects having to confront the same difficulties (see for instance (Missikoff, Navigli and Velardi, 2002)), and a be tter identification of tools and environments supporting Ontology building may have probably helped to identify earlier the problems and provided some directions for overcoming these problems. 5.3 Phases 3: Content population Two tentative of ontology content population can be distinguished: An initial content population that happened with the initial domain Ontology, and a final content population that happened with the new re-engineered Ontology. The initial tentative of ontology content population was done by different users with moderate level of expertise, that have tried to capitalized into the system using the DUI, with documents and information that they though would be useful. As already indicated, they
Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 61 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public found this capitalization process difficult and painful, first because the quality of the Ontology was bad (not the right concepts) and second because they did not have some guideline or style to follow. The result was not surprisingly of low quality: not homogeneous, and noisy (the end-users complaine d that the knowledge had “a lot of crap”), and almost unusable. The second (successful) tentative of ontology content population was accomplished only by the limited team (two-three) of people that elaborated (or reengineered) the main Ontology. This manner of proceeding had the advantage of better controlling the quality of this population, since experts were used for this purpose. It had also the advantage of making the processes of ontology elaboration and population happen concurrently and in an inte rrelated way with the design of the main Ontology itself, facilitating therefore the elaboration of this Ontology (populating the content helps in the identification the “good” concepts to be present in the ontology). Whilst the quality of the resulting Ontology and Ontology population was considered as good (the designer dedicated a lot of effort for this), this situation cannot be considered as satisfactory. Indeed, every person in the organization should be able to share his/her knowledge with others and contribute to “feed“ the electronic memory of the organization. We can however imagine (this can even become a research question to investigate in the future) that this process of collective content population would be easier once a minimum of Ontology population has been accomplished by the team of expert (users have good examples and styles to follow), in particular if it is followed by a set of accompanying measures (coaching, training, knowledge population review process, etc.) guarantying that the quality is preserved. What is the conclusion and lesson learned of this population? That as for Ontology-building, Ontology content population has appeared to be more difficult than what was expected. Besides, this problem do not seem to be related to the tools for Ontology population that have been designed in Ontologging, but more profoundly connected to Ontology theories and usage methodologies. Indeed, the quality of the content in Ontology-based system is more critical than in the case of traditional information systems, and the capitalization process inherently more difficult (because for instance of ambiguity (Garigue, 2003)). The answers to these questions would require some further investigations. The directions that have already been indicated for Ontology building appear also applicable and consist in more methodological work, as well as some work on tools and environments helping to support these methodologies. 5.4 Phases 4: Evaluating Ontologging “knowledge retrieval” The evaluation of the system from an end user perspective (knowledge retrieval) has distinguished two dimensions that will be presented in the next chapter: first, a basic
Page 1 and 2:
The ONTO-LOGGING Consortium consist
Page 3 and 4:
Evaluation report of the use of Ont
Page 5 and 6:
Executive Summary Evaluation report
Page 7 and 8:
Evaluation report of the use of Ont
Page 9 and 10: Evaluation report of the use of Ont
Page 53 and 54: • List of connected users to the
Page 59: Evaluation report of the use of Ont
Page 67 and 68: 5.5.1 Description of the evaluation
Page 69 and 70: 6 Discussion and Conclusions Evalua
Page 75 and 76: 8 Annex Evaluation report of the us
Page 93 and 94: esolver consultas. C42 Existen axio
Page 103 and 104: 1 The functionalities provided by D
show all

pdf 820Kb - INSEAD CALT

Create successful ePaper yourself

Delete template?

Save as template?