pdf 820Kb - INSEAD CALT

More documents

Recommendations

Info

Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 48 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public • Completeness. For instance, when trying to capitalize a given set of relevant documents, or other knowledge objects, what is the percentage of time that a concept or a relationship appears to be missing? Example of a scenario of a quantitative evaluation: Evaluate (with the criteria that have been indicated) the design of the sub-Ontology for the military sector at Indra. This Ontology has to be able to be used to elaborate proposals ranging from 500000 Euros to 5 million Euros. The size of the existing data to be captured in this sector is in the order of magnitude of: 250 documents, 400 people, 80 projects, and 120 topics. The first version of this Ontology will have to be available in a maximum 2 months and a half; the definitive version will have to be available in 8 months. 4.4.3.2 Ontology content population The second quantitative evaluation could relate to Ontology content population, and could assess the effort necessary to populate the content of an Ontology with the Ontologging tools. Different quantitative indicator could be used here: • The time to populate the content of the Ontology. How many hours, days, or weeks would be necessary to design the main Ontology, or some sub-ontology? • The richness of this ontology population. Number of instances, average number of relationships between the instances and more generally complexity and nature of the resulting network (according to graph theories). • Quality of the resulting Ontology population (noise, navigability, etc.). • Completeness. Percentage of knowledge considered as relevant that has been captured. • Level of knowledge sharing in the organization. For instance average num ber and nature of the distribution, of the sharing of documents and other knowledge item by the population of the knowledge workers. Example of a scenario of a quantitative evaluation: Evaluate (with the criteria that have been indicated) the population of the sub-Ontology for the military sector at Indra that has been described in the previous phase. Note: some shorter evaluation experience tests could also be considered here such as: Time necessary to capitalize a set of 10 documents. Time necessary to model 5 people instance. Time necessary to model 3 projects. 4.4.3.3 Usage
Evaluation report of the use of Onto-Logging platform in the user site Deliverable ID: D8b Page : 49 of 110 Version: 1.0 Date: 27 january 2004 Status: Final Confid.: Public The usage quantitative evaluation objective is to assess the capability of the Ontologging system to support knowledge related activities, and in particular to improve the productivity of the knowledge worker (time to complete a task, quality of the work). Different quantitative indicator could be used here: • The time to identify the knowledge items (project, people, documents, etc.) in the initial phase of a knowledge management task (for instance, the elaboration of a new tender). • Level of quality of the retrieve information (precision and recall). • Level of cognitive load (could be connected to the precision). • Level of use of the different knowledge items (statistical distribution in term of level of access to the different items). Example of a scenario of a quantitative evaluation: Evaluate (with the criteria that have been indicated) the elaboration of the answer to a set of tenders in the military sector. An idea could also consist in the comparison of the effort to answer to a proposal with and without the Ontologging tools. This category of experience may however appear not to be Attainable (or realistic) at Indra for human and deontological reasons. 4.4.4 Concluding remarks on the quantitative evaluation As we have seen, some level of quantitative evaluation would have been possible in this project. However, we can observe that it is more difficult with this category of evaluation to capture aspects that are difficult to formalize, such as for instance qualitative practices (such as the use of navigation instead of search) or level of knowledge sharing that however appear to be key elements in this project. If we had done so, consequence would probably have been an over-emphasis and some bias on knowledge retrieval methods, and the overlook of the elements that we can consider as the most important in this projects (in particular a semantic web orientation). If in the longer term, and as a further step, we can however consider that a more quantitative evaluation would be very desirable (and actually almost mandatory in order to validate “scientifically” the knowledge generated), in particular if we manage to define some quantitative indicators able to measure realistically the most important elements of the semantic web “vision”.
Page 1 and 2: The ONTO-LOGGING Consortium consist
Page 3 and 4: Evaluation report of the use of Ont
Page 5 and 6: Executive Summary Evaluation report
Page 47: Evaluation report of the use of Ont
Page 53 and 54: • List of connected users to the
Page 67 and 68: 5.5.1 Description of the evaluation
Page 69 and 70: 6 Discussion and Conclusions Evalua
Page 75 and 76: 8 Annex Evaluation report of the us
Page 93 and 94: esolver consultas. C42 Existen axio
Page 99 and 100:
Evaluation report of the use of Ont
Page 101 and 102:
Page 103 and 104:
1 The functionalities provided by D
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
show all

pdf 820Kb - INSEAD CALT

Create successful ePaper yourself

Delete template?

Save as template?