Proceedings of the 12th European Conference on Knowledge ...

Tony K. M. Lo and Patrick S.W. Fong
content <strong>of</strong> <strong>the</strong> lessons learned documents. “Learners” represent <strong>the</strong> audiences addressed, <strong>the</strong>
readers <strong>of</strong> <strong>the</strong> lessons learned, <strong>the</strong> users <strong>of</strong> knowledge and <strong>the</strong> action-takers <strong>of</strong> <strong>the</strong> recommendations
proposed. Participants, authors and learners are among <strong>the</strong> main stakeholders <strong>of</strong> <strong>the</strong> lessons
learned.
1.2 Lessons learned and knowledge quality
While <strong>the</strong> term “quality” has many definitions in management, <strong>the</strong>re are two qualities in lesson
learning: process quality and content quality. Process quality refers to <strong>the</strong> quality pertaining to <strong>the</strong>
lessons learning process, such as <strong>the</strong> efficiency <strong>of</strong> recording <strong>the</strong> information <strong>of</strong> events or <strong>the</strong> reliability
<strong>of</strong> <strong>the</strong> IT infrastructure. Content quality, which is <strong>the</strong> main concern <strong>of</strong> this paper, is about what
elements make for good quality in <strong>the</strong> content <strong>of</strong> a lesson learned. Issues regarding data and
information quality have been extensively studied, and criteria for evaluating data and information
management systems have been suggested (Madnick 2009; Redman 2005; Pipino et al. 2005; Eppler
2006). By contrast, little research has been conduced on knowledge quality. Evaluations <strong>of</strong> lessons
learned systems in terms <strong>of</strong> knowledge quality have not been widely studied. Yoo et al. (2010)
describe <strong>the</strong> dimensions <strong>of</strong> knowledge quality and propose quantitative instruments to measure it.
Davenport & Prusak (1998) state that <strong>the</strong> thing delivered is more important than <strong>the</strong> delivery vehicle.
A case study reveals that lessons learned systems are seldom reused because <strong>of</strong> concerns over
lesson quality (Milton 2010). There exists a gap in <strong>the</strong> study <strong>of</strong> <strong>the</strong> knowledge quality and hence a
need for exploring <strong>the</strong> quality <strong>of</strong> lessons learned.
One <strong>of</strong> <strong>the</strong> approaches <strong>of</strong> evaluating <strong>the</strong> quality <strong>of</strong> lessons learned is by result, i.e. by determining
whe<strong>the</strong>r learners have learned anything from <strong>the</strong> lessons that has resulted in a change in behaviour.
In o<strong>the</strong>r words, we can say a lesson learned is <strong>of</strong> good quality if, when <strong>the</strong> events described in <strong>the</strong>
lessons happen again, learners can repeat <strong>the</strong> success or avoid making <strong>the</strong> same mistakes by reapplying
<strong>the</strong> knowledge <strong>of</strong> those lessons. It is a “fit-for-use” approach. However, in that case, we can
only judge a lesson learned after its application. Although modifications <strong>of</strong> <strong>the</strong> content are always
possible, it would be more useful if an acceptable level <strong>of</strong> lessons learned quality could be ensured
before <strong>the</strong>y are disseminated to learners. The o<strong>the</strong>r way is to construct a list <strong>of</strong> attributes as criteria to
assess <strong>the</strong> quality <strong>of</strong> lessons learned, such as comprehensiveness, relevance, viscosity (Holsapple
2003; Boisot 1998; Szulanski 2003; and Zhao & Anand 2009). While <strong>the</strong> attributes provide a basis for
evaluation, <strong>the</strong> actual meanings <strong>of</strong> <strong>the</strong> attribute are usually vague, subjective and difficult to operate.
Fur<strong>the</strong>rmore, unlike information, good knowledge is expected to <strong>of</strong>fer more substantial meanings in
terms <strong>of</strong> reflection or in-depth thinking. The question is: what are <strong>the</strong> proper dimensions for evaluating
<strong>the</strong> quality <strong>of</strong> knowledge?
2. Research design
2.1 Research approach and process
While quantitative instruments or metrics have been proposed to measure knowledge quality (Rech
2007; Yoo 2011), this research attempts a different approach by proposing a questions checklist for
judging <strong>the</strong> quality <strong>of</strong> <strong>the</strong> content <strong>of</strong> lessons learned.The questions in <strong>the</strong> checklist are qualitative in
nature, which means that <strong>the</strong>y are not measurable in terms <strong>of</strong> numerals. We do not hold that good
lessons learned have to give answers to all <strong>the</strong> questions in a checklist. Nei<strong>the</strong>r should <strong>the</strong> checklist
be exhaustive or definitive. Not all <strong>the</strong> questions in a checklist are relevant for a particular lesson
learned. The key is to provide a framework for assessment <strong>of</strong> <strong>the</strong> content from various facets.
Depending on <strong>the</strong> real situation, attention has to be paid to how to strike a balance between <strong>the</strong>
comprehensiveness and conciseness <strong>of</strong> <strong>the</strong> questions. There are many facets that can be employed
for evaluation. This paper selects a couple <strong>of</strong> <strong>the</strong>m that are considered important components related
to <strong>the</strong> content <strong>of</strong> lessons learned, namely, context richness, experience sharing and reflective
thinking. They will be described in <strong>the</strong> following sections. A few questions for each component will be
provided to form a checklist based on literature review in each <strong>of</strong> <strong>the</strong> components mentioned above.
With this checklist <strong>the</strong> content <strong>of</strong> an au<strong>the</strong>ntic lesson learned is <strong>the</strong>n analyzed. Finally <strong>the</strong> results are
discussed, toge<strong>the</strong>r with <strong>the</strong> conclusions and implications <strong>of</strong> our findings.
2.2 Qualitative content analysis
In this paper, we use <strong>the</strong> method <strong>of</strong> content analysis for case study. There are variations <strong>of</strong> <strong>the</strong>
definitions <strong>of</strong> content analysis. In <strong>the</strong> broadest sense, it is a technique that examines <strong>the</strong> content <strong>of</strong>
1050