Proceedings of the 8th International Conference on Intellectual ...

Mohd Zali Mohd Nor et al.
Ano<strong>the</strong>r MAS prototype is similar to KM-Mantis, but uses different agents to capture knowledge from
different source documents and activities. Rodriguez et al. (2004) developed a MAS architecture with
5 main agents to monitor KMS activities in SM processes. However, <strong>the</strong> above prototypes do not
cover <strong>the</strong> full spectrum <strong>of</strong> SM processes, especially <strong>the</strong> Helpdesk, Maintenance Request (MR)
planning and Business Domain knowledge repository, to store and share SM knowledge and Domain
knowledge. These are covered by SmCAT.
3. Methodology
To evaluate SmCAT effectiveness as a KMS tool, <strong>the</strong> evaluation method shall need to answer
whe<strong>the</strong>r <strong>the</strong> tools really help users and maintainers to share and access <strong>the</strong> knowledge and
information easier. For this, SmCAT shall be benchmarked against a Rodriguez KMS prototype which
was also developed for SM environment. Users and maintainers were informed to use SmCAT for a
month, followed by Rodriguez KMS. Results on both tools could not be compared quantitatively, due
to different processes used and different Agents activities in SmCAT and Rodriguez KMS. In overall,
<strong>the</strong>re are more Agents’ activities in SmCAT compared to Rodriguez KMS, due to more activities in
helpdesk, MR and development areas. A qualitative method is required to gain a better perspective in
comparing KMS activities for SmCAT and Rodriguez KMS. Based on Wu and Wang’s KMS success
model (Wu and Wang, 2006), five main areas were addressed, as follows:
System Quality – defines how good <strong>the</strong> KMS is in terms <strong>of</strong> its operational characteristics, such as
ease <strong>of</strong> use, user friendly, response time and stable.
Knowledge or Information Quality – how good <strong>the</strong> KMS is in terms <strong>of</strong> its content quality and
Context quality. Content quality includes ease <strong>of</strong> creating document, consistent, logical words and
phrases, important and helpful, meaningful and practical, clear and unambiguous. Meanwhile,
context quality includes applicable to work, complete knowledge portal and good expert directory.
User Satisfaction – overall feelings <strong>of</strong> pleasure or displeasure regarding KMS. This includes
overall satisfaction, meeting knowledge needs, KMS efficiency and KMS effectiveness.
Perceived KMS Benefits - valuation <strong>of</strong> <strong>the</strong> benefits <strong>of</strong> <strong>the</strong> KMS by users. For example, helpful in
acquiring new knowledge, effectively manage and store knowledge, efficient, enhance job
performance and improve life quality.
System Use – The extent <strong>of</strong> <strong>the</strong> KMS being used. This includes helpful in making decision,
recording knowledge/information, communicating information, sharing general knowledge and
sharing specific knowledge.
Wu and Wang’s model provides <strong>the</strong> validated questionnaire constructs and measures. This makes it
easier for future studies to use this model. Questionnaire survey was distributed to <strong>the</strong> 15 users and
maintainers to gauge <strong>the</strong>ir opinion on <strong>the</strong> above areas. Users and maintainers who have used both
SmCAT and Rodriguez KMS were asked to fill <strong>the</strong> questionnaire. Since <strong>the</strong> questionnaire items are
based on Likert scale, Rasch analysis shall be used to estimate <strong>the</strong> probability <strong>of</strong> agreeableness on
each item. Rasch is a probabilistic model that uses ‘logit’ as <strong>the</strong> measurement units, by transforming
<strong>the</strong> ordinal data and raw scores into a linear scale (Bond and Fox, 2007). The data were first
inspected for reliability and fitness, to ensure that <strong>the</strong> data could be used for fur<strong>the</strong>r analysis.
To test if SmCAT is significantly better than Rodriguez KMS, pair-wised t-test shall be used, as
follows:
t =
β − β
S R
2 2
S σ R
σ +
n
Where βS = average measure for SmCAT, βR = average measure for Rodriguez KMS, σS = Standard
deviation for SmCAT and σR = Standard deviation for Rodriguez KMS.
The t-test results are compared against <strong>the</strong> Student’s t-test distribution table. T-test value exceeding
<strong>the</strong> critical value shall conclude that SmCAT is better than Rodriguez KMS. In addition, detail analysis
on each KMS success components could also be a valuable. As such <strong>the</strong> data shall be plotted onto a
Common Person Equating (CPE) chart (Bond and Fox, 2007). CPE plots <strong>the</strong> SmCAT KMS items
measures against Rodriguez KMS Items measures to determine <strong>the</strong> items (KMS success
377