27.06.2013 Views

Proceedings of the 8th International Conference on Intellectual ...

Proceedings of the 8th International Conference on Intellectual ...

Proceedings of the 8th International Conference on Intellectual ...

SHOW MORE
SHOW LESS

You also want an ePaper? Increase the reach of your titles

YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.

Mohd Zali Mohd Nor et al.<br />

Table 4: Rodriguez KMS - summary <str<strong>on</strong>g>of</str<strong>on</strong>g> 29 measured items<br />

Raw Score Count Measure Model Error<br />

Infit Outfit<br />

MNSQ ZSTD MNSQ ZSTD<br />

MEAN 41 15 1.51 0.83 0.87 -0.10 0.96 0.20<br />

S.D. 5 0 1.72 0.51 0.39 0.70 0.77 0.90<br />

MAX. 53 15 5.26 2.62 2.06 1.80 3.63 2.40<br />

MIN. 31 15 -2.12 0.48 0.05 -1.40 0.01 -1.20<br />

REAL RMSE 1.00 Adj.SD 1.40 Separati<strong>on</strong> 1.41 Item Reliability .66<br />

MODEL RMSE .98 Adj.SD 1.42 Separati<strong>on</strong> 1.45 Item Reliability .68<br />

S.E. OF Item MEAN = .33<br />

The summary <str<strong>on</strong>g>of</str<strong>on</strong>g> items’ measures for SmCAT and Rodriguez KMS is listed in Appendix A.<br />

4.1 Data reliability and fitness<br />

Based <strong>on</strong> <str<strong>on</strong>g>the</str<strong>on</strong>g> above summary tables, <str<strong>on</strong>g>the</str<strong>on</strong>g> SmCAT pers<strong>on</strong> reliability and item reliability are 0.90 and<br />

0.68, which are very good and fair, respectively. The Cr<strong>on</strong>bach-Alpha pers<strong>on</strong> raw score reliability is<br />

0.92. These indicate that SmCAT resp<strong>on</strong>ses are reliable for fur<str<strong>on</strong>g>the</str<strong>on</strong>g>r analysis. Similarly, <str<strong>on</strong>g>the</str<strong>on</strong>g> Rodriguez<br />

KMS pers<strong>on</strong> reliability and item reliability are 0.94 and 0.66, respectively, with Cr<strong>on</strong>bach-Alpha <str<strong>on</strong>g>of</str<strong>on</strong>g><br />

0.95. The Infit and Outfit Mean Square (MNSQ) for Pers<strong>on</strong> and Item means are within <str<strong>on</strong>g>the</str<strong>on</strong>g> limit <str<strong>on</strong>g>of</str<strong>on</strong>g> 0.6<br />

to 1.4 logit (Wright & Linacre, 1994). This indicates that <str<strong>on</strong>g>the</str<strong>on</strong>g> data fits <str<strong>on</strong>g>the</str<strong>on</strong>g> Rasch rating scale model,<br />

and could be used for fur<str<strong>on</strong>g>the</str<strong>on</strong>g>r analysis.<br />

4.2 T-Test<br />

To determine if we could c<strong>on</strong>clude <str<strong>on</strong>g>the</str<strong>on</strong>g> above hypo<str<strong>on</strong>g>the</str<strong>on</strong>g>ses, several t-tests are performed (at 95%<br />

c<strong>on</strong>fidence level), with <str<strong>on</strong>g>the</str<strong>on</strong>g> following results:<br />

Table 5: T-Test Results<br />

Mean Mean SD SD t-test df= t-value<br />

Segment<br />

SmCAT Rodriguez SmCAT Rodriguez value (n-1) @ 95%<br />

H1 - Overall 0.69 1.40 1.36 1.72 1.74 26 1.71<br />

Based <strong>on</strong> <str<strong>on</strong>g>the</str<strong>on</strong>g> above t-test results, SmCAT, in overall, is <strong>on</strong>ly slightly significantly better than<br />

Rodriguez KMS. However, since Rasch <strong>on</strong>ly provides unidimensi<strong>on</strong>al measures, we could not<br />

c<strong>on</strong>clude if SmCAT is also better than Rodriguez KMS in different segments <str<strong>on</strong>g>of</str<strong>on</strong>g> Knowledge Quality,<br />

User Satisfacti<strong>on</strong>, Perceived KMS Benefits, System Quality and System Use.<br />

4.3 Comm<strong>on</strong> Pers<strong>on</strong> Equating plot<br />

Whilst <str<strong>on</strong>g>the</str<strong>on</strong>g> t-tests may reveal <str<strong>on</strong>g>the</str<strong>on</strong>g> overall and segments results, <str<strong>on</strong>g>the</str<strong>on</strong>g> Comm<strong>on</strong> Pers<strong>on</strong> Equating (CPE)<br />

plot (see Figure 5) is an alternative method to evaluate <str<strong>on</strong>g>the</str<strong>on</strong>g> individual items distributi<strong>on</strong> for both<br />

SmCAT and Rodriguez KMS. CPE plots <str<strong>on</strong>g>the</str<strong>on</strong>g> items’ measures for SmCAT against items’ measures for<br />

Rodriguez KMS. Two standard deviati<strong>on</strong> upper and lower limits are also drawn. The items outside <str<strong>on</strong>g>of</str<strong>on</strong>g><br />

<str<strong>on</strong>g>the</str<strong>on</strong>g> limits are c<strong>on</strong>sidered items with significant differences (at 95% c<strong>on</strong>fidence interval). The items<br />

above <str<strong>on</strong>g>the</str<strong>on</strong>g> upper limit are items which Rodriguez KMS are significantly better. Meanwhile <str<strong>on</strong>g>the</str<strong>on</strong>g> items<br />

below <str<strong>on</strong>g>the</str<strong>on</strong>g> lower limit denote <str<strong>on</strong>g>the</str<strong>on</strong>g> items which SmCAT are significantly better than Rodriguez KMS.<br />

Based <strong>on</strong> <str<strong>on</strong>g>the</str<strong>on</strong>g> CPE plot, Rodriguez excels in four KMS success indicators; KQ2 - Words And Phrases<br />

Are C<strong>on</strong>sistent, KQ3 - C<strong>on</strong>tent Representati<strong>on</strong> Logical and Fit, KQB - Provide C<strong>on</strong>textual Knowledge<br />

and SU1 - Making Decisi<strong>on</strong>s. Meanwhile, SmCAT is significantly better in KQ9 - Complete Knowledge<br />

Portal, KQA - Accurate Expert Directory, PB2 - Effectively Manage Knowledge, PB5 – Improves<br />

Quality, US2 - Efficient and SU5 - Share Specific Knowledge.<br />

5. C<strong>on</strong>clusi<strong>on</strong> and future works<br />

Wu and Wang’s KMS success model was used to evaluate whe<str<strong>on</strong>g>the</str<strong>on</strong>g>r SmCAT really help users and<br />

maintainers to share and access <str<strong>on</strong>g>the</str<strong>on</strong>g> knowledge and informati<strong>on</strong> easier. Based <strong>on</strong> <str<strong>on</strong>g>the</str<strong>on</strong>g> survey results,<br />

SmCAT, in overall, is significantly better than Rodriguez KMS. SmCAT is also significantly better than<br />

Rodriguez KMS in Knowledge Quality, User Satisfacti<strong>on</strong> and Perceived KMS Benefits. In individual<br />

comp<strong>on</strong>ent level, SmCAT is still lacking in several key KMS comp<strong>on</strong>ents such as C<strong>on</strong>tent<br />

379

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!