WWW/Internet - Portal do Software Público Brasileiro

ISBN: 978-972-8939-25-0 © 2010 IADIS4.1 Website A Evaluation Results and AnalysisGroup G1 used the Nielsen heuristic set whereas group G2 used the customised guidelines. Each evaluator inG1 found a number of problems in Website A by using HE. The final list consists of 10 problems, which aresummarised in Table 1; the ‘Total Problems by group G1’ column refers to the total number of problemsidentified by the four evaluators together. They were able to identify 5 major problems and the remainingproblems were minor or superficial problems. In general, group G1 was able to reveal less than a quarter ofthe identified usability problems by both groups in this study. The evaluators agreed from their questionnaireresponses that the website is highly usable and has taken into account usability guidelines.HETable 1. Number and percentage of usability problems uncovered on Website A by group G1.Evaluator1Evaluator2Evaluator3Evaluator4Total Problemsby group G1Total problemsdiscovered by bothgroups (G1, G2)Total 10 2 5 3 10 42% ofTotalProblems23.80% 4.76% 11.90% 7.14% 23.80% 100%Group G2, who performed the customised guidelines method, were able to identify 40 usability problems(summarised in Table 2), which was significantly more than the number of the problems identified by groupG1. The results from Table 2 show the percentages of the identified problems, which range from 35.71% to61.90%. As a result, this evaluation, which is based on the customised guidelines, obtained percentages thatare in line with findings of similar studies (Jakob Nielsen, 1994, 1990), which show that a single evaluatorusually finds 20% to 50% of the problems in a system. All the evaluators in group G2 were pleased to beusing the customised guidelines method as this method reminds them of a number of usability aspects theymight have forgotten to think about during the evaluation. Also, according to the evaluators’ answers in thequestionnaire, they believe that the customised guidelines method encourages them to be thorough in theevaluation, which is seen as a drawback of HE, providing more opportunities for finding usability issues. Theevaluators’ answers in group G1 state that HE did not help much as it is so general, does not cover all theusability aspects, and does not offer any clues for the evaluation. Although, the website was claimed to be ahighly usable, group G2 was able to uncover different usability problems, including 8 major problems and 25minor problems.CustomisedGuidelinesTable 2. Number and percentage of usability problems uncovered on Website A by group G2.Evaluator5Evaluator6Evaluator7Evaluator8Total Problemsby group G2Total problemsdiscovered by bothgroups (G1, G2)Total 21 26 15 19 40 42% of Total 50% 61.90% 35.71% 45.23% 95.23% 100%Problems4.2 Website B Evaluation Results and AnalysisGroup G1, who used the Nielsen heuristics in the first study, used the customised guidelines method in thisstudy to offer more validity to the study. Each evaluator in group G1 found a number of problems in WebsiteB. The final list of the identified usability problems consists of 38 problems, which are summarised in Table3. The results from Table 3 show the percentages of the identified problems, which range from 39.47% to74.47%. As a result, this evaluation, which is based on the customised guidelines, obtained percentageswhich are in line with and even higher than the findings of similar studies (Nielsen, 1990), which show that asingle evaluator usually finds 20% to 50% of the problems in a system. They discovered 5 major usabilityproblems and 20 minor usability problems. The evaluators attributed this to the usage of the customisedguidelines method as they preferred to use it rather than to use the HE method. They justified this as theformer method clearly specifies what should be evaluated and it is also a useful technique of building a172