SEKE 2012 Proceedings - Knowledge Systems Institute

composed by a set of well-defined activities that are used to
evaluate the system’s usability. UEMs are divided into two
categories: (a) user testing, in which empirical methods,
observational methods and question techniques can be used to
measure usability when users perform tasks on the system; and
(b) inspections, which make use of experienced inspectors to
review the usability aspects of the software artifacts [2]. In this
research, we focus on usability inspections as they can lower
the cost of finding usability problems since they do not need
any special equipment or laboratory [14].
The main generic UIMs that can be used to increase the
system’s usability and therefore its quality are: the Heuristic
Evaluation [8], the Cognitive Walkthrough [11], and the
Perspective-based Usability Inspection [15].
The Heuristic Evaluation, proposed by Nielsen [8], assists
the inspector in usability evaluations using guidelines. The
evaluation process consists of a group of e valuators who
examine the interface using heuristics, which are rules that
seek to describe common properties of usable interfaces.
The Cognitive Walkthrough, proposed by Polson et al.
[11], is a method in which a set of reviewers analyze if a user
can make sense of interaction steps as they proceed in a predefined
task. In order to identify usability problems the
inspectors ask questions to answer if: (a) the action is
sufficiently evident; (b) the action will be connected with what
the user is trying to do; and (c) if the user will understand the
system’s response.
According to Zhang et al. [15], it is difficult for an
inspector to detect all kind of problems at th e same time.
Consequently, they proposed a usability inspection technique
based on perspectives (Usability Based Reading - UBR). In
the UBR, the inspector focuses in a s ub-set of questions
according to the usability perspective to find problems.
In the following Section we show how we carried out the
extension of the systematic mapping on UIMs for the Web.
III.
EXTENSION OF THE SYSTEMATIC MAPPING ABOUT
UIMS FOR WEB APPLICATIONS
Fernandez et al. [2] presented a systematic mapping on
UEMs for Web applications. However, in order to thoroughly
describe how UIMs for the Web had been applied, it was
necessary to independently analyze them. In this Section, we
briefly explain the planning and execution of the extension of
the systematic mapping in [2]. In this extension, we extracted
specific information from [2] regarding the new UIMs for
Web applications. We used the obtained results to answer the
following research question: “What new Usability Inspection
Methods have been employed to evaluate Web artifacts and
how have these methods been used?” Readers must take note
that a thoroughly described version of the execution process of
this extension can be found in [12].
Selection Process: Fernandez et al. [2] analyzed 206
papers about usability evaluation methods for Web
applications and classified them into categories. We have used
this classification as a starting point in the selection of papers.
We selected papers that, according to [2], described new
inspection methods for Web applications. From this initial set
of papers we only selected the studies that thoroughly
described UIMs at a mature stage. Consequently, we discarded
papers that met at least one of the following exclusion criteria:
Papers presenting usability problems and no
methodology to identify them.
Papers describing only ideas for new research fields.
Papers presenting techniques with no description of
their execution process.
Categorization of Studies: We cr eated a set of research
sub-questions to better address the state of a rt of U IMs for
Web applications. We used the answers to these sub-questions
to categorize the analyzed papers. Table I shows our research
sub-questions, their motivation, and the possible answers that
can be obtained when analyzing a selected research paper.
TABLE I. RESEARCH SUB-QUESTIONS, POSSIBLE ANDSWERS AND
MOTIVATIONS FROM THIS SISTEMATYC MAPPING EXTENSION.
Research Sub-Questions and
Answers
Q1. Theoretical Basis Method:
(a) Heuristic Evaluation
(b) Cognitive Walkthrough
(c) Perspective Based
(d) Other Basis
Q2. Type of Evaluated Artifact:
(a) HTML code
(b) Model
(c) Application/Prototype
Q3.- Type of Application
Evaluated by the Inspection
Method:
(a) Generic
(b) Specific
Q4.- Use of Inspectors in the
Inspection Process:
(a) Yes
(b) No
Motivation
To discover whether the Usability
Inspection Methods for the Web have
been developed considering well known
Generic Usability Inspection Methods or
whether they have been using new basis.
To discover which is the most commonly
evaluated artifact in Usability Inspection
Methods for the Web.
To discover whether the Usability
Inspection Methods for the Web have
been crafted to find generic usability
problems or usability problems of a
specific type of Web application.
To discover whether the Usability
Inspection Methods for the Web have
been automated to a point where
inspectors are no longer necessary.
Execution: Fig. 1 shows how we executed this systematic
mapping extension. From the initial set of 206 pa pers in
Fernandez et al. [2], we selected 37 papers that, according to
the classification in [2], presented new usability inspection
methods for Web applications. However, as 5 p apers were
unavailable for download, we reduced the initial set to 32.
Figure 1. Execution process of this Systematic Mapping Extension.
After reading each study, we discarded 6 studies for
meeting the exclusion criteria we defined in the selection
process stage. The Selected Primary Studies List in this paper
shows the 26 selected papers that we analyzed in this literature
review. In the next Sections we explain how we used the data
583