SEKE 2012 Proceedings - Knowledge Systems Institute

Following the experimental methodology (see Section 1),
we executed a feasibility study in September 2010 to evaluate
the first version of our technique [10]. Its goal was observe the
feasibility of the WE-QT technique regarding efficiency (the
ratio between the number of detected defects and the time spent
in the inspection process); effectiveness (the ratio between the
number of detected defects and the total number of existing);
and inspectors’ perception of our technique. Subjects were 12
undergraduate students attending a S oftware Engineering
course at the Federal University of Amazonas. The results
suggested that the WE-QT technique is feasible to assist novice
inspectors detecting usability defects. The feasibility study is
detailed in [10]. The results of this study were also used as
input to further improve the WE-QT technique, resulting in its
second version (WE-QT v2). Summarizing, the improvements
made in the technique were: (1) grouping questions related to
the same object by transforming them into multiple answer
questions; (2) removing redundant questions; (3) allocating
certain questions to be displayed only once during the
inspection flow; and (4) detailing certain terms of the
technique. Table II shows an extract of the WE-QT v2.
Set
1
2
TABLE II.
EXTRACT OF THE WE-QT TECHNIQUE - SECOND VERSION
Question
Does the page show any messages (error, warning
messages,...)?
Regarding the messages:
I cannot see the messages easily
I cannot understand the messages easily
The messages do not follow the visual pattern of the
application (same colors, text fonts,…)
Mapping
Yes No Next
2 7 ---
--- --- 3
7
The page:
Is not what I expected
Does not have a pleasurable interface, in general
Has texts, images, buttons that I cannot easily see
--- --- 8
Does the page inform you in which part of the application
8
you are at?
9 10 ---
9
Regarding your location:
I cannot see this information easily
I cannot understand this information easily
--- --- 10
10
The page:
Does not emphasize important information to reach my
goal
Emphasizes irrelevant information to my goal
Does not support frequent performed tasks
Does not provide an option to return to the previous page
(do not consider the browser option)
Does not provide an option to return to the home page
--- --- 11
11
Regarding the contents of the page:
The information, options and menus of the page are not
displayed in a natural and logic order
The words, texts, figures and symbols from the page are
not easy to be understood
The page does not provide the definition of images,
symbols and unusual words where they are displayed
The language of the page is not in accordance with its
topic
--- --- 12
IV. THE OBSERVATIONAL STUDY
Since the results obtained from the visibility study indicated
the WE-QT’s feasibility, we went one step further on following
the experimental methodology, in order to elicit the process
used by the usability inspectors when applying the technique
during a usability evaluation. With this purpose we performed
an observational study in October 2011 using the second
version of the WE-QT technique. Our aim was to understand
deeply the WE-QT process, so we did not compare the WE-QT
with any other technique. The observational study is d etailed
below:
A. Study Planning
The goal of this observational study, presented using the
GQM paradigm [22], can be formalized as:
Analyze: Web Evaluation Question-Technique (WE-QT v2)
For purpose of: understanding
With respect to: how the inspectors apply the WE-QT
technique
From the point of view: of Software Engineering researchers
In the context of: the evaluation of the usability of a real Web
application by undergraduate and postgraduate students.
In this study, we gathered two types of qualitative data:
observational and inquisitive data. The observational data was
collected during the inspection process, we used the Think-
Aloud Method that is an observation technique in which the
user is asked to 'think out loud' and describe the activities
he/she is performing as th ey are undertaken. This technique
helps understanding the difficulties experienced by the user and
also how they apply the technique [23]. We also used a
usability testing tool to capture the inspection section of each
inspector and to assist the collection of the perceptions of each
inspector during the evaluation. Inquisitive data was gathered
at the completion of inspection using follow-up questionnaires.
Despite the main goal of this study being eliciting the
process used by the inspectors when applying our technique,
we decided that executing quantitative analysis would provide
us with useful data. Therefore we used the effectiveness and
efficiency indicators, defined as: effectiveness (ratio between
the number of detected problems and the total of existing
problems); and efficiency (ratio between the number of
detected problems and the time spent in the inspection). These
indicators have been employed in other studies to evaluate Web
application usability inspection methods as well [9, 10, 24].
The object of this study was the Graduate student Portal
(www.ppgi.ufam.edu.br) of Federal University of Amazonas.
This application is used to support M.Sc. and PhD students as
well as teachers with academic matters. Three relevant use
cases were defined: (1) User authentication; (2) Registration in
two offered courses; and (3) Update personal data. Seven
subjects participated in the study, one undergraduate student
and six M.Sc. students. According to Nielsen [25], three to five
inspectors from each user profile are enough to detect most of
the usability defects.
During the planning stage we also elaborated the Inspection
Guide; the Consent Form; the Post Inspection Questionnaire to
collect the inspectors’ opinion about the technique; and the
Characterization Form to characterize subjects’ experience in
590