SEKE 2012 Proceedings - Knowledge Systems Institute

inspection and in usability. Table III shows the data obtained
from Characterization Form.
B. Study Execution
The inspection phase was carried out with each subject
individually. They were provided with the instruments to
accomplish the inspection and received instructions about the
evaluation by the moderator. Once the inspector understood the
procedures, the inspection process began. One researcher acted
as the observer, being responsible for conducting the detection
phase. It is worth to mention that the subjects did not receive
training on WE-QT technique, usability or inspections.
At the end of the inspection phase, a meeting attended by
the researchers and a control group formed by usability
specialists took place. A list of all usability problems identified
by the subjects was discriminated to classify these problems
into real defects or false positives. The authors of the technique
did not influence the discrimination.
V. QUANTITATIVE DATA ANALYSIS
As a result of the inspection, we identified a total of 85
usability defects. We computed the number of detected defects,
time spent during the inspection phase, efficiency and efficacy
for each inspector. Table III presents these results including
and their experience level.
N°
TABLE III.
Usability
Experience
SUMMARY OF THE INSPECTION RESULTS PER SUBJECT
Inspection
Experience
Defects
Time
(hours)
Defects
/Hours
% Founds
Defects
01 None None 25 1,85 13,51 29,41
02 None None 48 0,77 62,61 56,47
03 Low Low 26 0,67 39,00 30,59
04 Low None 27 0,92 29,45 31,76
05 Medium High 22 0,72 30,70 25,88
06 Low Medium 21 0,87 24,23 24,71
07 Medium Low 35 1,47 23,86 41,18
Table IV shows the averages for the time, and effectiveness
and efficiency indicators. Regarding the efficiency indicator,
inspectors detected an average of 31.91 defects per hour using
the WE-QT technique.
Total Known
Defects
TABLE IV.
DISCREPANCIES AND DETECTION TIME
Effectiveness
Average Indicator
(%)
Average Time
(Hours)
Efficience Average
Indicator
85 34,29% 1,04 31,91
Regarding the effectiveness indicator, each inspector found
an average of 34.29% of known defects. Considering the time
of inspection, each inspector spent an average of 1.04 hours (or
62 minutes) detecting the usability problems.
As mentioned before, despite the goal of this study being
eliciting the process used by the inspectors when applying our
technique, the quantitative analysis results provide an important
opportunity to evaluate the performance of the technique.
Considering the effectiveness indicator in the last study [10],
the WE-QT technique resulted in 29,37%, while the new
version of WE-QT technique resulted in 34,29% (see Table
IV). These results are an indication that the WE-QT v2 is more
effective than its first version. Still considering this indicator,
we compared our results to the results of a study presented in
[6]. According to Gomes et al. [6], the effectiveness indicator
of the WDP technique resulted in 13% and WDP-RT technique
resulted 29%. This indicates that our technique is considerably
more effective than WDP and lightly more effective than
WDP-RT. As the number of usability defects depends on the
application, it is not recommended to compare the efficiency
indicator with previous studies.
VI. QUALITATIVE DATA ANALYSIS
We analyzed the inquisition data using coding procedures
[12] in order to understand the subjects’ perception about the
technique. We a lso merged the inquisition data with the
observational data.
At the end of this analysis, the coding processes produced
altogether 21 codes which were associated to 3 categories:
Positive Aspects of the technique, Negative Aspects of the
technique and Improvements Suggestions. Although the
study’s goal was to identify how the inspectors apply our
technique, the Post Inspection Questionnaire had questions
about the appropriateness and ease of use of the technique,
which was useful to provide us with the subjects’ perception of
the technique.
Regarding the category “Positive aspects of the technique”,
we identified codes as: “It is simple and easy to apply the
technique”; “Using the technique contributes to detect a greater
number of usability problems” and “I would use and
recommend the WE-QT technique”.
Concerning the category “Negative aspects of the
technique” (Fig. 2) we identified the code “The questions have
ambiguous items”, we observed that the subject had doubt with
38% of the questions of the WE-QT technique. Another cited
code is “There is no reference to pop-ups and some elements
present in Web applications”, the observation data show that
the technique also does not cover elements such as logins
authentication, despite this element being evaluated as regular
data forms, there is n o specific evaluation criteria for its
features. The code “It was not trivial knowing what kind of
answer to provide to the questions” was considered as a delay
factor, and it also indicates that the technique requires more
guidance. The code “Using the technique is a little tiring” was
considered a harm factor to the inspector enjoyment when
using our technique.
591