SEKE 2012 Proceedings - Knowledge Systems Institute

In the following, we present the results of the
experiments run using the tools that we developed.
II. EXPERIMENTS
A.Pilot test using the EasyTheatre tool
First we ran a pilot test using the EasyTheatre
application [26]. EasyTheatre is an e-commerce
application for booking theatre tickets. One of the key
characteristics of the application is that it includes a
usability dashboard. Users can use the dashboard to enable
and disable functions corresponding to specific usability
mechanisms.
Apart from correctly capturing data, the goal of this
pilot study was to establish the soundness of the
application, as well as the type of data to be collected and
the reliability of the experimentation process.
We ran two experiments in which we tried to measure
the impact of global undo, help, system status feedback
and warning mechanisms on the efficiency and satisfaction
attributes.
The experiments were run with 2011 Universidad
Politécnica de Madrid Master in Software Engineering
students and 2010 Universidad Simón Bolívar Master in
Systems Engineering students. A total of 24 subjects
participated in the experiment. They were divided into two
groups: a control group (CG) that did not use usability
mechanisms and a test group (TG) that used built-in
usability mechanisms. Users were given use scenarios and
administered questionnaires used to collect the data in
writing. Users had to use the application to execute the
respective scenarios and then answer the survey questions
about the execution. The only difference between the
scenarios given to students belonging to the control and
test groups was that the control group members were not
asked to enable the mechanisms to be measured (global
undo, help, system status feedback and warning
mechanisms), whereas the test group members were. The
questionnaires were the same for both groups. Divided into
quantitative data: task time in minutes reported by users in
writing and qualitative data: responses on a 1-to-5 Likert
scale for each mechanism graded from Never to Always
[10][24][27].
Note that we use the difference (increase) in the data for
the test group compared with the control group in order to
measure the impact.
Data analysis
The datasets were analyzed depending on the type of
data they contained. The quantitative data were analyzed
using the Microsoft Excel . and SPSS 16 (Statistical
Package for the Social Sciences) [28] software tools,
whereas the qualitative data were analyzed using SPSS 16
only. SPSS was used to perform the non-parametric Mann-
Whitney U test [29] to check whether the differences
between the variables are statistically significant, at 95%
confidence level.
The aim of the experiment was to determine whether
the mechanisms have an impact on the already mentioned
usability attributes. The Mann-Whitney U test determines
whether there is a statistically significant difference
between the data reported by the CG and the TG. Not until
this statistical significance has been determined can we
infer the effect of each mechanism on the usability
attributes.
Quantitative data analysis
Table I lists the mean times used by the control and the
test group for each mechanism.
The results of the Mann-Whitney U test applied to the
gathered quantitative data show that the difference between
the mean times to complete a task in the test and control
groups was not statistically significant in any case. The
reasons for this could be any, all or a combination of the
following:
• Homogeneous group: all users were studying for a
computing-related master’s degree and had a
computing-related background.
• There were fewer than 20 subjects per group.
• Users reported time manually in minutes.
Although the Mann-Whitney U test results are
inconclusive, note that, in all cases except for the global
undo mechanism, there was a positive difference between
the means of the time indicator in minutes in the test and
control groups.
There is an explanation for result with respect to the
global undo mechanism, namely, users cannot undo actions
if the mechanism is not enabled. Control group users
cannot find a button to undo the action, so they
immediately give up. Accordingly, when we analyzed user
responses to the control group survey associated with this
mechanism, we found that all the users of this group
responded that it is never possible to undo actions.
TABLE I.
MEAN TEST DURATION (MINUTES)
MECHANISM CG TG Difference Increase (%)
Global Undo 3.36 3.88 -0.52 -15,4
Help 4.0 2.91 1.09 27,25
Warning 2.41 1.7 0.71 29,46
System Status Feedback 4.3 2.3 2.0 46,51
600