SEKE 2012 Proceedings - Knowledge Systems Institute

Qualitative data analysis
The Mann-Whitney U test applied over the
questionnaire responses, revealed that the differences in the
satisfaction attribute of both groups were statistically
significant at 95% confidence level.
The results show that the differences between the mean
responses to all the questions concerning the warning
mechanism are statistically significant. This applies to the
system status feedback indicators as well. But, the data for
the other mechanisms were not always significant. The
help mechanism indicators show that three out of the five
survey questions were statistically significant, whereas
only two out of the four questions for global undo were
statistically significant.
We analyzed the survey questions for these
mechanisms and concluded that this was a foreseeable
result because some of the questions were worded in such
a manner as not to signal the functionality that the
mechanisms actually provide.
Removing the incorrectly worded questions, we can
calculate the mean increase in user satisfaction [24] with
respect to the inclusion of each mechanism. Table II shows
the results. These results show that system status feedback
is the mechanism that has the great impact on the
satisfaction attribute
B. Experimental test using EasyFlight tool
As a result of the lessons learned in the pilot study, we
took corrective measures in order to gather more, and more
accurate and reliable data. So, we designed another
scenario-driven software application with automated
quantitative data capture to prevent any errors caused by
human task performance timing. We also implemented the
automatic capture of number of clicks or equivalent
navigational actions functionality. We revised and
corrected the survey questions associated with the different
mechanisms and automated survey-taking. Users of this
new application do not choose which mechanism to enable,
that is, the application automatically enables the
mechanism when they select which scenario to execute.
TABLEII.
MEAN INCREASE IN SATISFACTION
MECHANISM Increase (%)
Warning* 161.43
Global Undo* 213.99
Help* 223.01
System Status Feedback* 256.58
* Statistically significant
Thanks to the features added to the new application, the
experiment can now be run without the physical presence
of the participant users. This means that we can test many
more subjects from a heterogeneous group.
The new application developed was the EasyFlight
[30]. It had the following functionalities: perform airport
administration (i.e., add/edit airports), list and optionally
purge the list of expired bookings, check available flights
for an airline route or book a flight.
Recruiting process
The people that participated in the experiment were
recruited by email. Around 75 requests were sent, and 46
acceptances were received. Of the 46 participants, 24 were
categorized as general users (GU) and 22 as computingrelated
users (CRU). This categorization is necessary
because computer literate users might perform better than
general users, as they are likely to have frequent exposure
to controls and procedures that software applications
usually provide such as calendars, dropdown lists,
warnings and abort mechanisms and/or go back controls.
The categorized users were assigned to either the control or
test group. Each one of these two groups was composed of
23 people: 12 GU and 11 CRU. We assumed that response
time was random and considered it unnecessary to set up
any other randomization process to select the elements of
each group. To populate the control group, we went down
the list and picked the first 12 GU and the first 11 CRU.
We populated the test group similarly.
Task assignments
All 46 participants were given the same tasks related to
Global Undo, Progress Feedback, Structured Text Entry
and Go Back mechanisms. The tasks are “Undo airport
addition”, “Get list of expired bookings”, “Get flight
information” and “Book flight tickets”.
All participants were set the same set of four tasks.
Details on how to login into the system were provided
along with these tasks. Figure 1shows an example of some
of the steps of the scenario for the task “Book Flight
Tickets” and Figure 2 illustrates the corresponding
questionnaire.
Data analysis
As with EasyTheatre, the datasets were analyzed
depending on the type of data they contained. The detailed
results of these analysis can be found in [31].
601