SEKE 2012 Proceedings - Knowledge Systems Institute

USE SCENARIO
The application VuelosFácil is an e-commerce web application
developed for the acquisition of low cost airlines flight tickets. Task
description: Assume that you are a person visiting the airline’s website
because you are interested into book flight ticket as long as they fit
on your vacations plans.
In this particular case, from the available flights you are interested into
booking 2 tickets from Barcelona to Granada on the date November,
11 th 2011 at 10:30 AM. However, at choosing the flight you make a
mistake and select the one at 6:00 a.m. Thus, you want to modify the
flying time before submitting the booking.
You shall:
1. Once logged in, select scenario “Escenario #14” from the
scenarios’ list that is available at the upper left corner of the
screen.
2. Click on “Iniciar”.
3. Click on “Usuarios”.
4. Choose the flight at 6:00 horas from Barcelona to Granada on
November, 11 th 2011. To do so:
a. Input “BCN-00” as source airport.
b. Input “GRX-92” as destination airport.
c. Input ”2011-11-11” as flight date
d. Input 2 as the number of passengers.
5. Click on “Mostrar vuelos” button.
…….
Figure 1. Scenario for the task “Book flight tickets”
The corresponding usability attributes were measured
as following: 1) Efficiency: difference between the means
time in milliseconds to complete the task calculated as a
CG/TG ratio value expressed as a percentage and 2)
Relative Efficiency: Difference between the number of
clicks per group with respect to the minimum number of
clicks required to complete the task.
Table III below shows the results for the increase in the
efficiency attribute due to each mechanism. Notice that for
all mechanisms but go back, the efficiency in the control
and test group is statistically significant.
A similar procedure was used to calculate relative
efficiency, where 100% task efficiency was equivalent to
the minimum number of clicks required to complete the
task. Table IV lists the relative efficiencies of each group.
As Table IV shows, we cannot assess the impact of the go
back mechanism, as it is not statistically significant.
We find, on the other hand, that the progress feedback
and structured text entry mechanisms have a positive
impact on the efficiency attribute, where structured test
entry is the mechanism with the biggest impact. Regarding
the relative efficiency attribute for the global undo
mechanism, we can only calculate the relative efficiency of
the test group, as it is impossible to undo operations if the
mechanism is disabled, and once again the structure text
entry mechanism is the one with the biggest impact.
Qualitative data analysis
Collected data
Figure 2. Questionnaire for task “Book flight tickets”
Both quantitative and qualitative data were collected on
each participant. Quantitative data are composed of the
time taken to perform the task, and the number of click for
performing the task. These data were collected
automatically without participant intervention. Qualitative
data are participant responses to each task survey. There
was no means of collecting these data without participant
involvement. The data were collected by the developed
web application and stored in a database.
Quantitative data analysis
We calculated the following indicators for each
quantitative dataset:1) the mean number of click equivalent
actions and 2) The mean time to complete the task in
milliseconds. The Mann-Whitney U test [29] was used to
check whether the differences between the mean variables
for the control, and test group are statistically significant.
We also calculate the mean values for each question in
the questionnaire and group. We can calculate the mean
response to each question, as the survey questions were
rated on a 1-to-5 Likert scale [27]. We used these values to
calculate the means.
Question #1, Were you able to complete the task?, was
the same in all surveys. This question was designed to
measure the impact of each mechanism on the
effectiveness attribute. It is justified by the fact that,
according to [20], effectiveness can be measured according
to user perceptions, and one such measure is “percentage
of users successfully completing a task”.
Table V shows the results for Question #1 of the survey
for each evaluated mechanism. The impact of the progress
feedback mechanism is inconclusive, as it is not
statistically significant. We find that the global undo,
structured text entry and go back mechanisms have a
positive impact on effectiveness, where structured text
entry has the biggest impact.
602