SEKE 2012 Proceedings - Knowledge Systems Institute

Measuring the effect of usability mechanisms on
user efficiency, effectiveness and satisfaction
Marianella Aveledo M., Diego M. Curtino, Agustín De la Rosa H., Ana M. Moreno S.
Facultad de Informática
Universidad Politécnica de Madrid
Madrid, Spain
Abstract—In this paper we present the results of two
experiments designed to evaluate the impact of
particular usability mechanisms on software system
usability attributes. It focuses on mechanisms that
human computer interaction (HCI) and software
engineering (SE) researchers regard as having a
major impact on software development, like undo/
cancel or provide feedback. The experiments were run
using two different software applications that we
developed specially for the purpose. From the results,
we conclude that the inclusion of different usability
mechanisms has a positive impact on the usability
attributes of efficiency, effectiveness and satisfaction.
Keywords: usability, usability testig, usability attributes
I. INTRODUCTION
Usability is a software quality attribute listed in most
classifications [1][2]. According to ISO/EIC[3], usability is
an attribute of software system quality of use and is
defined as “The extent to which a product can be used by
specified users to achieve specified goals with
effectiveness, efficiency and satisfaction in a specified
context of use”. Usability means how well a product meets
stakeholders’ needs and achieves specified effectiveness,
efficiency and satisfaction goals in a specified context of
use. It is not surprising then that usability is increasingly
recognized as one of the critical factors for software
system acceptance [4].
It has been demonstrated that usability has implications
that go beyond the user interface and affect the system as a
whole [5][6]. In this context, Juristo, Moreno and Sánchez-
Segura [7] identify particular usability mechanism with a
major impact on the software functionality. They are:
progress feedback, system status feedback, warning, global
undo, abort, cancel, structure text entry and help.
Our aim is to study the effect of incorporating such
mechanisms in particular usability attributes like
efficiency, effectiveness and satisfaction. For that aim we
have conducted a literature review of the field of usability
evaluation and have found usability testing approaches
proposed by several authors [8][9][10][11] [12]. We have
studied the different approaches and proposals and found
that they all focus on the use of the continuous evaluation
of the designed interfaces in usability testing throughout
the iterative development of a software system. The goal
appears to be to reach the implementation stage with a
fairly mature interface model specifying the different
usability features established through the iterative
evaluation process. The different proposals evaluate a set
of ergonomic software criteria and/or usability features that
a well-designed user interface should have, such as, visual
clarity, consistency, compatibility and appropriate
functionality. Note that post-delivery evaluation proposals
focus on improving upcoming product releases. [9].
Several tools designed and developed for usability
testing have been reported in the literature since the 1990s
[13][14][15][16][17][18][19]. Notably, none introduce
methodological and/or tool proposals designed to evaluate
the software system interface from the viewpoint of how
mechanisms for improving different usability attributes can
benefit the system, considering effectiveness, efficiency
and satisfaction as usability attributes [2][3].
Recently we have designed two experimental
applications called EasyTheatre and EasyFlight. These
applications have been developed to capture data that
facilitate the measurement of the above impact [20]. These
applications are used to take quantitative and qualitative
measurements that provide sufficient data for calculating
the impact of the mechanisms on the attributes. The
quantitative measures taken are time to complete the task
and number of clicks or equivalent navigational actions.
The time to complete tasks is useful for measuring the
efficiency attribute, whereas the number of clicks and/or
equivalent actions is useful for measuring the impact on
relative efficiency [21][22]. The qualitative data that we
gather provide information on subjective user issues that
are used to calculate the impact on the satisfaction attribute
[21][23][24] through the mechanism/question/attribute
relationship [20]. So, the test performed can be defined as
summative [25], that is, a test whose goal is to perform a
competitive analysis to learn how much better (if any) the
application is perceived to be by users when the usability
mechanisms are implemented. Finally, the impact on the
effectiveness attribute is measured through information
supplied by users on whether or not they completed the
task successfully [2].
599