SEKE 2012 Proceedings - Knowledge Systems Institute

TABLE I.
COMPARISON OF THE ACCEPTABILITY OF OVERHEAD TIME
BETWEEN THE TWO GROUPS
Question Interactive Background P-Value
Please rate the
acceptability of the time
spent on recording links
Please rate the
acceptability of time
spent on applying rules
TABLE II.
Ave: 2.4
± 0.9
Ave: 3.0
± 1.3
Ave: 2.2
± 0.8
N/A
Note: Scale of 1 to 5, where 1= “Highly Acceptable” and
5=”Unacceptable”
0.341
N/A
LEVEL OF DISTRACTION IN LINK CAPTURE WITH ACTS
Question Interactive Background P-Value
Did you find the turning on
and off of the recording
button distracting?
Did you find the rules to be
distracting?
Ave: 2.8
± 1.2
Ave: 2.2
± 1.0
Ave: 3.9
± 1.1
N/A
0.027
N/A
Note: Scale of 1 to 5, where 1= “Very Distracting” and 5=”Not at all”
I. EVALUATION
To assess the extent to which ACTS addre sses challenges
from the three perspectives, we ran case studies in vari ous
contexts: lab settings, a proprietary project, and an open source
project. We focused on the following questions:
Q1. Is the cost associated with capturing a traceability
link acceptable to users? (Economic and social)
Q2. Are architecture-centric traceability links usable?
(Technical and social)
Q3. Would users use the tool in future projects?
(Social, economic and technical)
A. Case studies in laboratory setting
Case studies in a laboratory setting allow us to perform a
screen capture of the user actions with the tool for later
analysis. W e solicited feedb ack from 18 participants, 15 of
whom are PhD stude nts in either Co mputer Science or
Informatics at the University of California, Irvine. In addition,
seven of these participants ha ve held indus try positions ( e.g.,
programmer, architect, manager, and intern), allowing them to
assess the feasibility of the tool based on their roles in industry.
The participants were asked to use the AC TS tool in the
following task: capture trac eability links while editing a
structural design in A rchStudio and while viewing or editing
documentation files. Half of the user s were asked to
interactively apply rules and half applied the same rules in the
background. All users were also asked to retrieve the captured
links. Prior to the task, users were given a brief tutorial on how
to use the tool (about 5 m inutes). After t he task, users were
then asked to provide feedback via a questionnaire and semistructured
interview. Based on the screen capture, we also
measured the time for capt uring and retrieving traceabili ty
links.
Q1: Both groups concur that time spent is a cceptable (see
Table 1). An independent one-tailed t-test shows th at there is
no significant difference bet ween the ratings of both groups,
suggesting that both groups view the overhead as e qually
acceptable. Users commented that the tool “make[s] the lin king
job easier” and “[t]he tool saved me lots of time. Thanks!”
Using the recorded screen ca pture, we also measured the
time it takes for users to c apture traceability links and to
retrieve the captured links. The box plot in Figure 2 show s the
distribution of overall tim es between users who applied the
rules interactively and users who applied the rules in the
background. The box plot shows that most of the users are
more efficient when applying rules in the background. A onetailed
independent t-test co mparison between the two groups
yields a p- value of 0.005, strongly suggesting that appl ying
rules in the background is m ore efficient than interactive
application.
Meanwhile, the participants in the interacti ve rules group
felt that the manual application of rules was less ac ceptable
(average rating of 3 in Table I). These users felt that turning the
record button on and off was “somewhat distracting” while
users who applied the rules au tomatically in the background
found it less distracting (see Table II). An independent onetailed
t-test between the two groups on this question shows p =
0.027, indicating there is a significant difference between the
views of the groups. A user from the interactive rules gr oup
described turning the recording on and off to be “tedious.”
Q2: Linking artifacts to the architecture (or design) is a
useful feature to some of our pa rticipants. Five participants
(three with industry expe rience) identified this feature as the
feature they liked about the t ool. One of these participants, an
architect/programmer, reasoned that “ [i]n practice [it is]
necessary to link architectu re to doc uments (requirements
specs). [T]his makes the documents more useful.”
Q3: Over hal f of the particip ants indicated interest in
continued usage of the tool for their own software development
projects. Nine participants answered “yes”, and three
additional participants answered “yes” with conditions, such as
if the user is a system architect using ArchStudio or if the user
needed to link the documentation with the design. Three
additional participants answered “maybe”.
Most of the participants l iked the abilit y to link design
elements to documentation, the ability to link to specific points
within the documentation, and the ease of navigating to the
documentation from design. One participant liked the ability to
view the artifacts in their native tool: “Simplifies the process,
everything is integrated into one workspace.”
Figure 2: Time to capture and retrieve traceability links
415