SEKE 2012 Proceedings - Knowledge Systems Institute

equirement was considered “false positive” when it was
wrong (e.g. a misunderstanding about the business
processes) or in the case that it could not have been
extracted through business processes models. This
indicator is considered relevant because it prevents the
specification of requirements that are not in conformity
with the identified needs extracted from the business
processes.
The experiment goal using the GQM (Goal/
Question/Metric) paradigm [2] is presented in Table I.
TABLE I. GOAL OF THE 1 ST .EMPIRICAL STUDY
Analyze
The REMO technique
For the purpose of Characterizing
Effectiveness and adequacy of the
With respect to identified requirements using the
business processes
From the point of view Software Engineering Researchers
A requirements elicitation from a w eb
In the context of based system having as basis business
processes models.
Hypotheses: the study was planned and conducted in order
to test the following hypotheses (null and alternative,
respectively):
H01: There is no difference in t erms of effectiveness in
using the REMO technique to elicit requirements regarding
business processes.
HA1: The REMO technique presented a difference in the
effectiveness indicator, when compared to a traditional
approach.
H02: There is no difference in terms of adequacy of the
identified requirements when using the REMO technique and a
traditional approach.
HA2: The REMO technique presented different results
regarding the adequacy of the identified requirements, when
compared to a traditional approach.
Context: the study was carried out in M ay 2011 with
undergraduate students from senior-level undergraduate
Computer Science and Information System courses. These
students were attending Analysis and Design class at the
Federal University of Amazonas (UFAM). All the subjects had
tutorials about Business Process Modeling and Requirements
Elicitation. As the object study we used the business processes
modeling of part of the processes. This part described the
management activities of the discipline “Final Project”.
Preparation: 30 students played the role of system analysts
and signed a consent form. The subjects also filled in a
characterization questionnaire, with questions regarding their
practical experience. This questionnaire allowed us to identify
that, despite being undergraduate students; many of them had
experience in system analysis in the industry. An ordinal scale
was used to measure their experience: Low, Medium and High.
Three subjects were classified with High experience (having
participated in f ive or more development projects), sixteen
subjects were classified with medium experience, and the
remaining eleven with low experience (with theoretical
knowledge and no previous experience in industrial system
analysis). The categorization results are shown in Table II.
Execution: the subjects received: (a) an execution guide
with tasks, (b) the business context document containing the
BPMN processes modeling, and (c) requirements register
spreadsheet. Furthermore, the group that used the REMO
technique received the additional documentation: the
technique’s heuristics document. The group that did not use the
REMO technique used a traditional approach to identify the
requirements, using the knowledge used in the previous
trainings and the processes modeling, as suggested by [4].
The study was carried out by each group in different days.
The first day Group 1 used the traditional approach and the
second day Group 2 used REMO technique. Each group had
120 minutes to carry out the requirements elicitation based on
the business processes models. We collected 13 and 15
spreadsheets from the traditional approach and the REMO
technique approach respectively. Readers must take note that 2
students did not show up the first day, leaving the groups
unbalanced. We carried out an outlier analysis in order to
balance the groups. After finishing the study, we created a
unique requirements list, in order to discriminate the
requirements. The analyst responsible for the development of a
system for automating final projects control carried out the
discrimination process. The discrimination meeting was carried
out to identify which requirements were considered inadequate
(e.g. misunderstandings about the business processes).
Results and Quantitative Analysis: Table II p resents the
results of the requirements registers per group of subjects.
Readers must take note that we used a total of 29 known
requirements from the processes models, in order to calculate
the effectiveness degree of the requirements.
TABLE II.
Group 1 (TRADITIONAL)
REQUIREMENS RESULTS PER GROUP OF SUBJECTS
Group 2 (REMO)
Exp Sub IIR FP EI (%) AI (%) Exp Sub IIR \ FP EI (%) AI (%)
H
A09 14 5 31.03 64.29 H R03 21 12 31.03 42.86
A13 21 10 37.93 52.38 R04 21 9 41.38 57.14
A03 19 8 37.93 57.89 R06 24 9 51.72 62.50
A04 32 17 51.72 46.88 R07 14 4 34.48 71.43
M
A08 32 19 44.83 40.63 M R08 21 11 34.48 47.62
A10 26 13 44.83 50.00 R10 18 5 44.83 72.22
A14 21 6 51.72 71.43 R11 19 8 37.93 57.89
A15 28 10 62.07 64.29 R12 21 8 44.83 61.90
A01 28 18 34.48 35.71 R02 12 4 27.59 66.67
A05 7 6 3.45 14.29 R05 19 5 48.28 73.68
L A07 12 6 20.69 50.00 L R09 23 8 51.72 65.22
A11 29 17 41.38 41.38 R13 38 20 62.07 47.37
A12 23 11 41.38 52.17 R14 24 17 24.14 29.17
Legend: H – High; M – Medium; L – Low; Exp – Experience; Sub – Subjects;
IIR –Initially Identified Requirements; FP – False Positives; AI – Adequacy
Indicator; EI –Effectiveness Indicator.
In order to validate these data we used the Mann-Whitney
statistic method, which is supported by the SPSS Sta tistics
v17.0 1 tool, and the boxplots analysis. Fig.2 shows the
distribution of effectiveness per subject, per technique. The
boxplots graph shows only a slight difference between the
subjects who used the technique and those who did not use it.
When we compared the two samples using the Mann-Whitney
1 http://www-01.ibm.com/software/analytics/spss/
35