SEKE 2012 Proceedings - Knowledge Systems Institute

TABLE III.
Analyze
For the purpose of
With respect to
From the point of view
In the context of
GOAL OF THE 2 ND .EMPIRICAL STUDY
The 2 nd version of the REMO technique
Characterizing
Effectiveness and adequacy of the
identified requirements using the
business processes in the BPMN, when
compared to a traditional approach.
Software Engineering Researchers
A requirements elicitation carried out by
graduation students and based on
business processes models
Hypotheses: the study was planned and conducted in order
to test the following hypotheses (null and alternative,
respectively) regarding the effectiveness and adequacy
indicators:
H01: There is no difference in t erms of effectiveness in
using the REMO technique to elicit requirements regarding
business processes.
HA1: The REMO technique presented a difference in the
effectiveness indicator, when compared to a traditional
approach.
H02: There is no difference in terms of adequacy of the
identified requirements when using the REMO technique and a
traditional approach.
HA2: The REMO technique presented different results
regarding the adequacy of the identified requirements (not
considering false positives), when compared to a traditional
approach.
Context: We carried out the 2nd empirical study in
November 2011, with others senior-level undergraduate
students from Computer Science course (Federal University of
Amazonas). In this experiment, the students also had tutorials
regarding requirements elicitation and business processes
modeling. The business process model chosen as object of
study was part of the processes from the Academic Secretary of
the University. We selected four academic processes: (a) to do
enrollment adjustment; (b) to request recovery of academic
credits; (c) request grades correction; and (d) monitoring
application.
Subjects: 20 subjects agreed to participate in this study and
played the role of system analysts. These subjects were
classified and divided into two groups of 10 subjects. We
balanced each group regarding the subjects experience with
respect to their experience in software development and
requirements elicitation.
Instrumentation: the main instruments in this study were
the same type used in the first study: (a) an execution guide
with tasks, (b) the business context document containing the
BPMN processes modeling, and (c) requirements register
spreadsheet. Furthermore, the group that used the REMO
technique received the technique’s heuristics document.
Execution: the subjects carried out the study the same day
in different rooms. There was a moderator in each room to
guarantee that there was no communication among the
subjects. The subjects had 180 minutes to carry out the
requirements elicitation. At the end of the study, all subjects
returned the requirements lists and the follow up questionnaire
filled out.
We created a unique requirements list, in order to
discriminate the requirements. A specialist in requirements
elicitation carried out the discrimination process. The
discrimination meeting was carried out to ide ntify which
requirements were considered inadequate (false-positives).
Results and Quantitative Analysis: Table IV shows the
results per group of subjects. We calculated the effectiveness
indicator using 152 known requirements from the business
process models used as object of study.
TABLE IV. QUANTITATIVE RESULTS PER SUBJECTS
Technique
/ Subject
Experience IIR FP TRR EI AI
01 High 63 21 42 27.63% 66.67%
02 Medium 46 07 39 25.66% 84.78%
03 Medium 59 09 50 32.89% 84.75%
04 Medium 47 03 44 28.95% 93.62%
05 Medium 54 09 45 29.61% 83.33%
06 Low 31 07 24 15.79% 77.42%
07 Low 47 15 32 21.05% 68.09%
08 Low 37 10 27 17.76% 72.97%
09 Low 21 09 12 7.89% 57.14%
10 Low 29 05 24 15.79% 82.76%
11 High 52 05 47 30.92% 90.38%
12 Medium 48 16 32 21.05% 66.67%
13 Medium 25 10 15 9.87% 60.00%
14 Medium 26 02 24 15.79% 92.31%
15 Medium 26 05 21 13.82% 80.77%
16 Low 32 06 26 17.11% 81.25%
17 Low 43 06 37 24.34% 86.05%
18 Low 45 00 45 29.61% 100.0%
19 Low 48 03 45 29.61% 93.75%
20 Low 41 03 38 25.00% 92.68%
Legend: IIR – Initially Identified Requirements; FP – False Positives; TRR –
Total of Real Requirements; EI – Effectiveness Indicator; AI – Adequacy
Indicator.
In order to validate these data we used the Mann-Whitney
statistic method (supported by the SPSS Statistics v17.0 tool)
and the boxplots analysis. As we can see in Fig. 5, the boxplots
graph shows that the effectiveness indicator was similar among
the subjects who used the technique and those who used the
traditional approach. When we compared the two samples
using the Mann-Whitney test, we did not find any significant
differences between the two groups (p = 0.850 and = 0.05)
therefore, supporting the null hypothesis H01.
TRADITIONAL
REMO
Figure 5. Boxplots for the Effectiveness indicator of the requirements.
37