D.3.3 ALGORITHMS FOR INCREMENTAL ... - SecureChange

that the interview guide is accurate. Moreover, to reduce this threat we have used a
number of information sources and data collection techniques.
– Internal Validity:Internal validity is an inductive estimate of the degree to which
conclusions about causal relationships can be made (e.g. cause and effect), based
on the measures used, the research setting, and the whole research design. A threat
to internal validity in our case study was related to communication issues with the
ATM experts. First of all there was a communication gap between the requirements
analysts and the ATM domain experts: certain concepts have a different meaning
for them. A second issue was related to language barriers: the ATM expert had to
provide feedbacks not in their mother tongue language. For these reasons, some
useful feedbacks might not have been given to the requirement analysts.
– External Validity:External validity concerns the extent to which the (internally
valid) results of a study can be held to be true for other cases, for example to different
people, places or times. Since we have conducted a qualitative user study,
our objective was not to do any generalization of the results to other contexts, but
understand the change management process adopted when a new tool is introduced
in complex and critical systems such as air traffic management systems and to investigate
the applicability of the approach to model and reason on evolution. To
generalized the findings of this user study we are planning to consider other complex
systems in different application domains.
– Conclusion Validity:This threat is concerned with the degree to which conclusions
reached are justified. Our conclusions are based on the remarks and comments
given by the ATM experts. The video recorded discussions between the ATM
experts and requirement analysts have been analyzed to provide evidence in accordance
to the real context of the ATM domain. To have an effective discussion with
the ATM experts, the requirement analysts also had a prior meeting with them to
confirm the protocol for the discussion.
5 Analysis of the Results
This section summarizes the feedbacks that were collected during the two workshops
WS1 and WS2 and analysis of the feedbacks by the requirement analysts.
5.1 Feedbacks During the Experiments Execution
On the applicability to the ATM domain of the approach to model and reason on requirements
evolution the participants P5 and P12 have pointed out that the approach could
be very useful during the initial phase of the change management process adopted by
ANSPs. The approach can help managers, technicians and controllers to understand the
implications and impacts of a proposed change. P5 and P12 suggested that Evolution
Elicitation and Probability Estimation might be useful to identify the alternative operational
requirements associated with a proposed change. The Reasoning phase instead
can be used to support the decision makers in identifying the best solution at operational
level to be implemented.
A challenge in applying the approach to model and reason on requirements evolution
was reported by participants P6 and P9. They argued that due to the complexity of
10