Create

44
The work of the assessment panels will be
to judge the impact of each idea on each of
the 23 main system attributes. For this the
SP2 process requires a uniform ranking scale
defined both in abstract terms as well as in
numerical terms to allow mapping against
the top level issues. The CREATE team defined
the ranking scale as a numerical value with
a 5-point scale from: strongly positive at +3
points ,weakly positive at +1 point, neutral=0,
and similar marks on the negative side.
For the purposes of the CREATE assessment
process we consider a “neutral” ranking to be
the minimum value required for an idea to be
worthy of incubation for each criterion ranked.
While the set of criteria presented above was
designed to be as comprehensive as possible,
given the wide range of ideas likely to be
submitted to the IDEA Portal it is possible
that some criteria may not be significant
with respect to a specific idea. In this case
it is possible to rank these criteria as “not
applicable”. Basically, this special ranking
is similar to “neutral” but can be used to
normalise the overall rankings of ideas. Further
notes are at Appendices A & B.
It is evidently necessary to define for each
criterion a precise interpretation of the
marking. At Appendix B examples of the
recommended detailed descriptions for the
second level criteria are shown. Such a detailed
listing was found to be necessary during
the trial assessment activity when questions
arose regarding the semantics and intended
interpretations for the criteria. Certainly this
affected some criteria more than others and
this is why the lengths of the descriptions
differ greatly. In addition, for each criterion
there is an explanation given on how to
apply the ranking scale. The descriptions of
the criteria and the ranking instructions are
intended to remove personal bias from the
assessment as far as possible. However, there
may be cases in which the ranking instructions
will not fully match the particulars of a given
idea. In these cases the ranking instructions are
to be considered as guidelines outlining the
intentions behind a criterion.
6.6.5. Lessons Learnt
The assessment process and the criteria set
have evolved over the duration of the CREATE
project. In February 2010, the set of assessment
criteria presented above was tested twice by
being subjected to assessment activities. It was
first tested by a group of Bauhaus Luftfahrt
scientists and then by a session of academia
and industry stakeholders. These two test
activities produced useful feedback on the
criteria set as well as on the overall process.
Based on this feedback several issues were
identified that are particularly important.
Ethics as a criterion
The assessment criterion “Ethical constraints”
caused numerous discussions throughout the
assessment workshop. The discussions related
to the scope of applicability of this criterion as
well as to the exact application of the ranking
scale. Still, the discussions evolving around the
ethics of ideas were seen as an indicator that
a criterion of this kind must be included in
the criteria set. As a result of these discussions
we formulated an extended definition for
this criterion. We are well aware that our
definition is neither fully exact nor fully
comprehensive but should serve as a useful
guideline for assessment panel members.
Orthogonality of the criteria set
One problem of the early versions of
the criteria set was its inherent lack of
orthogonality. Orthogonality in this context
means that every aspect considered relevant for
idea assessment should be covered by exactly
one criterion. However, there were some
criteria that violated this condition. In order
to rectify this issue the criteria set was revised
after the assessment workshop. A number
of criteria that were found to be redundant
or otherwise unsuitable for the process were
dropped while some other criteria subsets were
merged into single criteria.
Assessment panel
The experience gained from the assessment
session suggests that an assessment panel
consisting entirely of aerospace engineers – as
might be expected in an aeronautics activity
– is not necessarily ideal. The assessment
body should incorporate a balanced mixture
of experts from different fields and also a
balanced mixture of visionaries and sceptics.
Preparation of the assessment panel
A diligent and thorough preparation of any
assessment session and of its participants is
of vital importance. This must ensure that
all assessors have a common understanding
of the assessment process, the assessment
criteria and the assessment objectives. The
short briefing held for the assessors on the
assessment objectives at the assessment
workshop turned out to be insufficient. The
goals of the CREATE process differ significantly