atw - International Journal for Nuclear Power | 10.2020

atw Vol. 65 (2020) | Issue 10 ı October
with the “correct” answers, which
were determined in advance. For the
ISV, we have decided to ask what the
relevant process aspects or process
parameters are in the current situation
and how these parameters will
change in the near future. Furthermore,
we have asked the crews about
their own tasks in the near future and
the goal of these tasks.
Due to the possible complexity
of processes, e.g. during incident
situations, it was clear that using free
test answers in the situation awareness
probes could lead large diversity
of answers which could impair an
appropriate assessment of their correctness.
This multitude of possible
answers might prevent any meaningful
evaluation.
Therefore a multiple choice test for
situation awareness was designed as
follows. Operators were asked for
process parameters or important
aspects which were relevant for the
decision on further procedures at the
moment of the simulator freeze. In
order to predefine the correct answers
for the freezes, we asked three experts
(2 simulator trainers and one process
engineer) independently of each
other, to specify in the freeze situations
which process aspects and
parameters are relevant for decisions
om future tasks and objectives in that
moment of the scenario. For some
cases, the experts had different views
on the relevant parameters. The final
set of offered parameters in the freezes
was based the highest degree of agreement
between the three experts.
For each freeze, six process para meters/aspects
were offered in a multiplechoice
test, three of which were correct
and three were incorrect. The incorrect
paramters/aspects (so-called “distractors”)
should not be obviously irrelevant
– then the test would be too easy.
But they should also not be too similar
to the relevant parameters either –
then the test would be too difficult. The
evaluation showed that we succeeded
in achieving a good level of difficulty
for the choices offered, but unfortunately
not in all cases.
Method 2 – Questions about own
tasks and colleagues’ tasks
During freezes, questions were asked
about future tasks and their objectives
(“When the current task is completed,
which will be your next task and its
objective”). These questions had to be
answered with regard to the own
tasks as well as with regard to the
tasks of the colleagues in the shift
crew (open-ended questions).
Method 3 – Status reports and
team briefings
A third method to assess situation
awareness was based on analysing
the content and thoroughness of
team briefs and status reports
which are a genuine element of the
team inter action in the control room.
Status reports were understood here
as summarized short descriptions
( mostly carried out by the SSV) of the
operational status of the plant or the
status of the most important parameters
and operating conditions.
Status reports were triggered by
external requests of the management
as part of the scenarios, when the
safety engineer entered the control
room, or at shift changes. Team briefs
were requested at strategy changes, or
before starting safety critical actions.
During status reports, the relevant
plant parameters and deviations from
normal operation at that time were
expected to be reported.
Procedure of the measurement
during freezes
During the three freezes, TO, RO and
SSV left their workplaces and moved
to an area outside the simulator. They
were then asked about (a) relevant
process parameters/aspects in the
present situation (multiple-choice:
choose at least three most important
parameters out of six parameters
given and rank order them according
to their importance for the present
situation), (b) the expected development
of these parameters (multiplechoice:
quick/slow decrease or
increase or no change). TO, RO and
SSV answered the questions presented
to them on a tablet computer
with keyboard in written form without
consulting their team members.
Their answers were compared against
the expected answers (based on
experts’ agreement) concerning the
relevant process parameters, tasks
and objectives. (situation awareness
measurement – Method 1).
Afterwards the operators were
asked about their current task
and their next task (coordination
measurement, see 3.5) as well as the
current task and the next task of
their colleagues, and the objectives
of the tasks (situation awareness
measurement – Method 2).
3.4 Communication
To evaluate communication, we have
identified between 13 and 31 measurement
points in each scenario
where communication was expected
to be important.
For these measuring points we
defined which content of communication
was expected (content of
communication) and what type of
communication was expected (quality
of communication). As for the quality
of communication, we distinguished,
between 2-way communication, 3-way
communication, briefing etc. For each
of these communication types, observable
behaviours were defined in order
to assess the quality of communication,
e.g. if the communication was
given in “face-to-face” manner, if
the receiver showed attention or if
the receiver showed a reaction or
expressed or showed understanding.
Both, content und quality of communication
were included in the
observation tool (see section 4) and
were checked by the observers during
a scenario trial.
At relevant points in the scenarios
the quality of communication was
evaluated not only according to
observable criteria, but also using an
overall assessment of the communication
process. A subjective assessment
by the observers was used here (rating
of poor, average or good).
3.5 Coordination
A satisfactory coordination was characterized
by the fact that all operators
knew what their colleagues were
doing (task awareness). For example,
the SSV should know which tasks TO
and RO are performing, the TO should
know which task the RO is performing,
and so on.
Data for coordination were col lected
via questionnaires during freezes.
After the questions concerning situation
awareness, questions for task
awareness of the operators (“Which
task is performed by your colleague
RO/TO/SSV?”) followed.
3.6 Workload
For workload assessment we used the
Bedford Workload Scale [6]. It is a
unidimensional scale that ranks
whether it was possible to complete
the task, if workload was tolerable for
the task, and if workload was satisfactory
without reduction. The scale
uses the concept of spare capacity to
define the levels of workload. A short
explanation of the scale before the
beginning of the scenario allowed
the operators to use it properly and to
rate their workload within seconds.
This allowed repeated measurements
of subjective workload during the
scenarios without too much intrusion
into the primary task. The workload
rating was announced via loudspeaker
ENVIRONMENT AND SAFETY 501
Environment and Safety
Are They Ready for Operation? How to Assess the Control Room System of a New NPP ı Rainer Miller, Rodney Leitner, Sina Gierig and Harald Kolrep