atw - International Journal for Nuclear Power | 10.2020

More documents

Info

atw Vol. 65 (2020) | Issue 10 ı October ENVIRONMENT AND SAFETY 500 2.5 Procedure Before crews started their first scenario, they were introduced to the ISV, they provided written consent not to give any information on scenarios to anyone not involved during the day. The crews then practiced handling the tablet computers: They filled in the questionnaires assessing the situation awareness and coordination on a test basis and practiced the workload rating. The tablet computers were then taken to the work places of the operators. In each trial before the simulation was started, all crew members were provided with two sheets. The first contained all information on the starting conditions for the scenario, such as actual nuclear power and unavailable components. The second sheet was provided for notes concerning any occurring inconsistencies during the subsequent trial, e.g. in the OM or on displays. The simulation started when the operators had settled in at their work places. The trial then followed the predefined target path. Workload was assessed every 20 minutes using the operators’ tablet computers at the work stations. For each scenario at three predetermined moments the simulation was briefly stopped. During these “freezes” the crews had to take their tablets and leave the FSS for about 10 minutes. In an adjacent room, the tablet computers were used to collect data on situation awareness and coordination. After the freezes, the operators went back to their workplaces and the simulation was resumed. The trials ended at predefined DIMs or plant states. After each trial, a systematic debriefing was carried out in the FSS, in which the test crews were asked about specific behavioural patterns and give their feedback on any inconsistencies concerning the OM or other system parts. The debriefing then continued without the operators to initially evaluate task performance and human error together with all observers. 3 Measurement of variables 3.1 Task performance Task performance was the primary evaluation aspect. It considered the correctness and completeness with which the shift crews fulfilled their tasks. To determine the task performance, global success criteria (which were valid for all scenarios), scenario specific success criteria and task performance key nodes were defined. Success criteria At the end of a scenario it was evaluated whether the following three global success criteria were met: (1) no unforeseen escalation of the scenario, (2) no damage of major equipment during the scenario, (3) and no relevant delay during the scenario should be caused by behaviour of the crew. In addition, three to five scenario specific success criteria were defined for each scenario. Task performance key nodes Furthermore, four to eight critical points or decisions in the process were identified (‘key nodes’) for each scenario to evaluate the task performance. The expected behaviour at these key nodes was predefined. An example of a key node and related scenario specific success criteria: During scenario B, an alarm occurred at a seal of a reactor coolant pump (RCP). The crew was expected to make the decision to manually shut down the RCP before an automatic shutdown occurred. The task performance key node here is “manual trip of the RCP”. The two scenario specific success criteria associated with this problem are “After trip of reactor coolant pump: start decreasing the power to 0 %” and “No attempt to restart the RCP”. The generic assessment questions as to task performance were: p Does the crew behave as expected at the task performance key nodes? p Are pre-defined scenario-specific success criteria reached? p Are the global success criteria reached? Data collection was done during the simulator sessions by observers for key node performance criteria and after the session during debriefing for scenario-specific and global success criteria. Data collection was done by operational experts and by human factors experts. If one of the success criteria was not met, the scenario trial was considered failed. 3.2 Human error Based on a phenotype-oriented approach of human errors, errors of omission and errors of commission were identified based on direct observation of the operators' be haviours and related to the predefined target path for the scenario. If an operator deliberately deviates from specifications, e.g. from the OM, then this is not evaluated as an error, but as a deviation (and thus a topic of task performance and not of human error). Beside the occurrence of errors, we observed whether the errors were detected by the operators and whether they were corrected. If an error was discovered and corrected in due time, it was not used for the evaluation of the pass/fail criteria. If errors occurred that were not detected and not corrected, the severity of these errors was classified by the validation team as high, medium, or low. The impact of each error was evaluated case by case, with focus on the probability of con sequences before the error was detected and corrected, and the consequences of the error on plant safety and integrity. No errors with high severity should have been left undetected and uncorrected by the shift team. The amount of errors with medium and low severity, which were not detected and not corrected had to be evaluated by the validation team if it was acceptable. For the evaluation of acceptability, the length of a scenario, the number of tasks and the type of operating procedure (NOP, AOP, EOP) had to be taken into account. 3.3 Situation awareness In safety research, the concept of situation awareness was developed to describe the adequate understanding of the present state and the foreseeable future as a prerequisite for any safe operation. Its assessment is either focused on an individual’s situation awareness or on the shared situation awareness between two or more agents, from which one can anticipate important future states. For the ISV, the concept of shared situation awareness was adopted, meaning that situation awareness was ensured as soon as at least one member of the crew was aware of the present state and was able to act accordingly. To assess situation awareness, a multimethod approach was adopted: Method 1 – SAGAT A comprehensive and well-established method for the assessment is the Situation Awareness Global Assessment Technique (SAGAT) [5] which allows for a real-time assessment by freezing a simulated environment and in the freeze ask agents about their understanding of the situation. After the questioning, the answers obtained in the test are compared Environment and Safety Are They Ready for Operation? How to Assess the Control Room System of a New NPP ı Rainer Miller, Rodney Leitner, Sina Gierig and Harald Kolrep
atw Vol. 65 (2020) | Issue 10 ı October with the “correct” answers, which were determined in advance. For the ISV, we have decided to ask what the relevant process aspects or process parameters are in the current situation and how these parameters will change in the near future. Furthermore, we have asked the crews about their own tasks in the near future and the goal of these tasks. Due to the possible complexity of processes, e.g. during incident situations, it was clear that using free test answers in the situation awareness probes could lead large diversity of answers which could impair an appropriate assessment of their correctness. This multitude of possible answers might prevent any meaningful evaluation. Therefore a multiple choice test for situation awareness was designed as follows. Operators were asked for process parameters or important aspects which were relevant for the decision on further procedures at the moment of the simulator freeze. In order to predefine the correct answers for the freezes, we asked three experts (2 simulator trainers and one process engineer) independently of each other, to specify in the freeze situations which process aspects and parameters are relevant for decisions om future tasks and objectives in that moment of the scenario. For some cases, the experts had different views on the relevant parameters. The final set of offered parameters in the freezes was based the highest degree of agreement between the three experts. For each freeze, six process para meters/aspects were offered in a multiplechoice test, three of which were correct and three were incorrect. The incorrect paramters/aspects (so-called “distractors”) should not be obviously irrelevant – then the test would be too easy. But they should also not be too similar to the relevant parameters either – then the test would be too difficult. The evaluation showed that we succeeded in achieving a good level of difficulty for the choices offered, but unfortunately not in all cases. Method 2 – Questions about own tasks and colleagues’ tasks During freezes, questions were asked about future tasks and their objectives (“When the current task is completed, which will be your next task and its objective”). These questions had to be answered with regard to the own tasks as well as with regard to the tasks of the colleagues in the shift crew (open-ended questions). Method 3 – Status reports and team briefings A third method to assess situation awareness was based on analysing the content and thoroughness of team briefs and status reports which are a genuine element of the team inter action in the control room. Status reports were understood here as summarized short descriptions ( mostly carried out by the SSV) of the operational status of the plant or the status of the most important parameters and operating conditions. Status reports were triggered by external requests of the management as part of the scenarios, when the safety engineer entered the control room, or at shift changes. Team briefs were requested at strategy changes, or before starting safety critical actions. During status reports, the relevant plant parameters and deviations from normal operation at that time were expected to be reported. Procedure of the measurement during freezes During the three freezes, TO, RO and SSV left their workplaces and moved to an area outside the simulator. They were then asked about (a) relevant process parameters/aspects in the present situation (multiple-choice: choose at least three most important parameters out of six parameters given and rank order them according to their importance for the present situation), (b) the expected development of these parameters (multiplechoice: quick/slow decrease or increase or no change). TO, RO and SSV answered the questions presented to them on a tablet computer with keyboard in written form without consulting their team members. Their answers were compared against the expected answers (based on experts’ agreement) concerning the relevant process parameters, tasks and objectives. (situation awareness measurement – Method 1). Afterwards the operators were asked about their current task and their next task (coordination measurement, see 3.5) as well as the current task and the next task of their colleagues, and the objectives of the tasks (situation awareness measurement – Method 2). 3.4 Communication To evaluate communication, we have identified between 13 and 31 measurement points in each scenario where communication was expected to be important. For these measuring points we defined which content of communication was expected (content of communication) and what type of communication was expected (quality of communication). As for the quality of communication, we distinguished, between 2-way communication, 3-way communication, briefing etc. For each of these communication types, observable behaviours were defined in order to assess the quality of communication, e.g. if the communication was given in “face-to-face” manner, if the receiver showed attention or if the receiver showed a reaction or expressed or showed understanding. Both, content und quality of communication were included in the observation tool (see section 4) and were checked by the observers during a scenario trial. At relevant points in the scenarios the quality of communication was evaluated not only according to observable criteria, but also using an overall assessment of the communication process. A subjective assessment by the observers was used here (rating of poor, average or good). 3.5 Coordination A satisfactory coordination was characterized by the fact that all operators knew what their colleagues were doing (task awareness). For example, the SSV should know which tasks TO and RO are performing, the TO should know which task the RO is performing, and so on. Data for coordination were col lected via questionnaires during freezes. After the questions concerning situation awareness, questions for task awareness of the operators (“Which task is performed by your colleague RO/TO/SSV?”) followed. 3.6 Workload For workload assessment we used the Bedford Workload Scale [6]. It is a unidimensional scale that ranks whether it was possible to complete the task, if workload was tolerable for the task, and if workload was satisfactory without reduction. The scale uses the concept of spare capacity to define the levels of workload. A short explanation of the scale before the beginning of the scenario allowed the operators to use it properly and to rate their workload within seconds. This allowed repeated measurements of subjective workload during the scenarios without too much intrusion into the primary task. The workload rating was announced via loudspeaker ENVIRONMENT AND SAFETY 501 Environment and Safety Are They Ready for Operation? How to Assess the Control Room System of a New NPP ı Rainer Miller, Rodney Leitner, Sina Gierig and Harald Kolrep
Page 1 and 2: nucmag.com 2020 10 ISSN · 1431-525
Page 3 and 4: atw Vol. 65 (2020) | Issue 10 ı Oc
Page 5 and 6: Kommunikation und Training für Ker
Page 35: atw Vol. 65 (2020) | Issue 10 ı Oc
Page 67 and 68: Kommunikation und Training für Ker

atw - International Journal for Nuclear Power | 10.2020

Create successful ePaper yourself

Delete template?

Save as template?