ANZCA Bulletin June 2011 - Australian and New Zealand College of ...

Assessing the
effectiveness
of training
In recent years, the Anaesthesia
and Pain Medicine Foundation has
awarded several research grants
to Associate Professor Jennifer
Weller, MBBS, MClinEd, MD, FRCA,
FANZCA, above, who is the head of
the Centre for Medical and Health
Sciences Education at the University
of Auckland. The grants are assisting
research essentially into the effects
and effectiveness of training. The
projects awarded grants are:
2009: A simulation grant of $34,812
for research entitled, “Validity of
performance in the anaesthesia
patient simulator as a measure of
performance in the operating room”.
2011: A project grant of $50,000 for
research entitled, “The effectiveness
of video-based training to improve
teamwork behaviours in acute care:
a randomised controlled trial”.
2011: A simulation grant of $22,800
for research entitled, “The impact of
assessment on life and learning”.
Here, Associate Professor Weller
reports on her research.
The project, “Validity of performance
in the anaesthesia patient simulator
as a measure of performance in the
operating room,” compares individual
anaesthetists’ teamwork behaviours in
clinical settings with their behaviours
in routine and crisis simulations.
We want to answer two questions: do
anaesthetists behave in the same way in
the simulated environment as they do in
the clinical setting; and does behaviour
in routine cases predict behaviour in a
clinical crisis This has implications for
using simulated environments to learn
teamwork skills and, furthermore, may
suggest the need to assess response to
crises in a simulator.
Our surgical colleagues have produced
a considerable amount of evidence
to support the validity of surgical
procedural simulators but with the
more complex integrated anaesthesia
simulators, this evidence is harder to
produce, although there is some support
for the validity of anaesthesia simulations
based on activity patterns. 1
In this study, we have videotaped 17
anaesthetists and anaesthetic assistants
in three settings: (1) an operating room
during a routine list; (2) a simulation
modelled on typical cases; and (3) a
simulated case with an intraoperative
crisis.
We have developed and validated a
coding framework to analyse interactions
between anaesthetists and the operating
room team and have now completed all
the coding – the analysis of our very
large data set has just begun and we look
forward to seeing what patterns emerge.
From participants’ perspective,
they considered their behaviours in the
simulator as realistic or very realistic, and
scored specific teamwork behaviours as
occurring at a similar frequency in the
simulations and the operating room.
We are now planning to rate all the
simulations again, this time using a
teamwork behavioural rater our group
developed and validated in a previous
ANZCA-funded grant. 2 We have managed
to find a willing group of assessors and
have the assessor training day arranged.
This will be no small task for them but
they should be able to tell us, among
other things, if behaviour in routine cases
predicts behaviour in a clinical crisis.
Previous work from our group has
focused on the development of robust
instruments to measure teamwork
behaviours, 3-5 in order, first, to see if we
can assess performance, and secondly to
see if interventions to improve teamwork
are effective. We have shown improved
teamwork in intensive care unit (ICU)
teams following simulation training, and
even identified improvements in specific
components of teamwork. 6
Following on from this work, our next
project is to assess “The effectiveness
of video-based training to improve
teamwork behaviours in acute care:
a randomised controlled trial”. Here
we want to drill down to very specific
behaviours associated with improved
teamwork, and see if we can teach these
using videos role-modelling desirable
models of communication.
54
ANZCA Bulletin June 2011