Take Two - Third Evaluation Report More than Words ... - Berry Street

differences between their HCA and that of the referrer.
There were 51 (22%) situations where the Take Two
clinician’s HCA differed from the referrers in terms of
the child’s past experience of abuse or neglect. Some
(n=6; 3%) of the differences were due to the child
suffering new experiences of abuse and neglect since
the previous HCA and some were due to new information
becoming available (n=7; 3%). However, according to
the Take Two clinicians, the majority of differences were
either due to Take Two undertaking a more detailed
assessment (n=12; 5%) or a difference of opinion or
interpretation of the information (n=26; 11%).
This is not to say that the Take Two clinician’s opinion
was more valid or accurate, but that any data collection
process based on opinion is subject to variability.
Strategies to increase validity included initial training
throughout the state regarding the referral document
and process, such as the need to review the client’s le
and to not just focus on the most recent experiences
of abuse and neglect. However, it is understood that as
with any data collection process, issues of time, recency
of training, focus on recent or most severe incidents
and other factors can in uence accuracy. Since this
exercise, adaptations have occurred within the HCA
including help text available within the computerised
version and further statewide training.
Despite limitations to the accuracy of the HCA, it is still
considered considerably more informative than other
available data sources. For example, data collected via
the DHS–CRIS system primarily report on abuse and
neglect that have been substantiated at the beginning
of each episode of child protection involvement, and
not on experiences that were substantiated within
an episode. The CRIS data regarding substantiated
abuse are available to Take Two through the CYPP
document and are considerably less detailed and less
comprehensive about the children’s experiences.
2.10.2 Measuring consequences of abuse and
neglect, such as behavioural, emotional and
developmental dif culties
Similar to measuring the occurrences of abuse and
neglect, measuring its consequences is multifaceted
and complex. In fact it is probably more so, as it is
a supposition in the rst place that the children’s
emotional, developmental, behavioural and other
dif culties are a consequence of these abusive
experiences. The limitations regarding the HCA also
apply to this question of consequences. In comparison
between the referrers’ and Take Two clinicians’ HCAs
undertaken in 2005, a similar level of difference was
found regarding consequences. Of the 66 (28%) HCAs
completed by Take Two clinicians that differed from
the referrers’, 20 (9%) were due to either an increase
or a decrease in dif culties since the time of referral.
The majority (n=34; 15%) were due to a difference
in assessment, which is to be expected given that a
common request to Take Two was to undertake a more
detailed assessment of the children’s dif culties. Eleven
HCAs (5%) showed a difference of opinion, according to
the Take Two clinicians.
Measuring children’s emotional, social and behavioural
dif culties as well as trauma symptoms is obviously a
major aspect of Take Two’s assessment role including
the use of standardised and other measures, such as
the SDQ, TSCC, TSCYC and the Social Network Map.
Each of these measures has speci c limitations, but
12
a general limitation is the reliance on questionnaire
approaches to ascertain emotional dif culties. There is
exciting research regarding the use of neurobiological
measures to ascertain levels of physiological arousal
and dysregulation, such as heart rate and cortisol
levels (e.g. Dozier, Peloso, Lindhiem, Gordon, Manni,
& Sepulveda et al., 2006; Fox & Card, 1999). However,
the complexities of including these measures in a
non-laboratory clinical program in addition to the
increased resources and training required would need
to be carefully weighed before deciding whether to
incorporate them in the Take Two research strategy.
A major strategy to minimise the limitations of the
questionnaire approach is to have multiple measures so
that the data can be triangulated.
2.10.3 Describing and measuring Take Two’s
work
Therapeutic interventions have proven a dif cult area
to describe and measure given the non-manualised
nature of Take Two’s approach. Two main approaches
have been used, namely a contemporaneous measure
where clinicians enter regular activity logs and the
retrospective measure of the clinicians’ survey that
asks questions regarding the nature of the interventions
used over the last year. The major limitation with the
activity logs is associated with common problems
associated with variable compliance with administrative
tasks. The major limitation with the clinicians’ surveys is
those associated with retrospective measures including
discrepancies with memory and interpretation.
2.10.4 Measuring outcomes and satisfaction with
the service
As noted earlier, a reliance on use of questionnaires has
limitations, especially when being used for measuring
outcomes. In addition, there are challenges to increase
rates by clinicians to use the measures and to increase
the likelihood of children, parents, carers and teachers
in completing the measures. McNamara and Neve
(2009) wrote of how to support clinicians to participate
in the research aspects of their clinical work and cite
Take Two as an example of how to achieve this goal.
Nevertheless, this remains an ongoing challenge and
commitment for Take Two.
Stakeholder surveys have been another important
means of gathering perspectives on whether positive
changes have occurred and on the level of satisfaction
with the program. All such surveys have limitations, such
as needing to restrict the number of questions in order
to be user friendly. This restricts the analysis to being
relatively super cial as there is only one question per
topic of interest. A design aw in the clinicians’ survey
meant that the date of completion was not recorded
and this led to some limitations in the analysis.
2.11 Summary
The design and methodology of evaluating the Take
Two program has developed alongside the program
itself. This chapter has put the evaluation methodology
within the context of the frameworks that informed its
foundation, including the evaluation framework and the
clinical outcomes framework. The chapter described the
range of sources for data collection and the multi-level
analyses of the work of Take Two with the individual
children and the program as a whole.
Frederico, Jackson, & Black (2010) “More than Words” – Take Two Third Evaluation Report, La Trobe University, Bundoora, Australia