aceUVi

The quality metrics used in Western Australia and Manchester assess both the
quality of the experience reported by audience members (eg, ‘inquisitiveness’,
‘captivation’, ‘connection’) and the quality of the product (eg, ‘excellence’ and
‘rigour’). The Manchester pilot found that there was ‘more variation in the public
[audience] scores for the more personal, subjective measures of “relevance”,
“challenge” and “meaning” than for the more technical dimensions of “presentation”
and “rigour”’ (Bunting and Knell 2014, 56). Such systematic discrepancies
suggest that the audience’s rating of experienced impacts on the one hand and
product quality on the other may need to be interpreted differently.
Boerner and Jobst (2013) and Brown and Novak-Leonard (2007) report contradictory
findings regarding the relative significance of impact and product quality
measures, so that the relationship between these two remains an important topic
for further research. The public’s less favourable scoring on the impact measures
(ie, ‘quality of experience’ measures) that were used in Manchester leads Bunting
and Knell to conclude, ‘It’s one thing to offer a polished, absorbing, highly
enjoyable cultural experience; it’s another to make a difference to how people
think and feel about the world’ (56).
The measures of experienced impact that were developed in Western Australia
and Manchester focus on the audience’s intellectual engagement with the performance.
Bunting and Knell note that some respondents in Manchester, in particular
the peer evaluators, wished there was a place to report their emotional
reactions. Rather than adopting additional quantitative impact questions to
capture emotional or social dimensions of the experience that have been included
in other frameworks, Bunting and Knell recommend adding some open-ended
qualitative questions to gather feedback on the emotional experience (Bunting
and Knell 2014, 62).
One of the major challenges of using assessment scores supplied by audience
members at the policy level is ensuring comparability across sites and across
various forms of cultural experience. As noted above, some researchers have
expressed concerns about comparing self-reported audience experiences across
different art forms and contexts (Belfiore and Bennett 2007, 258-60; Brown and
Novak-Leonard 2013, 7). However, one of the core objectives of both the DCA
and Manchester initiatives is enabling ‘comparisons between funding programs
and activities from different art forms’ (Chappell and Knell 2012, 16), and the
integration of peer-review and self-assessment in the evaluation frameworks are
intended as a buffer against idiosyncrasies of the audiences, the objectives of the
organisations, and other contextual factors.
CODA: COMBINING MEASURES OF EXPERIENCE AND QUALITY AT THE POLICY LEVEL 126
UNDERSTANDING the value and impacts of cultural experiences