The psychopathology of everyday art: a quantitative Study - World ...

Two weaknesses lie in the simple calculation of agreement; there is no account of
where in the table the agreement was and secondly we would expect some agreement
between raters by chance. A more reasonable answer is obtained by considering
agreement in excess of the amount by chance, which is only attempted by one study (39),
and lacks other figures. The best approach to this type of problem is that adopted by
studies 44 and 36, the kappa statistic, which may be interpreted as the chance corrected
proportional agreement, but it is important to show the raw data (which they don't). In
support of this statement, Neale's application of the DDS to children 208 , found a much
lower level of reliability than that reported by Mills 209 : only 12 variables reached
significance using the Kappa measure of agreement between 2 raters.
Conclusion
Generally, specific decisions are more reliable than global.
Content decisions were most reliable where objective, because they were more specific
and probably easier to define and rate. Most categories contained subjective decisions
and largely where these were global they were unreliable. This is particularly exemplified
in the category of Quality, where personal judgement was not reliable because experience
of art was clearly an influence on aesthetic appeal. Global subjective decisions became
more reliable when not referenced to the image, but suspicions must arise that the rating
208
E.L. Neale (1994) The Children's DDS, Art Therapy , V.11(2):119-26, but not included in this review
because the subjects were children.
209 Mills et al. (1993) op.cit.
162