The psychopathology of everyday art: a quantitative Study - World ...

studies did not even quote figures, 4 because they were too poor (50, 12, 10, 14). The
other 3 reported categories of 'very good' to 'acceptable' agreement (9, 11, 39; study 39
reported that 3 raters own criteria judgements of patient status, from 200 paintings was
10% better than chance but gives no other figures, although we know (study 14) that own
criteria judgements are based on widely differing individual values. Significant agreement
is reported, but with no indication of how this was arrived at). No study justified their
cut off points for 'good agreement' and these varied considerably from study to study.
Worse, perhaps than no figures, a good percentage of these reliability studies are
mis-analysed 202 . In particular, the correlation between the values reported by individual
raters or groups of raters is calculated in 9 studies (75, 35, 37, 38, 31, 10, 44, 21, 3), with
a high value of r interpreted as an indication of good agreement. Correlation is an
inappropriate analysis, firstly because the correlation coefficient is a measure of the
strength of linear association between two variables, not agreement. Agreement is
assessed directly. Secondly, there may be a high degree of correlation when the agreement
is clinically poor, as recognised by the actions of the authors of study 37, who used 7
terms with correlations below 0.31 because agreement was clinically high (but didn't drop
any clinically low ones). A high value of r can be obtained because, as for studies 3, 37,
38 and 31, there is large variation between subjects. The authors of study 37/38
recognised large differences between subjects in 37. They used a much more rigidly
defined group of subjects for study 38 and a higher cut off point for the correlation, to
202
Much of the information here is quoted from D.G. Altman (1994) Practical Statistics for Medical
Research , (4th reprint, original 1991), London: Chapman Hall.
159