The psychopathology of everyday art: a quantitative Study - World ...

How well does the method agree?
Table 16 shows a summary of the standard differences (S.diff) between the 6 trainee
raters and the author's score (column 1), plotted in Figs 1-15 275 (Appendix 2). The
scatter of differences remains constant and does not increase with the mean, so the
differences can be further investigated in the raw state 276 . A test of association would
be inappropriate at this stage as we cannot deduce that methods agree well if they are not
statistically different. There were large variations in the sample which may have
regressed the mean and may well lead to important differences seeming non-significant
(see tables 1-15 accompanying figs. 1-15, appendix 2).
The S.diff. can be used as a measure of agreement by itself, but it is more useful
to construct a range of values from it, which covers the agreement between the raters for
most cases.
For reasonably symmetric distributions we can expect the range mean +-2 S.D.
to include 95% of the observations. We can therefore take mean +-2S.Diff. as a 95%
range of agreement for individual paintings. This range defines the 95% limits of
agreement (shown in Table 16, column 2). For the present data, from the 95% ranges,
the author is expected to score within 1 point of scores from independent raters on drawn
line, orange and white; within 2 on yellow, black, space, and form, with a slight bias
towards higher scores; and within 3 points on a further 5 variables: red, green, blue,
intensity, and painted line, with discrepancies likely in either direction. 3 variables have
275 against Author+RATS/2.
276 Rather than translation to logarithmic scores.
222