The psychopathology of everyday art: a quantitative Study - World ...

2. Results are uninterpretable between studies using poor designs and
those with good.
Rather than making statistical compensation for poor studies, only studies which used
reliability measures for their terms were used. Design otherwise was equally poor.
3. Published research is biased in favour of significant findings
because nonsignificant findings are rarely published: the 'File
Drawer Problem'. 220
Checks were made on unpublished controlled studies through private correspondence;
there is every reason to believe that unknown unpublished studies of other types than
the two which are used in this review would show similar conflicting viewpoints, poor
design and statistical rigour as those published which are subject to peer review. There
does not seem to be a lack of published nonsignificant findings, so there may be little bias
in this field. A test was applied following procedures from Wolf (1986) 221 which
estimated how many additional studies with nonsignificant results would be necessary
to reverse the conclusion drawn, providing some estimate of the robustness and validity
of the findings.
4. Multiple results used from the same study bias or invalidate the
meta analysis and make the results appear more reliable than they
really are because they are not independent.
220
Not everyone is agreed on whether this point exists: a recent discussion of the various issues as to how
to estimate the proportion of conflicting evidence, in M.T. Bradley, R.D. Gupta (1997), Estimating the
Effect of the File Drawer Problem in Meta Analysis, Perceptual and Motor Skills , V.65(2_:719-22. I
followed Rosenthal's recommended checks (1984) op.cit. p.107-110.
221
Wolf (1986) Meta Analysis, op.cit. The 'fail-safe N' p.37-39, which was simpler than the calculation
from Rosenthal.
171