THE PLANTAR REFLEX - RePub

indicate consistent ratings. It is apparent that many observers fell short of
this ideal line. Cohen (1960, 1968) developed a statistical method to
measure observer agreement - a method which takes into account the
agreement to be expected by chance, and which also weighs the extent of
possible disagreements. If we apply this measurement to the intraobserver
variation between the first and the second presentation, displayed
in table V, we find that Cohen's factor 'weighted kappa' (ranging from 0
for chance agreement and l for full agreement) is 0.09 for film A and 0.28
for film B. These very low kappa values indicate that the usual clinical
interpretation of plantar reflexes can be meaningless in some cases.
The variation between different observers for the two films is indicated
in table V by the totals in the margin. Because individual inconsistencies
compensate each other to a large extent, the difference between ratings on
the first and the second presentation is much less marked for the group as
a whole. But the range of opinions is considerable, and the frequency
distributions for both films have some characteristics in common:
1. All ratings were distributed among four of the five choices offered: one
extreme of the scale was not opted for (unmistakably upward for film A,
unmistakably downward for film B).
2. About one half of the observers agreed about one particular choice
(possibly downward for film A, possibly upward for film B), while the
other half were divided among the three other categories.
3. The category 'neither upward nor downward' is underrepresented in
comparison with a symmetrical distribution.
This variation between observers is superimposed upon the intraobserver
disagreement which has already been proved to be extreme for
these films. The combined effect can only make the reproducibility of
interpretation even lower.
Effects of preceding information
The results for the 'information' group are shown in table VI. Like the
'no information' group, only a minority of the observers rated consistently
(within the diagonals). An important difference is, however, that here the
inconsistent ratings are not scattered symmetrically on both sides of the
diagonal, but are skewed towards one side, according to the combined
direction of bias. To investigate this further, all relevant differences
between the various mean ratings - shown in figure 3 - were subjected to
the t-test. For both films, I up resulted in significantly more pathological
markings than luown (tA = 3.10, degrees of freedom (d.f.) = 19, p