Association for Education and Rehabilitation of the ... - AER Online

Assessment Reliability and Fidelity
Table 3. Percentages, Point-by-Point Agreement: Procedural Fidelity
Assessment
Observer 1 Observer 2 Observer 3Observer 4
(Percent) (Percent) (Percent) (Percent)
1 89 91
2 100 98
3 100 98
4 100 100 96
5 100 86 100
6 35 39
7 98 98 100
8 100 90 100
9 68 64
10 98 98
11 89 89
12 100 99
13 95 90
14 65 72
15 100 98
16 98 98
17 98 95
18 89 89
Mean 90 93 91 88
Table 4. Percentages, Point-by-Point Agreement:
Reliability
Assessment
Observer 1
(Percent)
Observer 2
(Percent)
1 82
2 96
3 94
4 87 83
5 86 82
6 72
7 90 90
8 90 89
9 89
10 95
11 86
12 82
13 82
14 77
15 83
16 77
17 95
18 72
Mean 85 86
78 | AER Journal: Research and Practice in Visual Impairment and Blindness
the low-scoring educator assessment, a turn-taking
routine was not created and so could not be returned to
nor could a mismatch be added.
It did not always follow that low fidelity led to low
reliability. The assessment with the lowest reliability
score (72 percent) had a relatively high fidelity score
(89 percent), and an assessment that had a fidelity
score of 68 percent had a reliability score of 89
percent. However, all of the fidelity scores that were
90 percent or above had reliability scores of 82
percent or above. Therefore, it can be concluded that
high fidelity to quality indicators tends to lead to high
interrater reliability. This may reflect that some
assessors had a more thorough understanding of
the assessment tool both in terms of how to facilitate
best child performance and how to extract accurate
observations and conclusions.
Limitations
There are some limitations associated with the
study that could affect the extent to which results of
the study can be generalized to other settings or
individuals. The relatively small number of participants
in the study is a limiting factor; it should be
taken into account that all of the study participants