RESEARCH METHOD COHEN ok

RELIABILITY IN QUANTITATIVE RESEARCH 147
themselves between the test and the retest
times.
Reliability as equivalence
Within this type of reliability there are two main
sorts. Reliability may be achieved first through
using equivalent forms (also known as alternative
forms) of a test or data-gathering instrument.
If an equivalent form of the test or instrument
is devised and yields similar results, then the
instrument can be said to demonstrate this form of
reliability. For example, the pretest and post-test
in an experiment are predicated on this type of
reliability, being alternate forms of instrument to
measure the same issues. This type of reliability
might also be demonstrated if the equivalent forms
of a test or other instrument yield consistent results
if applied simultaneously to matched samples (e.g.
acontrolandexperimentalgrouportworandom
stratified samples in a survey). Here reliability
can be measured through a t-test, through the
demonstration of a high correlation coefficient
and through the demonstration of similar means
and standard deviations between two groups.
Second, reliability as equivalence may be
achieved through inter-rater reliability. If more
than one researcher is taking part in a piece of
research then, human judgement being fallible,
agreement between all researchers must be
achieved, through ensuring that each researcher
enters data in the same way. This would be
particularly pertinent to a team of researchers
gathering structured observational or semistructured
interview data where each member of
the team would have to agree on which data would
be entered in which categories. For observational
data, reliability is addressed in the training sessions
for researchers where they work on video material
to ensure parity in how they enter the data.
At a simple level one can calculate the interrater
agreement as a percentage:
Number of actual agreements
Number of possible agreements × 100
Robson (2002: 341) sets out a more sophisticated
way of measuring inter-rater reliability in coded
observational data, and his method can be used
with other types of data.
Reliability as internal consistency
Whereas the test/retest method and the equivalent
forms method of demonstrating reliability require
the tests or instruments to be done twice,
demonstrating internal consistency demands that
the instrument or tests be run once only through
the split-half method.
Let us imagine that a test is to be administered to
agroupofstudents.Herethetestitemsaredivided
into two halves, ensuring that each half is matched
in terms of item difficulty and content. Each half
is marked separately. If the test is to demonstrate
split-half reliability, then the marks obtained on
each half should be correlated highly with the
other. Any student’s marks on the one half should
match his or her marks on the other half. This can
be calculated using the Spearman-Brown formula:
Reliability =
2r
1 + r
where r = the actual correlation between the
halves of the instrument (see http://www.
routledge.com/textbooks/9780415368780 –
Chapter 6, file 6.6. ppt).
This calculation requires a correlation coefficient
to be calculated, e.g. a Spearman rank order
correlation or a Pearson product moment correlation.
Let us say that using the Spearman-Brown
formula, the correlation coefficient is 0.85; in this
case the formula for reliability is set out thus:
Reliability = 2 × 0.85
1 + 0.85 = 1.70
1.85 = 0.919
Given that the maximum value of the coefficient
is 1.00 we can see that the reliability of this
instrument, calculated for the split-half form of
reliability, is very high indeed.
This type of reliability assumes that the test
administered can be split into two matched halves;
many tests have a gradient of difficulty or different
items of content in each half. If this is the case and,
for example, the test contains twenty items, then
the researcher, instead of splitting the test into two
Chapter 6