What do students know and understand about the Holocaust?

24
Methodology
Of particular note were two large pilot studies
involving 900 students from nine schools.
Pilot study 1
In the first of these studies, 555 students (282
girls, 273 boys) from seven schools took part. The
students were from Years 7, 8, 9, 10 and 12. All
participants completed a paper version of the survey.
Predominantly, the survey comprised questions
we developed to explore students’ substantive
knowledge of the Holocaust, their recognition of
associated words and images, and the sources of
information and ideas about the Holocaust that they
were likely to draw upon. Additionally, the pilot survey
included attitudinal scales to tap into:
■■
students’ personal interest in other people at
local, national and international levels (to explore
individual agency and responsibility)
■■
students’ orientations towards school
■■
students’ attitudes towards learning about
the Holocaust.
The scale to measure students’ attitudes towards
learning about the Holocaust was created by the
research team. The other attitudinal scales were
developed using modified versions of questions
from the IEA Civic Knowledge and Engagement
Study (Amadeo et al. 2002).
For the knowledge-based questions, students’
responses were examined to determine the
extent to which they were able to provide ‘correct’
or ‘incorrect’ answers as judged against current
historiography. ‘Incorrect’ answers were examined
to give insight into common misconceptions about
the Holocaust. Total scores were calculated for each
student, to give an indication of levels of knowledge
across all of the questions in the survey. Mean
scores were also calculated to explore differences
in knowledge between year groups.
The attitudinal scales were assessed for reliability
using Principal Components Analysis (PCA) and by
calculating the Cronbach’s alpha. For each scale,
PCA was used to explore how many underlying
dimensions of the construct were being measured.
Cronbach’s alpha (α) was used to assess the internal
reliability of each scale, giving an indication of how
consistent each student was when responding to
the questions. We sought to include scales where
the Cronbach’s alpha was above 0.7, indicating an
acceptable level of reliability (Field 2013).
These analyses revealed some problems with the
individual agency and responsibility scale – namely
that students were answering the questions in an
inconsistent manner and the scale lacked internal
reliability (that is, the Cronbach’s alpha was below
0.7). Consequently, this scale was removed from
the survey and replaced with three scales that have
previously been used in other research. The Beliefs
in a Just World (Others) Scale was introduced to
the survey because it assesses the extent that
respondents think the world is a just place and so
gives insight into attitudes towards fairness. We
used the version that has been adapted for use
with adolescents and found to have good internal
reliability (Fox et al. 2010).
To tap into students’ attitudes towards perceived
‘out-groups’, we included scales to measure cultural
xenophobia and blatant racism (Elchardus and
Spruyt 2014). Elchardus and Spruyt reported that
both scales showed a high level of internal reliability
(both had Cronbach’s alphas above 0.8), however
they were originally used with university students.
In view of this, the questions in both scales were
modified and our second pilot study was used as an
opportunity to examine if this made them accessible
to secondary school students.
Pilot study 2
The scales to examine orientations towards
school and attitudes towards learning about the
Holocaust were retained in the second survey, but
some of the questions were refined to improve clarity.
The knowledge-content-based questions were
also kept in the survey, with just a small number
of amendments to improve the wording.
The first pilot study also highlighted that, where
possible, an online version of the survey was
preferable to a paper version as an online version
meant that a ‘question-skip facility’ could be used.
This enabled us to modify the questions that students
were presented with based on their earlier answers.
For example, if students indicated reading books
about the Holocaust, they were presented with a
list of books so they could identify which ones they
had read; if they had not read any books about
the Holocaust they were moved on to a different
question. This helped to reduce confusion for the
students as they completed the survey, because
they were not asked questions that were irrelevant
to them. Using an online survey also meant that data
could be uploaded from the survey platform (Survey
Monkey) into IBM SPSS for analysis. This reduced
errors that can occur through manual data entry.
In view of all of this, the aim of the second pilot
study was to review the data-collection process
when using the online version of the survey, as well
as evaluating the modifications made. For this pilot
study, 345 students (171 girls and 174 boys) from
two schools took part. The students were in years 7
to 12.
As for the first pilot study, descriptive statistics
were calculated for each scale and reliability