Evaluation - Scottish Screen

3.2 Design
The literature reviews by Buckingham (2004) and Livingstone et al. (2004) suggest that
longitudinal studies are required for proper evaluation of the impact of media literacy
initiatives and this was taken into account in developing the methodology outlined below.
Furthermore, both these reviews of the research literature suggest that rigorous research
needs to move on from a simple account of how media literacy is being taught in schools
towards a more thorough analysis of the impact of media education on students’ viewing and
reading of media in their daily lives and again account was taken of this suggestion. As
Livingstone et al. point out:
Researching media literacy faces some serious challenges.
Media literacy is concerned with people’s generally implicit,
yet complex and subtle understanding of the media, and
these are difficult to ask about directly. It may be concerned
with things they cannot do or have not seen the importance of…
(Livingstone et al., 2004, p56)
Consequently, both Buckingham and Livingstone et al. question the efficacy of
questionnaires and multiple-choice instruments in gathering the qualitative data necessary to
meet the challenges just described. The preferred method indicated by both reviews is the
use of focus groups. Within local authorities, however, the logistics of timetabled teaching
can often present difficulty for teachers to gather together in one place in order to form focus
groups. The evaluation team, therefore, were cognisant of the challenges presented by
Buckingham and Livingstone et al. when constructing observation and interview schedules
for teachers. Interview schedules were also constructed to ascertain lead practitioners’
perspectives on the initiatives. Focus groups remained the preferred means of involving
pupils and their parents in the evaluation. Observation of MIE classes, what lead
practitioners, teachers and pupils actually did, was considered an appropriate method of
triangulating what both groups said in interviews or focus groups.
Moreover, in the early stages of any initiative, the specific detail required in an effective
questionnaire may not yet be discernable to respondents. Therefore, to request data
concerning the long-term incorporation of the knowledge gained in the programme into the
classroom and curriculum could be asking teachers to reflect on something that has not yet
happened or has only happened to a limited extent. It seemed more appropriate at this stage
in the development of the initiative to ask teachers and their pupils to respond in three ways
to their experiences so far. First, they were asked to reflect on their engagement with MIE
both early in the project and at a substantially later stage; secondly to consider how this had
impacted on their enjoyment and reading of media texts at both stages; and thirdly to project
how what they learned through engagement with MIE could be used in the future in relation
to media and media texts and also in relation to the wider environment. In designing our
research in this way, our instruments paralleled the themes of the initiatives, namely learning,
teaching and sustainable change.
This methodology had three advantages. First, it responded directly to the immediate
priorities of Scottish Screen (including positioning moving image education within the four
capacities of A Curriculum for Excellence) and to those highlighted in the reviews of research
literature. Second, it built on the basis for a longitudinal study created by evaluation of the
Teacher Education Programme in session 2005 – 2006. The overriding concerns for the
12