a primer on verbal protocol analysis - Ammon Wiemers' Home Page

C35165/Schmorrow Page 339
A Primer on Verbal Protocol Analysis 339
of this segmentation scheme from the meteorology domain. As the illustration
shows, the length of each utterance can vary from a single word(s) (for example,
“850”) to fairly lengthy (for example, “clouds are going to be increasing for the
next 24 hours in western Ohio”). In this domain, we are generally interested in
the types of mental operation participants perform on the visualizations, such as
reading off information, transforming information (spatially or otherwise), making
comparisons, and the like, and these map well to our segmentation scheme.
The utterances “it’s at 700 millibars” and “clouds are going to be increasing for
the next 24 hours in western Ohio” would each be coded as one “read-off information”
event, even though they differ quite a bit in length. Of course, it would
be possible to subdivide longer utterances further (for example, “clouds are going
to be increasing // for the next 24 hours // in western Ohio”); however, according
to our coding scheme this is still one read-off information event, which would
now span three utterances. Having a single event span more than one utterance
makes data analysis harder. The two most important guidelines in dividing protocols
are (1) consistency and (2) making sure that the segments map readily to the
codes to be applied.
Although transcription cannot be automated, a number of video analysis software
programs are available to aid in data analysis, and some of these programs
allow protocols to be transcribed directly into the software. We have not found
the perfect protocol analysis program yet, though we have used MacShapa,
Transana, and Noldus Observer. Looking ahead and deciding what kind of analysis
software will be used will also save a great deal of time by reducing the need
for further processing of transcripts in order to successfully import them to the
video analysis software. Another option that we have successfully used is to transcribe
protocols in a spreadsheet program, such as Excel, with each segment on a
different row. We then set up columns for our different coding categories and can
easily perform frequency counts, create pivot tables, and import the data into a
statistical analysis program. This method works very well if the video portion
of the protocol is irrelevant; however, in most cases we are interested not only
in what people are saying, but what they are doing (and looking at) while they
are saying it. In this case, using the spreadsheet method is disadvantageous,
because the transcription cannot easily be aligned with the video. Video analysis
software has the advantage that video timestamps are automatically entered at
each line of transcription, thus facilitating synchronizing text and video.
Once data have been transcribed and segmented, they are ready for coding.
Unless the research question can be answered by some kind of linguistic coding
(for example, counting the number of times a certain word or family of words is
used), a coding scheme must be developed that maps to the cognitive processes
of interest. Establishing and implementing an effective and reliable coding
scheme lies at the heart of verbal protocol analysis, and this is usually the most
difficult and time consuming part of the whole process. In some cases, researchers
will have strong a priori notions about what to look for. However, verbal protocol
analysis is a method that is particularly useful in exploratory research, and
in these cases, researchers may approach the protocol with only a general idea.