Asking Questions - The Definitive Guide To Questionnaire Design ...

ASKING QUESTIONS THAT EVALUATE PERFORMANCE 233
of response required. In discussions related to developing rating
forms, it is easy to confuse the three and have seemingly conflicting
opinions about issues that are essentially distinct. In developing
such forms it is best to discuss these three dimensions separately.
1. Content. When deciding on the item’s content, it is important
to determine which elements of the course, instruction, and
learning areas need to be addressed. Questions constructed for the
course area should assess how well material was covered by instructors
and understood by students. Questions constructed for the
instruction area should assess instructor characteristics such as fairness,
clearness, willingness to interact, clarity, and so on. Finally,
those questions constructed for the learning area should address
issues such as a student’s satisfaction, perceived competency, and
desire to continue study in the field.
2. Inference. If student ratings will be used to produce measures
that require considerable inference beyond what is observed in the
classroom (such as “partial” or “fair,” “autocratic” or “democratic,”
“dull” or “stimulating”), then higher-inference measures are needed
(Rosenshine, 1970). Students should apply these measures when
they assess the instructor or the instruction.
If the purpose of the questions is to classify teaching behaviors
according to relatively objective categories, then low-inference
measures are needed. These measures are obtained as frequency
ratings of the instructor on such scales as “gesturing,” “variation in
voice,” “asking questions,” or “praise and encouragement.”
Ratings on high-inference items are particularly useful in exploring
new ideas, and they have generally yielded higher correlations
with overall instructor effectiveness than have the more
specific, or low-inference, behavioral measures. Yet because the information
in low-inference measures is easier to project to specific
behaviors, it is easier to use in instructional improvement programs.
3. Format. The questionnaire’s format can influence how carefully
students respond to the items and how accurately they will respond
with true feelings. Using open-ended (free-response) questions,