Education for a Digital World Advice, Guidelines and Effective Practice from Around Globe, 2008a

24 – Evaluating and Improving Your Online Teaching Effectiveness
My Sentence
by Student B—Friday, 2 September, 12:35 PM
In order to adequately address teaching effectiveness an instructor needs to use an effective tool to measure specific activities or deficiencies in student performance
by using techniques including but not limited to: surveys, analysis of performance, and questionnaires.
Re: My Sentence
by Instructor—Sunday, 4 September, 08:31 PM
This is a good start. WHEN does it happen? Keep in mind that the process does not end with using a data collection tool. There is analysis of the
process before the course begins, and after collecting the data. Also, WHERE does it happen? Is this online, in the classroom, or both?
-----
In order to adequately address teaching effectiveness [7 WHY] an instructor [1 WHO] needs to use an effective tool to measure specific activities
or deficiencies [2 DOES WHAT] in student performance [3 TO WHOM] by using techniques including but not limited to: surveys, analysis of performance,
and questionnaires [6 HOW].
Figure 24.4 Example one-sentence summary student submission with instructor’s reply
Evaluate online quiz or test results
If you use a learning management system (LMS), an
online workbook environment that comes with publisher
materials, or other online space that allows online
quizzes or tests, then you can use the results to identify
problem areas. LMS solutions like Moodle and ANGEL
provide tools to perform an item-by-item analysis to
evaluate several factors related to individual questions.
These factors include item facility, an indicator of the
question difficulty for students; standard deviation of
student responses on each question; and item discrimination,
an indicator of the difference between performance
by high-scoring students and low-scoring students.
Even if you can only get simple statistics, such as how
the class answered each question overall (e.g., 10 percent
picked “A,” 25 percent picked “B,” 65 percent picked
“C”), you can use this information to make adjustments.
One way to do this is to ask your students to take a pretest
or baseline quiz at the beginning of the course, and
then compare those results to the actual quiz results. In
face-to-face or hybrid course situations, you can use the
quiz results to address issues through quiz reviews or
changes in your lecture. Dr. Karen Grove from San
Francisco State University discusses how to use quiz
results to address learning gaps via a student preparation
module in the Orientation to College Teaching (http://
oct.sfsu.edu/implementation/studentprep/index.html).
If large numbers of the students get a question wrong,
the instructor can cover that topic more fully. The instructor
can also dispel misconceptions after seeing how
many students choose a particular incorrect answer.
Figure 19.5 is a sample of item analysis in the Moodle
LMS (version 1.5.4). Figure 19.6 is an example of much
simpler item statistics in the Blackboard LMS (version
6.2). The instructor can see how many people select each
answer. For essay questions, it provides a complete list
of all the essays.
Responding to formative
feedback with immediate,
minor changes
Collecting the student feedback is just the beginning.
However, you do not have to, and should not, wait until
the end of the term to start introducing changes as a
result of what you learn from the students. For instance,
students may use the different instruments to ask you to
be more flexible in the teaching approach, to maintain a
good attitude towards students and their ideas, to use
more appropriate assessment methods, or to add more
real-world application to the content.
To continue engaging students in the process, go over
the formative feedback results with them and solicit
suggestions for changes. I tell my students that, to me,
“constructive criticism” means that they must help construct
solutions to problems or issues that they find.
Together, the students and I look at survey results or
rubric scores, and any comments about a specific
teaching effectiveness criterion. If no comments address
372 Education for a Digital World