A COMPENDIUM OF SCALES for use in the SCHOLARSHIP OF TEACHING AND LEARNING

Summative Assessments
Summative Rubrics
Although APA lists several standardized assessments designed to measure various aspects of
communication (e.g., Collegiate Assessment of Academic Proficiency Writing Essay Test and
Writing Skills Test, WorkKeys Foundational Skills Assessment, and Collegiate Level Assessment),
and the Association of American Colleges and Universities (AACU) provides a Written
Communication rubric (https://www.aacu.org/value/rubrics/written-communication), none
focus on communication in psychology, or more globally, on scientific writing. Although
writing, specifically scientific writing, is typical in natural and social science curricula, few
standardized rubrics to evaluate such work have been developed, tested, and made accessible.
In this section, we identify empirically-tested rubrics for summative use in scientific writing that
can be easily accessed and implemented or adapted for classroom use in psychology.
Stellmack, Konheim-Kalkstein, Manor, Massay, and Schmitz (2009) created and evaluated the
reliability and validity of a rubric used to score APA-style research papers. The researchers
focused on only the Introduction section of the rubric; however, updated and expanded rubrics
for all sections (e.g., Introduction, Method, and Results/Discussion) are available at
http://www.psych.umn.edu/psylabs/acoustic/rubrics.htm. Students were instructed to write
an APA-style Introduction incorporating five sources. The rubric encompasses eight dimensions
that are fundamental aspects of this type of writing assignment: APA formatting, literature
review, purpose of study, study description and hypothesis, overall organization/logical flow,
sources, scientific writing style, and composition/grammar/word choice. Students earn up to 3
points for each dimension, and examples of the grade levels within each dimension are
provided. For example, if either the study description or the hypothesis is missing, then 1 point
would be earned for the study description and hypothesis dimension. Alternatively, if both the
description and hypothesis are provided, but either is unclear, then 2 points would be earned.
Scores for the Introduction section range from 0 to 24.
Stellmack et al. (2009) evaluated the interrater and intrarater reliability of this rubric. For three
graders, interrater reliability was defined both liberally (i.e., scores for a dimension were within
1 point across the three graders) and conservatively (i.e., scores for a dimension were equal
across the three graders). Agreement ranged from .90 (liberal) to .37 (conservative). For
intrarater reliability, graders re-evaluated a subset of their original papers 2 weeks after the
initial assessment. Agreement ranged from .98 (liberal) to .78 (conservative). In such
consistency estimates, values at or above .70 are viewed as acceptable (Stemler, 2004).
Beyond psychology-based writing, the Biology Thesis Assessment Protocol (BioTAP) provides a
systematic method to assess scientific writing in the biological sciences, and the components of
the rubric are applicable across natural and social science disciplines and could be adapted for
use within psychology. Reynolds, Smith, Moskovitz, and Sayle (2009) used BioTAP to evaluate
undergraduate theses in biology. The rubric has three categories: higher-order writing issues
(e.g., “Does the thesis make a compelling argument for the significance of the student’s
research within the context of the current literature?”), mid- and lower-order writing issues (“Is
59