Assessing Student Learning: A Collection of Evaluation Tools

Method:MaterialsMaterials are appropriate givenhypotheses and pilot tested and/orchecked for reliability. Materials aredescribed with enough detail that areader could replicate the study;materials should be appended if selfcreated,cited if not.Materials are appropriate but notcomplete (e.g., too few questions) ornot checked for reliability. Thedescription is adequate but could usemore detail. The measures areappended or cited, as needed.Materials are incomplete and notchecked for reliability, or they lackvalidity given the hypothesis. Theymay also be adequate but simplisticgiven the study goals. Thedescription is lacking in details butthe measures are appended or cited,as needed.Materials are incomplete and lackingin validity. They are not fullydescribed or included in anappendix.Method:ProcedureProcedure is appropriate and ethical.It is described, in order, with enoughdetail that a reader could replicatethe study; instructions and protocolare included. Condition assignmentsare clear; randomization andcounterbalancing are explained asnecessary.Procedure is appropriate and ethical.The description is primarilycomplete but some minor detailsmay be missing, or some proceduralaspects could be explained moreclearly.Procedure is appropriate and ethical.The description is not in order ordifficult to follow, or a few majordetails are absent.Procedure is not appropriate or notethical. The description is unclear,or many major details are absent.Data reductionMeasurement of the dependentvariable (i.e., scoring, quantification)is clear, and any procedures fordata treatment are explained (e.g.,reverse scoring is discussed ifnecessary; procedures for datacleaning or handling outliers arepresented). If necessary, a codingscheme is clear and appropriate andinterrater reliability is computed.Measurement of the dependentvariable (i.e., scoring,quantification) is clear and/or thecoding scheme is appropriate. Datacleaning and outliers may not bediscussed, or the discussion is notclear. Interrater reliability may nothave been addressed.Measurement if the dependentvariable is appropriate but notexplained clearly and/or the codingscheme is somewhat vague or doesnot cover all response possibilities(e.g., “maybe” in a Y/N task).The scoring/quantification of thedependent variable and/or the codingscheme is not appropriate for thedesign of the study. It may bedifficult to understand, even fromthe Results, how the data werescored/reduced.Results:DescriptiveStatisticsStatistics are appropriate (e.g.,means and SD; frequency) andcomputed accurately. Tables andfigures are correct, organized byrelevant variables, and called out intext.Statistics are appropriate andcomputed accurately. The figures ortables may have minor errors orconfusing aspects.Statistics are appropriate but may bemissing some relevant information(e.g., means but no SD). Figures ortables are redundant with text oromitted when necessary.Statistics are inappropriate (e.g.,means computed on categoricaldata) or computed inaccurately.Figures or tables are omitted whennecessary.Results:InferentialStatisticsInferential analysis is appropriate foraddressing each hypothesis. Eachfinding is stated in “plain English”and supported with statistics in APAformat.Results section includes correctlyused inferential statistics, but theymay be incomplete (e.g., lackingappropriate post hoc tests) or thefindings are unclear. Results maynot be linked to hypotheses.Results section includes inferentialstatistics, but they may be incorrector incomplete. Results do not seemlinked with the hypothesis of thestudy.Overall the inferential statistics donot address the hypotheses of thestudy. Results are reportedincorrectly, the wrong test is used, orsome critical information is missing.