Annual Performance Report (APR) Scoring Rubric - Arts Education ...

Research Methodand Design5. Clarity of purpose Research objectives,questions andhypotheses are notclearly stated oridentified.6. Explicitness ofresearch design7. Variety inresearch design8. Attention toreliability andvalidity1-Below Beginning 2-Beginning 3-Developing 4-AccomplishedOnly the goals and The goals and researchresearch objectives are objectives are stated, butstated. The questions or the questions orhypotheses related to hypotheses related to thethe evaluation of the evaluation of the project’sproject’s goals are not goals are only alluded tostated.and not explicit.Research methods andstudy populations are notstated or identified or arestated but not described,making replication orscaling up of the programimpossible.The research designincludes only onemethodology or datasource, or there isinsufficient information toidentify the types ofresearch designs used.There is no informationprovided about thepopulations studied andthe sampling proceduresused.Research methods andpopulations are brieflydescribed but may notsubstantially or logicallyrelate to the researchobjectives, questions, orhypotheses.The research designincludes more than onemethod and data sourcebut does not identifyprocedures for dataanalysis or the datasource and method arenot appropriate to theevaluation objectives.The report identifies thepopulations andsampling procedures,but they areinappropriate to ensurevalidity of findingsResearch methods andpopulations are describedbut are not clearly relatedto the research objectives,questions, or hypotheses.The design includesmultiple methods and datasources; data analysisprocedures are identified,but their appropriatenessis inconsistent. Thoughwarranted, the selectedmethods do not includeexperimental or quasiexperimentalmethods.The report identifies thepopulations and samplingprocedures. They aresufficient to ensureinternal or externalvalidity, but not both.The report clearly identifies the projectgoals and the research objectives,questions or hypotheses related to theevaluation of the project’s goals.Research methods and populations arefully and clearly described, and aresubstantially and logically related to theresearch objectives, questions, orhypotheses. The research design is explicitenough to support secondary analysis,replication, or the opportunity to further testfindings.The design includes multiple methods,multiple data sources, and consistent,aligned methods of data analysisappropriate to the purpose of theevaluation. The Research substantiallyreduces plausible competing explanationsfor results through the use of experimentalor quasi-experimental designs andtriangulation of multiple methods.The report identifies the populationsstudied and sampling procedures used.Sampling procedures are sufficient toensure reliability and internal and externalvalidity.© 2011 Arts Education Partnership 2

Arts EducationSpecific Criteria9. Teacher selfefficacy,comfortand confidencewith art making10. Attitudes towardsusing the arts inthe classroom11. Understanding ofbenefits of artslearning12. Establishment ofan arts educationprofessionallearningcommunity1-Below Beginning 2-Beginning 3-Developing 4-AccomplishedThe report and evaluation The report makes The report includes ONLY The report and evaluation include data thatmake no reference to general reference to post data on participating relates explicitly to the comfort andteacher comfort,participating teachers’ teacher’s self-efficacy, confidence levels of program participants inconfidence with teaching self-efficacy, self-image, self-image, comfort, and teaching or incorporating an art form. Theor incorporating an art comfort, and confidence confidence related to report includes pre- and post- data thatform, or self-efficacy or related to creativity and creativity and art making show explicitly the program’s effect onself-image related to art making oror performing.teacher comfort, confidence, self-efficacycreativity and art making performing, but theand self-image.or performing.statements are notThe report does notinclude data on teachers’attitudes towards usingthe arts in theirclassroom.The report does notinclude information ordata related to teachers’understanding of the roleof arts learning in studentcomprehension andachievement.There is no informationpresent regarding aprofessional learningcommunity that waseither established oremerged as a result ofthe professionaldevelopment program.supported by data.The report makesgeneral reference toteachers’ attitudestowards using the arts intheir classroom, but thestatements are notsupported by data.The report makesgeneral reference toteachers’ understandingof the role of artslearning in studentcomprehension andachievement, but thestatements are notsupported by data.The report makes vaguereference to interactionbetween programparticipants beyond theprogram, but does notidentify a professionallearning community.The report includes ONLYpost program data onteachers’ attitudestowards using the arts intheir classroom.The report includes ONLYpost program data onteachers’ understandingof the role of arts learningin student comprehensionand achievement.The report indicates theestablishment oremergence of aprofessional learningcommunity amongprogram participants, butdoes not go into detail onhow the community issupported or sustained.The report includes pre- and post- programqualitative data on teachers’ attitudestowards using the arts in their classroom.The report includes explicit pre- and postdataon teachers’ understanding of the roleof arts learning in student comprehensionand achievement.The report includes information on theestablishment of a professional learningcommunity of participants and explainshow the community is supported andsustained. Information is providedregarding participation in the communitybeyond the duration of the professionaldevelopment program.© 2011 Arts Education Partnership 3

Effectiveness ofEvaluation13. Recommendationsfor improvement14. EducatorOutcomes:Knowledge15. EducatorOutcomes:Application1-Below Beginning 2-Beginning 3-Developing 4-AccomplishedThere is no mention of The report includes The report includes The report includes recommendations forstrategies for improving some recommendations recommendations for improving the research methods used, asthe study or improvement for improving the improving the program well as recommendations for improving theof the program overall. program and/or the and/or the research program based on research findings.The findings are not research methods used, methods used; theInterpretation is sufficiently robust tointerpreted forbut therecommendations may be generate new questions or hypotheses toimprovement.recommendations are inconsistently related to guide further research.not logically related to the findings.There is no data orevidence related toparticipants’ gain inknowledge or skillsThere is no data orevidence showingparticipants’ use of newknowledge or skills.16. Student Outcomes There is no data orevidence that showschanges in studentlearning or achievement,or the data that purportsto measure the effects onstudents is inconsistentor incomplete.the findings.The report references toparticipants’ gain inknowledge or skills but itis not supported withdata or evidence.The report mentionsparticipants’ use of newknowledge and skills butdoes not support theclaim with data orevidence.The report includesinformation aboutstudent achievementthat is based solely onteacher perception butis not supported byother data or evidence.Student outcomes arenot mentioned explicitly.The report includes one ortwo forms of data orevidence related toparticipants’ gain inknowledge or skills.The report includes dataand evidence regardingparticipants’ use oflearned techniques andskills in their classroomsettings on a one-timebasis as a result of PD.The report includes dataand evidence that relatesto changes in studentlearning and achievement,but only includes data andevidence supporting onestudent outcome,cognitive OR academicOR personal.The report includes multiple, demonstrablyauthentic measures that may include bothquantitative and qualitative data andevidence related to participants’ gain inknowledge or skillsThe report includes specific data andevidence regarding participants’ use oflearned techniques and skills in theirclassroom settings on a continuous basisover time.The report includes specific data andevidence that relates to changes in studentlearning and achievement as a result ofteachers’ new learning. The report includesevidence related to varied studentoutcomes: cognitive, academic (measuredagainst rigorous academic standards), andpersonal.© 2011 Arts Education Partnership 4

Communication ofFindings17. Logical reasoning The reported findings donot follow a logical chainof reasoning and are notgrounded in evidencefrom the evaluation, orthe report over-reachesin describing the findings.18. Use of Data andAnalysis19. Transparency offindings1-Below Beginning 2-Beginning 3-Developing 4-AccomplishedThe report findings are The reported findingspresented but not exhibit a logical andsubstantiated by a chain evidence-based chain ofof reasoning. The reasoning, though theconclusions are stated conclusions reached arebut not justified.not explicitly justified orare inconsistent inData are not analyzed orpresented in the reportfindingsThe report makes noreference to validity orlimitations to the findings.20. Report Clarity The report is written andpresented withoutattention to clarity. Thereport contains jargonthat is not defined and isdisorganized. Thefindings are presented ina manner that makesthem difficult tounderstand.Quantitative andqualitative data areanalyzed, but onlyminimally or incorrectly.Findings from dataanalysis are notpresented.The report alludes tolimitations of thefindings but does not gointo detail about thelimitations or remainingthreats to validity orpossible competingexplanations.The report brieflydescribes the program,its context, purposes andfindings but in a mannerthat is only somewhatclear. It includesundefined jargon and/orspeculations not explicitlybound by themethodology andfindings.alignment.Quantitative andqualitative data areanalyzed inconsistentlyand/or findings arepresented with minimalinterpretation. Only apartial or incompletepicture of the programemerges.The report lists thelimitations to the findingsbut does not present themin detail or expand into theremaining threats tovalidity or possiblecompeting explanations.The report describes theprogram, its context,purposes, procedures andfindings of the evaluation.It may include somejargon but largely avoidsspeculations not bound bythe methodology andfindingsThe reported findings exhibit a logical andevidence-based chain of reasoning. Theconclusions reached in the evaluation areexplicitly justified, so that stakeholders canassess them.Quantitative and qualitative data areappropriately and systematically analyzedand presented in such a way that acomplete picture of the program emerges,with both types of data in support of oneanother.The report fully and candidly presents anylimitations to the findings, includingremaining threats to validity and possiblecompeting explanations.The report describes the program beingevaluated, including its context, and thepurposes, procedures, and findings of theevaluation, in a clear and concise mannerfreeof undefined jargon- in such a way thatall information is easily understood. Thereport avoids sensationalism orspeculations not bound by themethodology and findings.© 2011 Arts Education Partnership 5