After-School Initiative's Toolkit for Evaluating

More documents

Recommendations

Info

Toolkit for Evaluating Positive Youth Development 92 Appendix X. Determining Which Survey to Use: Post-only versus Pre-post Post-only survey question sets are recommended for use with the ASI programs for a number of reasons. • Post-only survey administration creates less burden on respondents and staff. • Measuring change using a pre-post format requires more response scale points to be sensitive enough to measure change, making its use more difficult with participants of younger ages or lower literacy levels who are better served by 3-point scales. • Post-only allows respondents to use their internal gauge to assess change rather than requiring that change be calculated by simple math. • Post-only is less susceptible to “ceiling” effects so that youth who consider themselves very strong at program start still can report improvement in behavior or attitude (even if the change is subtle). However, the post-only survey is considered to be weaker methodologically because it is more susceptible to bias and the promotion of positive responses. When youth rate their status (on a behavioral or attitudinal outcome) at the end of a program, that self-report is unlikely to be influenced by their memory of their rating several weeks earlier when the program began. The fact that youth are providing unprompted responses (independent of earlier self-assessment) makes their final self-rating more credible. Consequently, when using the pre-post design, be sure that youth are not reminded of their earlier ratings when they provide their post-program self-assessment. Program leaders will need to decide which method works best for their program. There are a number of factors to consider. Some are listed in the table below. Factors to Consider Age and literacy level of participants Program resources Outcomes of interest Need for more or less rigorous outcome data Stage of program development Questions to Consider Are your participants able to answer the questions on the pre-post instrument in a trustworthy way? (The pre-post version is only recommended for participants in grade 5 and above.) Do you have the staff resources to administer the survey twice? It may be particularly difficult if you frequently add new participants and must track each one for a unique pre- and post-period rather than enrolling groups at the start of each semester. Are many of your outcomes those that may be susceptible to a ceiling effect? (Outcomes for which participants report high levels at baseline will show less change in a pre-post format; e.g. honesty, caring.) Do you want stronger evidence? Do you need stronger evidence for your funders? Are you skeptical of selfreported post-only data? Has your program been around long-enough that you are ready for a more rigorous outcome evaluation? (New and developing programs might benefit more from the post-only format until they have stronger roots at which time they may move to pre-post format.)
Flowchart for Selecting Post-only or Pre-post Evaluation Design The flow chart below is designed to guide staff in determining whether a post-only or a pre-post method would work best for their afterschool program and participants. It provides one framework for determining evaluation design, however, programs may choose to adjust the order to fit their needs. Post- Only Design Developmental Stage of Program New Established Post- Only Design Youth Literacy Level < 5th Grade 5th Grade + Post- Only Design Program Resources Low Adequate Post- Only Design Ceiling Effect on Outcomes Assessed Likely No Post- Only Design Audience Requires Rigorous Assessment No Yes Pre-Post Design Toolkit for Evaluating Positive Youth Development 93
Page 1 and 2:
THE AFTER-SCHOOL INITIATIVE’S Too
Page 3 and 4:
Table of Contents INTRODUCTION ....
Page 5 and 6:
Introduction The After-School Initi
Page 7 and 8:
Mitigation of burden on participant
Page 9 and 10:
Using This Toolkit As part of their
Page 11 and 12:
Guarantee anonymity or confidential
Page 13 and 14:
• Who is responsible for gatherin
Page 15 and 16:
The following is a checklist of tas
Page 17 and 18:
The Survey Administration The surve
Page 19 and 20:
Problem situations during survey ad
Page 21 and 22:
Outcome Domain Outcome Indicators A
Page 23 and 24:
Academic Success Many after-school
Page 25 and 26:
Arts and Recreation Many youth prog
Page 27 and 28:
Cultural Competency As communities
Page 29 and 30:
Life Skills Life skills are profici
Page 31 and 32:
Positive Life Choices Although the
Page 33 and 34:
Section 2: Program Quality Toolkit
Page 35 and 36:
ASI requirement: All program qualit
Page 37 and 38:
Section 3: Youth Background and Pro
Page 39 and 40:
Section 4: Alternative to Post-only
Page 41 and 42:
Review of positive youth developmen
Page 43 and 44:
Reliability of Toolkit Domains Doma
Page 45 and 46: Appendix III. The Use of Incentives
Page 47 and 48: [ASI program staff may delete some
Page 49 and 50: Child Assent to Participate in the
Page 51 and 52: [ASI program staff may delete some
Page 53 and 54: Asentimiento de jovenes para partic
Page 55 and 56: Outcome Domain Community Involvemen
Page 57 and 58: Outcome Domain English Post-only Qu
Page 59 and 60: Outcome Domain Sense of Self Englis
Page 61 and 62: 7) Who will assist the survey admin
Page 63 and 64: Toolkit Outcome Domains Toolkit Out
Page 65 and 66: Toolkit Outcome Domains Toolkit Out
Page 67 and 68: Toolkit Domains Toolkit Outcome Ind
Page 69 and 70: TABLE 3 - Toolkit Outcome Domains L
Page 71 and 72: Toolkit Domains Toolkit Outcome Ind
Page 73 and 74: Outcome Domain Advanced Question a
Page 75 and 76: Outcome Domain Advanced Question I
Page 77 and 78: Background Information Advanced Que
Page 79 and 80: CSAP Tracking Form Student ID Readi
Page 81 and 82: Attendance Tracking Form Student ID
Page 83 and 84: Appendix IX. Complete Post-only Too
Page 85 and 86: [Academic Success - For ASI grantee
Page 87 and 88: [Life Skills - For ASI grantees cho
Page 89 and 90: 17) What grade are you in? ________
Page 93 and 94: [Life Skills - For ASI grantees cho
Page 95: 17) ¿En qué grado estás? _______
Page 99 and 100: Outcome Domain Post-Only Question P
Page 101 and 102: Outcome Domain Post-Only Question P
Page 103 and 104: Appendix XI. Complete Pre-Post Tool
Page 107 and 108: [Positive Core Values - All questio
Page 109 and 110: [NAME OF ASI PROGRAM/AGENCY] ENCUES
Page 111 and 112: [Cultural Competency - For ASI gran
Page 113 and 114: [Questions a and b are required if
Page 115 and 116: REFERENCES 1 The International Coun
show all

After-School Initiative's Toolkit for Evaluating

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?