Evaluation Toolkit for Community Youth Programmers - Bank of Ideas

AcknowledgmentsThe Offord Centre initiative to establish direct links withcommunity programmers across the country was theinspiration of the late Dr. Dan Offord, to whose memory thiswork is dedicated. As a child psychiatrist and the long-timedirector of a children’s summer camp, he had first-handexperience with children who needed help in achieving themost they could out of life. He understood how the righthelp at the right time can make an enormous difference toa child’s eventual health and happiness. He firmly believedthat the most effective way to help individuals was by usingcommunity programs to give an equal chance for healthydevelopment to all our children.We would like to express our gratitude to the hardworkingorganizations and groups named here for sharing theirexperiences in order to develop effective child and youthprogramming in Canada. Their co-operation and readinessto discuss practices, successes and challenges have madethis work possible. We hope that our collaboration resultedin a greater appreciation for the role of evaluation in theirprogramming. We would also like to thank Dr. Ellen Lipmanwho helped guide the Hands Across Canada project in Dr.Offord’s stead. The assistance from donations by Sun LifeFinancial over the two years of the project, on behalf of itsemployees and Clarica advisors, made this collaboration andlearning possible. We also deeply appreciate the interestand assistance of the Sun Life Community Relations staff, inparticular, Wilda Graham. Stephen Kingston of MediaDoc,Kitchener, deserves our thanks for his skill in designing thetoolkit for web site and publication.Copyright 2004, Offord Centre for Child Studies.Permission for downloading, reproduction or distributionof all or part of this document is granted for personal andnon-profit educational uses only. All other uses requirepermission from the Offord Centre. Please acknowledgeuse of the material with the following citation: Shaw,A. T & Racine, Y. A. Evaluation Toolkit for CommunityYouth Programmers. (2004, Nov.) Retrieved from theOfford Centre for Child Studies, McMaster Children’sHospital-McMaster University web site at: http://www.offordcentre.com/. Use of individual tools should includeacknowledgement of the original source agency cited.Printed copies of the toolkit are available free of charge bycontacting the Administrator, Offord Centre for Child Studiesat Patterson Building, Chedoke Site, McMaster University,1200 Main St. West, Hamilton, Ontario, Canada, L8N 3Z5.Supplies are limited.Aurelia ShawYvonne RacineNovember, 2004Participating organizationsAlbert Park/Radisson Heights Community Association,Calgary, ABChild and Adolescent Services Association, Edmonton, ABProgram for Adult Learning, Lac La Biche, ABWood’s Homes, Calgary, ABJohn Maclure Community School Society, Abbotsford, BCKi-Low-Na Friendship Society, Kelowna, BCKiwassa Neighborhood House, Vancouver, BCMarlene Street Tenants’ Association, Inc., Winnipeg, MBWinnipeg Boys and Girls Clubs, Inc., Winnipeg, MBNew Brunswick Association for Community Living,Fredericton, NBCommunity Youth Network for SouthwesternNewfoundland, Inc., Stephenville, NFAlexander Society for Special Needs, Wolfville, NSAbility Online, Toronto, ONCentre de sante communautaire Hamilton-Wentworth/Niagara, Hamilton, ONChristie Lake Community Centres, Ottawa, ONChristie Ossington Neighbourhood Centre, Toronto, ONDelta Family Resources Centre, Toronto, ONDoorsteps Neighbourhood Services, Toronto, ONFamily Counselling Centre of Sarnia-Lambton, Sarnia, ONFamily Services of Haliburton County, Haliburton, ONFrontier College (National Office) , Toronto, ONHaldimand-Norfolk Women’s Services, Simcoe, ONHouse of Friendship, Kitchener, ONLearning Disabilities Association of Peterborough and TrentUniversity, School of Education and Professional Learning,Peterborough, ONLennox and Addington Interval House, Napanee, ONKerry’s Place Autism Services, Kingston, ONLakeshore Area Multi-services Project, Toronto, ONNorth Hamilton Community Health Centre, Hamilton, ONOttawa Carleton New Beginnings for Youth , Ottawa, ONOverbrook Forbes Community Resource Centre, Ottawa, ONSt. Catharines Resource Service for Youth, St. Catharines, ONToronto Children’s Breakfast Club, Toronto, ONWoodGreen Community Centre of Toronto, Toronto, ONWyndham House, Guelph, ONYES Employment Services and Volunteer Youth in theMillennium, Thunder Bay, ONHillsborough River Area Development Corporation, MountStewart, PEFrontier College Saskatchewan, Saskatoon, Saskatchewan

Table of ContentsIntroduction iWho could benefit?♦ iiEvaluation as process ♦ iiiStep 1: Preparing the Groundwork for Evaluation 11.1 Start to plan early ♦ 11.2 Define the evaluation purpose and scope ♦ 21.3 Develop realistic expectations ♦ 61.4 Promote a pro-evaluation attitude ♦ 71.5 Include ethics and respect for individuals ♦ 81.6 Anticipate possible evaluation results ♦ 101.7 Identify how to use the evaluation ♦ 10STEP 2: Planning a Meaningful Evaluation 132.1 Select and work through a program logic model ♦ 132.2 Think about theory of change and strategies ♦ 142.3 Consider a broad range of objectives to evaluate ♦ 152.4 Consider the best timing to find expected results ♦ 152.5 Involve relevant partners and stakeholders ♦ 162.6 Determine methods for monitoring program results ♦ 17

STEP 3: Designing the Data Collection Process 193.1 Decide on evaluation methods ♦ 193.2 Seek out data sources ♦ 233.3 Make data collection accessible and convenient for collectors ♦ 243.4 Gain participant/parental support & protect confidentiality ♦ 27STEP 4: Developing Data Collection Tools 294.1 Review the link between data collection and objectives ♦ 294.2 Determine relevant data collection tools ♦ 304.3 Look for model questionnaires ♦ 324.4 Modify existing questions to suit respondents ♦ 334.5 Guidelines for writing new questions ♦ 364.6 Pre-test questionnaires ♦ 41STEP 5: Collecting Data 435.1 Ensure access to necessary data ♦ 435.2 Promote and explain data collection ♦ 445.3 Provide appropriate training to data collectors ♦ 455.4 Provide adequate time and resources ♦ 465.5 Conduct data collection ♦ 46STEP 6: Working through the Data 576.1 Drawing conclusions about program effects ♦ 576.2 Protect and preserve the data ♦ 586.3 Review data for errors ♦ 586.4 Organize qualitative data in relation to evaluation questions ♦ 596.5 Coding and data entry ♦ 616.6 Using statistics to describe data ♦ 626.7 Knowing about inferential statistics ♦ 67

STEP 7: Making Use of Evaluation Data 697.1 Review results with program staff and volunteers ♦ 697.2 Responding to the results ♦ 707.3 Share lessons learned with stakeholders ♦ 717.4 Support staff to act as program ambassadors ♦ 727.5 Closing the evaluation circle ♦ 72Glossary of Evaluation Terms 75Selected References and Resources 79Appendix – Examples of Data Collection and Evaluation Tools1 (appendix)

IntroductionCommunity programs providing opportunities or delivering services for youth havebeen developed by a variety of organizations: schools, private associations, clubs,community groups, government and non-profit non-governmental agencies, tenants’associations, municipal departments and more. In 2003 and 2004,The Offord Centrefor Child Studies was fortunate to work with staff from about 40 such agencies andcommunity groups across Canada as a resource for their evaluation planning, in afunding initiative of Sun Life Financial. All programmers face the need to examinetheir operations and programming, to evaluate their accomplishments in relation totheir objectives. The flood of information on the Internet and in libraries about howand why to ‘evaluate’ can be overwhelming.This toolkit supports what is already being done in a number of excellent programsacross the country to evaluate the impact of their activities on children and youth.It focuses on the needs of both fledgling and more established groups with limitedresources. Building on a range of community experience, it• incorporates basic yet comprehensive guidance in one place• addresses common concerns, and• provides examples of workable methods and toolsA key goal of the Offord Centre for Child Studies is to help promote practices thatwill improve our children’s chances for success in life. It is an organization committedto research and to the dissemination of that research into the community. Whereverpossible, research must be translated into actions that make a difference in society.The toolkit is an investment in community programming futures. Like any goodinvestment, we hope it provides returns in the type of quality programming that canchange lives. A growing bank of research in the United States, Canada and elsewhereshows that children can be helped to succeed even if they have less chance of successthan other kids. One method is access to programs that help them connect withpeople, develop their skills and give them hope for a future which has a place for themin it. Good quality, effective programming can help children who face hurdles in lifeand lack those opportunities get back on track. Good quality, effective programmingcan provide positive learning experiences for all children.i

Who could benefit?This basic toolkit offers some guidance about self-evaluation: a sequence of stepsfor planning and designing a program evaluation, developing useful data collectiontools, analyzing and using data, and communicating results. It also providesexamples of data collection tools currently in use in Canadian communities by avariety of programs both large and small. The ideas offered here would be relevantfor groups offering programs intended to improve some aspect of community life.Groups starting out to offer programs could benefit as the toolkit offers guidance tohelp develop an organizational mindset about evaluation and build it into programplanning. However, existing groups might also use the toolkit to compare itssuggestions with their own activities or to gain practical advice for incorporatingevaluation into a new session.Programs that informed this toolkit, though considerably varied, concentrated onyouth development strategies and included variations of elements to enhance skills,knowledge, attitudes and motivation.• Homework, academic, computer skills• Literacy• Pro-social skills, e.g., co-operation, teamwork, peaceful conflict resolution, bullyingawareness, empathy, cultural sensitivity• Positive family interaction and support• Athletic, active recreation• Creative arts, e.g., art, music, drama, native drumming• Life skills, e.g., cooking, gardening• Enrichment experiences: outings, presentations, family activities• Leadership training• Mentoring, counseling, perception/awareness training• Positive peer examples• Employment-related skills• Volunteer serviceThe appendix contains examples of data collection forms used in current programsthat may spark ideas for simple methods of data collection. As field-tested tools,they are often compromises between theory and practicality and may be modified tobetter suit individual situations. Selected resources and references to explore topics ingreater depth are provided in References and Resources.iiThe Offord Centre for Child StudiesHands Across Canada

Evaluation as process• Evaluation is best considered as an ongoing process:a) to provide data that aids decision-makingb) allows for periodic adjustments during programmingc) encourages coordination and cooperation of all stakeholders• The following steps concentrate on the development of criteria forexamining the effectiveness of programming over time.• In contrast to one-time evaluations at the close of a session, periodic,longer term or continuous evaluations, encourage programmers to:a) take note of the process,b) develop a broader perspective about the effects of a program—onparticipants, volunteers, families—andc) consider explanations for its varying levels of effectiveness.Evaluation Toolkit for Community Youth Programmers Introduction iii

Preparing theGroundwork forEvaluationSTEP1Start to Plan EarlyDefine the Evaluation Purpose and ScopeDevelop Realistic ExpectationsPromote a Pro-Evaluation AttitudeInclude Ethics and Respect for IndividualsAnticipate Evaluation ResultsIdentify How to Use the Evaluation1.1 Start to plan earlyIt is ideal to develop evaluation plans while designing a new program and workingthrough objectives, activities and related factors. However, managers may also need todevelop or modify an evaluation process for an existing program.Plan the evaluationas part of programdevelopmentNew programs• include evaluation as an integral part of the program during programplanning• discuss the role of evaluation in initial community discussions, needsassessment and planning meetings• consider what evaluation implies – i.e.,• how to monitor and evaluate what happens in the program,• determine if the program is worthwhile for the time, effort, energyand money involved;• learn about the program’s strengths and weaknesses• thinking about what and how to evaluate will help• clarify objectives and program activities and• prevent misunderstanding among partners and stakeholders1

• use planning tools, e.g.,• program logic model• theory of change model (See Step 2 for more details.)• include evaluation costs in program budgets, grant proposals and otherfunding requests.Existing programs• develop evaluation plans for the next session along with budgets, fundingproposals or new operation plans• factor in sufficient time and resources before next session• include partners and stakeholders in the evaluation planning wherefeasible• developing or modifying evaluation plans during a program session isdifficult but possible:• new/revised evaluation methods can be tested• preliminary results can inform ongoing practice1.2 Define the evaluation purposeand scopeCommunities are coming to understand that steps taken to help children’sdevelopment can have long lasting consequences for their health, happiness andsuccess as adults. Evaluations that look at the effects of programs launched to helpyouth can provide data to validate programming efforts and directions and guideprogramming change. Some types of evaluation do not consider program effects.What is the evaluation purpose? What is to be evaluated?This toolkit providesguidance for extendingevaluation to includeoutcomes.At the outset of planning, understand the different types of evaluations, what you andothers expect to achieve and set parameters accordingly to avoid later confusion andencourage a shared purpose among stakeholders.Types of evaluations include: service evaluation, process evaluation, organizationalreview, outcome evaluation.2The Offord Centre for Child StudiesHands Across Canada

For whom and why is the evaluation being done?Motivating FactorsMotivations may include:• funders’ requirements• public relations• to fulfill partnership demands or• for a research agenda.Evaluations of existing programs may also be called for• to encourage sustainabilityORIn response to• specific questions raised in the community• observations raised by staff, both positive and negativeIntended Benefits• trustworthy information that will be meaningful to staff, volunteers andfamilies• a transparent process• greater feeling of input from community stakeholders• community support• data may allow stakeholders to consider how programming benefits canrepay present, or offset future, costsWhat should be included?Whatever initiated theevaluation will dictateto some extent thequestions asked.“Be sure that whatever you do is useful to the project itself, not just anadd-on to keep funders happy. Think: what do we who are deliveringthe program want to know to help us get smarter faster? Build on that.”– Dr. Fay Martin, Executive Director,Family Services of Haliburton County.4The Offord Centre for Child StudiesHands Across Canada

Building-block data for evaluation• description of program activities, times, duration, venues• participant data: registration, full name, contact information• volunteer data: application data, training material• program materials• costs (actual and donated)• outcomes data (See Step 2)Record what and howprogramming wasdelivered in order tolearn more about whattypes of activities andstrategies can be mosteffective.Monitoring program costsMonitoring and regulating costs are aspects of accountability: determining how mucha program costs to deliver benefits to the community.Consider the total cost of remounting a program.• Track actual spending and the value of donated goods and services, e.g.,• resources• facilities• volunteer time• discounts and subsidized costsPrograms will not becopied and operatedin other areas,despite evidence ofeffectiveness, if they aretoo expensive for mostcommunities to run.Donated contributions reflect the value of staff and community support to yourprogram:• useful knowledge to aid fundraising• show areas of vulnerability and strengths that could have a bearing onsustainabilityEvaluation Toolkit for Community Youth Programmers Step 1 5

1.3 Develop realistic expectationsBe realistic about thetime and help you willneed, the availableresources, costs andlimitations of theevaluation process. Plancarefully to make theevaluation as efficientand meaningful aspossible.Constraints on Evaluation• levels of knowledge and expertise available• priorities for resources for program delivery• program size and numbers• sources and types of data• time and funding available• attitudes and perceptions within and outside the programTime DemandsProgram managers sometimes underestimate the time required for evaluation—andto decide how to explain, report and use findings for program improvement. Theanalytical stage may also require more time than allotted.Need an example?Sorting and analysis of data:• Quantitative data (coded or numerical) may need to be analyzed by subgroups toanswer questions, for example, about whether girls and boys fared differently orwhether the program was finding and keeping the target group.• Qualitative data, e.g., from interviews, focus groups• may actually take longer to work through than quantitative results• need repeated review by independent readers to discern patterns and intensity ofcommentsHow to reduce limitations?Weigh the pros andcons of having anexternal evaluationor outside helpto conduct theevaluation, particularlyif objectivity isan importantconsideration. Atransparent processand well-designed datacollection methods canincrease objectivity.• involvement of partners, external evaluators or outside assistance toincrease expertise and resources• restricting the evaluation to key questions• planning evaluation in stages over time• sampling multi-site programs• use of existing or readily available data• use of ready-made data collection tools and instruments• adequate preparation and careful timing to collect sufficient data withminimal call backs (See Step 5)6The Offord Centre for Child StudiesHands Across Canada

Remember that evaluation is likely to be an ongoing or at least repeated process inyour program, a series of steps toward continued program improvement.1.4 Promote a pro-evaluationattitudeStart with key issuesand feasible questions.It is possible toadapt to changingcircumstances, learnfrom experience andamend your evaluationmethods as neededover time.Stakeholders may agree on the type and purpose of an evaluation and still do sowithout any measure of enthusiasm or commitment.Understand the role of volunteers and staff in evaluation“Do an attitude check - look at evaluation not as a choreor burden or something to be afraid of, but as a learningopportunity and a chance to analyze your program andits effectiveness and make changes to improve it.”– Doris Martin, Family Services Program DirectorHouse of Friendship, Kitchener, Ontario.Potential volunteer and staff concerns• evaluation may be potentially threatening to volunteers and staff• time and paper work for little value• irrelevant to daily needs and routines• imposed from above or outside• not responsive to staff/volunteer concerns or inputPotential remediesStart with an inclusive evaluation planning process• staff and volunteers are more likely to understand the value of evaluation,treat it seriously and feel more comfortableConsider an outcomes evaluation as a mutual search for knowledge andunderstanding about how best to help participants.Value the role of volunteers and staff in conducting the evaluation• as a continuation of volunteers’ and staff desires to help youth throughtheir involvement• volunteers are vital to evaluation as a program’s eyes and earsEvaluation Toolkit for Community Youth Programmers Step 1 7

Unbiased dataInformed consent may seem at odds with unbiased data.Informants may be reluctant to voice any criticism of programs if they think:• negative information will be linked to their children, with negativerepercussions• don’t have confidence in personnel• don’t understand how data will be used• valued programs may be jeopardizedRaised expectationsBe aware of what an evaluation can change. Volunteers, participants and families mayrespond enthusiastically toward an evaluation, particularly if there is a therapeuticor service element to the program, but have unrealistically high expectations ofresults. They may expect some follow up to improve their lives or situation orprogram expansion, which leads to disappointment in the program when it does notmaterialize.Ethical policies• Plan data collection carefully to obtain meaningful data and avoidpotential problems (see Step 4)• Plan clear, open communication and develop relationships of trust withstaff and sources of data• Obtain informed consent from families to obtain necessary data• Plan safeguards to assure privacy and confidentiality of information (SeeStep 5)• Ensure staff is aware how releasing information, personal stories, pictures,data for publicity might affect participants’ lives. Parents may not be awareof potential risks.Need an Example?After a club identified and publicly praised a youth member as a former gang member whohad turned his life around with the help of their youth program, former gang associatesattacked both him and family members. The club then revised its policy about identifyingindividuals, even with permission.Evaluation Toolkit for Community Youth Programmers Step 1 9

1.6 Anticipate evaluation resultsWho will need to see the results?Consider when planning the evaluation what stakeholders need to know, what variousresponses could indicate and their importance to particular groups. A well designed,well prepared evaluation will be useful to programmers and meaningful to all whocontribute towards itCan results be explained?A positive attitude toevaluation includes,paradoxically, beingprepared for negativeresults.Evaluation may not show what you anticipated. Results may be negligible or negative.Plan for success but be ready to explain weaknesses and what they may say aboutassumptions, strategies, activities and procedures.Some motivations for evaluation, like raising community support and fundraising,assume favourable results. But negative results can also motivate staff and guideprogram improvement1.7 Identify how to use theevaluationIdentified uses• defined by factors that initiated the evaluation: e.g.,• program improvement• giving direction to community development• publicity, community awareness• raising community support• fundraising• staff development• volunteer education• reports to funders• relationship-building with participants/stakeholders10The Offord Centre for Child StudiesHands Across Canada

Be open to unanticipated uses for dataNeed an Example?Data collection by a social services agency for a recreation program, documenting levels ofactivity of youth in local areas was used to inform local government and communities aboutinequalities and helped spur initiatives by groups as varied as local art galleries and thepublic health unit.There will usually be multiple audiences for evaluation results:• staff and volunteers,• funders, partners and other supporters,• participants and families• the local community and• perhaps a larger community with specialized interests.Directed messagesDirect summarized or excerpted results to specific partners or segments of thecommunity using usual channels of communication and readily available media toaddress specific needs or in response to their participation. (See Step 7).Sharing results with the research communityProgrammers may share researchers’ interests to some extent in the mechanismsof a program strategy—why it worked as it did to affect participants. Their focusis on knowledge to improve programming and increase its benefits.An ever-growing body of data is being amassed to help us all learn what can helpchildren and youth. Much of that data is from programs in the United States,which may or may not be equally representative of Canadian findings. Developingpartnerships for evaluation purposes with academic centres and researchers willhelp promote Canadian knowledgeAn underlyingmessage of this toolkitis that the broadercommunity, includingacademic researchersand communityprogrammerscan benefit fromknowledge gainedthrough individualoutcome-basedprogram evaluationswhen well designedand well executed.Evaluation Toolkit for Community Youth Programmers Step 1 11

Planning a MeaningfulEvaluationSTEP2In Step 1, relevant parties agreed on the motivation, scope and resources to carry out theevaluation, and considered ethical constraints, attitude and the need to manage expectations.Step 2 begins the process of designing a particular evaluation tailored for your program, itsobjectives and strategies with the aid of a logic model.Select and work through a program logic modelThink about theory of change and strategiesConsider a broad range of objectives to evaluateConsider the best timing to find expected resultsInvolve relevant partners and stakeholdersDetermine methods for monitoring program results2.1 Select and work through aprogram logic modelThe recommended model was adapted by the Offord Centre from a program logicmodel created by the Harvard Family Research Project. (See R1 in Appendix)Alternative models may not address ways of evaluating program impact.A simple one page model that connects objectives to anticipated results is useful• for volunteer training• to ensure that staff and supporters fully understand the direction andthinking behind a program• at presentations and community meetings to provide a framework for acommon understanding, and• allows program developers and other community residents to challengetheir assumptions and reach a common understandingWriting an evaluationplan forcesprogrammers to clarifytheir thinking about aprogram and provides away of expressing thatthinking explicitly tostaff and partners.13

2.2 Think about theory of changeand strategiesStrategies for bringing about changeIsolate the strategy or strategies that explain how various program elements areexpected to bring about proposed changes.Need an example?For instance, in a program to increase positive self concept and self esteem in native fosterchildren, one activity may be to bring in native guests as positive role models.Two distinct strategies are involved:1) to build a sense of belonging to First Nations’ culture and to share its ethos by bringingadmired elders to teach traditional wisdom.2) to showcase guests who are successful in modern, mainstream activities to help youthrealize future possibilities for themselves.The theory of change—how and why a program is expected to work—prompts specificquestions to test the theory.Consider indicators of change, timing and contributing factors, e.g.,• How is change expected to occur?• When should/does change occur?• Do changes occur in steps or stages?• Is each stage recognizable?• By what signs is each stage recognizable?• What influences would have a bearing on the changes, either enhancing orlimited them?Your evaluation can provide information of importance to the larger field of youthdevelopment. Sometimes differing strategies reinforce one another and increase theeffectiveness of programming, particularly when multiple aspects of children’s livesare included, such as family and school.14The Offord Centre for Child StudiesHands Across Canada

2.3 Consider a broad range ofobjectives to evaluateWhat is most meaningful to evaluate?Sometimes outside agents such as funders or government agencies identify what theywant you to evaluate. However, you may suspect or observe unanticipated effects.Broaden your thinking about might be expected to result and allow room for newdiscoveries.Need an example?The obvious evaluation question in a program putting volunteers into schools to tutorchildren with learning problems would be: “Did the tutored children improve their readingskills?” However, the programmers also decided to look for effects in tutors— students inteacher training—after working closely with children who struggled with reading difficulties.They found that student volunteers gained considerably in their appreciation of specificliteracy problems and became more knowledgeable about terminology, conditions andstrategies for helping children. Because the program had such a beneficial effect on thestudent teachers it was continued as an integral part of the teaching curriculum.2.4 Consider the best timing to findexpected resultsProximal effectsImmediate, direct effects generally of primary interest to community programmers.e.g.,• for a basketball program, an increase in skills related to playing basketball,but also possible improvements in teamwork, discipline or feelings ofcompetenceDistal effectsMore indirect effects or ones that develop in the longer term e.g.,• for the basketball program, distal effects could be health benefits if theperson was motivated by the program to continue playing basketballregularlyFollow up evaluationsometimes shows anincrease in effects for atime after the programends. More typically,effects wash out ordiminish over timeafter a program endsunless there is somereinforcement of whatparticipants gainedduring the program.Evaluation Toolkit for Community Youth Programmers Step 2 15

Time the evaluation measurement at the most reasonable time period to expectbenefits from your program. e.g.,• effects of training in pro-social skills or anger-management may buildfrom the end of the program as the family learns, practises and adjusts tonew ways of interacting2.5 Involve relevant partners andstakeholdersPrograms with multiple sites and/or partnerships have stressed the importance ofopen, consistent communication strategies, especially when deciding what effects toinvestigate.• maintain a regular flow of information• assign specific roles and acknowledge expertiseBenefits from early involvement with community• resources to provide expert advice, contextual information and historicalknowledge of community activities• stakeholders help establish what indicators and degree of change wouldestablish convincing evidence of effectiveness• challenges and any later necessary adaptations more easily understood, toavoid alienation or disillusionment16The Offord Centre for Child StudiesHands Across Canada

2.6 Determine methods formonitoring program resultsA number of factors influence the choice of evaluation methodology. e.g.,• evaluation purpose,• time and resources available• level of expertiseEvaluation approachesEvaluation approaches include experimental, quasi-experimental or non-experimental,longitudinal or single-time, formative and/or summative, comparative or noncomparative(See Glossary.) They can use quantitative methods, qualitative methodsor a combination of both. Some methods provide what is considered more reliableinformation in scientific terms, but even this depends on numbers and quality ofevaluation.There is a perception that some evaluation methods are intrusive and incompatiblewith program philosophy, e.g.,• a program for disaffected teenagers fears that using questionnaires,typically associated with authority, could jeopardize their developingrelationships with the program.Test your assumptions. e.g.,• involve participants in discussions about evaluation and how it canimprove programming to gain their interest and cooperationCAUTION: Detailed knowledge about what programmers are assessing and whatresults they hope to find could influence participants’ responses and bias results.Meet this challenge by using:• a variety of evaluation techniques• concrete indicators that measure behaviour and attitudes as well asqualitative assessments of attitude and satisfaction• careful question design. (See Step 4.)Evaluation Toolkit for Community Youth Programmers Step 2 17

Designing the DataCollection ProcessSTEP3Step 2 asked you to produce a logic model that included clarifying the strategy or theory ofchange connecting the objectives, activities and expected results of the program. The model isthe basis for a list of changes or indicators that can be observed or measured.Step 3 discusses methodology and means of collecting data about indicators of change andrelevant factors in order to answer questions about program implementation and effects. Moredetail appears in subsequent steps.Decide on evaluation methodsSeek out data sourcesMake data collection accessible and convenient for collectorsGain participant/parental support & protect confidentiality3.1 Decide on evaluation methodsThere are several alternative methods for evaluating the effectiveness of a particularprogram or program element. Determine which are most suitable for the programsituation.Experimental studiesthe ideal for social science researchWith rigorous standards that include:• randomized selection of people to receive a program,• matched group characteristics of participants and non-participants,• a degree of blindness so that participants do not know what elementsare being tested and researchers do not know whether data come fromparticipants or non-participants.19

Encouraged for research purposes• not yet common for evaluating community prevention and interventionprograms• usually involves the assistance of academic institutions or researchdepartments of large agenciesNeed an example?• In cases such as agencies with sufficiently high case loads or schools, it has beenpossible to randomly select people to participate, or not, in a program. Ensuring thatevaluators are blind to whether data apply to participants or non-participants is still aconcern.Sampled studies• when large numbers are involved, as in national, multi-site programs,it may be necessary to sample participants, i.e., select a part of thepopulation to represent the whole population. (See Glossary for Statisticalsamples)Comparing participantsto non-participants isproblematic because ofpossible selection bias:whatever factors madeparticipants decide tojoin a program in thefirst place may makethem different from thecomparison group ofnon-participants.Quasi-experimental studies• more common for community groups—compares a group of participantsto another group of people with similar characteristics outside theprogram.Group comparisons• appropriate within a multi-site program if program curriculum, deliveryand characteristics of participant groups are the same.Programs may compare different elements of a program, e.g., when a programintroduces a bullying awareness presentation for one session but not to a comparablegroup in the same program.Quantitative and qualitative researchQuantitative research• uses measurable, quantifiable data and mathematical formulas or statistics• Community groups most commonly use basic statistics, e.g.,percentages and averages that can be turned into graphs andcharts to easily show patterns of characteristics and behaviour.20The Offord Centre for Child StudiesHands Across Canada

• basic tools are data collection forms such as questionnaires, standardizedtests and scales• requires careful data collection, analysis and statistical methods• findings can be used to predict similar outcomes in similar circumstancesQualitative research• uses descriptive explanations and narrative to describe phenomena thatcannot be measured easily• provides context of a particular place and time• may not be comparable or generalized to other or larger groups of people• basic tools: interviews, open-ended questions, focus groups andobservation notes• gives respondents more freedom to bring up points or issues in their ownwords which are important to them but may not have been considered bythe program developer or evaluators• especially helpful for development or pilot-testing of programs, processesor questionnairesContent analysis• technique for systematically pulling meaningful data from qualitative datasources,• e.g., volunteer observations, interview notes, focus groupdiscussions, personal narratives and participants’ comments.• readily available computer software packages systematically searchout meaning beyond the easily recognizable anecdote• manual analysis possible with smaller amounts of dataQuantitative andqualitative research canboth be used to goodeffect in communityprogram evaluationMethods:1) look for topics and patterns to emerge from the material itselfOR2) develop a list of topics or themes first, then code and sort answersaccordingly, adjusting topics to suit the information and level ofdetail wanted.Ideally, two or more people discuss the probable topics then analyze dataindependently and compare results to overcome individual bias about importance andprioritiesEvaluation Toolkit for Community Youth Programmers Step 3 21

Time-dependent data collectionSome program effectsmay be short lived,some more long lasting,others may be delayedor perhaps evenincrease over time.• techniques for measuring the extent of change in a participant• assesses where a participant rates in terms of objectives at the end of aprogram• commonly a pre- and post-test method, where programmers obtain thesame type of data about participants at the start of a program and againafter it ends• may include additional data collection, periodically during long programsand/or additional follow-up,• may also assess further characteristics of participants which could alsohave shown change• data collected with a standard set of questions in an interview, a customquestionnaire, standardized scale or a combination of all threeFormative and summative methods• complement to pre-post-test methods• formative methods compile data over time to show development• summative evaluation is a one-time test of the final outcomeFollow up evaluationThink carefully aboutthe type and timingof effects which youcan expect fromyour program, andthis will guide thetype of evaluationmethodology whichwill work best for you.• evaluations that follow participants or gather data about them over alonger time frame after the end of the program, or over several sessions• likely to increase evaluation knowledge• can be difficult and costly to trace and re-establish contact withparticipants• try to establish a reliable method for contacting people at the outsetSome evidence suggests that effects of programs are generally short lived unless theyare somehow reinforced by booster programs or other interventions.22The Offord Centre for Child StudiesHands Across Canada

3.2 Seek out data sourcesData assessmentIt is helpful to prepare a plan showing the process of evaluation and means of datacollection to determine what you have and what you need. (See R5 & R6 in Appendix)Assess potential data in terms of:UsefulnessWhat information will be provided?How will the data be used to answer an evaluation question?Can the data be used to corroborate or strengthen data from othersources or fill gaps?Ease of collectionWhat do you have to do to get the data?Is the data in a form that makes it easy to extract the necessaryinformation?CostIs there a cost for obtaining, handling or transforming data?e.g., amalgamating records from multiple sites, photocopying,electronic data entry, programming?Sensitivity Will permission be needed to obtain data or safeguards be needed foruse by volunteers/staff?CredibilityIs the data accurate and seen as credible by others, especiallystakeholders?Usefulness of dataConsider what can be learned from existing program records:• attendance records of participants recorded by individual name provide:• overall sustained participation rates,• consistency and duration of attendance,• data that should be associated with levels of effectiveness• registration records provide• demographic information for comparison purposes and targetting• contact information for follow up.Evaluation Toolkit for Community Youth Programmers Step 3 23

• intake interviews provide• baseline information about extent of knowledge, achievementprior to program participation for comparison with post-test.• intake questionnaires asking why participant joined or how they learnedabout the program can provide data for comparison with non-participants.• attendance records correlated with data about age and gender• can show overall patterns among sessions and multiple sites aboutage groups attracted or not attracted to the programNeed examples?Examples of ongoing or periodic record-keeping• attendance records by full name• checklists of participation in activities• checklists, badges, tests, lists of program achievement , e.g., skill levels• volunteer notes from observations, with systematic categories, e.g., social skills,“following instructions”, “volunteering assistance”, “conflicts with peers”• tutors’ workbooks, logs• notes of regular volunteer/staff meetings to discuss observations3.3 Make data collection accessibleand convenient for collectorsUsefulness of records for data collection depends not only on planning but on theircompleteness, consistency and accuracy.Simplify the collection process• ensure as much consistency as possible,• provide sufficient time and staff to allow timely, on-site recording of data,observations etc.The old computeradage of ‘garbagein, garbage out’applies equally todata collection andevaluation.Use trained collectors• provide training for special skills, e.g., interviewing techniques, facilitatingfocus groups,• or hire experienced volunteers/personnel24The Offord Centre for Child StudiesHands Across Canada

Ask for regular feedback• what is working well• what procedures need to be modified to ensure good data collection.Monitor data collection• check regularly that data is being collected in the form requested• make timely adjustments if difficulties or gaps are apparentExternal data sources• What data would be helpful?• to evaluate broader program objectives or• to add to or confirm data from internal sources, e.g., family,classroom, neighbourhood characteristicsA welcomingatmosphere, timeto chat with parentsafter the program,approachable, multicultural,multi-lingualvolunteers and staff,a suggestion box,telephone ‘progress’chats, inclusiveprogramming, outingsand get-togethers withparents are all methodsused by some programsto include parents andmake it easier to includethem in programevaluation.• Assess the factors mentioned previously (usefulness, ease of collection,cost, sensitivity and credibility)• Regarding ease of collection, consider convenience for informants as wellas data collectors• timing of interviews, surveys, data pick-up, venues, permissionfrom parents and authorization of superiors, other authorities(principal, social agencies.) Agencies that depend on teachers tocomplete questionnaires find data collection more difficult if thetiming coincides with periods when teachers are busy with examsand marking.• police or other municipal bodies may have useful data to useas baselines for neighbourhood profiles but it may be timeconsumingto find the right channels and obtain access.Evaluation Toolkit for Community Youth Programmers Step 3 25

Need examples?Examples of possible external data and uses• school attendance records could show program effects re. increased schoolengagement leading to improved attendance, fewer skipped classes• school promotion/report data could show improved academic performance, possiblyrelated to, e.g., literacy, homework completion, tutoring programs• school intake data could show community rates of transiency for context with programsbuilding school-community links• community data e.g., fire and police calls for rates of community dysfunction,disturbance related to community-building, youth pro-social programs• tenants’ association, condominium association data, e.g., incidents of vandalism asabove• public health data re. incidence rates of problems, health concerns related to specificinitiatives, e.g., teen smoking, eating disorders, teen pregnancy• national youth survey data – context/comparison figures on a variety of characteristicsrelated to family functioning, recreation, substance use, education, employment etc.(Ontario Child Health Survey (OCHS), National Longitudinal Survey of Children andYouth (NLSCY).)• census data can provide aggregate information on neighbourhoods26The Offord Centre for Child StudiesHands Across Canada

3.4 Gain participant/parentalsupport & protect confidentialityProvide appropriate information• at the beginning of a program or each session• explain what information will be collected (and questionnaires they and/or their children will be asked to complete)• that activities will be monitored as part of program evaluation• information you wish to collect outside the program about individuals,(type of information, informants, purpose, eventual destination of data )Obtain consent• tailor explanations or translate written material so it is easily understoodby all participants and/or families.• Check requirements for data collection:• schools may require written parental consent before releasing anydata or permitting teachers to comment on students• the active support of leaders in an organization, such as schoolprincipals, and of opinion leaders in a community can increasecooperation for data collection• parental cooperation is more likely if parents know what to expectand are assured that their opinions and data about their familieswill be kept confidential and/or anonymous.Protect confidentiality• develop a system to provide security and protect confidentiality forany data collected e.g. password protection on computers, locked filingcabinets (See Step 6)Evaluation Toolkit for Community Youth Programmers Step 3 27

Developing New DataCollection ToolsSTEP4Step 3 discussed methodology and means of collecting data relevant to indicators of change inorder to answer questions about program implementation and effects.Step 4 provides ideas for developing new data collection tools, primarily questionnaires, toprovide specific information for the evaluation.Review the link between data collection and objectivesDetermine relevant data collection toolsLook for model questionnairesModify existing questions to suit respondentsWrite new questions (See guidelines)Pre-test questionnaires4.1 Review the link between datacollection and objectivesFirst, take time to review the purpose of the evaluation.• What precise information is necessary to evaluate program results:• about participants• families• volunteers• community features?• Is new data needed to fill a gap in existing sources?• Do you want to obtain similar data from more than one source?29

Need an example?Data from more than one source provide more information about participants and possibleprogram effects. Parents may observe that children are showing stronger social skills andless conflict behaviour at home after taking part in a program. Your confidence in this resultwould increase if teachers, who did not necessarily know about the program, reported similarimprovements in behaviour at school.• Decide how you can best obtain the information with your resources andthe time available, keeping in mind timing, confidentiality, likelihood ofcooperation from sources• A chart to show the need for information, sources and methods will helppoint to gaps in the process (See R5 & R6 in Appendix.)4.2 Determine relevant datacollection toolsDistinguish between the need for standardized instruments and non-standardizeddata collection tools such as questionnaires and scales.Standardized instruments have beendeveloped by specialists to measure psychosocialconcepts or qualities, e.g., self-esteem,self-control, sense of belonging, senseof community, family functioning. Theyhave been tested in various versions, someabbreviated, in different languages, settingsand populations to determine their level ofreliability and validity. Using a standardizedinstrument conveys some advantages:you can be confident of its findings andmay compare them to findings in othercomparable populations.Non-standardized data collection tools,especially questionnaires, are created tomeasure specific effects of a program, e.g.,homework completion, use of spare time,reading habits, recreational skills. Theymay be borrowed or adapted from otherprograms, with due credit given, but have noestablished level of reliability or validity.No questionnaire is perfect, i.e., every question understood and answered as intendedby every informant to provide a completely accurate picture of reality in a givensituation. There will always be a certain degree of error in results.Non-standardized questionnaires have an unknown degree of error and thus resultsbased on them from different times or venues should be compared with caution. Theycan only suggest similarities or differences among programs or groups and should besupported by other evidence.30The Offord Centre for Child StudiesHands Across Canada

Using standardized instrumentsIf you are interested in investigating effects on the psychological well-being ofparticipants, it may be possible to use one of many standardized questionnairesor instruments used for academic research. Using questionnaires that have beendeveloped, tested and critiqued by others can save considerable time. Online databases are good sources of information about tests and instruments, many availablethrough university or public libraries. Several published volumes also containcatalogues of tests and questionnaires. Most instruments are copyrighted. It isnecessary to contact authors and/or publishers for permissions to use all or part of astandardized instrument. Some may be ordered as kits for a price, others are free withproper attribution given to the author in a credit line on the form. Consider consultingsomeone experienced in using such instruments, e.g., in research settings such ascolleges and universities or larger social service agencies.Suitability for community programming• designed to suit specific populations and may not be transferable toother groups, e.g., adult to children, urban to rural, U.S. demographics toCanadian.• sometimes, questions or specific wording may not be entirely suitable forthe group or purpose• must be used precisely as written and formatted,• using certain questions and omitting others invalidates thereliability and validity of the instrument, making it unsuitable forcomparison purposes.• Some standardized tests, however, have been used for decades and provenin a variety of population groups.• It is possible to add extra questions to a standardized instrumentwithout compromising the instrument’s use for comparative purposes,if you analyze the instrument questions separately from the additionalquestions.• It may be helpful to borrow ideas and phrasing from questionnaires thatare not held in copyright, or use them and relevant articles as resourcesto help clarify your own ideas about concepts without claiming the samelevel of reliability for results.It is useful to do aliterature search byInternet for appropriatestandardizedinstruments and articlesabout their use. TheAmerican PsychologicalAssociation (APA)has a webpage ofinformation for peoplelooking for suitablestandardized tests.See http://www.apa.org/science/faqfindtests.html. See also:TestLink at http://www.ets.org/testcoll/index.html), Test Locator fromERIC/AE at http://www.ericfacility.net/ericdigests/ed385604.html and the Healthand PsychologicalInstruments (HAPI)database through alibrary site. Test Reviewsare online at theMental MeasurementsYearbook at: http://buros.unl.edu/buros/jsp/search.jsp.Evaluation Toolkit for Community Youth Programmers Step 4 31

4.3 Look for model questionnairesA variety of non-standardized questionnaires have been developed by agencies andcommunity groups, which can provide models or be used intact with permission.Questionnaires from community groups are likely to have a mix of question types tocollect quantitative and qualitative data.Advantages• already been tested and used• development work is done regarding• how much to ask• question wording• placement• questionnaire formatCharacteristics of good model questionnaires• contain a description about confidentiality• provide some explanation for the purpose and use of the questionnaire• easy to read and follow from question to question• reasonably consistent in format and layout of questions• give precise, comprehensible instructions• differentiate instructions clearly from questions (e.g., font size, type,boldface)• express appreciation for the respondent’s time and assistance• have easily understood, unambiguous questions• are appropriate for the purpose and the respondent in tone and content• unemotional, impersonal and reasonably unintrusive• provide opportunities for identifying date, collection place and codedidentification as needed• provide simple ways to minimize the amount of writing an informantneeds to do, e.g., tick boxes, lists of responses to circle• provide prompts to assist specific answers and avoid unclear responses(“yr/month/day” instead of “date”; birthdate instead of “age”)• use scales to identify the intensity of the response (e.g., a 5-point scalewith each point labeled)32The Offord Centre for Child StudiesHands Across Canada

SuitabilityEven in good model questionnaires, some questions may not be suitable as they areworded.• omit unnecessary or unsuitable questions unless the intention is tocompare the results of the model questionnaire as a whole to your own• make every effort to contact the originator of the questionnaire and obtainconsent to use itDetermine the utilityof data to be gatheredfrom a question beforeincluding it in the finalquestionnaire.4.4 Modify existing questions to suitrespondentsConsider both the ability and willingness of respondents to answer the questions, interms of, e.g.,• language, reading level and specialized vocabulary• knowledge• cultural experience, attitudes• community contextIt is importantthat questions areinterpreted the sameway by all respondents,so they are allanswering the samequestion—the one youmeant.Language and vocabularyTranslationWhen language comprehension is a problem and a written questionnaire is necessary:• translation of questions into the most familiar language for informants is agood option but may introduce new errors.• a helpful check is to have a second person translate questions back into thefirst language (without seeing the original)• compare results of re-translation and discuss implications of anydifferences.Other optionsOptions followed by youth programs• have an oral interview conducted by a staff person or volunteer whospeaks the same languageEvaluation Toolkit for Community Youth Programmers Step 4 33

• children often act as translators for parents in community programs• someone else may be able to explain the questionnaire at the time it isbeing completed• errors may creep in because there is no way of ensuring the explanationsare sufficiently correct or completeInappropriate vocabularyInappropriate language may have a negative effect on respondents’ willingness toanswer a question• possible problems in vocabulary include:• jargon or specialized meanings• convoluted phrases open to more than one interpretation• juvenile language that undermines the serious intent of thequestions• culturally offensive termsGenerally a grade 4-6 vocabulary level is considered simple enough without seemingtoo juvenile. (Available software programs can determine vocabulary level.)KnowledgeClarify whether aquestion should askabout attitudes orabout behaviourand which is morelikely to provide themost meaningfulinformation. Examplesof behavioural changecan be a strongindicator of programeffectiveness. Reportingon behaviour mayalso be less subjectiveor influenced by arespondent’s supportfor the program.Questions aboutobserved behaviourcan be asked of severalsources to build acomposite picture andstrengthen data.View each question from the viewpoint of the respondent, e.g., youth participating inthe program, parent, teacher, volunteer or tutor.• Can each question be answered easily, without much mulling over andsecond-guessing?• Is it equally easy for all informants to answer the question?• Does the respondent have the knowledge to answer the question?• Does the question ask about a concept that needs a definition or someexample?• Does the question ask about something outside the informant’s sphere ofexperience, e.g., asking parents about how children are behaving in theclassroom or teachers about conditions or behaviour in a child’s home?• Is the question culturally appropriate or sensitive to cultural, religious orethnic differences? For example, the terms synagogue, mosque or simply“religious services” might be included to expand a question about churchattendance.• Does the question ask for secondhand information?34The Offord Centre for Child StudiesHands Across Canada

Need an example?For example, “How did your child/student enjoy the program?” really means,“How do you think they enjoyed the program?” The reliability of the response could varydepending on the parent or teacher’s level of knowledge and relationship with the child.Firsthand information can be obtained by reworking the question, e.g., “Has your child/student ever expressed any feelings about the program? Yes/ no If yes, what did they say?”Community contextWords may assume different meanings in different contexts, providing stumblingblocks to confuse data analysis. Specific terms may be unknown in a differentjurisdiction. It is helpful to check terms with a similar audience to the informants andconvert them to local usage. Obvious examples are U.S. compared to Canadian usage,in currency, terms such as food stamps vs. food banks or government programs suchas the U.S. National School Lunch Program.Questions should be unambiguous. Less obvious problems may be arise fromquestions that are inappropriate for the local context. Asking a general or open-endedquestion allows a fuller range of responses that can be classified later. Focus groups ormore general discussions with respondents before writing a questionnaire also helpsdetermine the full range of options to be considered.Need an example?For example, a researcher inquiring about relationships with non-school activities andviolence asked youth in a particular area about membership in non-school clubs. Since therewere few non-school clubs available in the area, the number of positive responses led theresearcher to suspect that some youth had considered gangs to be non-school clubs! It canbe useful to add examples. In the above case, adding, “e.g., Boy Scouts, 4-H, Boys and Girlsclubs” might have given the researcher more confidence in the data. What if youth also haveaccess to athletic clubs for golf, tennis, swimming, curling or gymnastics? Is membership inthose clubs any different from membership in the YM/YWCA, which respondents would likelynot consider a “club”? Beginning with a more general question, then asking for specifics isanother way of avoiding ambiguity.For example, “Do you belong to a non-school group or club where you spend time outsideschool? If yes, please specify what it is.”______________________________________.Evaluation Toolkit for Community Youth Programmers Step 4 35

4.5 Guidelines for writing newquestions:Question typesOpen-ended questions• allows the informant to answer freely• useful when you cannot predict all possible responses and want to knowthe range of available options• more time-consuming for the respondent, who may then choose to skipthem• handwritten responses more time-consuming to record and analyse• may be difficult to decipher or ambiguousClosed-ended questions• gives a set number of responses to choose from• useful when the question writer knows what range of responses isavailable• adding an “other” category with a “please specify”, takes care of unforeseenoptions but is more time-consuming to code.Response optionsChecklists• include instructions to “check ONLY one”, or “check all that apply”.• if the latter, any box left blank by accident would be coded as a “NO”response• avoid that problem by providing options for every item, e.g, yes, no; never,sometimes, often, alwaysScalesSince devising good scales is more difficult than it seems, try to find scales that havebeen used somewhere else previously.• providing a scale for a response e.g., from 1-5 or from poor to excellent, iscommon36The Offord Centre for Child StudiesHands Across Canada

• options need to be mutually exclusive and not confusing• options cover the full range of likely replies and are evenly spaced• generally include an odd number of options, (often 5 or 7 points) to allowfor a neutral mid-point• omitting a neutral midpoint forces respondents to state an opinion by notgiving them the “I don’t know” option• a Likert Scale asks for an opinion about a statement, e.g, “Please read thestatements below and then circle the one number on the right that bestdescribes how much you agree or disagree with each statement.”• keep response scales in the same order from question to question (1-5)not (5-1), e.g., “I feel comfortable with my volunteer tutor.” (1-5 from totaldisagreement to total agreement) and “I am afraid of being bullied by olderchildren in the program.” (1-5 from total disagreement to total agreement)(When coding the latter question, keep the scoring in the same positivedirection as the first question by reversing the numbers. See Step 6.)• positive statements are usually preferable unless a negative will give morerelevant dataProvide sufficient information to informants• provide definitions of standard terms (e.g., wages, household income) onthe questionnaire or• have staff give definitions while distributing questionnaire• avoid defining words related to opinions or feelings• informants may feel inhibited or restricted to a particular range ofresponses• provide examples instead of definitions to suggest types of feelings oropinionsAssumptions• review questions to look for hidden assumptions• discuss the meaning with others to suggest other interpretations• informants can misunderstand and misconstrue questions that writersthink are obvious• e.g., Does the question specify a clear time frame or frequency, e.g., “in thepast 12 months”, “this school year?”Evaluation Toolkit for Community Youth Programmers Step 4 37

Neutral questionsAvoid questions that anticipate one option more than another or do not provide a fullrange of options. For example, asking a child, “How much did you like being in theprogram, a lot or a little?” provides no possible response for someone who didn’t likeit at all. Giving more options, with a broader range, provides more precise, thereforemore meaningful data.One concept per question• avoid multiple ideas in a question (often containing “and”), where arespondent can answer “yes” to one part of the question and “no” toanother part• divide such questions into two separate questions that can be coded andcompared separatelyFollow-up questions• consider following up questions about attitudes or opinions by askingabout the intensity with which they are held.• link opinion to behaviour to help evaluate the strength of people’sresponsesNeed an example?A question asking how much someone liked reading a certain book, might have a follow-up,“Did you like the book well enough to recommend it to someone else?” or even better, ”Didyou recommend it to someone else afterwards?” or “Have you looked for other books sincethen by the same author?”Verifying responsesIt is helpful to verify the consistency or accuracy of responses by asking more than onequestion about a certain attitude or behaviour in a different way, where space allows.For example, it wouldlikely be more helpfulfor data analysis toask an open-endedquestion, “How manytimes did you attendthe program thissession?” insteadof giving a range ofoptions such as “ 10.”Encourage precision• avoid grouping data or collapsing categories for essential questions sinceyou may need more precise data later on.• there is a trade-off between missing opportunities for analysis and askingfor so much precision that informants can’t or won’t want to provide it• pretesting can help determine the best balance for each questionnaire.38The Offord Centre for Child StudiesHands Across Canada

Personal questionsOne major exception to precision concerns certain personal data. Respondents mayskip questions they find too personal and possibly not complete the questionnaire.There may be less intrusive ways of finding out the information, e.g., on the subjectof income, providing a checklist of options with a fairly wide range (>$5,000.) willimprove response rate.• Other information can stand in for income, such as educational level andhome ownership, which are good indicators of income and less likely tobother respondents.The ‘not applicable’ categoryA full range of responses may include “not applicable.” Questions that cannot beanswered because they do not apply to respondents can be irritating and possiblyinfluence their attitude towards completing the questionnaire.Skipped responses (non-responses) may have been skipped on purpose or by accident,creating ambiguity for analysis Think about all possible interpretations when writingquestions and as needed add, “Please explain.” or ask a filter question.Need an example?Consider the question, “Does your child enjoy reading at the library?” Adding an option“not applicable” would cover respondents whose children do not read at the library. But arespondent might also check it to mean “I don’t know”, “my child does not use a library” oreven “my child does not know how to read.”Filter questions• a preliminary question to filter out respondents who lack some knowledgeor experience necessary to answer a question• e.g., “Do you like playing checkers?” implies that the respondent knowswhat checkers is and has played it. Ask respondents first, “Have you everplayed checkers?” and have those who answer “NO” skip to anotherquestion.Evaluation Toolkit for Community Youth Programmers Step 4 39

Skip patterns• instruction to skip a response after a question filters out some respondents• can be confusing and distracting• consider using arrows if there are several skips• best placement is directly after the response, to prevent the informantfrom overlooking the instruction• phrase the skip as a positive statement, e.g.,1. Do you take dance lessons?1 Yes ___2 No ___ go to question 3instead of “If NO, do not answer question 2”Make forms easy to complete• try for consistency in typeface, layout of similar questions• more white space on forms, especially between questions, increasesreadability and is more inviting for respondents• reduce the amount of writing respondents need to do, e.g., provide checkboxes, circles or ( )• provide sufficient space for written answers• create logical order and flow to questions• provide instructions for each question or each different type of question• make instructions more noticeable with a different font or boldface• keep instructions and questions together, not on separate pages• start each sentence on a new line for easy readabilityProvide information about confidentiality (also See Step 5)Explain the purpose for collecting data on the form or in a verbal introduction(preferably both). Tell informants about confidentiality: that their responses willnot be reported individually or with their names. It may be necessary to reassureparticipants that the program welcomes both positive and critical comments in orderto identify gaps in service and strengthen programming. If personal questions arebeing asked, the need for confidentiality, and confidence that it will be maintained,are especially important. Some projects may require more safeguards aroundconfidentiality than others. Since programs that want to follow individuals overtime within a program or over various sessions, need to be able to identify the40The Offord Centre for Child StudiesHands Across Canada

participants who provided data, it is important to have some identifier, either nameor another unique identifier like gender coupled with date of birth. Check that theidentifier is unique. Data may be kept anonymous to all but one person who tags eachquestionnaire with an identification code instead of a name, records it on a master list,then links it to other data records by the same code. The master list should be keptsecure and accessible to only one person on the evaluation team.Even if names are collected, information provided in reports should not be traceableto an individual either by name or by specific identifying information. A personalstatement such as a testimonial or quotation could be published with permission fromthe individual (and/or parents). However, families and programs may not anticipateall the possible negative consequences of releasing personal information that isdistributed publicly.4.6 Pre-test questionnaires• a ‘practice run’ using a few people similar to informants in language,culture and education• before pre-test, practise coding and recording data by preparing a dummytable of information to ensure neither gaps nor unnecessary questions.(See Step 5)• watch for any hesitation and ask about the respondent’s reaction• after pre-test, discuss the informant’s interpretation of questions toresolve possible misunderstandingEvaluation Toolkit for Community Youth Programmers Step 4 41

Data CollectionSTEP5Step 4 discussed the difference between standardized and non-standardized instruments, howto evaluate and modify existing data collection tools and ideas for developing new questionsStep 5 provides ideas to improve the quality of information gathered for evaluation purposes.Ensure access to necessary dataPromote and explain data collectionProvide appropriate training to data collectorsProvide adequate time and resources for data collectionConduct data collection (Tips on focus groups, observations,interviews, questionnaires)Data collection is the backbone of the evaluation process. Data collectors needadequate information, resources, time, supervision and support.5.1 Ensure access to necessary dataAdvance preparation• confirm the feasibility of getting the data required• streamline the process• obtaining permissions• confirming or adjusting timetables• fostering co-operation• providing information on importance of the evaluation and itspotential benefitsIf programs involveyouth in service workin the community,volunteering orworking at otheragencies, schools etc.,data collection will beacross organizations.43

Consultation with organizations providing data• consult appropriate level in organizational hierarchy (e.g., managers,school principals) to ensure buy-in to data collection (e.g., a letter ofsupport, staff meeting)• review circumstances and type of data to be collected• address concerns about confidentiality or the ethics of disclosure• utilize relevant expertiseNeed an example?The school principal can shed light on what teachers might be expected to observe duringrecess, which could affect the type of data they could provide.5.2 Promote and explain datacollectionModel professional, objective attitudes toward evaluation and systematic methods ofinquiry.Promote the need for data collectors to• encourage open disclosure on the part of informants• be objective and neutralData collectors within a program may be tempted to filter negative informationfor management consumption or, more likely, inadvertently influence informantstowards more positive answers by their manner and the way they phrase comments.Participants, parents and community residents may be hesitant to discuss programshortcomings openly, out of loyalty and friendship with staff. They too need tounderstand that criticism, rather than jeopardizing the future of a program, is morelikely to help sustain and improve it.44The Offord Centre for Child StudiesHands Across Canada

5.3 Provide appropriate training todata collectors• awareness of confidentiality policy and procedures• commitment to the value of data collection for evaluation• knowledge to aid understanding of respondents (cultural, socioeconomic,scheduling)• understanding role of specific data in the evaluation (purpose, focus)Need an example?For example, if programs are evaluating change at the level of the individual youth, it will benecessary to link his/her specific pre-test and post-test data. Programs may also want to linkdata about each participant’s attendance and perhaps level of participation and achievementin the program.• familiarity with the final version of forms to be used and concerns thatarose during retesting (See Step 4)• written instructions appropriate for different collection techniques,stressing importance of full, accurate records, correct identification ofresponse data,• e.g., when and how to record identity of respondent• stress on the importance of consistent explanations, methods across sites,sessions• practice sessions and role-plays are useful:• interviewing practice,• explaining and completing forms and questionnaires• making observations and using observation logs.• access to timely assistance from program supervisor for any questions orproblemsSince consistency ofdata collection is vitalto evaluation, staff needadequate informationand training to collectdata in the same wayacross sites or sessions,even if there are onlytwo data collectionperiods - pre- andpost-tests. Consistentmethods of datacollection and handlingcan produce morereliable and credibleresults. Qualitativetechniques also needto be conducted ina systematic andthoughtful way toprovide accurateresults.Evaluation Toolkit for Community Youth Programmers Step 5 45

5.4 Provide adequate time andresourcesStaff and resources forevaluation are oftenlimited, especially incommunity programs.Some programs budget5-10% of funds forevaluation, whichallows additionalstaff to be hired orcontracted to help withdata collection tools,data collection, analysisand report preparationPrograms need to build in time for record-keeping before or after each activity sessionor have enough volunteers to allow records to be maintained in real-time insteadof recalled after the fact. Naturally, records and observations are likely to be morecomplete if they are not rushedThe amount of time necessary for data collection depends on:• the collection method (review of existing written sources, observations,focus groups, personal interviews or questionnaires)• whether it is conducted in person, self-completed (completed by therespondent), completed on-site or off-site, online, by telephone or by mail• time can range from at best a few hours to many hours over a period ofweeks for each round of data collection.5.5 Conduct data collection(Tips on focus groups, observations, interviews, questionnaires and review ofwritten sources)Focus groups• a gathering of people who share a common interest or experience andengage in a discussion about it guided by a trained or experiencedfacilitator• focus groups are less structured than interviews• telephone conference calls can be used if a meeting is not possible• participants should be assured of confidentiality at the outsetFacilitator’s roleFacilitator has a general script with a few specific open-ended questions• follows up interesting side topics that may provide useful information• training needed because individuals can affect group dynamics, swayingpeople by the force of their personality to agree, disagree or keep theiropinions quiet46The Offord Centre for Child StudiesHands Across Canada

• facilitators do not express their own opinions or engage in debate• facilitator tries to ensure a balanced, comprehensive discussion• introduces topics• asks follow-up questions• gauges the degree of agreement and dissent• probes the strength of people’s opinions• limits discussion where necessary• tries to bring everyone into the discussion to air all viewpoints• brings the focus group to a close at allotted time• summarizes main points• asks about agreement or clarification of those points and any otherissues not raisedTiming• usually 1-2 hours in length• refreshments commonly provided• childcare for siblings is helpful• allows parents to talk freely without sense of urgency to leave• focus group of parents is sometimes held while children are participatingin a program session elsewhereRecording discussionSince it can be difficultto find a convenienttime and venue fora focus group, somecommunity programshave successfullycombined focus groupsessions with specialevents like luncheons,potluck dinners orgame days• facilitator does not take notes• programs preferably have one or more staff people take notes in detail andthen compare notes afterwards• no names should be attached to comments in notes• confidentiality observed in follow-up reports as well• focus group discussion can also be videotaped or recorded, with theconsent of participants• videotapes of discussions kept in-house• analyzed without attaching names or other identification tocommentsEvaluation Toolkit for Community Youth Programmers Step 5 47

ObservationsAll observers should have a shared understanding of programming and context beforeconducting observations.Levels of observationCommunity programs use various types of observation for evaluation.• external observers use qualitative techniques• view program activities on a single occasion or periodically overthe course of the session• record subjective highlights• choose a case study• develop a narrative of activities, or• monitor agreed-upon indicators through observation• internal observers: program leaders, staff or program volunteers observeactivity sessions on a more regular basis. They focus their observations onparticular aspects, e.g.,• consistency of program operations, accordance with planning• participant characteristics, participation, interaction and successof curriculum activities,• varying skill levels, development and observable effects over timeNeed more detail?Data collection can be improved by providing written instructions so that all observersunderstand what to focus on, e.g.,• program activities and resources• suitability of venue• safety• knowledge and quality of communication• volunteer-participant interaction• peer interaction, bullying• adherence to specific curriculum, procedures• specific indicators of effect48The Offord Centre for Child StudiesHands Across Canada

Advantages• observations can prompt and inform questions to be posed in subsequentinterviews or questionnaires, and• identify aspects and patterns that are unrecognized or unappreciated bystaff, or need further explorationObserver effectsThe act of observing affects what is being observed. Effects vary depending on whoor what is being observed and how the observation is carried out. Effects may beshort-lived, particularly when those being observed are children who are busy andinterested in something other than the observer. Repeated observation becomesroutine and so unremarkable.Familiarity and objectivityInternal observers are less likely to disrupt programming. They can differentiatebetween unusual situations and normal behaviour more easily than external observersand may bring an additional layer of knowledge to observations of individualbehaviour and relationships. However, their greater familiarity with personnel andparticipants may make objectivity difficult.Periodic, low-keyand/or unobtrusiveobservations providea more representativeview of a program thansingle observations andallow observers to seechanges that may beoverlooked on a dailybasis.Observers may want to talk to volunteers or staff after the observation period todiscuss their ideas, ask follow up questions and learn about the representativenessof that particular session. When volunteers or program leaders are observers, timelyfeedback and objective discussion with peers in the program about their observationsis important.Clear guidelines for observations will help direct feedback away from personaldiscussion.ConsistencyTo ensure consistency of observation techniques and perceptions among differentobservers, programs can hold trial-runs with role-play scenarios, then havethem compare and discuss their own observations. To improve the reliabilityof observations, different observers should try to reach consensus, minimizingdifferences in what they report and their interpretation of events.Need an example?For example, in an after-school program, if a child jumps up and shouts out for thevolunteer’s attention, one observer might note that as disruptive, another might interpret itpositively as excitement and engagement and a third might see it as normal behaviour notworth recording.Evaluation Toolkit for Community Youth Programmers Step 5 49

Recording observationsExpand notes andfill in details as soonas possible after theobservation.Clearly identify thetime, date, location,session, name ofobserver and names ofsources.Observers may not want to make detailed notes during the observation, preferring towatch unobtrusively and be fully engaged with what is going on during the session. Anabbreviated checklist of what should be observed can be helpful to:• focus on the evaluation’s areas of concern• allow quick note-taking either during or immediately after the session• provide a consistent format for a volunteer logInterviewsInterviews can be conducted in person or by telephone. An interview may have thefeel of a conversation but should follow a prepared script or list of questions.AdvantagesSoftware for monitoringtelephone interviewingand coding data(CATI ComputerassistedTelephoneInterviewing) isavailable for largeprojects but probablyneither necessary norcost-effective for mostcommunity programs.• interviews provide more flexibility than questionnaires for wide-rangingdiscussion• can explore opinions, changes in attitudes, feelings about experiences andexplanations about behaviour• interviewers can ask follow-up questions or prompt informants to providemore details or clarification• interviewers have the advantage of spontaneity in following importantunanticipated comments; however,• should resist adding new questions or varying wordingInconsistent data from one interview to another cannot be easily compared oranalyzedFactors to consider• language• literacy level• culture• gender of interviewer vis-à-vis informant• setting for the interview• timing• need for childcare and• expected length of the interview• attitudes of the interviewer50The Offord Centre for Child StudiesHands Across Canada

The above factors, which relate to either interviewer or informant, can affect thesuccess of an interview. It calls for careful planning to get the most information fromthe interview process.Role of the interviewerIntroductions• interviewer politely introduces herself/himself,• explains the general purpose of the evaluation• how the interview fits into the process,• the time it will take, and any other circumstances such as inducements toparticipate ( e.g., payment, reimbursement for transportation costs, giftsor extras like entries for a prize draw)• reiterates the confidential nature of the interview, (that names will not beused with any comments or quotations without prior permission)Recording interviews• ask for permission before tape recording a conversation• review promptly to create a full transcript of replies• issues of security and confidentiality are as applicable torecordings of interviews as to other data formatsTaking notesIf an informant prefers not to be recorded, trained interviewers:• write down responses verbatim, word for word• write neatly• carefully link responses to questions so the responses can be readand correctly analyzed• review notes during pauses and immediately after the interview• look for incomplete or illegible notes, while the informant is stillpresent and able to clear up problemsEnsure there is sufficient space on interview forms for complete responses oradditional work pages.Evaluation Toolkit for Community Youth Programmers Step 5 51

Guiding the interviewWhen informants givea very brief answer toopen-ended questions,interviewers can probeby saying, e.g., “Isthere anything else?”or “Could you explainthat a bit more?” beingcareful to use thesame wording for eachinformant.Interviewers need to guide responses to open-ended questions (questions withouta set of optional responses)• to give each interviewee an equal opportunity to say what isimportant to them without interruption• clarify ambiguity• resolve contradictions• probe for thoughtful not ill-considered answers• ensure consistency in probing questionsNeed an example?It’s also the responsibility of the interviewer to look out for contradictory or confusinganswers and probe to resolve misunderstanding. They may read back a response they havewritten down and say, “Did I write that down correctly?” in order to try to eliminate confusionarising from a response. Interviewers have to avoid being overly friendly since there is adanger that informants will try to please by telling the interviewer what they think theinterviewer wants to hear.Voice and body languageInterview questions are ideally simple, unambiguous, and unbiased (not morelikely to lead to one particular response than to another.)However, an interviewer can affect the neutrality of questions by• tone of voice,• inflections or• mannerismsNeed an example?When reading a list of responses, for instance, a trained interviewer avoids emphasizing oneoption over another, by voice, facial expression (smiling, frowning or eye-rolling) or evenbody language (leaning forward for what they think might be more acceptable options.)Interviewers should look attentive and listen closely to responses, but avoid giving anyfeedback on the validity of a response.52The Offord Centre for Child StudiesHands Across Canada

QuestionnairesQuestionnaires share with other data collection methods the need for an appropriateintroduction, on the form and preferably repeated orally to explain the place of thequestionnaire in evaluation, the confidentiality policy and the value to the programof full, honest responses. When carefully written and pretested, questionnairesare simple and easy methods of obtaining high-quality data. Questionnaires canbe completed by the informant (self-complete) or completed with assistance at theprogram, completed off-site and returned, mailed, or completed online.Need an example?Online questionnaires are gaining in popularity and are easy to complete and return for thosewho can access and use the technology. They may also include built-in procedures that checkresponses and refuse inappropriate ones (e.g., checking more than one option) or promptfor missing values. The U.S. Scouting movement and other large organizations use onlinequestionnaires in evaluation.For pre- and post- test data collection• arrange consistent administration of questionnaires to all participants• absent participants should have pre-test as close to the original time aspossible• for post-tests, anticipate timing and venue to encourage completion ofquestionnaires• make arrangements to obtain questionnaires not completed or returnedNeed an example?Youth programs may have some special activities, celebrations or graduation ceremonieson the last day of a program, which makes it more problematic to administer a post-testquestionnaire. Some programs either administer post-tests before the last day or at thebeginning of the last day before any social activities. Sometimes rewards are offered forreturned questionnaires such as entry into a lottery to win a prize.Evaluation Toolkit for Community Youth Programmers Step 5 53

Completion ratesCompletion rates vary depending on the method of administering questionnaires and,depending on circumstances, also the length and complexity of the questionnaire.• have clear, consistent procedures for tracking questionnaire completionstatus, e.g., a tracking chart for individual questionnaires and variousstages of follow-upIt is important to try to get as high a response rate as possible. If a large percentage ofinformants do not return questionnaires, the question arises whether respondents andnon-respondents differ in some appreciable ways, which would bias the survey results.Need an example?Particularly forpre- and post-tests orquestionnaires, it isimportant to obtain ahigh completion ratefor matched pre- andpost-tests to be ableto measure changesand possible programeffects over time.Some community groups have had success by handing out questionnaires to parents atspecial events such as program graduations. However, it can be only a small portion ofparents who attend such events. If those who attend are more committed to the programthan those who did not attend, their responses may not accurately reflect all the parents. It isimportant to try to obtain responses from as many respondents as possible and to considerhow non-respondents might differ from respondents—perhaps by circumstance, educationor background.Factors• individuals who know the project and people involved are more likely toreturn questionnaires• self-complete questionnaires handed out during ‘class time’ or similargatherings to be handed back immediately, have a high rate of return• questionnaires handed out as people are leaving a class or event have alower completion rateMailed questionnaires• relatively inexpensive way of surveying a large number of informants orsample a population such as community residents or youth• lowest rate of return initially• require additional time to obtain correct names and addresses• may require extensive follow-up to increase numbers54The Offord Centre for Child StudiesHands Across Canada

Need more detail?Recent research suggests that response rate is improved when the data collection procedureis perceived as professional. Survey questionnaires sent to specifically identified people inorganizations after a telephone call explaining the survey and alerting them to its arrivalresulted in considerably higher rates of return than average mailed surveys to personnel inorganizations. (Hagar, Wilson, Pollack and Rooney, 2003)There is a common protocol for conducting mail surveys called the Dillman Method, asfollows:.1) sending a professional-looking survey with an informative covering letter explaining thesurvey and importance of response and a stamped business reply envelope;2) sending a coloured, printed postcard after about 10 days, thanking people who replied,reminding others to reply and asking the person to contact the program if they need anew questionnaire;3) after another 3-4 weeks, sending a second mailing to non-respondents with a differentcoloured questionnaire and reply envelope; and following up with a phone call or emailreminders and/or questions about problems/concerns. This provides an opportunityto find out why people have not responded and determine if they are different in anysubstantial ways from respondents. A lower response rate is acceptable if there seems tobe no discernable difference between respondents and non-respondents.Phone surveysUnless prearranged, phone surveys using standard questions are problematic. They arehindered by the widespread use of answering machines, necessitating time-consumingcall-backs and follow up.Review of other written sourcesOther written sources may provide evaluation or comparison data related to indicatorsof program effects. These could include community data, e.g., crime statistics, publichealth data, or records (at a more micro level) such as fire calls in a tenant housingarea, school attendance figures and library usage. They may also include demographicdata and statistics from other surveys such as the National Longitudinal Survey ofChildren and Youth or provincial child health surveys.Evaluation Toolkit for Community Youth Programmers Step 5 55

Working through theDataSTEP6Step 5 provided ideas to improve the quality and quantity of data gathered for evaluationpurposes, including use of standardized and non-standardized instruments, qualitative andquantitative techniques.Step 6 talks about what to do with the raw data after it has been collected, how to deal with itphysically as well as how to uncover patterns related to program impact by simple statisticalor qualitative analysis. Many community programmers are knowledgeable about methodsof handling and analyzing data. The basic information and examples in this section will helpprovide instruction for junior or inexperienced staff and volunteers.Drawing conclusions about program effectsProtect and preserve the dataReview data for errorsOrganize qualitative data in relation to evaluation questionsCoding and data entryUsing statistics to describe dataKnowing about inferential statistics6.1 Drawing conclusions aboutprogram effectsIn community programs, it is becoming common to compare data about individualsfrom before and after the program. This can provide persuasive evidence about thequality and likely effects of that program on participants, but cannot prove that theintervention caused the effects. Some research into longer-term interventions hasproduced evidence suggesting a causal relationship because the effect increases withthe extent of participation, similar to what physicians would call a ‘dose-response’relationship. Longer programs that can collect data about the same individualsregularly over time have more likelihood of showing such a relationship.In evaluation studies,a greater range ofquestions can be askedand answered wheninformation is collectedabout individuals, thenanalyzed by groupingresponses intocategories.57

Even without making a claim for cause-and-effect, community programmers canlearn much about their programs by looking at the extent of change across individuals,comparing amounts of change and looking at the full range of results to findfrequency patterns, how often certain results are obtained, when and by whom.6.2 Protect and preserve the dataPrimary rules• keep the raw data, composed of notes, logs, checklists, questionnaires,interview reports, etc. intact and secure• make sure the secure place is recorded and known to more than one staffmember, with safeguards are in place to monitor access to the originaldata• produce an identification number for each original data collection form tomake it easier to identify data, while preserving the informant’s anonymity• work from copies of the original documents• when scanning or entering data to a computer, keep the original copyintact so that it will never be changed or overwritten• identify each working file to keep track of progress, corrections etc. andmake backupsTypes of identification• range from a simple sequential list of numbers (for computer entry addzeros to allow sufficient digits to accommodate the largest number)• to serial numbers identifying the particular form (focus group, pre-test,post-test questionnaire etc.)• numbers rather than combinations of letters and numbers are preferablefor computer analysis6.3 Review data for errorsSources of errorQuality of data is improved if errors can be corrected at the source, e.g. when selfcompletedquestionnaires are turned in during a program session.58The Offord Centre for Child StudiesHands Across Canada

Review forms immediately for obvious errors e.g.,• missing data• several boxes checked instead of one• illegible writingAsk the respondent to correct the problem.Other sources of error• inaccurate transcription of data• misreading replies and• keying errors when coding data into a data base or spread sheet foranalysisThe best strategy is to try to solve problems in the data before tabulating andanalyzing responses.6.4 Organize qualitative data inrelation to evaluation questionsThe prime motivation for data collection and analysis is to learn more about what isgoing on in programs so they can be improved. As in other elements of communityprogramming, it is important to set priorities for what can be accomplished withinbudget and resource limits.After reading through at least some of the qualitative data collected, start to thinkabout ways of relating the data to evaluation questions and indicators.Asking a few importantquestions of the data,and doing that analysiscarefully, is the best useof time and resourcesfor evaluationBecome familiar with the range of responses and ideas raised. Data will raisequestions that seem important to try to analyze, ideas that may be generalizableacross respondents, (extent to which responses reflect the whole group or a specificsubgroup.) By organizing and coding qualitative data, a researcher can compareresponses among sub-groups (e.g., by gender or other known demographic data) oramong responses and variables.• Looking first for information needed to answer questions raised in theplanning stage:• increases the speed of handling data and interpreting results• avoids the risk of being overwhelmed by reams of data, charts andtablesEvaluation Toolkit for Community Youth Programmers Step 6 59

• Develop priorities for which questions are most important to the originalintent of the evaluation.The disadvantage of a narrow focus is that you can fail to notice unexpected outcomes.Questions to ask while reviewing the data:1. What information do the responses give about what the evaluation was assessing?2. What ideas are common to responses?3. How can responses be grouped to say something meaningful about the program—meaningful to people in the program, people running the program and the broadercommunity?4. Do responses say anything surprising?5. Do responses from one question support or contradict those from other questions andfrom other data sources data.Working through qualitative data: An exampleA community recreation centre offering a recreation/homework program might askparents an open-ended question: “Why did you enroll your child in this program?”and obtain the following responses:a) to give him something to do instead of video games at homeb) to keep her out of mischief till I get home from workc) to relieve my mother who looks after the babyd) to learn English moree) access to a computer for schoolworkf) for homework helpg) to make sure he does his homework before dinnerh) to get more physical exercisei) the teacher suggested it to help with homeworkj) to make friendsk) he thought it would be fun instead of staying homel) for the sports activitiesm) an opportunity to socialize with other childrenn) to get her away from the televisionFirst, think about the reasons for asking this question. Perhaps programmersdeveloped the program to keep latch-key children off the streets while their parentsworked. They advertise the program as providing a positive alternative to hanging outon the street. Only response (b) specifically supports the contention that the program60The Offord Centre for Child StudiesHands Across Canada

helps latch-key children. Two other responses (c) and (g) might be categorized as‘parents wanting adult supervision for child,’ which is similar, but clearly thosechildren would not be out on the street without the program.Programmers want to assess the fit between what they offer and parents’ needs inorder to better serve the community. They also wish to find out what elements attractparticipants so they can emphasize those in advertising and presentations to increasetheir numbers.With those questions in mind, programmers decide how best to categorize the data tofocus the analysis.A category like ‘seeks productive activity for child’ would fit almost allthe responses but tells little about how to improve the fit with parents’needs or attract more children. Looking at the responses with thoseobjectives in mind, one might group and code the responses as 1. “toprovide academic help” (d,e,f,i), 2. “to provide supervision” (b,c,g), 3.“for outgoing social activity” (j,k,m,), 4. “to increase physical activity”(h,l). Responses (a) and (n) do not provide enough information to putthem in either of the last two categories though one might interpretthem as fitting in one or the other. They could go into a catch-allcategory 5. “other”, or “alternative to passive activity at home.”The level ofinterpretation to beused with the datais set by the qualityand amount of datacollected, as well asoriginal discussionswith stakeholdersabout what they wantto learn from theevaluation.An alternative, using less interpretation, is to collapse fewer responses, giving each aseparate coding number except for those with almost identical wording. That wouldmean collapsing only (f) and (i), ‘homework help’ and (j) and (m) ‘to make friends.’6.5 Coding and data entryMaking sense out of data entails counting responses and tabulating how frequentlyeach occurs, which requires data to be coded and organized.Coding• code individual responses in accordance with a master coding sheet to alloweasy computer identification, e.g., male = 1, female = 2; no = 0, yes =1.• responses already in the form of numbers, as for age or grade or numericscales (1-5 from least… to most…), are entered as is• blank responses are coded as well, to show where the informant failedor refused to answer the question. The number used is one that will notappear in other coding, like 99, if this is well outside the range of otherpossible responsesIn evaluation studies,information is collectedabout individuals, thenanalyzed by groupto find patterns andtrends.Evaluation Toolkit for Community Youth Programmers Step 6 61

ScalesThe response scale for a question on a form or in an interview is usually writtenfrom least to most, with the numbers increasing accordingly. Sometimes, however,the ‘least’ option is the most positive response (least dissatisfied, least bullied.) Itis important that all the coded scales are in the same direction (all positive or allnegative) to enable data from different questions to be compared or cross-tabulated.This reordering has to be done at the coding stage rather than on the data collectionform to avoid confusing informants.Data entryData can be entered onto coding forms by hand or directly into a computer.• To find and reduce coding errors:• have two people enter data separately then compare results(double-entry)• or have someone verify data entry by spot-checking for errorsagainst the original coding form or questionnaires.Common errors to look for are:• numbers outside the range of codes (6 for a 1-5 response option)• transposed numbers (1.52 instead of 1.25.)Automatic range checks are provided by most commercial data entry programs.6.6 Using statistics to describe dataAnalytical toolsBasic data analysis can be accomplished• with a computer software package (necessary for a large data base)• by putting data into table form using a spreadsheet as in Excel or• with the table function in a word processorOther software at varying levels of complexity is available free or for purchase on theweb.62The Offord Centre for Child StudiesHands Across Canada

Types of analysisAnalysis can take two forms. The first describes the data, shaping the results to bringout patterns and trends that may be hidden. This process is easy for non-statisticiansusing fairly basic mathematics and is the logical follow-up to data collection. Thesecond more complex type of analysis applies mathematical tests to give a statisticalestimate of the level of confidence one can have in the accuracy of the findings. Thatlevel of statistical analysis is only touched on here.Descriptive statistics• percentage (the number of responses for each option/variable as apercentage of all responses)• mean or arithmetic average (the sum of scores divided by the number ofresponses to the question)• median or mid-point response with an equal number of responses aboveand below it)• mode (the most frequently occurring response)• range (the amount data are dispersed: difference between the highest andlowest values, or range of categories with at least one response)• standard deviation (the average of the distances between each value andthe mean of all the values. See Glossary.)How to use descriptive statisticsMost data used in descriptive statistics will either consist of• named categories: individual items, either one thing or another (e.g.,male/female or English/French/Italian)• a choice that falls somewhere on a quantifiable spectrum of options (e.g.,smaller to larger, less to more)Categorical or nominal dataCategorical data are best described by counting how many informants’ responses fallwithin each category (frequency distribution.) The following example shows how basicdescriptive statistics can allow programs to see, and show, such patterns.Evaluation Toolkit for Community Youth Programmers Step 6 63

Example:Consider the following question:Which activity in the community recreation program do you most like participatingin?No one would ask,“what is the averagegender of participants?”or “what is the averagelanguage spoken inthe class?” You cannotcreate an averagefor discrete, namedcategories. Nominal orcategorical data canbest be described bycounting how manyinformants’ responsesfall within eachcategory (frequencydistribution.)a) art activitiesb) computer skillsc) music workshopsd) potterye) reading buddiesf) gym gamesNumber of responses (N=23, 12 girls and 11 boys)Responses: a = 2, b = 4, c = 5, d = 3, e = 3, f = 6Since the responses are different from one another, but don’t lend themselves to anyorder, counting or measurement, they are categorical (or nominal.) Calculating anaverage (mean) or median would be meaningless for these responses, but the programcan examine the range (which responses were selected ) and distribution (howfrequently responses were selected) by calculating percentages or proportions (thenumber of times each response was given, divided by the total number.)The results (arranged in descending order ) are: F = 6 (26.086)C = 5 (21.739)B = 4 (17.391)D = 3 (13.043)E = 3 (13.043)A = 2 (8.695)Although readers can determine the relative position of responses from a list likethis, plotting a bar graph of percentages will make it easier to show staff differences inresponse levels and ask for feedback. Initial results show that f) is the modal response,(the most frequently selected option) meaning that gym is the most enjoyed activity.However, this initial analysis may raise other questions. For example, did boys andgirls have different preferences? Since there are almost matching numbers of girls andboys, looking at those results could provide further information.Ordinal dataWhen the options given for a question can be arranged in some order (one is bigger,better or more of something than another), it is an ordinal scale. An example wouldbe questions with word options like: very happy, happy, neither happy nor unhappy,unhappy, very unhappy, which have a definite order but no equal or even definitedistance from one option to the next. Because they lack a measurable, mathematicalinterval between them, calculating a mean or average level of happiness for the groupis also not really appropriate.64The Offord Centre for Child StudiesHands Across Canada

Median, mode and rangeInstead of mean or average, a programmer can ask about the median, the person whois in the middle of the group in terms of attitude, with half the range of responses onone side and the other half on the other side. This is useful because it shows you thetrend of the responses.Looking at the following example:8. How do you feel about playing with other children in the recreation program?neither happyvery unhappy unhappy nor unhappy happy very happy1 2 3 4 5Q8, N = 15, Responses from coding sheetplaying with other children 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Totalsvery happy x x x x x x 6happy x x x x 4neither 0unhappy x x x x 4very unhappy x 1To find the median if there are not a lot of responses• put all responses in order then count to find the mid-point.For this question and program, the median response is “happy.’The mode for this data set (the most frequent response) is “very happy.”If two options had both had the most frequent number of responses, they would bothbe modes and the data would be called bi-modal. (If all categories have the samenumber of responses, there is no mode.)Both the median and mode for these results would be encouraging for programmers.Distribution of responsesAs in this example, responses may not be evenly distributed across all the responseoptions. Often they will be clustered at one or more typical responses, with only a fewpeople giving quite different responses.Evaluation Toolkit for Community Youth Programmers Step 6 65

The pattern of distribution has more effect on the mean value than on the median.A very few responses that are quite different from the majority can skew the meaneither up or down, and provide a less than accurate picture of results. Because of this,the median is usually a more useful statistic and you may want to compare all three:mean, median and mode.The range, another useful measure, can be easily illustrated in a table or bar chart.In the example given, the majority of respondents are happy playing with others inthe program. However, results show a broad range, with fully one-third of childrenunhappy to some extent. A narrower range of responses with no child selecting thebottom two or three response options would have been preferable. Since the data referto a specific and small group of children, the results may provide some clues aboutthe operation of the program, relationships with volunteers or the dynamics of thatparticular group. It may raise possible questions about cliques or bullying that may bemore fully explained by qualitative data from observations.Quantifiable dataThe full range of descriptive statistics already described can be used for quantitativeordinal data. In this type of data, each response option is ‘so many units more thananother’ or ‘so many times more than another’ on the scale being used.Examples from community programming data collection would be:1) quantitative questions (how much, how many, how often?) that provide ascale with numbers or2) questions that ask for measurements like height, weight, or test scores.It is common to dividedata into quartiles, theresponses at the 25thand 75th percentilesthen plot responseson a curve. This can bedone simply by firstfinding the median,then finding themedian of each groupof responses on eitherside. See Glossary for amore detailed formula.These types of data can provide more precise information and can be described inmore ways:• by finding the mean response• the median response (and quartiles)• the mode• range ( determined by subtracting the smallest value from the largestvalue)• standard deviation (a measure of variability of responses not often used incommunity programming evaluations)Handling ‘extreme’ responsesStatisticians often treat results that lie at the extreme ends of a distribution asexpendable, especially when working with large numbers of responses. In smallerscalecommunity evaluations, results that lie at the bottom of a distribution, reflecting66The Offord Centre for Child StudiesHands Across Canada

dissatisfaction or less positive results, may also provide information about programareas needing improvement. They may need to be viewed as red flags or challenges toseek further for explanations.Cross-tabulationAnswers to the original evaluation questions are often found with simple tabulationof responses and descriptive statistics as explained above. However, there may alsobe a need to look at subgroups among respondents and compare results from certainquestions. Cross-tabulation, which examines the relationship between responses fromtwo questions, allows for more complex ways of looking at the data. For example, across-tabulation might look at the attendance/participation records of respondents ina skills development program compared to changes in skill levels before and after aprogram. This is an easy operation for simple statistical software, some of which canbe downloaded without cost from the Internet (See References and Resources.)6.7 Knowing about inferentialstatisticsAs already mentioned, a second level of statistical analysis can help determine howmeaningful the data are, (i.e., how well they represent reality, rather than being theresult of chance) and how well they might pertain to another population.Such statistical analysis can be done by community groups but how and when itshould be done depends on a variety of factors related not just to the data, but to thewhole research design, e.g., the type of data, number and selection of respondents andtheir characteristics compared to the larger population. Such analysis should be doneby staff, volunteers or community partners with specific knowledge and experiencewho can help design data collection from the outset.However, it is helpful to have some knowledge of the jargon to understand whatexperts are able to provide and to aid in assessing research related to similar programsor interests.The most commonly accepted way to express the odds of some finding in the databeing true (an accurate reflection of reality), rather than the result of chance, (i.e.,the finding was the result of some other incidental factors) is the p-value. Individualevaluators can decide what level of p-value is appropriate for the data beinginvestigated but generally p

Making Use ofEvaluation DataSTEP7Step 6 provided basic techniques for handling and analyzing the raw data after it wascollected, through simple quantitative and qualitative methods.Step 7 looks at what happens after the evaluation, when program leaders are faced withquestions about how best to use the results. The final stage should be a catalyst for action andan opportunity to re-engage with the community.Review results with staff and volunteersResponding to the resultsShare lessons learned with stakeholdersSupport your staff to act as program ambassadorsClosing the evaluation circle7.1 Review results with staff andvolunteersThe sequence of planning and conducting a self-evaluation is an opportunity forprogram managers to either set the agenda of discovery or play a major role indoing so. It is natural for managers, staff and stakeholders to expect that evaluationresults will be immediately relevant to their work, since the framework of topics andquestions was built upon fundamental program concerns. At first glance, however,some results may seem puzzling, ambiguous or counter-intuitive, while other resultsmay seem obvious and trite.Unfortunately, research teaches us that,a) thinking something is not the same as having data to support it,b) results may engender alternative possibilities of interpretation,c) sometimes results provide only partial answers and raise additionalquestions.69

To gain additional value from evaluation, program managers share analyzed data withstaff and volunteers familiar with day-to-day program operations—utilizing theirknowledge and experience to help interpret patterns in the data and what they sayabout programming. Such consultation may vary from informal one-on-one meetingsto group workshops or presentations, depending on the size and structure of theprogram.An in-house review of results prior to any public disclosure:• allows staff and volunteers to see the culmination of the evaluationprocess• how their input can influence decision-making and programdirection• how their roles may influence program effectiveness• provides an in-house opportunity to air all viewpoints• ensures that staff know how results are being interpreted by peers andsupervisors• helps staff/volunteers understand what effort is expected of them and whyit is important• helps build an in-house culture receptive to follow-up evaluation• helps to make program leadership responsive to concerns and accountablefor taking action• provides an opportunity to explain factors allowing or preventing programchange7.2 Responding to the resultsAfter in-house review and input into interpretation of results, managers respond andgive the whole exercise meaning by:Dream bigStart smallAct now• Absorbing lessons• Supporting promising practices• Developing approaches to overcome problems• Formulating new questions• Taking action• Communicating results70The Offord Centre for Child StudiesHands Across Canada

Evaluation is no different from other projects embraced by community programmers.It is a process that begins from a broad perspective. It can engender big dreams toimprove the overall concept and operation of activities. Yet, to succeed, it should befocused first on particular, immediate and manageable issues. Otherwise, limitationsin resources can discourage and bog down community groups overloaded withinformation. Sometimes, making even limited changes in reaction to evaluationresults can have long-lasting impact.Need an example?Evaluation results in a tutored literacy program described earlier turned up an unexpectedeffect: children who had been engaged in the program and benefited from it weredisappointed and unhappy when the tutors left at the end of the academic session. Sincetutors were university students who would not be returning to the program, there wasan abrupt break in all the fledgling relationships that had developed. A procedure wasimmediately added to soften this parting so that children’s attitudes towards the readingexperience were not coloured by it. Each tutor wrote a special goodbye letter that explainedwhy the tutor was leaving, talked in detail about what they had done together, praised thechild’s effort and accomplishments and set out a plan for the future, e.g., ‘I hope you keeppractising what we have done together and read before bedtime every night.’ Tutors alsoshared their reading strategies with parents to enable them to continue what the childrenenjoyed. That small change improved attitudes toward the program, left children happierand more motivated to continue their efforts and increased communication betweenparents, teachers and tutors.7.3 Share lessons learned withstakeholdersEvaluation results can be categorized according to their relevance and importance for:1) program operation (programming, procedures, staffing),2) interaction with the ‘program community’ (participants, family, partners),3) broader community (funders, supporters, potential for both.)Managers can tailor how they will deliver evaluation results to the variousstakeholders. Not all stakeholders need or will want the same information or level ofdetail but will want information about questions they raised initially.Engaging the communityThe final stage of evaluation creates an opportunity to re-involve participants andfamilies, letting them see the connection between their participation in providing dataand the overall picture. By promoting and explaining evaluation results, programs canbuild links in the community, acknowledge what is working and why, and ask for bothnew ideas and assistance to overcome challenges.Evaluation Toolkit for Community Youth Programmers Step 7 71

Methods of information-sharingA variety of existing opportunities can be used to share evaluation results, e.g.,presentations, annual general meetings, volunteer recognition ceremonies. Variedformats can reach different segments of the community: newsletters, flyers, reports,web sites, community newspapers. As with other communication, information shouldbe accessible in tone, vocabulary and language to those it is meant to inform.Community responsibilityBad news will leak out eventually, especially if the evaluation is in response to somepublic criticism. It is beneficial for community relations if programs seize the initiativefrom a potentially disheartening review to examine how problems can be addressedand solved. Community residents and youth programmers have the same basic aims—to help the community’s children. This can provide the impetus for the support,goodwill and compromise needed to move forward with a strengthened program.7.4 Support your staff to act asprogram ambassadorsA tactic of successfulcommunity groupsis to empower andencourage staff andvolunteers to spread apositive vision of theircommunity programs,acting as informalambassadors whereverthey go.Publicize positive findings, both inside and outside the organization, e.g.,• promising effects of programs on children’s skills, knowledge, behaviourand attitudes and• success in recruiting and helping specially targeted children in thecommunity.An evaluation can also be a mechanism for advocacy, for publicizing a program,widening community support and targeting funding.7.5 Closing the evaluation circleEvaluation as processThe results of an evaluation process lead back to its original purpose. If those resultssatisfy the original questions, program managers and staff have a direction chartedfor future program operations. Evaluation can be extremely satisfying. It can showthat basic assumptions are sound and can confirm a theory of change. Sufficient72The Offord Centre for Child StudiesHands Across Canada

documentation of procedures allows consistent duplication of the program in anothersetting or with another population. However, a promising evaluation is not the finalword on a community program’s effectiveness.Community programs contend with change in every session: different participants, oftendifferent personnel, which means that the same level of effectiveness is not guaranteedanother time.Programmers need to repeat the same questions and pose new ones.They need to be ready to adjust programs—look for new solutions to unforeseenchallenges—capitalize on unexpected bonuses—learn from lessons— and expand elementsthat seem most promising.Rethinking evaluationResults may not satisfy the questions you originally posed. Results occasionallyfind that programs are counter-productive or ineffective in meeting their originalobjectives.Factors can relate to the program itself, the theory behind it or the quality of theevaluation.• Changes might be too subtle to be readily observed or respondents mightnot have adequate knowledge of either the child or circumstances wheresome change is noticeable.• Learning happens in small increments and may not be evident in the earlystages or when there are small numbers of participants and informants.Results may raise other questions about longer-term or delayed effects that demandsome follow-up or refinement of the evaluation methods— more data from anothersession, more children, different sources, closer observation, different questions. Theevaluation process is iterative, leading back on itself, engendering more questions andthe need for deeper understanding.Results, for instance,can be influenced byan indicator that isunsuited to capturingthe anticipatedchange becausethe measurementtimeframe isinappropriate, i.e.either program effectsbuild slowly over alonger period after theprogram ends or wereshort-lived and are nolonger evident.Learning from others’ evaluationsProgrammers can learn from others’ evaluation results as well as from their own. Morecommunity groups are engaging in some form of systematic evaluation and posting resultson websites and in databases of research associations. Programmers can compare theirmethods and evaluation results. Following these or similar guidelines—asking appropriatequestions, carefully collecting and analyzing data, and filtering it through experience— willenable community programs to build up a bank of knowledge, with realistic solutions to helpboth programs and children succeed.Evaluation Toolkit for Community Youth Programmers Step 7 73

Glossary ofEvaluation TermsComparativeA type of research that compares the results of one groupreceiving a treatment (or intervention) to another receivingeither no treatment or a different one. (See quasi-experimentaland experimental.)Convenience Samples A sample developed not from a random selection of a givenpopulation but from an available group, e.g., ‘man-in-the-streetsurveys.’ Likely occurs when the full extent and characteristicsof the population of interest are unknown or random samplingis not feasible. Results from a convenience sample cannot begeneralized to represent a larger or different population.ExperimentalFormativeLongitudinalThe most rigorous method of evaluating effects of sometreatment or intervention: by selecting two groups of people,comparable on various potentially relevant characteristics,then randomly assigning them to participate in a particularprogram or not. The results are compared by researcherswho are blinded to (unaware of) which people participated.If no other relevant common factors influence the groupof participants during the research period, any measurablechanges after the program to the ‘participant’ group alone,can be viewed as a program effect. Randomly matchedexperimental studies require certain resources and expertiseand sufficient numbers in each group to meet statistical testsfor reliability of results.A type of evaluation resulting from a compilation of datafrom various sources over time to show development of theperson or thing being evaluated. An example would be anemployment skills program that evaluated students basedon a series of practice lessons, work assignments, in-classparticipation and teacher observations based on interaction.A study in which the same group of people are studiedperiodically over a long time and compared with themselvesto determine the occurrence of change in relation to specificvariables.75

MatchingNon-comparativeNon-experimentalQuartilesProcess of making the study group and comparison groupcomparable with respect to certain factors, e.g. age, sex.A study or examination of subjects that draws no comparisonswith others outside the group. It may examine aspects fromhistorical data (retrospective), describe a current single-timesituation or track subjects periodically over time (prospective).Similar to non-comparative studies, having none of theattributes of experimental studies, e.g., random selection intomatched groups that are then treated differently, with blindedcomparative analysis of effects.A set of values that mark dividing points in a set of numericaldata, used to help describe the dispersion of the data. The firstor lower quartile has 25% of the values equal to it or lesserin value. The top quartile has 75% of the values either at orless than it in value. The first quartile may be approximatelycalculated by placing a group of values in order from smallestto largest, determining the median of the values, then findingthe median for the values on either side of that mid-pointvalue. (e.g. 2,3,3, 4,5,5,6,7,7,9,10) For an odd number of values,exclude the median when calculating the quartiles, for an evennumber, include it.More precise calculation for small data sets:To calculate the position of the quartile values.1. Rank the data set observations from smallest to largest.2. To find the lowest quartile, add one to the number of datavalues, rounding to the nearest whole number, then divideby four, (n+1/4). If the resulting number is halfway betweentwo whole numbers, round up to the next number. Thisgives you the rank position of the lowest quartile value.3. To calculate the highest quartile, first multiply the numberof data plus one by three then repeat the above step. Theobservation with this rank represents the upper quartile. If3(n+1)/4 falls halfway between two integers, round down.Based on UNESCO site for computer training at http://203.162.7.85/unescocourse/statistics/35.htm76The Offord Centre for Child StudiesHands Across Canada

Quasi-experimentalStudies that generally compare groups of participantsand non-participants that share some important similarcharacteristics but are not matched and randomly selectede.g., one classroom of children in a specific grade in oneschool participate in a program and are compared to asecond class in the same grade and school who did notparticipate. This is done more frequently when standardizedinstruments (questionnaires or scales) are used to collectdata about a specific characteristic or question such as selfesteem,substance use habits or bullying. Such standardizedinstruments have accepted levels of reliability and validity.Such studies may or may not be blindedSingle-time evaluation A one-time snapshot assessment of a program, or intervention,to provide a description of the state of a particular indicator (orcharacteristics of the program or participants). Often used atthe conclusion of a program to determine levels of satisfaction.Standard deviationStatistical samplesSummativeThe size of the standard deviation is a measure of the variabilityof quantitative results. With regard to ordinal, interval and ratioscale data, it is a numerical expression of the average of howfar each value differs from the mean of a data set. Simple steps:1. Calculate the difference between each data value and themean of the set. (individual value – mean value.)2. Square each difference, then sum them and divide the totalby the number of data values to find the variance.3. Take the square root of the variance to find the standarddeviation.The selection of a smaller group to represent a larger knowngroup (the population) using random selection so thateveryone in the population has a chance of being selected.The sample group should then be representative of the wholepopulation.A type of single-time evaluation showing the final outcomefrom a single measurement or assessment. A single test at theend of a course to pass the level is a summative evaluation.Evaluation Toolkit for Community Youth Programmers Glossary 77

References andResourcesRecommended sources for more in-depth informationon topics relevant to community program evaluation;the majority are available online.Data collection methodsConducting Survey Research. (1999 version 2.0).Retrieved from the University of Toronto, TheHealth Communication Unit at the Centre forHealth Promotion website: http://www.thcu.ca/infoandresources/publications/Surveys_Master_Wkbk_V2_Formating%2008.09.03_Content%2003.31.99.pdfDillman, D. A., Clark, J. R., & Sinclair, M. A. (1995).How prenotice letters, stamped return envelopes, andreminder postcards affect mailback response rates forcensus questionnaires. Survey Methodology, 21, 1-7.Hager, M.A., Wilson, Sarah, Pollak, T.H., Rooney, P.M.(2003). Response Rates for Mail Surveys of NonprofitOrganizations: A Review and Empirical Test. Nonprofitand Voluntary Sector Quarterly, vol. 32, no. 2, 252-267.Woodward, C., Chambers, L. and Smith, K. (1982).Guide to Improved Data Collection in Health andHealth Care Surveys. Ottawa: Canadian Public HealthAssociation.Evaluation-related data basesEvaluating Community Programs and Initiatives. PartJ. The Community Toolbox web site: http://ctb.ku.edu/tools/en/tools_toc.htmThe Evaluation Exchange. Out-of-School TimeLearning and Development Project/OST EvaluationDatabase at the Harvard Family Research Project website: http://www.gse.harvard.edu/hfrp/eval.htmlHealth in Action. Online Action to Health Promotionand Injury Prevention Information in Alberta web site:http://www.health-in-action.org/evaluations/rt.shtmlKids Count. (2004) Baltimore, MD: The Annie E. CaseyFoundation web site: http://www.aecf.org/kidscount/Making Connections, A Neighborhood TransformationFamily Development Initiative: Baltimore, MD: TheAnnie E. Casey Foundation web site: http://www.aecf.org/initiatives/mc/mcid/indicator_main.phpMeasures and Indicators Database: MakingConnections, A Neighborhood Transformation FamilyDevelopment Initiative. Baltimore, MD: The AnnieE. Casey Foundation web site: http://www.aecf.org/initiatives/mc/mcid/measures_main.phpOutcome Measurement Resource Network. UnitedWay of America web site: http://national.unitedway.org/outcomes/Voluntary Sector Evaluation Research Project. Ottawa:Carleton University web site: http://www.vserp.ca/about.html79

Evaluation guidesEvaluation Cookbook. Institute for Computer BasedLearning. (web version 1999). Learning TechnologyDissemination Initiative. Retrieved from Heriot WattUniversity: Edinburgh, UK web site: (for reference only:please see legal notice re. copying data) http://www.icbl.hw.ac.uk/ltdi/cookbook/contents.htmlFarell, K., Kratzmann M, McWilliam, S, Robinson, N,Saunders, S., Ticknor J., & White K. (2002). Evaluationmade Very easy, Accessible and Logical (EVAL).Retrieved from Dalhousie University web site: http://www.acewh.dal.ca/eng/reports/EVAL.pdfGilliam, Aisha. (n.d.) Evaluating HIV PreventionPrograms using Qualitative Methods, Booklet #9.Retrieved from US Centers for Disease Controland Prevention, Division of Adolescent and SchoolHealth web site: http://www.cdc.gov/HealthyYouth/publications/hiv_handbook/hiv_booklet9/index.htmKirkpatrick, S. (2001). The program logic model:What, why and how? Retrieved August, 2003 fromCharity Village web site: http://www.charityvillage.com/charityvillage/research/rstrat3.htmlWestat, J.F., Frierson, Henry, Hood S., Hughes,G,Katzenmeyer, C. (Jan. 2002). The 2002 User FriendlyHandbook for Project Evaluation. Retrieved fromNational Science Foundation: Division of Research,Evaluation and Communication web site: http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5.pdfW.K. Kellogg Foundation Evaluation Handbook (1998).Retrieved from W. K. Kellogg Foundation, BattleCreek, MI, Evaluation Unit website: http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdfWong-Reiger, D., & David, L. (1994). A hands-on guideto planning and evaluation: How to plan and evaluateprograms in community-based organizations. Ottawa:Canadian Hemophilia Society (Available from theNational AIDS Clearinghouse in Ottawa; 613-725-3769).YouthARTS: A Step-by-Step Approach for EvaluatingYour Arts Program’s Outcome. (n.d.) Retrievedfrom Americans for the Arts website: http://www.americansforthearts.org/youtharts/evaluation/approach.aspMesaros, W. ( 2001). Making Evaluation Integral toYour Asset-Building Initiative: Employing a Theoryof Action and Change. Available to view at SearchInstitute web site: http://www.search-institute.org/research/knowledge/MakingEvaluationIntegral.htmlPorteous, Nancy L., Sheldrick, Barbara J., Stewart,Paula J. (n.d.) A Program Evaluation Toolkit. Universityof Ottawa Community Health Research Unit.Can be ordered through web site: http://www.medicine.uottawa.ca/epid/chru/toolkitdesc.htmThe Scottish Arts Council e-Tool for Evaluation.(2003). Edinburgh, Scotland. Web site at http://www.evaluationforall.org.uk/introduction/introduction.htmlUNICEF Guide for Monitoring and Evaluation. (n.d.)Retrieved from UNICEF: Division of Evaluation, Policyand Planning website at: http://www.unicef.org/reseval/mander.html80The Offord Centre for Child StudiesHands Across Canada

Needs assessmentA Practical Guide to Family Literacy. (n.d.) (includesa good section on needs assessment and examplesof community program evaluations) Retrieved fromFamily Literacy Action Group web site: http://www.nald.ca/CLR/Pgtfl/contents.htmNational Adult Literacy Database web site: http://www.nald.ca/index.htmCommunity assessment, agenda setting and choice ofbroad strategies. Part B. Community Toolbox web site:http://ctb.ku.edu/tools/en/tools_toc.htmDoing it right: A needs assessment workbook. (1993).Edmonton Social Planning Council Edmonton, AB.Can be ordered for $5.00 online at http://www.edmspc.com/publications.htmlKeating, D.P. & Hertzman, C. (eds). (1999).Developmental health and the wealth of nations: Social,biological, and educational dynamics. New York: TheGuilford Press.Richmond Neighborhood Indicators Project (n.d.)PolicyLink web site: http://www.policylink.org/EDTK/Mapping/PartnershipsMatheson, Jane. (2000). Making sense of partnershipsin the nonprofit sector: do we really know what we aredoing? A paper written as the basis for a presentationto the National Healthcare Leadership Conferenceand Exhibition in Ottawa, Ontario, June 18-21, 2000.Retrieved from the Woods Homes website: http://www.woodshomes.com.Questionnaire constructionConstructing the Survey. Bill Trochim’s ResearchMethods Knowledge Base, The Centre for SocialResearch Methods, Cornell University, USA web site:http://www.socialresearchmethods.net/kb/survwrit.htmDesigning a Questionnaire. (n.d.) from ASA series,What is a Survey? Retrieved from American StatisticalAssociation web site: http://www.amstat.org/sections/srms/brochures/designquest.pdfOrganization of Research Co-ordinators & Assistants.(1994). Blueprint for a Data Collection Tool. Faculty ofHealth Sciences: McMaster University: Hamilton, ON.Woodward, C., and Chambers, L. (1980). Guide toQuestionnaire Construction and Question Writing.Ottawa: Canadian Public Health Association.Standardized test retrievalInformation about selecting standardized testsavailable on the American Psychological Association(APA) web site: http://www.apa.org/science/faqfindtests.htmlTest Reviews are online at the Mental MeasurementsYearbook at: http://buros.unl.edu/buros/jsp/search.jsp.TestLink provides descriptive information at http://www.ets.org/testcoll/index.htmlStandardized instrument database on AustralianCentre on Quality of Life web site: http://acqol.deakin.edu.au/instruments/instrument.phpStatisticsNorman, G. R., Streiner, D. L. PDQ Statistics. (1986).Toronto: B.C. Decker Inc.Simple Interactive Statistical Analysis. Onlinestatistical computation available at web site: http://home.clara.net/sisa/Evaluation Toolkit for Community Youth Programmers References and Resources 81

Strategies to engage youthInvolveYouth: A guide to involving youth in decisionmaking.(2004). Retrieved at City of Toronto InvolveYouth Campaign, Toronto, Ontario web site: http://www.city.toronto.on.ca/involveyouth/pdf/guide.pdfOrganizational Assessment Checklist. 14 Points:Successfully Involving Youth in Decision Making.Somerville, MA, USA. Retrieved at Youth onBoard web site: http://www.youthonboard.org/organizationalassessment.pdfYouth Action Network. UK. Various strategies andresources available at website: http://www.youth-action.org.uk/info/goodpracticeprinciples.htmlVoluntary sectorCentre for Voluntary Sector Research andDevelopment, Carleton University, Ottawa web site:http://www.cvsrd.org/eng/index.htmlCharity Village. Canadian non-profit organization withresources and links to other non-profit groups. Website: http://www.charityvillage.com/Voluntary Sector Evaluation Research Project.Examples of community-based research linkinguniversities and members of the community involvedin program delivery. Voluntary Sector EvaluationResearch Project web site: http://www.vserp.ca/links/linksComm.htmlYouth developmentElke Geiger, E., Britsch, B. (n.d.) Out-of-School-TimeProgram Evaluation: Tools for Action. Education,Career, and Community Program. Retrieved fromNorthwest Regional Educational Laboratory web site:http://www.nwrel.org/ecc/21century/publications/ost_tools.pdfHufnagel,G,L,& Blasiar, D.(eds.) (2001). Lessons inMentoring: A Guide to Working with Youth. Centre forSubstance Abuse Prevention. Retrieved from SouthwestCAPT (Center for the Application of PreventionTechnologies) web site: http://www.swcapt.org/pdf/mentoring/mentoring_book.pdfIssue by Issue: Program Quality. (n.d.) Retrieved fromThe Forum for Youth Investment web site: Little,P,D,& Traub, F. (May 2002) Evaluating MunicipalOut-of-School Time Initiatives— Learning From theField. Retrieved at the Harvard Family Research Projectweb site: http://www.gse.harvard.edu/~hfrp/projects/afterschool/resources/municipal.htmlPittman, K, Yohalem, N & Wilson-Ahlstrom, A.(Issue 1, Oct. 2002). Out-of-School Research MeetsAfter-School Policy. The Forum for Youth Investment.Retrieved from The Forum for Youth Investmentweb site: http://www.forumforyouthinvestment.org/comment/ostpc1.pdf (See also related issues at thesame site.)Reports from evaluations of youth programming crimeprevention initiatives. Government of Canada, PublicSafety and Emergency Preparedness Canada web site:http://www.prevention.gc.ca/en/projects/index.aspScales, Peter C., & Roehlkepartain, Eugene C. (2003).Boosting student achievement: New research on thepower of developmental assets. Search Institute Insights& Evidence, 1 (1). Retrieved from the Search Instituteweb site: http://www.search-institute.org/research/Insights/82The Offord Centre for Child StudiesHands Across Canada

AppendixThe following tools are a selection of resources and data collection tools provided by participatingorganizations and groups for reference or possible use by other community youth programmers.Some have been modified for generic use.Program resourcesR1 Evaluation planning model and completed example ♦ 1R2 Example of alternate program logic model (literacy and homework help) ♦ 5R3 Example of participants’ outcomes chart (employment skills) ♦ 7R4 Example of indicators and measurement chart ♦ 9R5 Data collection methods schedule ♦ 10R6 Evaluation process form: data collection ♦ 11R7 Example of participant’s performance indicators (employments skills) ♦ 12R8 Example: introductory letter to principal re. evaluation ♦ 13R9 Sample recognition certificate (achievement-based, literacy) ♦ 14R10 Instructions for questionnaires/interview forms (for use with T2, P7 & P8) ♦ 15R11 Community programming evaluation checklist. ♦ 18

Data Collection(source of data)CommunityC1 Recreation participation survey (rural) ♦ 20C2 Community needs assessment (household) ♦ 24C3 Community needs assessment (child recreation data) ♦ 27Parents/guardiansP1 Registration and permission form ♦ 34P2 Parent mid-program telephone survey ♦ 35P3 Parent pre-program telephone interview (child assessment) ♦ 36P4 Parent post-program telephone interview ♦ 38P5 Parent post-program evaluation ♦ 41P6 Parent mid or post program questionnaire ♦ 42P7 Parent post-program interview form ♦ 46P8 Parent group interview form ♦ 51ParticipantPt1 Participation form ♦ 57Pt2 Participant exit interview ♦ 58Pt3 Focus group questions after program ♦ 59Pt4 Participant social functioning questionnaire ♦ 60Partner agenciesA1 Volunteer service questionnaire for agencies ♦ 64

TeachersT1 Post-tutoring teacher survey ♦ 67T2 Teacher post-program questionnaire ♦ 69T3 School staff post-program evaluation ♦ 71Volunteers (leaders/tutors etc.)V1 Sample volunteer application form (web-based program) ♦ 73V2 Sample volunteer application form ♦ 75V3 Registration and attendance form ♦ 79V4 Attendance log: participants and volunteers ♦ 80V5 Attendance tracking sheet ♦ 81V6 Observations pre-test (special needs) ♦ 82V7 Observations mid-program (special needs) ♦ 83V8 Observations post-program (special needs) ♦ 84V9 Volunteer program evaluation #1 ♦ 86V10 Volunteer program evaluation #2 ♦ 88V11 Facilitator’s assessment (objectives) ♦ 90V12 Facilitator’s feedback post-program ♦ 91V13 Staff and volunteer post-program questionnaire ♦ 93V14 Tutor monthly letter to parents re progress ♦ 97V15 End of tutoring summation ♦ 98V16 Jr/Sr leader mid-program self-evaluation ♦ 99

R1 (pg 1 of 4)Evaluation Planning Model1Evaluation Planning Model andCompleted ExampleProgramOutcomes1. problem and desired results(what and for whom?)2. contributing conditions 6. indicators of changeand causes(what evidence, change to what,(what helps & hinders?)to whom, how much?)3. strategies(how will it work?)4. planned activities 5. program(what, when, bymeasures/outputswhom, for whom?)(what was accomplished?)adapted from: Family Research Project, Harvard University, 2002Please use the following questions to complete the modelling activity starting on the attachedblank activity page. A completed example appears after the questions. The model is useful toconfirm clear objectives for the development of evaluation procedures and measures. Thedescription of indicators and measurement tools can point to the need for questionnaires,attendance and activity logs, and baseline data that might otherwise be overlooked.1. Problem to be Addressed and Desired Results:What child/youth/community problem(s) does the program aim to resolve or reduce?2. Conditions and Causes:What factors affect or contribute to the situation, both positive and negative?3. Strategies:What strategies or theory underlie the initiative, to help overcome negative factorsand/or strengthen positive ones? (E.g., youth development, increased competence,mentoring, service learning, role modelling)Offord Centre for Child Studies, 2003 1Evaluation Toolkit for Community Youth Programmers Appendix 1

R1 continued (pg 2 of 4)24. Activities:Describe specific activities that support the strategy (ies).5. Program Measures:What will be used to monitor the progress of the program as planned? Considermeasures of effort or output, e.g., number of activities, participants and volunteers, rateof individual’s attendance over time, levels of satisfaction.) These will help to show howwell the program is being implemented.6. Indicators:What could indicate some effect on an individual/family/community that relates toprogram goals and might result from program opportunities? When would changes likelyappear and how could they be measured? Data about individuals can be grouped foranalysis by gender, age or other pertinent factor. Change is measured from baselinedata: knowledge/evidence of individual status at the beginning of the program. Notethat the model is an iterative loop, with data about results feeding back into anexamination of how the original situation is affected and whether the program needs tobe adjusted accordingly.EXAMPLETo give a basic example of how the model can be used, let's consider a communitysituation around certain teenagers.1. There is local concern about a number of teens congregating on neighbourhoodstreets and in malls at all hours. Some have dropped out of school, some may beliving on the streets and some are involved in negative activities. Results desired bythe community might be: i) a reduction in negative activity by teens, ii) moreproductive use of time by teens, iii) redirection of teens’ interests from negative topositive behaviour. Another approach might be to address underlying communityproblems: suspicion of teenagers, lack of recreation facilities, employmentopportunities, etc.2. An investigation of the situation uncovers the extent of the problem (averagenumbers of teens, areas affected, crime/accident reports) and uncovers factorsrelated to cause of the situation: e.g, family dysfunction, psychological/behaviouralproblems, early school leaving, poverty, racial and ethnic inequality in the communityand lack of opportunities for that age group. When asked, the teens themselves mayidentify problems they would like addressed such as poor literacy skills,dissatisfaction with school programming, lack of marketable skills and/or a lack ofjob-finding skills.3. The chosen strategy of this particular program is to encourage unemployed teensto return to school and/or upgrade employment skills and knowledge outside ofschool in order to find work. The target group is youth who are on the street, lackhigh school diplomas or are unemployed.The underlying theory would be youth development and community connectedness,implemented through: outreach to target teens skill and knowledge enhancement through out-of-school teaching, workshopsand job-related experience in the communityOfford Centre for Child Studies, 2003 22The Offord Centre for Child StudiesHands Across Canada

R1 continued (pg 3 of 4)3encouragement to think about the future and one’s ability to have a positiveimpact on the community.4. Activities include: literacy tutoring, volunteer placements, job coaching, resumewriting, service learning and motivational activities.5. For each type of activity the program will have performance measures, ways ofassessing:how successfully it was implemented, no. of sessions, no. of hours, no. of targeted participants and volunteers -recorded with attendance logs and record of participation in each activity, satisfaction of participants and volunteers/community partners with program -assessed through observations, interviews and/or questionnaires and6. Indicators of what impact it made on the participants, i.e., acquisition of skills, experience or knowledge and putting that learning to productive use (e.g., by returning to school, gainingemployment or continuing volunteer work) attitude/behavioural changes directly related to the problems preventing theteens from finding employment or returning to school, and the relatedproblem.What concrete indicators or evidence can the program identify and measure?Objectiveliteracyupgradingvolunteerexperienceemploymentskills beforeand afterparticipation inthe programIndicator (whatchanged)reading more difficultmaterialsustainedparticipation involunteer serviceresumé writing/knowledgeeffort in job searchpreparationpresentation effortMeasurement tool, MeasurementTimes and Rater(evidence of change, how much)- reading and completing sampleapplication forms more easily than atstart of program (pre-post)– rated byprogram leaders using checklist- questionnaire completed by serviceplacement/facilitator rating hourscontributed, behaviour of volunteer,satisfaction with effort of volunteer(after service)- completed resume, either first-time orrevision(pre-post)- completed plan to facilitate search:researching opportunities, appropriatecover letters etc. (record of activitiescompleted by program leaders/selfreport)- appearance, conduct, confidence inpractice interview (rated by observers,self-report)Offord Centre for Child Studies, 2003 3Evaluation Toolkit for Community Youth Programmers Appendix 3

R1 continued (pg 4 of 4)4Example ‘continued’Objectivepro-social attitude,behaviour (reliableattendance,responsible, cooperativebehaviour)future planning/productive activityIndicator (whatchanged)sustained participationin activities, being ontime, fulfilling agreedupon tasks, meetingexpectations willinglyreturning to school orobtaining work;continuing to volunteeror other productiveactivity compared tobaselineMeasurement tool, MeasurementTimes and Rater(evidence of change, how much)- program attendance andparticipation log completed byprogram leaders- questionnaire/interview ofprogram leaders, volunteer servicementor- enrollment in school- more employment (pre-post)- more specific plans for future(pre-post)through follow-up structuredinterview or questionnaire forparticipantIndicators and Relationship of Change to OriginalProblem/ObjectiveIndicators should lead back to the initial problem. In this case, it should be possible todetermine how many participants who were part of the target group have: (a) acquiredskills, knowledge, favourable behaviour and experience that will help them get work, (b)obtained employment or (c) returned to school, and d) have some more positive plan fortheir future than before the program. These successes should reduce the number ofyouth on the street to some measurable extent if the premises and target group for theprogram were correct. It might also be possible to determine how many youth were inthe original target group before the introduction of the program if the community beingserved is small. If that is so, it will be possible to determine how much of an effect theprogram has made on that specific community problem.Offord Centre for Child Studies, 2003 44The Offord Centre for Child StudiesHands Across Canada

R2 (pg 1 of 2)Example of Alternate Program Logic Model Organization:Project Name:Time Frame:Situation: Integrated program for families- adult literacy classes and homework support for childrenContext: (problem & factors) Immigrant families with limited English-speaking cannot support children’s homeworkand school experience because of lack of literacy skills, lack of knowledge about and understanding of educationsystem, lack of communication with children’s schools, teachers and staff.How? What We Want to Occur How Do We Know?InputsWhat do youprovide to theprogram?Means of FundingStaffVolunteersEquipmentSuppliesProgram spaceTransportVolunteer TrainingCommunicationPartnershipsActivitiesWhatactivities willform theprogram?ParentLiteracyClasses- group lessons& coaching bytrainedinstructorOutputsHow many activities,sessions, events?(# & hr.).Participants/volunteers/ratio?X number of activitiesX number of sessions, per wk anddurationX number of hoursX no. of volunteersVolunteer/participant ratioShort-TermOutcomes- Adults able toread andunderstand simpleschool-relatedmaterial- Adults able towrite simple notese.g. Notes to theteacherIntermediateOutcomes- Adults able toread newspaperor magazinearticles.- Adults able towrite simpleparagraphLong-TermOutcomesParentsimprove theirreading andwriting skillls.Children areable to dohomeworkindependentlyand look forhelp whenneeded.Adults able tosupport theirchildren and/orlook forrequiredsupport.Families fullyaware ofimportance ofreadingactivities andintegrate theminto theirroutine. Adultsand childrenconfident andparticipating inschool andcommunityevents andactivities.MeasuresPre-post testsof reading &writing skills,based onschoolinformation.Volunteer logsReports fromteachers.InterviewsInterviews,questionnairesAdapted from example by Delta Family Resource Centre, Toronto, ON, 2003Evaluation Toolkit for Community Youth Programmers Appendix 5

R2 continued (pg 2 of 2)InputsWhat do youprovide to theprogram?ActivitiesWhatactivities willform theprogram?Child TutoringProgram- individualhomeworkhelp- themedgroupactivities(art/craft)- jointparent/childhomeworkrelatedactivitiesFamilyReadingCircles- intergenerationalreading- discussion/- conversationOutputsHow many activities,sessions, events?(# & hr.).Participants/volunteers/ratio?Short-TermOutcomes- Increased interestin homework- Improvements inhomeworkcompletionrate- Children feelingmore positiveabout school- Children feelingmore positiveabout homeworkChildren feeling morepositive aboutthemselves- Increase inparents andchildren readingactivities- Overall interest forreading isincreased.- Informal networksinitiatedIntermediateOutcomes- Interest inhomework andhomeworkcompletionmaintained.- Children ask forhelp whenneeded andshow signs ofindependenceand confidencewhen doinghomework.Report cardreflectsacademicimprovements.- Family readingactivities arepart of the familyroutines andactivities.- Parents madeat least one ortwo friends.Long-TermOutcomesMeasuresVolunteer logsPortions ofreport card;teacherquestionnairePre- and postinterviewsParentInformationSessions- field trips- guests- Acquisition ofinformation on theeducation systemand communityresources- accessto at leastone resource.- Parents attendparent-teacherinterview- contact teacherswhen needed.- Parents feelcomfortablecommunicatingwith the schooland teachers.- Parents knowwhat to do to gethelp andadvocate fortheir chidlren.- Familyparticipate incommunityactivities.Focus groupswith parents.Adapted from example by Delta Family Resource Centre, Toronto, ON, 20036The Offord Centre for Child StudiesHands Across Canada

R3 (pg 1 of 2)(Employment Skills Program)Example of Outcomes Chart for ParticipantsOutcomes Planned Actual How Measured CommentsCompletion ofProgram80% Entrance and graduation datesentered into membershipdatabase.Reasons youth left theprogram.Certified and/orLicensed Training.70% ofProgramGraduatesNumber of youth who achieve alicense and/or certified trainingduring program.Types of trainingachievements.Summer Employment80% ofProgramGraduatesNumber of youth who achievesummer employment.Details on types andlength of work.Continuing with anEducation Program inSeptember80% ofProgramGraduatesNumber of youth who continueeducation in September.Details on educationlevels.High School Diploma90% ofProgramGraduatesNumber of program graduatesthat receive their high schooldiploma.Comparison tograduation rates forother youth in thesecommunities.Enrollment in aPost-SecondaryProgram70% ofProgramGraduates.Number of youth who enroll inpost-secondary programDetails on types ofeducation programsparticipants enroll in.EmploymentNumber of youth who attainemployment instead of schoolOR attain employment incombination with school.Details on types ofemployment.Non-EngagedLess than10% of allyouthserved.Number of youth who areneither in an education programor working a year aftergraduation ie. high school dropout; unemplolyed; socialassistanceDetails on barriers tosuccess for theseyouth.UnknownNumber of youth who theagency loses contact with.Evaluation Toolkit for Community Youth Programmers Appendix 7

R3 continued (pg 2 of 2)Other Measures Used:Connecting youth with positive adult relationships; new experiences and newskills. Documented in youth interviews.Demonstration of employment skills through portfolios and completion ofassignments. Rubrics used to assess skills and ensure good instruction anddevelopment. (See example of resume rubric.)Attitudes toward learning and completing school. Interview statements made preand post program.Increased knowledge of careers and post-secondary opportunities. Statementsmade pre and post program.Number of youth with criminal conviction pre-program and post-programFrom Winnipeg Boys and Girls Club, Winnipeg, Manitoba, 2004.8The Offord Centre for Child StudiesHands Across Canada

R4Chart of Indicators and Measurement MethodsIndicator Measurement In place CompletedLiteracy classReading, writing, andunderstanding of simplematerialInformation sessionsIncreased knowledgeand understanding ofschool programs, reportcards, parent-teacherinterviews, andhomework.Attendance at parentteacherinterview andask questions as neededContact teacher asneededIncreased informationon community resourcesand access to themReading CirclesIncreased interest in andnumber of familyreading activities.Increase number offriends to two or more tosocialize or look forhelpTutoring SessionsStudent come preparedand completeshomework.Student attendsregularlyReading and Writingskills assessmentPre-post surveyParents report/ focusgroupParents report/ focusgroupPre-post surveyPre-post surveyPre-post surveyTutor’s check list /logAttendance sheetChildren feel positiveabout school andhomework achievementsTestimonials / FocusgroupDeveloped by Delta Family Resource Centre, Toronto, ON, 2004.(family literacy and tutoring activities project)Evaluation Toolkit for Community Youth Programmers Appendix 9

R5Data Collection Methods SchedulePurpose of data:Question?Method of datacollectionDemographic info. Registrationform,IntakeinterviewParticipantactivitiesParticipation rate attendancerecordSourceof dataParticipantparticipantVolunteer log volunteer/tutorvolunteer/tutorI.D. #(foreachform)Prepared by PretestdataAchievements/skilldevelopmentQuality ofexperience forparticipantsQuality ofexperience forvolunteersMeeting parents’needsEffects onparticipantspre-post testvolunteer logParticipantvolunteerinterview participantsinterview,experiencelog,questionnaire,focus group?interview,questionnaire,focus group?interview,questionnairevolunteerparentsParticipantsParentsteachersOfford Centre for Child Studies, McMaster University-Hamilton Health Sciences, Hamilton, ON, 2004DateFollow-upadministered Post-test Other10The Offord Centre for Child StudiesHands Across Canada

R6Evaluation Process FormData CollectionAvailable Forresources developmentCompletionscheduleContacts for DataCollectionPreliminaryStagesSchedulefor useReleventObjectiveResponsibilitye.g., registration forms, intake interviews, daily attendance records by name; participation checklists; volunteer logs re. behaviour,achievements; tests; quizzes, homework sheets, interview forms, focus group outlines, questionnaires, data from social serviceagencies, police; school data re. attendance, teachers’ meetings, tenants’ association data re. police and fire calls.Evaluation Toolkit for Community Youth Programmers Appendix 11

R7List of Participant’s Performance IndicatorsFacilitators are responsible for assessing and documenting the following outcomes andindicators for each participant.Changes in attitude, skills, knowledge.Document new experiences, feelings, and accomplishments.Spotlight areas for improvementConcrete ItemsSIN NumberBank AccountDebit CardResumeCover LetterFilled Out Job ApplicationsTaped InterviewsWorkplace VisitsVolunteer WorkCertified TrainingHours WorkedParticipation RatesOutcomesExpected outcomes for the program include: Participants will be able to demonstrate job-hunting skills, such as using the Internet,job banks, papers, and personal contacts to find work. Participants will be able to accurately complete an employment application. Participants will develop a personal resume. Participants will demonstrate acceptable interviewing skills. The interviews arevideotaped and critiqued. Participants will be exposed to a variety of careers and workplaces. Participants will interview and/or job shadow a number of successful professionals. Participants will learn about relevant education and training programs Participants will perform volunteer work for their club and the community Participants will learn about workplace safety and employee rights in general Participants will learn communication and conflict resolution skills Participants will leave with concrete items such as a resume, S.I.N. number, bankaccount, wages etc. Participants will be connected with employment and/or continued education programs.The primary goal is to support youth to finish high school and move on to postsecondaryeducation.Adapted from the Winnipeg Boys and Girls Clubs, Step-Up program, Manitoba, 200412The Offord Centre for Child StudiesHands Across Canada

R8Example: Introductory letter to Principal and teachersre. after school evaluationJune 15, 2004Dear [Teachers’ and Principal’s names]:An After-School Program has been provided at the _____________________by ______________________this school year. We are conducting a programevaluation to help determine the success and impact of this program andprovide direction for future programming.We would greatly appreciate your feedback to assist us in our evaluationprocess. The enclosed questionnaire will take no more than 15 to 20 minutesto complete (one questionnaire per teacher, not one per child) and willprovide essential information for the After-School Program. We haveobtained written consent from the parents of the children who participated inthe After-School Program to give you the names of their children for thepurposes of this evaluation. [attach list related to each class] We realize thisis an extremely hectic time of year for teachers and are very grateful for yourassistance with the evaluation! Thank you very much!Please hand in your completed questionnaire to the school office byWednesday, June 30, for one of our staff to pick up. If you have anyquestions or concerns, please contact me at tel. ________________ or byemail _________________________________.Sincerely,Adapted from House of Friendship, Kitchener ON, 2004.Evaluation Toolkit for Community Youth Programmers Appendix 13

R9This certificate is awarded to:____________For participating in a FRONTIER COLLEGE Reading Circle.Congratulations!You have practiced your reading skills and have successfully read _____books!John O’LearyPresidentDate: March 31, 2004Tina WhalenSaskatchewan Regional Coordinator14The Offord Centre for Child StudiesHands Across Canada

R10 (pg 1 of 3)AFTER-SCHOOL PROGRAM EVALUATION(INSTRUCTIONS FOR QUESTIONNAIRES/INTERVIEW FORMS T2, P7 & P8)General InstructionsFor all the interview forms, make sure you identify your program site (nameof community centre) at the top – place an X or check mark beside your programname. If you use the individual parent questionnaires that parents fill out, checkto make sure the program site is identified – if not, please fill it in yourself. Alsofill in the date at the top of the form and similarly check the parent form,providing the date or approximation of the date it was completed.)Specific instructions on how to record the responses to each question are givenafter each question – follow these carefully, making sure you recordnumbers of responses where asked for and comments where asked for.For open-ended questions, write down all the responses you get; if morethan one person gives the same response, record in brackets after thatresponse how many people gave it.PROBE AND CLARIFY!!! ASK PARTICIPANTS TO BE AS SPECIFIC AS THEYCAN. If someone gives a vague answer, question them further to obtain moredetail – e.g. ask them gently, “What do you mean by that?” or “Can you be morespecific?” or “Can you give me an example?”E.g. for #2 on the Participant Group Interview Form, if a participant says thatwhen they didn’t come to the After-School Program, they were “hanging out”,ask “By yourself or with friends? Where did you hang out? What were youdoing?”, etc. – and write down this information. Or for #3, if a participantsays that they came because their parents “made them come”, ask why theirparents wanted them to come? (Try to find out whether this means that therewas no one at home to look after them after school, or perhaps theirparent(s) wanted them to get out of the house and do something active.)Remember: the more detailed and specific the information we can get,the more accurate and valuable our evaluation results will be.Use the back of the page if you need to – just mark clearly which question theinformation pertains to and any other required identification to ensure that theperson analyzing the data can clearly follow and interpret what you’ve written.Specific Instructions1) Parent Questionnaire/Interview FormNote that there are 2 versions of the parent questionnaire – one is for groups(called “Parent Group Interview Form”) and the other is for individual parents tofill out on their own (called “Parent Questionnaire”). You decide which to use – youcan use both as needed.Please give the individual questionnaire only to those parents who aremotivated enough and have adequate literacy skills to complete it on theirown!The forms look long (5 or 7 pages), but they aren’t terribly long in terms of numberof questions – just a lot of space for written responses.Evaluation Toolkit for Community Youth Programmers Appendix 15

R10 continued (pg 2 of 3)2The individual “Parent Questionnaire” includes a line for the parent to writehis/her initials so that we can assign an I.D. code and compare the parents’responses with their children’s.Here are detailed instructions for the “Parent Group Interview Form”. Thenumbers that follow correspond with the questions on the interview form.‣ For tracking purposes, assign each group and each parent in the group anI.D. code.‣ Near the top of the page you’ll see “Group No. _____.” Assign a number (1, 2,3, etc.) to each group of parents you interview – a different number or code foreach group.‣ On a separate piece of paper, at the start of a group interview, list all the names ofall the parents in that group and assign a “parent code” to each of them – e.g. inGroup 1, Sally Smith is “Parent 1” and is assigned the code “P1”, Betty Brown is“Parent 2” and is assigned the code “P2”, etc. Be consistent and keep it simple.Then enter each parent’s code in the appropriate space in #1.‣ Read the bold italics “Introduction text” out loud to the group before you toexplain the purpose of the interview.‣ Question #1 allows space for up to 6 parents in a group. If you have morethan that, add them below “Parent 6” or on the back of the page.‣ Ask each parent how many children s/he had in the program, and the gender(M/F – or you can use B or G for boy or girl), age and grade of each of theirchildren in the program. Make sure that the information you record foreach person is the correct information for their I.D. code – i.e. theinformation about Sally Smith’s children must correspond with herI.D. code. If a parent had more than one child in the program during theyear, write down the information for all of his/her children in the program inthe right-hand space, putting a comma or semi-colon (;) between each child’sinformation. You may use the right-hand margin or write on the back of thepage if needed – just make sure it’s easy to identify which informationbelongs with which parent.‣ Question #2. Write the number of children in the appropriate category.Each parent may have children in several categories, depending on how manyof his/her children attended the program. E.g. if one parent had 3 children inthe program, one of whom attended from September to June, one fromDecember to June, and one from January to March, you would put a mark inthe “September to June” space, one in the “December to June” space, andunder “Other”, write in From “January” to “March”. You can use tick marks toindicate number of children in each category, then add them up for eachcategory.‣ Question #3. We are again looking for number of children. An individualparent may have children in different categories.‣ For the rest of the questions, follow the directions carefully regarding whereto record the number of responses, and where to write down respondents’comments (for open-ended questions.)from House of Friendship, Kitchener-Waterloo, ON, 2004216The Offord Centre for Child StudiesHands Across Canada

R10 continued (pg 3 of 3)‣ If you get more than one of the same comment for any open-endedquestion, indicate in brackets following that comment the number of similarcomments or responses you get.3‣ Remember to thank parents at the end!2) Participant Group Interview FormRecord the date and enter the name(s) of the person(s) doing the groupinterview.As with the Parent Group Interview Form, assign an I.D. code to each group ofchildren you interview. Fill in the “Group No.” in the line indicated near the topnumbering the groups “1, 2, 3, 4”, etc.). On a separate piece of paper, at thestart of each group interview, list all the names of all the participants in thatgroup and assign a “participant code” to each individual – e.g. in Group 4, JeffJones is “Participant 1” and is assigned the code “P1”, Tom Taylor is “Participant2” and is assigned the code “P2”, etc. Enter each child’s code in the appropriatespace in #1. Then write the age, grade, and gender of each child beside his orher I.D. code, making sure you enter the right data for each respectiveparticipant. If you have more than 6 participants in a group, add theirinformation underneath the others or on the back.After you fill in the Participant Information, read the bold italics “Introductiontext” out loud to explain the purpose of the interview.Only one format (group interview) is provided for participant interviews. If youend up interviewing any participants individually, use the same form, but insteadof recording the number of responses, just place a check mark or X beside theparticipant’s response. Remember to assign an I.D. number (participant code) toeach participant, whether interviewed in a group or individually – and keep arecord of the names corresponding to the I.D. numbers on a separate sheet ofpaper.Follow the directions carefully regarding where to record the number ofresponses, and where to write down participants’ comments for open-endedquestions. If you get more than one of the same comment for any open-endedquestion, indicate in brackets following that comment the number of similarcomments or responses you get.Remember to thank the participants at the end!from House of Friendship, Kitchener-Waterloo, ON, 20043Evaluation Toolkit for Community Youth Programmers Appendix 17

R11 (pg 1 of 2)Community Program Evaluation Checklist1. Does the program have some form of program logic model or evaluationmodel with consideration of outcomes and impact of programming?2. Does it have a target group and plan for outreach, recruitment and sustainedparticipation of all individuals?3. Does it have a feasible theory of change underlying programming? 4. Does it have a written description of activities?(e.g., details of delivery, lesson plans, resources needed, scheduling.)5. Is there a specific process for evaluation, agreed upon by necessary parties? 6. Is evaluation developed from the program outset to be periodic and ongoing? 7. Is there a realistic schedule for collecting and analyzing data for reporting and programadjustment?8. Are specific questions and/or concerns addressed by the evaluation? 9. Does it measure outcomes by looking at observable indicators? 10. Does the evaluation include qualitative data from participants, volunteers,parents and others?11. Does it compare data from individuals over time to capture change? 12. Is it possible to find or create an appropriate comparison group? 13. Are participants involved in developing evaluation methods as appropriate (age, ability)? 14. Is attention paid to confidentiality and ethics in order to protect participants,families and other sources of data?15. Do staff appreciate the importance of evaluation? 16. Do staff have necessary training, resources and support to collect data? 17. Are data collection methods and tools carefully designed and appropriatefor the informants?18. Is data collection monitored? 18The Offord Centre for Child StudiesHands Across Canada

R11 continued (pg 2 of 2)19. Is there scheduled ongoing communication among staff to bring forward andresolve issues?20. Are original data kept secure and protected from misuse? 21. Do the data answer the original questions posed in evaluation planning? 22. Do managers include staff in discussions of data analysis? 23. Are managers & staff open to unexpected results and how they can be utilized? 24. Are results used for timely program modification and improvement? 25. Are results and ramifications shared inside and outside the program and madeunderstandable to all audiences?26. Are staff and volunteers sufficiently informed to share positive results about the program with the community?Evaluation Toolkit for Community Youth Programmers Appendix 19

C1 (pg 1 of 4)Community data: #1 .RECREATION PARTICIPATION SURVEY: GRADE 9Part A: Something about you.1. The name of the teacher in whose classroom you are completing this survey:2. Where you live:a) In which township do you live?b) Do you live in ? (check one only) [list villages in rural area]Part B: What activities do you do?Here are some questions about what activities you do DURING THIS SCHOOL YEAR. Make an X in theappropriate box.Mark only one box for each statement. Never less than 1 to 3 4 orDURING THIS SCHOOL YEAR once a times a moreweek week times a1 Outside of school hours, I take part in sports witha coach or an instructor.2 Outside of school, I play sports or do physicalactivities WITHOUT a coach or instructor.3 Outside of school hours, I take lessons or attendgroups in art, dance or music.4 Outside of school hours, I take part in art, danceSor music activities WITHOUT adult supervision.I take part in clubs or groups such as GirlGuides, Boy Scouts, Cadets.6 I have a job (Le., work for which you are paid)Please specify:7 Outside of school, I play computer or videogames.8 I watch TV.9 I listen to music.week20The Offord Centre for Child StudiesHands Across Canada

C1 continued (pg 2 of 4)Community data: #1Please answer the same questions about what activities you did LAST SUMMER? Make an X in theappropriate box.Mark only one box for each statement. Never less than 1 to 3 4 orLAST SUMMER once a times a moreweek week times a10 I took part in sports with a coach or an instructor.11 I played sports or did physical activitiesWITHOUT a coach or instructor.12 I took lessons or attended groups in art, dance ormusic.13 I took part in art, dance or music activitiesWITHOUT adult supervision.14 I took part in club or group activities sponsoredby Girl Guides, Boy Scouts, Cadets.15 I had a job. Please specify:16 I played computer or video games.17 I watched TV.week18 I went to over-night camp. No 1 week 2 weeks more19 I went to day camp. No 1 week 2 weeks more3. On average, about how many hours a day do you watch TV (Mark only one box for each column)a) DURING THIS SCHOOL YEAR?o less than one hour a dayo 1-2 hours a dayo 3-4 hours a dayo 5-6 hours a dayo 7 or more hours a dayb) LAST SUMMER?o less than one hour a dayo 1-2 hours a dayo 3-4 hours a dayo 5-6 hours a dayo 7 or more hours a day4. How often do you read for fun (not just for school)? (Mark only one box for each column.)a) DURING THIS SCHOOL YEAR?o every dayo a few times a weeko once a weeko a few times a montho almost neverb) LAST SUMMER?o every dayo a few times a week oonce a weeko a few times a montho almost neverEvaluation Toolkit for Community Youth Programmers Appendix 21

C1 continued (pg 3 of 4)Community data: #129 Ice Skating YES NO YES NO32 Martial arts (e.g., Karate, judo) YES NO YES NO33 Mechanics (e.g., fixing cars) YES NO YES NO34 Model-drawing or model building YES NO YES NO35 Painting / drawing YES NO YES NO36 Photography YES NO YES NO37 Playing a musical instrument YES NO YES NO38 Skate boarding YES NO YES NO39 Ski-ing, snow-boarding YES NO YES NO40 Snowmobiling YES NO YES NO41 Snowshoeing YES NO YES NO42 Soccer YES NO YES NO43 Swimming YES NO YES NO44 Tennis YES NO YES NO45 Toboganning, tubing YES NO YES NO46 Volleyball YES NO YES NO47 Wall or rock climbing YES NO YES NO48 Name any other activityYES NO YES NOYES NO YES NOYES NO YES NOYES NO YES NOYES NO YES NOYES NO YES NONOW... 6. You've reviewed a long list of activities, and we would like you to CHOOSE from all the thingsyou would LIKE to do, those which are your FAVOURITES. Close your eyes, mentally review all theactivities you've considered, and decide on your MOST FAVOURITE, SECOND FAVOURITE, ANDTHIRD FA VOURITE. Put the NUMBERS 1, 2, AND 3 beside your favourites. They are organizedalphabetically to help you locate them.THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU!FOR TAKING PART IN THIS SURVEY.From Family Services of Haliburton County, Ontario, 200322The Offord Centre for Child StudiesHands Across Canada

C1 continued (pg 4 of 4)Community data: #129 Ice Skating YES NO YES NO32 Martial arts (e.g., Karate, judo) YES NO YES NO33 Mechanics (e.g., fixing cars) YES NO YES NO34 Model-drawing or model building YES NO YES NO35 Painting / drawing YES NO YES NO36 Photography YES NO YES NO37 Playing a musical instrument YES NO YES NO38 Skate boarding YES NO YES NO39 Ski-ing, snow-boarding YES NO YES NO40 Snowmobiling YES NO YES NO41 Snowshoeing YES NO YES NO42 Soccer YES NO YES NO43 Swimming YES NO YES NO44 Tennis YES NO YES NO45 Toboganning, tubing YES NO YES NO46 Volleyball YES NO YES NO47 Wall or rock climbing YES NO YES NO48 Name any other activityYES NO YES NOYES NO YES NOYES NO YES NOYES NO YES NOYES NO YES NOYES NO YES NONOW... 6. You've reviewed a long list of activities, and we would like you to CHOOSE from all the thingsyou would LIKE to do, those which are your FAVOURITES. Close your eyes, mentally review all theactivities you've considered, and decide on your MOST FAVOURITE, SECOND FAVOURITE, ANDTHIRD FA VOURITE. Put the NUMBERS 1, 2, AND 3 beside your favourites. They are organizedalphabetically to help you locate them.THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU!FOR TAKING PART IN THIS SURVEY.From Family Services of Haliburton County, Ontario, 2003Evaluation Toolkit for Community Youth Programmers Appendix 23

C2 (pg 1 of 3)(Children’s Skills Through Activities and Recreation program)COMMUNITY NEEDSASSESSMENTHOUSEHOLD FORMHOUSEHOLD 1.0.UNIT NUMBERNAME OF SURVEYORSFrom Christie Lake Kids (formerly Christie Lake Community Centre, Ottawa, ON, 2003.24The Offord Centre for Child StudiesHands Across Canada

C2 continued (pg 2 of 3)Community needs assessment (household data)Program Needs AssessmentHousehold Form1. Form # 0012. Household ID #1 2 34 5 63. Name of respondent4. Address5. Date of interview day day month yearmonthIyear7 8 9 10 1 1 126. Number of Adults (aged 17+) who arecurrently live in the home88 not applicable 99 do not know13 147. Number of children aged 11 to 16 yearswho are currently living in the home.88 not applicable 99 do not know8. Number of children aged 6 to 10 whoare currently living in the home.88 not applicable 99 do not know9.Number of children aged 0 to 5 who arecurrently living in the home.88 not99 do not knowapplicable10.Ages and names of all children whoare currently living in the home ages 0 to5 years.88 not99 do not knowapplicable15 1617 1819NameAgeNameAgeNameAgeEvaluation Toolkit for Community Youth Programmers Appendix 25

C2 continued (pg 3 of 3)Community needs assessment (household data)NameAgeNameAge11.Please describe your home situation.Please mark appropriate code.20 21a) biological mother and father 001b) biological mother alone 002c) biological father alone 003f) biological mother & step father 004g) biological father & step mother 005h) biological mother & partner 006I) biological father & partner 007j) adoptive father 008k) adoptive mother 009I) foster mother & father 010m) other (specify) 01112.How long have you lived in the area? # of months # of months# of monthsnot applicable 99 do not know22 23 2413.What language is spoken most in thehome? Please mark one applicable code.a) English 001b) French 002c) Somali 003d) Arabic 004e) Chinese 005f) Other (specIfY) 00625 2626The Offord Centre for Child StudiesHands Across Canada

C3 (pg 1 of 7)COMMUNITYNEEDS ASSESSMENT FORMCHILD FORMHOUSEHOLD 1.0.UNIT NUMBERNAME OF SURVEYORSFrom Christie Lake Kids, formerly Christie Lake Community Centre, Ottawa, ON, 2003.Evaluation Toolkit for Community Youth Programmers Appendix 27

C3 continued (pg 2 of 7)Community needs assessment (child data)Child Form1. Form # 0022. Household ID #3. Child ID #4. Name of child5. Relationship of respondent to child. Pleasemark code that applies.a) Biological Mother 001b) Biological Father 002c) Grandmother 003d) Other relative 004e) Adoptive Mother 005f) Adoptive Father 006g) Foster Mother 007h) Foster Father 008I) Other (specify) 06. Birthdate of Child7. Current agedaymonthyearyearsmonths8. .Sex M F9. Has this child ever attended program camp?Yes0912 13 14 1518 19 201 2 22No1 2 23.28The Offord Centre for Child StudiesHands Across Canada

C3 continued (pg 3 of 7)Community needs assessment (child data)10.lf Yes, How many years has this childattended the Camp?Years88 notapplicable99 do not know24 2511.Are you familiar with the S.T.A.R. program? Yes No1 2 2612.Has this child ever participated in the S.T.A.R.program? Yes No1 2 2713.1f yes, how many years has this child Years Yearsparticipated in the S.T.A.R. program?88 notapplicable99 do not know28 2914. In what activities did this child participate inS.T.A.R. Mark all that apply.88 not applicable 99 do not know01 computers Yes No1 2 3002 swimming Yes No1 2 3103 martial arts Yes No1 2 3204 cooking Yes No1 2 3305 tutoring Yes No1 2 3406 rockers Yes No1 2 3507 arts and crafts Yes No1 2 3608 skating Yes No1 2 3709 hockey Yes No1 2 38010 guitar Yes No1 2 39011 winter camp Yes No1 2 4015.Did this child participate in any coached orsupervised sports activities outside of school inthe past year? 88 not applicable 99 do not know Yes No1 2 4116.lf yes, what activities were they? Mark all that Yes Noapply. 88 not applicable 99 do not know.Evaluation Toolkit for Community Youth Programmers Appendix 29

C3 continued (pg 4 of 7)Community needs assessment (child data)02 skating Yes No1 2 4303 soccer Yes No1 2 4404 hockey Yes No1 2 4505 basket ball Yes No1 2 4606 football Yes No1 2 4707 other (specify) Yes No1 2 4817.Did this child participate in any supervisedarts activities outside school in this past year?88 not applicable 99 do not know Yes No1 2 4918.If yes, what activities were they? Mark allthat apply. 88 not applicable 99 do not know01 art Yes No1 2 5002 dance Yes No1 2 5103 pottery Yes No1 2 5204 crafts Yes No1 2 5305 choir Yes No1 2 5406 other (specify) Yes No1 2 5519.Did this child participate In a supervisedcommunity organization in the last year? (e.g.boys scouts, girl guides).88 not applicable99 do not knowYes No1 2 5620.If yes, what organizations were they?Mark all that apply.01 Boy Scouts Yes No1 2 5702 Girl Guides Yes No1 2 5803 Boys and Girls Club Yes No1 2 5904 YMCA Yes No1 2 6005 Church group Yes No1 2 6106 other (specify) Yes No1 2 62.30The Offord Centre for Child StudiesHands Across Canada

C3 continued (pg 5 of 7)21.What sports activities would you like to seethis child involved in with S.T.A.R.? Mark all thatapply. 88 not applicable99 do not know01 swimming Yes No1 2 6302 hockey Yes No1 2 6403 skating Yes No1 2 6504 Rockers (games & activities) Yes No1 2 6605 martial arts Yes No1 2 6706 basketball Yes No1 2 6822.What supervised art activities would you liketo see this child involved in with S.T.A.R.?Mark all that apply.01 dance Yes No1 2 6902 art Yes No1 2 7003 pottery Yes No1 2 7104 arts and crafts Yes No1 2 7205 music Yes No1 2 7323.What other activities would you like to seethis child involved in with the S.T.A.R. program?01 leadership in training (aged 15 +) Yes No02 winter camp Yes No03 tutoring Yes No04 cooking Yes No05 computers Yes No06 other (specify) Yes No24.What barriers have you experienced in tryingto get your child involved in these sports, artsand community activities? Please mark all thatare applicable. 88 not applicable 99 do not know01 financial Yes No1 2 741 2 751 2 761 2 771 2 781 2 791 2 80.Evaluation Toolkit for Community Youth Programmers Appendix 31

C3 continued (pg 6 of 7)02 transportation Yes No03 behavioural difficulties Yes No04 racial and/or religious Yes No1 2 811 2 821 2 8305 difficulty motivating child to participate or attendprograms Yes No1 2 8406 child is behind in skill levels Yes No1 2 8507 0ther (specify) Yes No1 2 8625.What school does this child attend? Pleasemark one that applies. 88 not applicable 99 do not .Know. (List.)a)b)c)d)e)f)g)01 87 8802 89 9003 91 9204 93 9405 95 9606 97 9807 99 100h) Other (specify)08 101 10226.Has your child ever repeated a grade? 88 notapplicable 99 do not know Yes No1 2 10327.1f yes, what is the first grade your childfailed? Please mark appropriate code. 88 notapplicable99 do not knowa) grade one 01 104 105b) grade two 02 106 107c) grade three 03 108 109d) grade four 04 110 111e) grade five 05 112 113f) grade six 06 114 115g) grade seven 07 116 117.h) grade eight 08 118 11932The Offord Centre for Child StudiesHands Across Canada

C3 continued (pg 7 of 7)Community needs assessment (child data)I) grade nine 09 120 121j) grade ten 10 122 123k) grade eleven 11 124 12528.Have they now or ever been in a special classfull or part-time? Yes No1 2 12629.lf yes, please specify. Please markappropriate code.a) Full time currently 0 127 12801b) Part-time currently 002c) Full time in past 003d) Part-time in past 0043D.Has this child ever been charged, arrested ormade an appearance in court?88 not applicable 99 do not know Yes No1 2 12931.Do you and this child currently receive orhave you in the past received any support fromany social service agency? 88 not applicable 99 donot know Yes No1 2 13032. If yes, please mark which agency you andyour child are currently or have in the past beeninvolved with.a) Children's Aid Society 001 131 132b) Youth Services Bureau 002 133 134c) Crossroads 0d) Hospital (specify) 0e) Other (specify) 003 135 13604 137 13805 139 14033.Does this child have any learning disabilitiesthat may require special attention whileattending our programs?88 not applicable Yes No99 do not know 1 2 14134.Does this child have ADD, ADHD, emotionalor any other behavioural problems that mayrequire any special attention while attending ourprograms?88 not applicable99 do not knowYesNo1 2 142.Evaluation Toolkit for Community Youth Programmers Appendix 33

P1REGISTRATION FORM(children’s program)Program Name:Session Dates:Name of Child:(First)Date of Birth:Address:(Middle)(Last)M/FTelephone Number:Name of Parent/ Legal Guardian:Name of Emergency Contact:Telephone Number:Grade:Name of School:Name of Teacher:Languages Spoken at Home:Family's Country of Origin:If family is not originally from Canada, number of years in Canada:Income Level: 20,000/yr___ 20,000- 40,000/yr___ 40,000/yr and up___Please indicate any medical concerns (e.g., food allergies)I give permission for my child to attendSignedDateParent/GuardianI give permission for my childunaccompanied at the end of the evening (Yes / No).to walk homeSignedParent/GuardianDateI give permission for photographs to be taken of my child for evaluation, education and publicitypurposes only.SignedParent/GuardianDateAdapted from Haldimand-Norfolk Women’s Services, Ontario, 2002.34The Offord Centre for Child StudiesHands Across Canada

P2(girls’ camp experience, mid-program)Parent Mid-program Telephone SurveyCamper's Name:Parent's Name:Date of contact: (yr/mo/day)Remind parent/guardian about o/n and what to bring (pyjamas, blankets, sleepingbag, change of clothes, pillow, toiletries, etc.). Remind parents of our rules andexpectations. Double check the emergency contact number for the week end.In an effort to improve our program, we would appreciate you taking a couple ofmoments to answer a few questions.1.What has your daughter's impression been so far? What has she told you aboutthe program?2. What activities has she enjoyed?3. What has she not enjoyed?4. Do you have any other feedback for us?5. Do you or your daughter have any questions or concerns about theovernight camping coming up?Adapted from Haldimand-Norfolk Women’s Services, Ontario, 2003.Evaluation Toolkit for Community Youth Programmers Appendix 35

P3 (pg 1 of 2)PARENT’S SURVEY: Getting to Know Your ChildChild’s Name:Your relationship to the child:Child’s age:Has your child participated in past sessions of this program?***********Date:1. We are interested to know how you would describe your child’s overall developmentallevel, unique characteristics and special needs.2. Your responses to the following questions will provide background information aboutyour child’s movement ability and experience with various art mediums.MUSIC:Does your child:- Listen to music at home?- Listen to different kinds of music?- Play with rhythm instruments at home?- Like to sing?- Like to dance?- Have a strong interest in music?- Other comments?Commentsfrom Alexander Society for Special Needs, Nova Scotia, 2002.”36The Offord Centre for Child StudiesHands Across Canada

P3 continued (pg 2 of 2)ART:Does your child:- Like to colour?- Like to draw?- Like to paint?- Like to play with play dough?- Use other art materials at home?- What is his/her favourite art activity?STORIES/DRAMA:Does your child:- Like to be read to?- Like to dress up in play costumes?- Participate in imaginative play activities?- Act out different pretend characters with his/her toys?CommentsCommentsMOVEMENT:Does your child:- Like to clap or tap out a beat to a song?- Like to skip or hop?CommentsIs your child able to follow short visual directions for hand/arm movements?How would you describe your child’s physical coordination?3. What are your expectations of this program for your child?4. Do you anticipate any effect on you and/or your family from having your child attendthis program?from Alexander Society for Special Needs, Nova Scotia, 2002.”Evaluation Toolkit for Community Youth Programmers Appendix 37

P4 (pg 1 of 3)(post-program phone interview)Parents' Survey –Time TwoWe are asking you, once again, to kindly answer some questions as partof our final round of program evaluation for this session of the creative play program.Your cooperation is greatly appreciated.Child's Name:Date:Your relationship to the child:We are interested to know how participation in this program has affected your child in positive ornegative ways.1. Have you noticed changes in any of the following areas related to Music?- Interest in and enjoyment of music?- Concentration and listening skills?- Confidence to express his/her self through music?- Forms of self-expression through music?- Other changes related to musical expression and interest?2. Have you noticed changes in any of the following areas related to Art?- Interest in and enjoyment of art activities?- Observation skills?- Awareness of and interest in colour? _- Coordination and control of art materials?- Use other art materials at home?- Ability and/or interest to express his/her self through art?- Other changes related to artistic interest and self-expression?From Alexander Society for Special Needs, Nova Scotia, 2002 138The Offord Centre for Child StudiesHands Across Canada

P4 continued (pg 2 of 3)3. Have you noticed changes in any of the following areas related to stories and drama?- Interest in hearing stories or in story telling?- Interest to engage in imaginative play?- Confidence to engage in imaginative play?- Other changes related to stories and imaginative play?4. Have you noticed changes in any of the following areas related to Movement?- Improvement in fine motor skill?- Co-ordination?- Improvement in gross motor skills?- Rhythmical skills and ability?- Body awareness?- Other changes related to movement and control?5. Have you noticed changes in any of the following areas related to Social andPersonal Development? .- Awareness of self?- Self control? ( e.g. patience)- Increased independence?- Awareness of others?- Social skills?- Development of new friendships?- Other changes related to social and personal growth?From Alexander Society for Special Needs, Nova Scotia, 2002 2Evaluation Toolkit for Community Youth Programmers Appendix 39

P4 continued (pg 3 of 3)6. What do you feel are the principal changes you have observed in your childthat directly relate to their participation in the program?7. Do you feel you and/or your family have been affected in any ways as a resultof your child's participation in this program? If so, please elaborate.8. Do you have any recommendations or feedback for the facilitators of thisprogram? Or any additional comments you would like to make?Thank you!From Alexander Society for Special Needs, Nova Scotia, 2002 340The Offord Centre for Child StudiesHands Across Canada

P5Parents’ Post-program EvaluationDate:_________________(children’s social skills program)day/mo/yr(one evaluation form per child please)1. Did your child like being in the group?2. What did your child tell you about the group? What did your child likedoing and not like doing?3. Were there any noticeable changes in the behaviour of your child duringor after the program?Parents’ Post-programDate:____________(children’s social skills prog(one evaluation form per c1. Did your child like bein4. What did you think about the day, time, size, or activities of theprogram?2. What did your child tedoing and not like doing5. Would you like any other follow-up services for your child? Please explain.3. Were there any noticeor after the program?6. Any other comments?Parent’s name or ID _____________________________From Overbrook-Forbes Community Resource Centre, Ottawa, 20024. What did you think aboprogram?Evaluation Toolkit for Community Youth Programmers Appendix 415. Would you like any oth

P6 (pg 1 of 4)I.D. # ________Parent questionnaire(mid or post-program)Part 1: General QuestionsDate: _____/______ /______day / month / yearInstructions: Please circle the answer that is best for you.1) How did you hear about the program?a) a friendb) my child’s teacherc) my child’s friendd) an advertisement(where exactly did you see it?)______________________________e) other (please say how or where you found out)2) Why did you decide to let your child come to the program?(circle all the reasons that are correct for you)a) it is a free programb) it has the 'walking school bus' to pick my child up from schoolc) it gives my child somewhere safe and supervised to go after schoold) my child will learn more Englishe) my child will get help with homeworkf) my child will learn about art and craftsg) my child will be active in sports and gamesh) my child will receive a nutritious snacki) my child will learn about musicj) my child will have other children to play withk) my child will learn to get along with others3) Have you ever told another parent, a child or one of your friends they should come to theprogram? yes no4) Have you learned about other community resources through the program? Places to go forassistance, information or other activities for you or your child? yes noPlease explain.From the Christie-Ossington Neighbourhood Centre, Toronto, 2003. 142The Offord Centre for Child StudiesHands Across Canada

P6 continued (pg 2 of 4)I.D. # ________Part II: Your Opinion of the ActivitiesFor the following questions, please circle the number which best describes your opinion.1 = my child never enjoys this activity2 = my child hardly ever enjoys this activity3 = my child sometimes enjoys this activity and sometimes doesn’t4 = my child enjoys it very much, most of the time5 = my child always enjoys it a lot1. On the scale of 1-5, how does your child enjoy the reading and writing activities at theprogram? (books, reading circle, mail, the newspaper etc.) 1 2 3 4 5Comments?_______________________________________________________________2. How does your child enjoy getting help with homework at the program? 1 2 3 4 5Comments? _______________________________________________________________3. How does your child enjoy making Art projects at the program? 1 2 3 4 5Comments? _______________________________________________________________4. How does your child enjoy the game and sports at the program? 1 2 3 4 5Comments? _______________________________________________________________5. How does your child enjoy the music workshops at the program? 1 2 3 4 5Comments? _______________________________________________________________Part III: Behaviour Changes You Have NoticedFor the following questions, please circle the number which best describes your opinion.1 I have seen no improvement since my child started coming to the program2 I have seen a very small improvement3 I have seen a fair improvement4 I have seen a big improvement5 I have seen an excellent improvement1. By circling a number on the scale below, please rate any changes in the your child'sability to read, write, understand or speak English.1 2 3 4 5Comments?_______________________________________________________________from the Christie-Ossington Neighbourhood Centre, Toronto, 2003. 2Evaluation Toolkit for Community Youth Programmers Appendix 43

P6 continued (pg 3 of 4)I.D. # ________2. Please rate on the scale any change in your child's interest in arts and crafts and creatingnew things.1 2 3 4 5Comments?_______________________________________________________________3. Please rate on the scale any change in your child's interest in playing sports and beingactive.1 2 3 4 5Comments?_______________________________________________________________4. Please rate on the scale any change in your child's interest in healthy eating.1 2 3 4 5Comments?_______________________________________________________________5. Please rate on the scale any change in your child's interest in music and making music.1 2 3 4 5Comments? ?_______________________________________________________________6. Please rate on the scale any change in how your child values himself (or herself) and actstowards others.1 2 3 4 5Comments? ___________________________________________________________________________________________________________________________________________________________________________________________________________________Part IV: How Can We Improve?1) What changes would you make to the program, if any? How would they improve it?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________2) What features and activities do you like most about the program? Why are they good?____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________3) What features and activities do you like least about the program? What are your concerns?__________________________________________________________________________________________________________________________________________________from the Christie-Ossington Neighbourhood Centre, Toronto, 2003. 344The Offord Centre for Child StudiesHands Across Canada

P6 continued (pg 4 of 4)I.D. # ______________________________________________________________________________________________________________________________________________________________4) Would you recommend the program to your friends or other parents? Please explain why orwhy not.______________________________________________________________________________________________________________________________________________________________________________________________________________________5) Any additional comments or suggestions?____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Thank you for answering these questions.They will help us to make a better program for your children.from the Christie-Ossington Neighbourhood Centre, Toronto, 2003. 4Evaluation Toolkit for Community Youth Programmers Appendix 45

P7 - See instructions R10 (pg 1 of 5)PARENT Interview Form(AFTER-SCHOOL PROGRAM)Program Site: __________________________________Date ______________________Day/ month/ yearYour initials: ________We are asking the parents of children who attended the After-School Program thisyear what they thought about it and if there are ways we can make it better fornext year. We won’t identify your comments by name and your answers will begrouped together with those from other parents to find out general ideas aboutthe program.1. How many children did you have attending the After-School Program? ___________a) What are their genders, ages and grades in school?Gender (M/F) Age Grade in School 2003/04Child #1 _______ _______ ________Child #2 _______ _______ ________Child #3 _______ _______ ________Child #4 _______ _______ ________2. How long did your child(ren) attend the After-School Program?Child #1: From _______________ to ________________Child #2: From _______________ to ________________Child #3: From _______________ to ________________Child #4: From _______________ to ________________3. How often, on average, did your child(ren) attend the After-School Program? (indicatenumber of children for each category)Once in a while _________Once a week _________Twice a week _________Three times a week __________Four times a week _________Every day __________b) If your children didn’t attend the After-School Program every day, what did they do afterschool (3:30 to 5:30 p.m.) on the days that they weren’t in the After-School Program?____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________from House of Friendship, Kitchener, ON, 2004 146The Offord Centre for Child StudiesHands Across Canada

P7 continued (pg 2 of 5)4. Why did you want your child(ren) to attend the After-School Program?______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________5. Do you think the After-School Program has had any effect on your child(ren)?Yes ________ No _________a) If yes, what kinds of things have you noticed?__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________6. Has your child(ren)’s school performance changed since they began attending the After-School Program?Yes __________ No __________a) If yes, in what ways?___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________b) What do you think are the reasons for this?____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________from House of Friendship, Kitchener, ON, 2004 2Evaluation Toolkit for Community Youth Programmers Appendix 47

P7 continued (pg 3 of 5)7. Have you seen any changes in your child(ren)’s ability to get along with others sincethey began attending the After-School Program?Yes _______ No _______a) If yes, please describe these.__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________b) What do you think are the reasons for this?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________8. What do you feel your children have learned from participating in the After-SchoolProgram?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________9. What do/did you like about the After-School Program?_____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________from House of Friendship, Kitchener, ON, 2004 348The Offord Centre for Child StudiesHands Across Canada

P7 continued (pg 4 of 5)10. Did your child(ren) ever talk to you about what goes on at the After-School Program?Yes _______ No _______a) If yes, did they say what they liked about the program? Please be as specific as youcan._____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________b) Did they say what they didn’t like about the program? Please be specific._____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________11. What would your child(ren) be doing or have been doing after school (3:30 to 5:30p.m.) if they weren’t participating in the After-School Program?_____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________12. Have your children made any new friends in the After-School Program?Yes _______ No _______ Don’t know _______a) If yes, how many? ___________b) Do your children play with friends from the After-School Program when they’re not atthe program?Yes _______ No _______ Don’t know _______from House of Friendship, Kitchener, ON, 2004 4Evaluation Toolkit for Community Youth Programmers Appendix 49

P7 continued (pg 5 of 5)13. How many people in the neighbourhood could your child(ren) turn to if they neededhelp? _____________________________________________________________a) How sure are you that these people would be willing to help if your child(ren) askedfor help?Very sure ________Fairly sure _______Not sure at all ________14. Do you have any suggestions for changes or improvements to the After-SchoolProgram?_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________15. Do you plan to send your child(ren) to the After-School Program next fall?Yes _______ No _______ Not sure ________a) Why or why not?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________16. Do you have any other comments - is there anything else you’d like to tell us about theAfter-School Program?______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Thank you for your response!from House of Friendship, Kitchener, ON, 2004 550The Offord Centre for Child StudiesHands Across Canada

P8 - See instructions R10 (pg 1 of 6)PARENT GROUP INTERVIEW FORMProgram Site: _____________________________________Group No. _______Date ___________Day/mo/yrName(s) of staff person(s) conducting interview: __________________________________________________________[Introduction text:] We’re talking to the parents of kids who attended the After-School Program this year to find out what they thought about it and if there areways we can make it better for next year. We won’t write your name on yourcomments, and we’ll group your answers together with answers from otherparents to find out general ideas about the program.1. How many children did you have attending the After-School Program, and what are theirgenders, ages and grades in school?Participant No. of Children attending Gender/Age/Grade of each childCode After-School Program attending After-School ProgramParent 1: __________ ___________ ______________________Parent 2: __________ ___________ _____________________Parent 3: __________ ___________ _____________________Parent 4: __________ ___________ _____________________Parent 5: __________ ___________ _____________________Parent 6: __________ ___________ _____________________2. How long did your child(ren) attend the After-School Program (from when till when)?(indicate number of children for each category)From September to June ___________From October to June ____________From November to June __________From December to June __________From January to June ___________From February to June ___________From March to June _____________From April to June ____________From May to June ___________Other (specify):From __________ to ____________From __________ to ____________3. How often, on average, did your child(ren) attend the After-School Program? (indicatenumber of children for each category)Once in a while _________Once a week _________Twice a week _________Three times a week __________Four times a week _________Every day __________from House of Friendship, Kitchener, ON, 2004 1Evaluation Toolkit for Community Youth Programmers Appendix 51

P8 continued (pg 2 of 6)(For the remaining questions, list all the things parents say; if more than oneparent says the same thing, indicate in brackets after the comment how manyparents made that comment)b) If your children didn’t attend the After-School Program every day, what did they do afterschool (3:30 to 5:30 p.m.) on the days that they weren’t in the After-School Program?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________4. Why did you want your child(ren) to attend the After-School Program?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________5. Do you think the After-School Program has had any effect on your child(ren)?Yes (indicate number of responses) _________No (indicate number of responses) _________a) (If yes) What kinds of things have you noticed?__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________6. Has your child(ren)’s school performance changed since they began attending the After-School Program?Yes (indicate number of responses) __________No (indicate number of responses) __________from House of Friendship, Kitchener, ON, 2004 252The Offord Centre for Child StudiesHands Across Canada

P8 continued (pg 3 of 6)a) (If yes) In what ways?(Remember to list all parents’ comments; if more than one parent repeats acomment, put in brackets behind that comment the number of parents makingthat comment)________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________b) What do you think are the reasons for this?____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________7. Have you seen any changes in your child(ren)’s ability to get along with others sincethey began attending the After-School Program?Yes (indicate number of responses) _______No (indicate number of responses) _______a) (If yes) Please describe these.__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________b) What do you think are the reasons for this?______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________from House of Friendship, Kitchener, ON, 2004 3Evaluation Toolkit for Community Youth Programmers Appendix 53

P8 continued (pg 4 of 6)(Remember to list all parents’ comments; if more than one parent repeats acomment, put in brackets behind that comment the number of parents makingthat comment)8. What do you feel your children have learned from participating in the After-SchoolProgram?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________9. What do/did you like about the After-School Program?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________10. Did your child(ren) ever talk to you about what goes on at the After-School Program?Yes (indicate number of responses) _______No (indicate number of responses) _______a) (If yes) Did they say what they liked about the program? Please be as specific as youcan.___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________b) Did your children say what they didn’t like about the program? Please be specific.______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________from House of Friendship, Kitchener, ON, 2004 454The Offord Centre for Child StudiesHands Across Canada

P8 continued (pg 5 of 6)(Remember to list all parents’ comments; if more than one parent repeats acomment, put in brackets behind that comment the number of parents makingthat comment)11. What would your child(ren) be doing or have been doing after school (3:30 to 5:30p.m.) if they weren’t in the After-School Program?___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________12. Have your children made any new friends in the After-School Program?Yes (indicate number of responses) _________No (indicate number of responses) _________Don’t know (indicate number of responses) __________a) (If yes) How many? (indicate number of responses for each category)One ___________Two ___________Three __________Four or more _________b) Do your children play with friends from the After-School Program when they’re not atthe program?Yes (indicate number of responses) ___________No (indicate number of responses) ___________Don’t know (indicate number of responses) ___________13. How many people in the neighbourhood could your child(ren) turn to if they neededhelp? (indicate number of responses for each category)One _________Two _________Three ________Four or more __________a) How sure are you that these people would be willing to help if your child(ren) askedfor help? (indicate number of responses for each category)Very sure ________Fairly sure _______Not sure at all ________14. Do you have any suggestions for changes or improvements to the After-SchoolProgram? Yes (indicate number of responses) __________No (indicate number of responses) __________from House of Friendship, Kitchener, ON, 2004 5Evaluation Toolkit for Community Youth Programmers Appendix 55

P8 continued (pg 6 of 6)(Remember to list all parents’ comments; if more than one parent repeats acomment, put in brackets behind that comment the number of parents makingthat comment)a) (If yes) What suggestions do you have?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________15. Do you plan to send your child(ren) to the After-School Program next fall?Yes (indicate number of responses) __________No (indicate number of responses) __________a) (If yes) Why?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________b) (If no) Why not?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________16. Do you have any other comments - is there anything else you’d like to tell us about theAfter-School Program?_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Thanks for talking to us, we appreciate you spending the time to help usfind out more about the program’s impact.from House of Friendship, Kitchener, ON, 2004 656The Offord Centre for Child StudiesHands Across Canada

Pt1(employment skills program)Participation Form The following form needs to be completed before a participant can receive their training allowance.Month:Name:Centre:__________________________________________________________________________________________________________________________________________________________________DateProgramHoursActivitiesParticipantInitialsCoordinator’sInitialsWinnipeg Boys and Girls Clubs, Manitoba, 2004.Evaluation Toolkit for Community Youth Programmers Appendix 57

Pt2Participant Exit Interview Date: yr/mo/day______________Name or ID : _________________(employment skills program)1. What did you like best about being part of theprogram?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________2. What did you find frustrating about theprogram?____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________3. What things have you learned that will help you findwork?___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________4. What do you think employers expect from theiremployees?___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________5. What are some new things you got to try or do because you are in theprogram?_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________6. What careers interest you?___________________________________________________________________________________________________________________________________________________________________________________________________________7. What were the three most important things you learned during this program?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Adapted from the Winnipeg Boys and Girls Clubs, Step-Up program, Manitoba, 200458The Offord Centre for Child StudiesHands Across Canada

Pt3Focus Group Questions After Camp TripDate: (yr/mo/day)Name of interviewer:Name(s) of participants:1. Tell me about yourself.2. Do you feel your training prepared you for the adventure trip?3. How well did you get along with the other members of your group at camp? Yourcounselor? People at the campground? Scout Interpreter (ie. Name person)?4. When you think about the camp, what are three feelings that you think about?5. How did the camp make you feel about yourself?6. What were the most and least favorite activities? And why?7. Do you feel you could generalize or take what you learned back to the campus?8. How was the length of the trip? Was it too long, too short or just right?From Wood’s Homes, Calgary, AB, 2004Evaluation Toolkit for Community Youth Programmers Appendix 59

Pt4 (pg 1 of 4)Participant questionnaire: pre- and post-programSECTION A: FRIENDS AND FAMILYThe following statements are about your friends and family.Please answer each question.Mark your answers like this1. I get along with kids easily. False Mostly false Sometimes false / Sometimes true Mostly true True2. About how many days a week do you dothings with friends outside of school hours? Never Less than once a week 1 day a week 2-3 days a week 4-5 days a week 6-7 days a week3. Other than your parents or guardians, arethere any other adults who you can talk to andwho help you? Yes Go to question 4 NoSkip the next few questions and goto Question 8From Overbrook-Forbes Community Resource Centre, Ottawa, ON, 2003. Based on selected questions supplied byOfford Centre for Child Studies.60The Offord Centre for Child StudiesHands Across Canada

Pt4 continued (pg 2 of 4)4. What is their relationship to you?(Mark everyone you feel you can talk toabout yourself and your problems.) Grandparents Older brother or sister Aunt, uncle or other relative A friend of the family Sitter or babysitter Parent’s boyfriend / girlfriend Teacher Coach or leader (e.g., scout or church leader) Someone else (write down who)_____________________________________________________________________5. Of these people, who do you do most thingswith?(mark only one box) Grandparents Older brother or sister Aunt, uncle or other relative A friend of the family Sitter or babysitter Parent’s boyfriend / girlfriend Teacher Coach or leader (e.g., scout or church leader) Someone else (write down who)_____________________________________________________________________From Overbrook-Forbes Community Resource Centre, Ottawa, ON, 2003. Based on selected questions supplied byOfford Centre for Child Studies.Evaluation Toolkit for Community Youth Programmers Appendix 61

Pt4 continued (pg 3 of 4)6. How often do you spend time with thisperson ? (mark only one box) Most days A few times a week, about once a week Less than once a month About once a month7. What kind of things do you do with thisperson?(mark all the things that you do?) Play sports or do other physical activities Take part in arts and crafts, music or danceactivities Go on outings (e.g., to the zoo, movies, sportsevents, etc.) Do other things (write down some of thesethings)_____________________________________________________________________8. During the past 6 months, how well haveyou gotten along with other children such asyour friends and classmates? Very well, no problems Quite well, hardly any problems Pretty well, occasional problems Not too well, frequent problems Not well at all, constant problems9. During the past 6 months, how well haveyou gotten along with your brothers andsisters? Very well, no problems Quite well, hardly any problems Pretty well, occasional problems Not too well, frequent problems Not well at all, constant problemsFrom Overbrook-Forbes Community Resource Centre, Ottawa, ON, 2003. Based on selected questions supplied byOfford Centre for Child Studies.62The Offord Centre for Child StudiesHands Across Canada

Pt4 continued (pg 4 of 4)SECTION : MYSELFThe following statements are about you and how you deal with problems.Please answer each question.Mark your answers like this1. If another child frustrates me, I am mostlikely to: Talk to the other child about it Take a deep breath and move somewhere else Ignore him/her Try to get others to dislike him/her Attack the other child, by yelling or hitting2. I offer to help other children (friend, brotheror sister) who are having difficulty with a task. False Mostly false Sometimes false / Sometimes true Mostly true TrueFrom Overbrook-Forbes Community Resource Centre, Ottawa, ON, 2003. Based on selected questions supplied byOfford Centre for Child Studies.Evaluation Toolkit for Community Youth Programmers Appendix 63

A1 (pg 1 of 3)Volunteer Service Questionnaire for AgenciesAs part of our ongoing program evaluation, we are asking agencies and other sponsors with whom weplace volunteers to answer the following questions. Please have the person who would be mostknowledgeable about the performance of the selected volunteer complete this questionnaire. Youranswers will help us monitor the progress of our clients and adjust our program to reflect their needs.Name of Volunteer being assessed_________________________________________Your Program Name: _____________________________________Date: [ ]/ [ ] / [ ]day/ mo./ yr.Filled out by Name: __________________________________Position: _________________________________Tel: [ ][ ][ ][- [ ][ ][ ] ex. [ ][ ][ ][ ]1. How many hours has this participant worked as a volunteer for your program? [ ][ ][ ]2. Is this placement a one-time only event? yes [ ]no [ ]3. Please circle the total duration of the volunteer placement.1 = 1 day but less than 54 = 6 to 20 days5 = >20 days4. Did the volunteer arrive when he/she was scheduled?(Please circle the number next to your answer on the following scale. For one-time-onlyplacements, please circle 1 for no and 5 for yes.)1 = never2 = sometimes3 = frequently4 = almost always5 = always5. Did he/she stay for the full duration of the scheduled time?(Please circle the number next to your answer on the following scales. For one-time-onlyplacements, please circle 1 for no and 5 for yes.)1 = never2 = sometimes3 = frequently4 = almost always5 = alwaysFrom Youth Employment Services and Volunteer Youth in the Millennium, Thunder Bay, ON, 2003.64The Offord Centre for Child StudiesHands Across Canada

A1 continued (pg 2 of 3)6. Did he/she have difficulty following instructions as given?1 = never2 = sometimes3 = frequently4 = almost always5 = always7. Did the volunteer ask any questions that were needed in learning to perform the dutiescorrectly?1 = never2 = sometimes3 = frequently4 = almost always5 = always8. Was the volunteer able to work cooperatively with staff and other volunteers in the program?1 = never2 = sometimes3 = frequently4 = almost always5 = always6 = not applicable9. If the volunteer had contact with people other than staff, was he/she able to interact with themin a satisfactory manner?1 = never2 = sometimes3 = frequently4 = almost always5 = always6 = no contact10. If the volunteer had some problem interacting with other people, please explain, (e.g.,language skills, attitude)________________________________________________________________________________________________________________________11. Have you made arrangements with the volunteer to return for additional work? yes [ ]no [ ]12. Were you satisfied with this volunteer? yes [ ]no [ ]2Evaluation Toolkit for Community Youth Programmers Appendix 65

A1 continued (pg 3 of 3)13. Do you have any comments that would help the volunteer improve his/her performance?______________________________________________________________________________________________________________________________________________________________________________________________________________________________14. Would you welcome him/her back for another placement if you were to accept volunteers infuture? yes [ ]no [ ]15. Are you willing to accept any volunteers in future? yes [ ]no [ ]We appreciate your taking the time to help us with our program evaluation.From Youth Employment Service and Volunteer Youth in the Millennium, Thunder Bay, ON, 2003 366The Offord Centre for Child StudiesHands Across Canada

T1 (pg 1 of 2)(literacy tutoring program)Questionnaire for Resource Teachers:Post- tutoring survey of help/support to tutorsDate of contact:Name of School Resource Teacher/Teacher:Name of School:Length of time involved in the program:Grade(s) taught:Our research with the tutors is showing that they learned a great deal about reading instruction,reading strategies, and strategies for children with learning disabilities. We don't know just howor from whom they learned these things. For instance, how much was hands on experience tromworking with the child and using the tutor manual, and how much resulted from guidance orresource material they gained trom the school resource teachers, or other staff associated withthe program.1. Think about your interactions with individual tutors and what the tutors might have learnedform you at those times. Could you describe the nature of those interactions even if they werevery informal?2. Did you provide any handouts, suggestions for activities that you might have provided toindividual tutors? What were the titles of the handouts?3. Did you have any group meetings and if you did what was the nature of those meetings?4. Could you describe any other learning that took place with the tutors that we have notcovered? Were there any specific materials or reading ‘special needs’ informationcommunicated? .From Learning Disabilities Association of Peterborough and Trent School of Education and ProfessionalLearning, Peterborough ON, 2002.Evaluation Toolkit for Community Youth Programmers Appendix 67

T1 continued (pg 2 of 2)Questionnaire for Resource Teachers cont.5. What were the most common strategies or materials that you gave to tutors to use with theirchildren?6. What are your thoughts about how the tutors learned so much?Thank you very much for taking the time to answer these questions.From Learning Disabilities Association of Peterborough and Trent School of Education and ProfessionalLearning, Peterborough ON, 2002.68The Offord Centre for Child StudiesHands Across Canada

T2 - See instructions R10 (pg 1 of 2)TEACHER QUESTIONNAIRE(AFTER SCHOOL PROGRAM)SCHOOL __________________________________Date: _______________YR/MO/DAYGRADE _____________1. Did you know before this which of your studentsparticipated in the After School Program? Yes _______ No _______2. Looking at the list provided, what percentage of childrenin your class attended the After School Program?(please circle the appropriate choice)a) less than 25%b) 25-50%c) over 50%3. How aware are you of the activities that take place in the After School Program?a) completely unawarea) somewhat awareb) reasonably awarec) very aware4. Based on what you have heard parents say, what is the reason most children attendthe After School Program?(Circle all responses that apply)a) because of a lack of sufficient after-school care in the communityb) because of the homework and computer help availablec) because of recreational sports, arts and musical activitiesd) because of opportunities to play with other childrene) because of adult supervision availablef) don’t know5. Have you ever referred parents to the After School Program for their child?Yes _______No _______SCALE FOR NEXT QUESTIONS:1. no improvement noticed2. minimal improvement noticed3. fair improvement noticed4. great improvement noticed5. remarkable improvement noticed6. During this school year, did you see any change in how frequently After SchoolProgram participants completed their homework?Please circle the appropriate number (referring to above scale.)Scale: 1 2 3 4 57. Did you see any change in After School Program participants’ ability to do theirassignments and class work?Please circle the appropriate numberScale: 1 2 3 4 5from House of Friendship, Kitchener, ON, 2004 1Evaluation Toolkit for Community Youth Programmers Appendix 69

T2 continued (pg 2 of 2)8. Did you notice any change in After School Program participants’ social interaction?e.g. ability to get along with others, make friends, include others, co-operate withothers, attempts to resolve conflict peacefully, etc.Please circle the appropriate numberScale: 1 2 3 4 5Comments:___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________9. Have you noticed any other changes in After School Program participants at school?Yes _______ No _______[If yes] What were they?_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________10. Have you heard any comments from children about the After School Program? (e.g.,activities, experiences, relationships)Yes _______ No _______[If yes] Could you tell us what they were?_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________11. Have you heard any comments from parents about the After School Program?Yes _______ No _______[If yes] Could you tell us what they were?_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________12. Do you have any suggestions about ways to improve the After School Program?Yes _______ No _______[If yes] Please tell us what they are.______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Thank you for helping us evaluate our After School Program!from House of Friendship, Kitchener, ON, 2004 270The Offord Centre for Child StudiesHands Across Canada

T3 (pg 1 of 2)SCHOOL STAFF POST-PROGRAM EVALUATIONCommunity After School Program(CASP)In order for CASP to meet the program goals and better serve children in the community,feedback is needed from teachers and the school administration. Thank you for your timeand input to the CASP program. Please answer the following questions based on yourobservations of your students in the after school program.1I do not know2Strongly Disagree3Disagree4Agree4Strongly AgreePlease circle the appropriate number according to the scale above.1. I feel the children at this school benefit from the after school program. 1 2 3 4 5Please comment on your reasons..____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________2. There is a need for after school recreational opportunitiesfor children at this school. 1 2 3 4 5Please comment. _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________3. Children’s confidence and self esteem have improved as a result 1 2 3 4 5of this program. (e.g. asks questions in class, participates in class discussions)Please comment.________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________4. The CASP program is valued at our school. 1 2 3 4 5(e.g., staff and administration talk positively about or assist the program)Please comment. _________________________________________________________________________________________________________________________________________________________________________________________________________Adapted from the Albert Park/Radisson Heights Community After-school Program, Calgary AB,2003.Evaluation Toolkit for Community Youth Programmers Appendix 71

T3 continued (pg 2 of 2)5. I have seen improved academic skills in thechildren in my class since being in CASP. 1 2 3 4 5Please specifyhow.____________________________________________________________________________________________________________________________________________________________________________________________________________________6. From my knowledge of families, some of these children would be leftunsupervised without CASP. 1 2 3 4 5Please comment. _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________7. Additional comments to help us improve the program?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Adapted from the Albert Park/Radisson Heights Community After-school Program, Calgary AB,2003.72The Offord Centre for Child StudiesHands Across Canada

V1 (pg 1 of 2)Volunteer Application(web-based youth program)NameHome AddressPostal CodeHome TelephoneWork TelephoneFax NumberEmail address..Current Occupation/Job TitleDescriptionOrganization(and type of organization)Years in positionHighest level of educationDate completedSchool Name and LocationProfessional certificates orGraduate education completedOther InformationPlease list any languages (otherthan English) that you speakWhat are some of your personalhobbies or interests?Adapted from Ability Online, Toronto ON, 2002.Evaluation Toolkit for Community Youth Programmers Appendix 73

V1 continued (pg 2 of 2)Emergency ContactAddressHome TelephoneWork TelephoneRelationshipHealth StatusPlease inform us of anysignificant health problemswhich may affect your work.How did you hear aboutthis program?Tick any of the boxes to theright which apply to the areasyou would most like to beinvolved with ?How many hours per week willyou be available?Please supply names, addresses,phone numbers and emails (ifavailable) of two people who canprovide a reference for you.From a friendFound on the InternetFrom my doctor/hospitalAdvertisingOther (please explain):Host/Hostess of online conference ______Host/Hostess in any online conference thatneeds helpTraining other new volunteersSupporting fundraising initiatives forRepresenting the program at publicevents1. Work/School2. Personal (non Family member over 18 years of age)If you are under 18, please have 3. Parent or Guardian (if applicant is under 18)your parent or guardian signhere to show they approve ofyour application.Adapted from Ability Online, Toronto ON, 2002.74The Offord Centre for Child StudiesHands Across Canada

V2 (pg 1 of 4)SAMPLE VOLUNTEER APPLICATION FORMThis is a sample volunteer application form that can be used by youth to help describe what their maininterests are and why they want to volunteer.Name: ________________________________________Phone: _______________________________Address: _________________________________________________________________________________________________________________E-mail: ______________________________________________________Emergency Contact: Name:Relationship to you:Phone:1. Tell us about yourself. (Please circle your answer.)a) Are you between the ages of :14-16 17-24 25-29b) Do you have a valid driver's license?No Yes Class _______c) What level of schooling have you completed? ________d) Are you currently in or out of school? _______e) Have you taken any other courses or training? (if yes, pleaselist)__________________________________________________________________________________f) Why are you applying for a volunteer placement? (check all thatapply)Gain work experience/skills __Make me more employable __School requirement __Help out my community __Use my current skills __Other (Please explain)______________________________________________________________________________________________________From Youth Employment Services and Volunteer Youth in the Millennium, Thunder Bay, ON, 2003.Evaluation Toolkit for Community Youth Programmers Appendix 75

V2 continued (pg 2 of 4)2. Availabilitya) At what times are you available to volunteer?I am flexiblePrefer weekdaysPrefer eveningsPrefer weekendsPrefer daysOther (Please specify)____________b) When are you available to start a volunteer placement?______c) Do you have a preference as to where you do volunteer work?No __ Yes __d) Do you have any physical limitations or are you under any course of treatment which might limityour ability to perform certain types of work?No ___ Yes __Please explain. _______________________________________________3. Skills and Interestsa) What special skills, interests or hobbies do you have?b) What work or volunteering experience do you have?c) What did you like most about those experiences?d) What did you like least about them?From Youth Employment Services and Volunteer Youth in the Millennium, Thunder Bay, ON, 2003.76The Offord Centre for Child StudiesHands Across Canada

V2 continued (pg 3 of 4)e) What is the main benefit you want to obtain from volunteering?f) What are your career objectives?4. Preferences in Volunteeringa) Is there a particular type of volunteer work you are interested in doing (check all thatapply)?[ ] Working one-on-one with another person[ ] Working directly with a staff person as an assistant[ ] Helping around the office in general administrative duties[ ] Doing research or an individual project[ ] Doing public speaking, fund raising, special events, etc[ ] Working on group projects[ ] No preference[ ] Other (Please specify) ___________________________b) Is there a person or group you are particularly interested in working with? (check all thatapply)[ ] Adults[ ] Seniors[ ] Children[ ] Other youth[ ] People with disabilities[ ] Agency staff[ ] Men[ ] Women[ ] Animals[ ] No preference[ ] Other (Please specify)c) Is there any group that you would not be comfortable working with?No __ Yes __ Please comment. _________________________________From Youth Employment Services and Volunteer Youth in the Millennium, Thunder Bay, ON, 2003.Evaluation Toolkit for Community Youth Programmers Appendix 77

V2 continued (pg 4 of 4)5. Referencesa) How did you learn about the Program?[ ] Brochure[ ] Employment Counsellor[ ] Radio[ ] Newspaper[ ] Friend/family[ ] Poster[ ] School[ ] Other (Please specify)b) Have you ever been convicted of a criminal offence?No__ Yes __ Please comment.c) Have you ever been charged with neglect, abuse or assault?No__ Yes __ Please comment.d) Please list two non-family references whom we might contact:Name:Name:Phone:Phone:Date:Signature:From Youth Employment Services and Volunteer Youth in the Millennium, Thunder Bay, ON, 2003.78The Offord Centre for Child StudiesHands Across Canada

V3(School-based reading program)Registration & Attendance FormLocation:Dates:Times:Site Leader:___________________Elementary SchoolWednesdays (October – March)3:00pm – 4:30pm_____________________ Session # ______PLEASE USE A SEPARATE SHEET FOR EACH SESSIONName: Date: Teacher ContactPhone #AgeGradeAdapted from Frontier College reading circles, Saskatoon, Saskatchewan, 2004.Evaluation Toolkit for Community Youth Programmers Appendix 79

V4(Example from Program providing literacy training and homework assistance for families)Attendance log: participants and volunteersActivityLiteracyClassesTutoringSessionsTotal # ofadults andattendance rate#%Total # ofchildren andattendance rate#%Total # ofvolunteers /attendancerate#%# offacilitators#ReadingCirclesInformationsessionsChild Care forsiblingVisit to thelibrary andlocalcommunitycentresDeveloped by Delta Family Resource Centre, Toronto, ON, 2004.80The Offord Centre for Child StudiesHands Across Canada

V5Attendance Tracking SheetEach program leader/volunteer monitors attendance to provide attendance rates and makes personal contact withparticipants soon after an absence to encourage participation and determine underlying program-related issues.ParticipantNameAttended program (days)(Mark absence with A)1 2 3 4 5 6 7 8 9 10Contactedre. absenceComments re.absence/reminders/strategyContact info.(phone/email)Additionalcomments:Developed from practices of Christie Lake Kids, Ottawa, ON, 2002.Evaluation Toolkit for Community Youth Programmers Appendix 81

V6(special needs program)VOLUNTEERS’ SURVEY: Observations pre-testChild’s Name: Date:Are you currently attending high school? or university?Do you have prior experience in working with children with special needs?Have you volunteered with this program before?********1. We are interested to know more about why you decided to get involved with thisprogram. What are your goals and hopes for yourself in volunteering with thisprogram?2. Please take a few minutes to briefly sketch your first impressions of the child you areworking with. Please consider his/her social skills, unique characteristics, and specialneeds.Thank You. Please return this completed questionnaire next week, [date].From Alexander Society for Special Needs, Nova Scotia, 2002.82The Offord Centre for Child StudiesHands Across Canada

V7VOLUNTEERS’ SURVEY – ObservationsDate: y/mo/d ____________Dear Volunteer:Once again we are asking you to kindly answer the following survey questions.This is part two of our on-going evaluation of this program. Through working closelywith the children each week, your comments can provide valuable insights into anychanges in the children.Today we are asking you to take 5 minutes before you leave to answer thequestions below. Your cooperation is greatly appreciated!Name of the child you work with:Now that you have gotten to know this child better, please describe what you feel to bethe biggest challenges that this child is working through as he/she participates in theprogram. Please describe any changes you have noticed in the child over the course of theprogram.Thank You.Please return this completed questionnaire before you leave today.from Alexander Society for Special Needs, Nova Scotia, 2002.Evaluation Toolkit for Community Youth Programmers Appendix 83

V8 (pg 1 of 2)Volunteer Observations and Assessment(post-program survey, special needs program)Date: day/mo/yr _________________Dear Volunteer:Once again, we are asking you to kindly answer the following survey questions.This is the final stage of our on-going evaluation of the program for this term. This timewe are asking you to answer two primary questions: first, how this program has affectedthe child you have worked with, second, how the program has affected you.Please return the completed questionnaire next week. Your cooperation is greatlyappreciated!Evaluation ResearcherYour name: ______________Name of the child you worked with most frequently this term: ___________1. Have you observed any changes in this child over the course of the program (likelyunrelated to the programming)?a) First, in the area you see as a major challenge for this child.b) Second, in terms of this child’s development in the following areas: social,cognitive, emotional and physical. Please explain.Adapted from the Alexander Society for Special Needs, Nova Scotia, 2003 184The Offord Centre for Child StudiesHands Across Canada

V8 continued (pg 2 of 2)2. How do you feel the program has affected the child, in either positive or negativeways?a) First, in the area you see as a major challenge for this child. Please explain.b) Second, in terms of this child’s development in the following areas: social,cognitive, emotional and physical. Please explain.3. In reflecting on your decision to volunteer with the group, do you feel yourparticipation has been worthwhile? Please explain.a) What effect has your involvement had on you? Please elaborate.4. Do you have any additional comments to help us improve the program?Adapted from the Alexander Society for Special Needs, Nova Scotia, 2003 2Evaluation Toolkit for Community Youth Programmers Appendix 85

V9 (pg 1 of 2)Volunteer Program EvaluationEvaluation # 1Program Name:Start Date:Evaluation Date:Name: (optional):____________________________________________________________________________________________________________________________________To be completed by volunteers upon the completion of first session of the program.Please answer the questions to the best of your ability and give as much detail aspossible. This is confidential and we appreciate your honesty.1. Please explain in a few sentences the program you are becoming involved in.Include details such as time, place, and dynamics of the group (volunteers /children.) Include as much information as you have.2. Do you feel adequately prepared to be a part of this program? Touch on aspects ofthe training provided so far, leadership of the program director, communication &support.From Frontier College, Saskatoon, Saskatchewan, 2004. 186The Offord Centre for Child StudiesHands Across Canada

V9 continued (pg 2 of 2)Volunteer Program EvaluationEvaluation # 13. Have you volunteered in the past? Please give a brief description of your pastvolunteer commitments and experiences if any.4. How do you feel you will make a difference in this program? What are some ofyour personal goals, for the group, for yourself?5. Please add some final comments here in terms of suggestions, concerns orquestions.Thank you for your time!From Frontier College, Saskatoon, Saskatchewan, 2004. 2Evaluation Toolkit for Community Youth Programmers Appendix 87

V10 (pg 1 of 2)Volunteer Program Evaluation # 2To be completed by volunteers upon completion of the last session of the program.Program Name: _________________________________Start Date: _________________________________Evaluation Date: _________________________________Name: (optional): _________________________________Please answer the questions to the best of your ability and give as much detailas possible. This information will be confidential. We appreciate your honesty.Please use the following scale for your ratings.1 very dissatisfied2 dissatisfied3 barely okay4 satisfied5 very satisfied1. How satisfied were you overall with your experience in the program? ___Please comment______________________________________________________________________________________________________________________2. How satisfied were you with the experience provided to the childrenoverall by the program? _____Please comment_________________________________________________________________________________________________________________________________________________________________________________3. Do you feel you made a meaningful connection with any or all of thechildren involved? Please comment.__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Adapted from Frontier College Reading Circles, Saskatoon, Saskatchewan, 2004.88The Offord Centre for Child StudiesHands Across Canada

V10 continued (pg 2 of 2)4. In reflecting on the program, what are some of the highlights for you?Think about such things as educational value, your own volunteer goals,the dynamics of the group…_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________5. Thinking about communication between volunteers and the supervisor,how satisfied were you with what you experienced this session? _____Please comment______________________________________________________________________________________________________________________________________________________________________________________________________________________________6. Do you have any suggestions or comments that might make thisprogram better for future participants? Think about organizationalthings such as volunteer-child ratio, the set up of the program, space,time etc…________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________7. Will you continue to support your community in terms of doingvolunteer work in the future? Yes ____ No _____ Uncertain ___8. Has this experience changed your perspective on community service inany way? Please comment.________________________________________________________________________________________________________________________________________________________________________Adapted from Frontier College Reading Circles, Saskatoon, Saskatchewan, 2004.Evaluation Toolkit for Community Youth Programmers Appendix 89

V11(creative play program)FACILITATOR’S SURVEY1. In your opinion, what is the overall goal of the program?2. In your view, what are the program objectives and hopes for the childrenin each of the following areas? Please pay particular attention to the areaof the program you oversee.Story Telling/ Drama:Music:Movement:Drawing:Painting:Handiwork:Social Skills:Other:[individual plans can be developed for each child as needed]From Alexander Society for Special Needs, Nova Scotia, 200290The Offord Centre for Child StudiesHands Across Canada

V12 (pg 1 of 2)Facilitator’s Feedback FormPlease share your observations about the program by answering the questionsbelow. Your answers will help us identify what is working and what needsimprovement. Thank You!Section OneOn a scale of 1-5, please circle the answer that best describes how you feel.Overall, how satisfied are you with your job as facilitator?1 2 3 4 5Not Satisfied Very SatisfiedOverall, how do you feel about the program’s effects on youth participants?1 2 3 4 5Not Beneficial Very BeneficialOverall, how effective was communication among employees involved in the program?1 2 3 4 5Not Effective Very EffectiveSection TwoPlease answer the following questions.Have you observed any changes in participating youth during the program? If yes, canyou give some examples of what you’ve observed?Adapted from the Winnipeg Boys and Girls Clubs, Winnipeg, Manitoba, 2004.Evaluation Toolkit for Community Youth Programmers Appendix 91

V12 continued (pg 2 of 2)What suggestions do you have for improving the program?What were some things we did that helped you in your role as facilitator?What are some things we might do that would make “the job” of facilitator easier?What did you like best about the program?Adapted from the Winnipeg Boys and Girls Clubs, Winnipeg, Manitoba, 2004.92The Offord Centre for Child StudiesHands Across Canada

V13 (pg 1 of 4)STAFF AND VOLUNTEER QUESTIONNAIRE(AFTER-SCHOOL PROGRAM)Program Site: ____________________________Staff ______Volunteer ______Date ______________________Day/ month/ yearWe are asking the staff and volunteers of the After-School Program what theythought of the program and if there are ways we can make it better for next year.Your answers will be grouped with answers from other staff and volunteers andwill not be singled out on an individual basis. Please be as specific as you can inanswering the questions.1. How long did you work or volunteer in the After-School Program this year?From __________________ to ____________________How many days a week (on average) did you work or volunteer in the After-SchoolProgram? _______ days a week2. Did you observe any changes in After-School Program participants in any of thefollowing areas? If so, please describe.a) Social skills:____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________b) Other learning, skills and knowledge:__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________c) Participants’ sense of “connection” to the community (e.g. participants talking abouthaving more friends in the neighbourhood or spending more time with friends, talkingabout what’s happening on the street or in other parts of the neighbourhood, makingcomments that would indicate a sense of pride in their neighbourhood, or saying thingsthat would indicate increased knowledge of opportunities and available resources in thecommunity, etc.). Please be specific.____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________from House of Friendship, Kitchener, ON, 2004 1Evaluation Toolkit for Community Youth Programmers Appendix 93

V13 continued (pg 2 of 4)#2 cont. Did you observe any changes in After-School Program participants in any ofthe following areas? If so, please describe.d) Participants’ involvement in organized activities in their neighbourhood or in the widercommunity – outside of the After-School Program (e.g. clubs, lessons, sports, visits tolibraries, etc.):____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________e) Participants’ willingness to start doing their homework without prompting during theAfter-School Program:____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________f) Participants’ need for help with their homework:____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________3. Have you observed any other changes (not identified in #2) in children who participatedin the After-School Program? If yes, please describe them.______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________4. Are there ways you think the After-School Program has impacted the community orneighbourhood? If yes, what are they?______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________from House of Friendship, Kitchener, ON, 2004 294The Offord Centre for Child StudiesHands Across Canada

V13 continued (pg 3 of 4)5. What comments have you heard from parents about the After-School Program?______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________6. What comments have you heard from the children (participants) about the After-SchoolProgram?________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________7. What did you like about the After-School Program?______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________8. What suggestions do you have for changes or improvements to the After-SchoolProgram?_____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________9. Did you receive enough support to do your job in the After-School Program?Yes _______ No _______a) If not, what would you need in order to feel adequately supported?____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________from House of Friendship, Kitchener, ON, 2004 3Evaluation Toolkit for Community Youth Programmers Appendix 95

V13 continued (pg 4 of 4)9. b) Do you have any suggestions for ways we could help you do a better job?______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________10. Briefly describe one highlight of the After-School Program for you (this may be anindividual story or something about the program that particularly impressed you, etc.).___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________11. Other comments:___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Thank you for your response!from House of Friendship, Kitchener, ON, 2004 496The Offord Centre for Child StudiesHands Across Canada

V14Literacy tutor monthly letter to parents re. progress[Date]Dear [Parent],[Something personal about your enjoyment of working with your child.] In this past month,we've being doing lots of things in our tutoring sessions. [Child] really enjoysand. Below are some of the things we've done during this time.Books read:Things we are working on:Strategies [child] is familiar with:Some accomplishments since we began [or, the last letter] are that [child] can . . . .Yours support at home including the daily reading makes a big difference! Now that [child]can, you could encourage this in the daily reading.I meet with [child] on [day] at [time] and [day] at [time]. I'd be pleased if you could join us toobserve a tutoring session. If this is possible, please call [School Resource teacher], theSchool Resource Teacher, at [school phone number] to schedule a time.I look forward to meeting you.Yours sincerely,[sign your name legibly, or print afterwards]TutorFrom the Learning Disabilities Association of Peterborough and Trent University School of Education andProfessional Learning, Peterborough, ONEvaluation Toolkit for Community Youth Programmers Appendix 97

V15Child's fIrst nameChild's schoolEnd of Tutoring SummationPlease summarize characteristics of the child you tutored, noting behavioursthroughout the year as well as a statement of where the child is at this point intime. Please make special note of any changes over the year. Child's strengths: Child's likes / interests: Child's dislikes: Child's attitudes: Child's area(s) of need: Child's progress:Favourite books: Instructional levels:How is this child best-served?Tutor name (please print)Date (yr/mo/day)SignatureFrom the Learning Disabilities Association of Peterborough and Trent University School of Education andProfessional Learning, Peterborough, ON, 200398The Offord Centre for Child StudiesHands Across Canada

V16 (pg 1 of 2)JUNIOR & SENIOR LEADERSHIP PROGRAMMID-POINT SELF-EVALUATION FORMName of Jr./Sr. Leader: ________________________________Name of Evaluators: ________________________________Date of Evaluation: _________________________________Please circle the number, which you think best reflects your skills in each area.(#1=not doing job at all #3=doing well most of the time #5 needs NO improvement,doing excellently)Do you…….Arrive at work on time or early 1 2 3 4 5Participate in all activities that thepre-teens are doing (e.g., swimming)or activities at the park 1 2 3 4 5Phone if you will be late / absent 1 2 3 4 5Start games and activities with pre-teensand children on your own initiative 1 2 3 4 5Seek assistance and support from 1 2 3 4 5leaders when you are uncertainTake full responsibility for dutiesand follow through getting them done 1 2 3 4 5Show lots of energy/enthusiasm at work 1 2 3 4 5Support other junior leaders if they need it 1 2 3 4 5Focus on the pre-teens & children, withoutgetting distracted by other leaders/volunteers 1 2 3 4 5Do jobs without being asked, (think aheadto what needs to be done and do it) 1 2 3 4 5Communicate ideas and instructionswell to the pre-teens and children 1 2 3 4 5Handle unexpected problems well 1 2 3 4 5Treat the children and pre-teens you leadfairly and appropriately 1 2 3 4 5From Kiwassa Neighbourhood House, Vancouver, B.C., 2002Evaluation Toolkit for Community Youth Programmers Appendix 99

V16 continued (pg 2 of 2)Please identify your biggest strength.Please identify a challenge you have had in the program and how you dealt with it.What is your goal for next month?If you could make changes to the junior leadership program what would they be?From Kiwassa Neighbourhood House, Vancouver, B.C., 2002100The Offord Centre for Child StudiesHands Across Canada

Hands Across Canada Project Offord Centre for Child StudiesChedoke Site, Patterson Building, 1200 Main Street West. Hamilton, Ontario, Canada L8N 3Z5Toll-free Number: 1-888-541-KIDS. Email: info@offordcentre.com. Web: www.offordcentre.com