Evaluation - Event Hub


Evaluation - Event Hub

INTRODUCTIONThis guide’s aim is to suggest how arts organisations can begin tomonitor their work, evaluate what they do, and share results withpartners, funders and their own workers. We feel that Monitoringand Evaluation is not only increasingly important, but can becreative and enjoyable too. For the best results, try to buildmonitoring into your project at the start, and treat it as an integralpart of the project.CELFYDDYDAU GWIRFODDOL CYMRUVoluntary Arts WalesPO Box 200WelshpoolPowys SY21 7WNtel 01938 556455fax 01938 556451email info@vaw.org.ukweb vaw.org.ukVAW acknowledges funding from the Arts Council of Wales and theWelsh Assembly Government. Voluntary Arts Wales is an initiativeof the Voluntary Arts Network which is registered in Scotland asCompany No. 139147 and Charity No. SC 020345. VANacknowledges funding from the Carnegie UK Trust, the ArtsCouncils of England, Scotland,Wales and Northern Ireland.ISBN 1-899687-45-9Designed by Arts FactoryTel: 01443 757954The pack is designed for you to pull out and use what you want atany time.You may need to know different things when monitoringone project than you will in the next, and may have differentresources. Above all, this pack is far from complete – there aremore techniques than we can mention here. We’re hoping that you’lltreat this as a starting point, and integrate our material and ideaswith your own. More than anything, we’d like you to contact usabout your own successes and problems. By sharing comments aboutthe pack’s strengths and shortcomings, we can continue to improvethe advice and materials we offer.For the purposes of this guide, monitoring is information gatheringwork that takes place throughout the life of the project. All thoseinvolved should be aware of what you are trying to monitor andwhat the targets are. The knowledge you gather now will be crucialto a successful evaluation. Evaluation is the assessment of theproject, usually after the event. It is done to improve future projectsand management. It can show how well funds have been used andthe benefits to all those involved. This process should be open andhonest for the best results. Most importantly, the results should beseen to be acted on – otherwise the process becomes meaningless.01

THIS SECTION IS ABOUT SELF-EVALUATION, THINKING ABOUT HOWWE ARE DOING AND WHAT CAN BEDONE TO IMPROVE OUR WORK.WE BELIEVE THAT ALL ARTSORGANISATIONS AND ARTISTS NEEDTO DEVELOP THE EVALUATIONPRACTICES THAT SUIT THEIR OWNNEEDS AND RESOURCES, REFLECTCHANGES IN THE WORLD AROUNDTHEM AND IN THEIR OWN WORKAND SUPPORT THEIR CORE AIMSAND OBJECTIVES.TAKING A CLOSER LOOKLOOKING AT WHY WE DO WHAT WE DOEvaluation is a learning process. It involves owningup to our failures and weaknesses as well asrecording and celebrating success. Evaluation canbe formative (carried out while work is in progress)or summative (carried out at the end of a projector afterwards). It always demands reflection andanalysis, and should lead to changes in how we dothings. There is absolutely no point in gatheringdata unless it answers questions that we needanswered, unless we have the resources to dosomething with it, and unless we can use it toimprove and develop our work.03

ffffWHY AND WHEN?Evaluation needs to be at the heart of allour activity, not a project-by-project ‘addon’.It should be reflected in our policies,business plans, job descriptions andinduction materials. We need to develop a“self-evaluation framework” and actionplan which everyone can understand andshare in, and systems for passing on thelessons learned. As we go on, “reflectivepractice” (a constant examination of ourwork to see what can be learned at eachstage) will become more explicit andconscious. Evaluation is about valuingwhat we do and should be part of a sharedprocess, creating a greater sense of worthfor everyone involved in our work – whetherthey are staff, volunteers, artists or otherpartners.WHO, WHAT AND HOW?Evaluation processes can be organised andled in different ways, according to thenature of the work. You could choose:To use an internal evaluator (someone whois already involved in your work like acommittee member or employee).To take advantage of external support forinternal evaluators (for example by askingVoluntary Arts Wales to help you design anevaluation process to suit your needs).To co-opt participant evaluators (ask peoplewho are involved as participants to giveyou their opinion so that you can get aninside view of how the work is going).To appoint a specific ‘external’ evaluator(organisations like VAW sometimesundertake evaluation on behalf of others.Some organisations might choose to employa consultant to undertake this work).fffffBut which one is right for you? In order tofind out you need to think about:Who is the audience for this evaluation andwhat kind of information will they require?Do they want facts and figures or more ofa feel of the organisation’s work?What kinds of evaluation methods areappropriate for the individuals andorganisations that will contribute to it?There is no point in choosing a lengthyquestionnaire (or even a short one!) ifpeople have a short attention span, aredropping in and out of sessions or havedifficulties in reading. One-to-one interviewsmay be very difficult if individuals do noteasily communicate verbally or feeluncomfortable in this kind of context.What kind of sample is needed to gathera reliable level of information? Do youneed to survey everyone or would a goodchat with a few people give you a clearerpicture?Has the organisation considered all theimplications of the methods you havechosen? How much will it cost? Doeseveryone need to know what is happening?Is the organisation looking to evaluate theeffectiveness of its work, the impact of itswork, or the quality of its work, or allthree of these factors?Answers to these questions will help youdecide which methods of self-evaluationare most appropriate your needs.HOW TO DO YOUREVALUATIONYou can use lots of different methods inyour evaluation processes as this packpoints out. The methods you need willdepend on what you are trying to find out.There will often be existing statistics orother information from national and localagencies and project planning reports,which can be very useful in clarifying theframework of your work. For example, youmight want to find out about the populationof the town or village where your festivaltakes place, how many school age children,how many elderly people etc. The localauthority may well be able to help you withthis. Quantitative and qualitative methodswhich have been tried and tested by otherorganisations working in the same fieldcan often be adapted for your work. Findout what has worked for other festivals orevents? VAW has a range of examples usedby other organisations which you could tryout. The simple techniques on show in thispack may be useful as examples althoughit’s much better to see them as templatesto adapt to your group’s own needs andresources than just importing themthoughtlessly. Creative methods likephotography, film-making, drama andstory-telling can often be worth a thousanddry words.SAVE YOURSELF SOMETIME AND EFFORTA lot of organisations feel they have enoughto do delivering the work and making sureeveryone is clear about what they are doingwithout adding another separate processcalled evaluation. We need to think aboutcreative ways of ensuring evaluation is notan add-on but part of everything we do.ffffffffffffThis could mean:Designing workshop sessions or specialsessions within a project or programme inways that encourage evaluation of thecreative process or product – for examplemaking a piece of theatre or collage aboutthe work itself and asking everyone tocontribute.Welcoming opportunities (such ascontributing to a radio programme orpresenting to other groups) which allow allthose involved to reflect on the journeythey have taken.Designing organisational evaluations sothat there are opportunities for artists andparticipants to contribute, analyse andrecommend and to be kept informed ofresults and outcomes.If you are paying arts workers, ensuringthat the additional time required fromthe artist for self-evaluation activity iscontained within his/her contract andproperly paid for and that this is not anactivity that takes place in isolation fromthe rest of the project work.Once you have thought about all this youneed to write it down in an evaluation plan.An ‘organisational evaluation’ frameworkmight then have the following headings:Name of project or name of aspect of ourworkLinks to our aim/core purposesObjectives of the project or aspect of workPerformance indicators to be monitoredEvaluation themes or questions to beexploredMethods to be usedResources requiredWho is responsible for ensuring the workis carried out04 05

WHY BOTHER?MONITORING AND EVALUATION CAN OFTENSEEM A A CHORE, A A TIME-CONSUMING ANDUNREWARDING ADD-ON TO TO THE ARTS ACTIVITYTHAT WE REALLY CARE ABOUT. LET’S BE BEHONEST –– WHY SHOULD WE BOTHER?HERE ARE A A FEW REASONS:f Reviewing what we’ve donef Measuring 1 Reviewing progress what and we’ve identifying done failuresf Making improvements to our workf Showing the impact on our community2 Measuring progress and identifying failuresf Showing volunteers and staff the value of their workf Developing the group’s strengths3 Making improvements to our workf Keeping control of our financesf Impressing our fundersf Involving4 Showingstakeholdersthe impactand partnerson our community56789Showing volunteers and staff the value of their workDeveloping the group’s strengthsKeeping control of our financesImpressing our fundersInvolving stakeholders and partners07


HERE ARE OUR SUGGESTIONS FOR ANEVALUATION PROCESS, FROM THE FIRSTMINUTE TO THE LAST.BEFORE THE EVENT...DURING THE EVENT...1 O’CLOCK:WHAT IS THE DEMAND?Have you asked people in the community ifthey want or need this event? Will theytake part as an audience or participants?Do similar things already go on? Can youshow a lack of provision in the area? Haveprevious surveys identified what is goingon, and what might be needed? Are therepotential partner organisations you shouldapproach? Most funding applications willask for evidence like this before giving youa grant.2 O’CLOCK:WHAT ARE OUR TARGETS?These could include getting people involved,reaching out to new audiences, developinga series of events, teaching new skills,developing partnerships. Targets can bequantitative (eg increasing the number offolk dance groups by 15%) or qualitative(improving the quality of life for disabledpeople in the area). Funders and otherpartners often want to know what benefitswill come from their investment in you.3 O’CLOCK: WHATRESOURCES HAVE WE GOT?How much money? What equipment orvenues are available? Do we havevolunteers and staff with the skills todeliver what we need? What partners areavailable to help, and what can they bringto the project? When you’ve got this far,you’ve established the baseline level fromwhich the project is starting.4 O’CLOCK:HOW MUCH DO WE NEED?What are your estimates for project oractivity costs? Costs could includeexpected expenditure on equipment, feesfor paid workers, volunteer expenses,promotion, rent, venue hire, stationery andphones. If you keep a record of anythingyou’re being offered for free, such as use ofa hall, or volunteers giving their time, itcan be used as ‘match funding’ when youapply for grants. Keep a check on yourincome and expenditure throughout theproject. Compare this with your originalestimates and make necessary adjustments.5 O’CLOCK:WHO’S INVOLVED?Who is taking part, and how many of themare there? You can collect data on wherethey come from, how they got there, age,gender, ethnicity, disabilities, how they foundout about the event, whether they’ve beeninvolved before and if they’ll come again.Ask questions that reflect what is mostimportant to you.You can devise ideas thatuse stickers, clicker devices, ticket sales,box office records, police and stewardestimates, photographs and much more.Details will come later in the pack.6 O’CLOCK: HOW GOOD DOPEOPLE FEEL?Numbers don’t capture the essence of anevent, and they don’t tell you what peoplehave gained from it. Qualitative data cancome from photographs, videos, talking topeople, listening to their comments, ideasbooks, graffiti walls, press releases,pinboards and post-its. Open questionswithin questionnaires can also draw out alot of information. More information aboutall these techniques, and also their prosand cons in different situations, will also begiven later in this guide.7 O’CLOCK: WHO’SCONTRIBUTING WHAT?Simple records can give you informationabout how much time your staff andvolunteers gave, what tasks they undertook,and the problems they faced. You can alsokeep track of the ‘help in kind’ you receive,whether it’s voluntary work, discounts,organisation time, or gifts of facilities.Remember that every person in themonitoring process is giving a great deal –their time, their feelings and their opinions.Try to ensure that the process is rewardingfor them too, and allows them to expressthemselves.Then the process becomeseverybody’s property. Monitoring is a humanand creative activity for all of us, if we getit right.8 O’CLOCK: FEEDBACK TIMEAsk your workers, artists and otherparticipants for their considered responsesas soon as possible after the activity comesto an end. Ask local businesses, schools,residents groups or social clubs for theirthoughts. There’s a range of interviewformats, forums, collaborative criticismtechniques and questionnaires that can helphere – details and examples to follow.10 11

AFTER THE EVENT...9 O’CLOCK: DID WE MEETOUR TARGETS?‘Hard’ data first (statistics) – how did thenumbers involved relate to your estimates?Did you achieve the outputs you setyourselves, whether it was more trainedsamba drummers, a higher proportion ofolder people or more Welsh-speakers takingpart? ‘Soft’ data too – does your materialshow the quality of the experience? Thismight include increased ambition amongstparticipants, or new skills and confidenceamong the organisers. Did the widercommunity respond as we thought? This iswhen monitoring becomes evaluation – thetime for a detailed examination of thematerial you’ve gathered, and reflect onwhat it’s telling you.10 O’CLOCK: HOW MUCH DIDIT ALL COST?A detailed comparison of your real incomeand expenditure against your estimates willtell you a lot about the way the projectdeveloped, and how you can ensure valuefor money in the future. Where were theunexpected windfalls, and why did thebucket develop holes where it did?11 O’CLOCK:TIME FOR CHANGES?What do we deserve a pat on the back for?What were the high points of the project,and what were the lows? And what do weneed to change and put right, so we can goforward stronger? Now the committee needsto see the drafts you’ve all been workingon, and approve development plans for thefuture.12 O’CLOCK: TIME TO SHARETHE INFORMATION OUTThe best evaluation package in the world isuseless unless you share it. Make sure thatit’s distributed to your committee, staff,volunteers, and the artists or leaders whosetalents you drew on. Your partners in theproject will appreciate being involved.Sharing an honest evaluation with yourfunders will increase their respect for you,and show them that you mean business. Ifit’s going to be widely read, it’s importantthat it’s as lively, colourful and professionallyproduced as possible. Use pictures, straplinesand quotations as well as tables anddry text.12

TOOLS FOR THE JOBNOW THE WORK’S MAPPED OUT, AND WE’VEDECIDED WHICH ASPECTS OF EVALUATION WEWANT TO CONCENTRATE ON, WE NEED TOIDENTIFY THE BEST TOOLS. OFTEN A MIXTUREOF TECHNIQUES WILL WORK BEST, AND THEINHERENT BIASES OF ONE FORM WILL RECTIFYTHOSE OF ANOTHER.fffffffffMEASURING ATTENDANCEPAPER LABEL STICKERSPROCESS: Cheap, low-tech, and capableof all sorts of variation. Use them to countpeople attending an event – just subtractyour empty spaces from the number on thepacket.PROS: Different sizes or colours can beused to differentiate – children from adults,men from women, returnees from first-timearrivals, and so on. A friendly system.CONS: Very few disadvantages. Childrenmay stick them on each other, etc – but hey!NUMBER CLICKERPROCESS: For counting entries through agate or turnstile.PROS: Even simpler than the stickers – justread the number. Can be reset when desired.CONS: Not as adaptable nor as mobile.Cannot distinguish people who leave andthen return.CAMERA AND VIDEOPROCESS: For estimates of attendance,especially in free events when othertechniques are unavailable.PROS: Quick and easy, and easily fitted inwith a number of other jobs.CONS: It’s much harder to get an accurateoverview with a camera than with the eye.ffffffBOX OFFICE ANDGATE RETURNSPROCESS: For counting entries at thegate, also postal and on-line bookings.Numbering tickets is the simplest option,but the technology extends to computerisedand web-based systems.PROS: The more advanced your technology,the greater the range of information thiscan give you, down to block bookings,occupations, family size and post-codes.CONS: Only tells you who came throughthe system – not over the wall, in with theperformers, or without charge for otherreasons. Can demand permanent staff. Nogood for free events.STEWARDS AND POLICEPROCESS: For estimates of attendance,especially in free events when othertechniques are unavailable.PROS: These people have experience anda good eye for numbers. In a rural area,ask the Young Farmers to help – whenyou’re used to counting and controllingsheep, humans are a piece of cake.CONS: They’ll have other things to thinkabout, and the best you can hope for is anestimate – but you can repeat the processat different times during an event.15

ffffffMEASURING RESPONSES AT THE TIMEQUESTIONNAIRE (CLOSEDQUESTIONS)PROCESS: Tick-boxes and yes-no answers,basic questions of fact, or asking for ascore or rating out of five or %. Manyquestionnaires will lead on from open toclosed questions. These static techniquesoffer people the chance to make animmediate response on their own terms.PROS: Provides a range of basic informationand can be answered quickly. Try linkingthe questionnaire to entry for a prize draw.Relatively easy to process. The ‘Snap’software programme now makes analysismuch easier. ‘Snap’ T: 01454 280 820 orwww.snapsurveys.comCONS: Often resented, messy, takes up timeand space, and rates of return are oftenpoor – 25% is high. Those who do fill themin may not represent the whole group. Badquestionnaires can lead to confusion andmisleading information.QUESTIONNAIRE (OPENQUESTIONS)PROCESS: Questions that encourage afuller and more considered response. Manyquestionnaires will lead on from open toclosed questions.PROS: Gives more scope for people toexpress their feelings and encourages a senseof participation and ownership. Can providephrases that ‘sum up’ an experience, and canhelp to develop long-term connections. Canbe done at the time (more immediate) orlater – including by post or email.CONS: Slower to complete, and often evenfewer are returned. Also much harder todesign, process and evaluate. All questionnairesare inevitably rather impersonal.16ffffffINTERVIEWPROCESS: Individual or group, recordedon paper, video or audiotape. Like aquestionnaire, can offer open or closedquestions or a mixture.PROS: Human and face-to face, morespontaneous than many other techniques.Can draw responses from those who mightnot write comments or answerquestionnaires. Mobile and adaptable,and friendly to the less literate.CONS: Disruptive in the wrong situation.Very dependent on the quality of theinterviewer. Questions and answers can bemisunderstood, and personal factors mayaffect the response. Taped interviews canbe difficult and time-consuming toprocess. Avoid leading questions.CAMERA AND VIDEOPROCESS: Photographs or video footageby designated person, or a number ofparticipants.PROS: Vivid and colourful, providinginsights into the atmosphere and feelingsgenerated. Avoids exclusion of the lessliterate. A professional can concentrate onspecific areas, and amateurs provide acollage of incidents. Great for futuremarketing and enriching dialogue withfunders.CONS: The least objective of all thesetechniques. The camera only sees what it’spointed at. Does not involve others indeveloping the monitoring process. Videofootage is harder to evaluate, distributeand integrate with other materials thanstill pictures are.ffffffGRAFFITI WALL, COMMENTSBOOK AND POST-IT NOTESPROCESS: These static techniques offerpeople the chance to make an immediateresponse on their own terms.PROS: Simple, open, informal, andrelatively private. Often elicits honest andunexpected responses.Very good forcapturing witty, catchy comments and forincreasing a sense of participation. Demandslittle staffing.CONS: No control over the sorts ofstatements that will be made. Not arepresentative method – many people wouldnever put themselves in this position.MEASURING RESPONSES AFTERWARDSGROUP FEEDBACKSESSIONSPROCESS: A means of consulting allparticipants soon after an event or activityends.PROS: Delivers considered responsesfrom people on the inside, whose futurecommitment is essential. The group situationshould stimulate broad discussion, but itmay be necessary sometimes to consultdifferent groups separately.CONS: It’s vital that these sessions areprepared and facilitated, and do notbecome an arena for competing views.All points raised should be recorded, thegroup should not be pushed towardsunanimity, and the recorder’s opinionsmust not intrude.ffffffARCHERY TARGET ANDSTICKERS ETCPROCESS: Targets can be divided intosections to reflect different elements. Asimple,‘gut-feeling’ response tool.PROS: Simple, low-tech, tactile and fun touse – especially but not exclusively withyoung people. Easy to process afterwards.CONS: No room for qualified or consideredresponses.Not very private – answers maybe affected by peer pressure and otherfactors. The ‘pin-board questionnaire’ is amore formal variation on this theme.PUBLISHED AND VIRTUALCONSULTATIONSPROCESS: Magazines, newsletters, letters,e-mail discussion groups, and web-basedchat facilities.PROS: Provides feedback over a longertime-scale than any other form. Gives peopletime, privacy and freedom to participateas they see fit. Useful in formulating newpolicies, managing debates and drawing innew members.CONS: Lacks immediacy. Only attractscertain types of people and responses.Hard to integrate into the deadlines of anevaluation process.Tends to be heavilymediated, responding to the opinions ofthe organisers.Examples of many of these techniques can be found in the resources section that follows, together with a case study of anevaluation package that combined several. They’re included to give you ideas, and to stimulate your own new ones.17

More magazines by this user
Similar magazines