Training Scientists to Make the Right Moves - Howard Hughes ...

More documents

Recommendations

Info

Chapter 9 Evaluating the Training At a Glance “” QA At a Glance Quantitative Versus Qualitative Data Quantitative data are finite, clearly delineated, countable numbers. Examples are yes/no answers, numerical rankings, or range-of-quality scales. These measures allow you to generate graphs and figures that tell your story with visuals. They also make it easier to compare sessions from year to year since you have actual numbers. Another advantage of quantitative data is that most people will answer the questions. If you ask only for open-ended feedback, fewer people are likely to respond. An example of a quantifiable probe is “This session helped me learn how to write a letter of application for a faculty position” accompanied by a four- or five-point scale, for example: Disagree, Somewhat Disagree, Somewhat Agree, Agree A scale from 1 to 5, with 1 being Strongly Disagree and 5 being Strongly Agree Note that the range-of-scale measure will give you important shadings of satisfaction that cannot be captured with a simple yes/no response format. However, interpreting these and other quantifiable measures correctly will likely require hiring an evaluation consultant or asking someone at your organization who is experienced in statistical analysis to help with this task. Qualitative data (e.g., comments drawn from open-ended text boxes and interviews) are less concrete and often resist packaging in tidy numerical form. This can make them harder and more time-consuming to collate, yet they can deliver insights unobtainable through questions that yield quantitative data. Verbatim comments can not only underscore observations drawn from the quantitative data but also add punch to brochures and final reports. Qualitative data can also spark new ideas for subsequent training events. Most evaluation instruments ask questions that will produce at least some qualitative responses. Keep in mind that the use of open-ended questions and comment boxes is more feasible with small numbers of participants than with large groups. Chapter One What are some examples of questions that elicit qualitative data? One program planner finds the following questions productive: What did you find most helpful about today’s program? What, if anything, would you change about the program? What topics would you like to see offered in future workshops? 83
Training Scientists to Make the Right Moves Tips for Developing an Effective Evaluation Questionnaire The evaluation questions should be linked to the goals and objectives of the training event. Consider using a mixture of short answer, multiple choice, and open-ended comment boxes in the questionnaire (see appendix 4 for an example of a mixed-format form used for the 2005 BWF-HHMI Course in Scientific Management). Evaluation pros offer some additional advice on designing the questionnaire: Don’t ask the question if you won’t do something with the data. Stick to questions that will help you determine whether you met your objectives. Avoid fishing expeditions—asking questions about things over which you have no control serves no purpose. Steer clear of questions about intangibles. An example of such a question is “How would you rate the university’s commitment to training?” Evaluation is all about meeting your objectives; direct your questions to those specific aims. Avoid ambiguous wording. An example is “Did this lecture (workshop, session, etc.) meet your expectations?” If you don’t know what the respondents’ expectations were before the training, you can’t evaluate their responses. Ask questions that get at the root of purported benefit. For example, you might ask participants to explain why they think they changed a particular behavior entirely because of a specific training session—why they could not or would not have made the change otherwise. Keep the questions short and to the point. Remember that no one really likes to fill out surveys, and if time and finances are tight, minimize the number of open-ended text boxes that elicit qualitative data. Be aware of question bias. Avoid framing the question to lead to the answer you want. For example, by asking “How do you think this workshop will improve your skills?” respondents are being prompted to talk only about positives. A better way to phrase the question might be “Please describe which of your skills have been improved by taking this workshop.” The answer may require more work to analyze, but you will get a better idea of how respondents view the training and it may help them crystallize their own opinions about how well their time was spent. Even if you have obtained a suitable survey from a colleague or other source, it is a good idea to step back and consider whether it might benefit from tailoring to the specifics of your situation. Remember that whether or not a question is poorly framed depends on the objective of the question (see figure 9.2 for examples). 84
Page 1 and 2:
TRAINING SCIENTISTS TO MAKE THE RIG
Page 3 and 4:
© 2006 by the Howard Hughes Medica
Page 5 and 6:
Training Scientists to Make the Rig
Page 7 and 8:
Page 10 and 11:
PREFACE In July 2002, the Burroughs
Page 12:
ACKNOWLEDGMENTS This guide represen
Page 15 and 16:
Page 17 and 18:
Page 20 and 21:
INTRODUCTION Preparing postdoctoral
Page 22 and 23:
Chapter 1 GETTING STARTED Deciding
Page 24 and 25:
QA At a Glance Chapter One What is
Page 26 and 27:
At a Glance “” QA Gathering Ide
Page 28 and 29:
Chapter 2 OBTAINING SUPPORT AND ASS
Page 30 and 31:
Chapter 2 Obtaining Support and As
Page 32 and 33:
Page 34 and 35:
“” QA Chapter 2 Obtaining Supp
Page 36 and 37:
Page 38 and 39:
At a Glance At a Glance “” QA A
Page 40 and 41:
Chapter 3 DECIDING WHAT,WHEN, AND W
Page 42 and 43:
QA Chapter 3 Deciding What,When, a
Page 44 and 45:
At a Glance Chapter 3 Deciding Wha
Page 46 and 47:
QA At a Glance Chapter One Chapter
Page 48 and 49:
At a Glance At a Glance “” QA C
Page 50 and 51:
Chapter 4 DEVELOPING A BUDGET AND G
Page 52 and 53: Chapter 4 Developing a Budget and
Page 54 and 55: Chapter 4 Developing a Budget and
Page 56: Chapter 4 Developing a Budget and
Page 59 and 60: Training Scientists to Make the Rig
Page 66 and 67: Chapter 6 FINDING AND WORKING WITH
Page 68 and 69: Chapter 6 Finding and Working with
Page 70 and 71: Chapter One Chapter 6 Finding and
Page 72: Chapter 6 Finding and Working with
Page 82 and 83: Chapter 8 MAKING IT HAPPEN In This
Page 84 and 85: Chapter 8 Making It Happen At a Gl
Page 86 and 87: QA Chapter 8 Making It Happen At a
Page 88 and 89: Chapter 8 Making It Happen Typical
Page 90 and 91: Chapter 8 Making It Happen Reprodu
Page 92 and 93: Chapter 8 Making It Happen What w
Page 94 and 95: Chapter 8 Making It Happen At a Gl
Page 96: At a Glance At a Glance “” QA A
Page 99 and 100: QA Training Scientists to Make the
Page 101: Training Scientists to Make the Rig
Page 114 and 115: Appendix 1 BUDGET PLANNING Sample C
Page 116 and 117: Appendix 1 Budget Planning: Sample
Page 118 and 119: Appendix 2 THE BWF-HHMI COURSES IN
Page 120 and 121: Appendix 2 The BWF-HHMI Courses in
Page 136: Appendix 2 The BWF-HHMI Courses in
Page 144 and 145: Appendix 4 COURSE SUMMARY EVALUATIO
Page 146: Appendix 4 Course Summary Evaluati
show all

Training Scientists to Make the Right Moves - Howard Hughes ...

Create successful ePaper yourself

Delete template?

Save as template?