In-Basket Design: Developing and Implementing an On-Line ... - IPAC

More documents

Recommendations

Info

Step 10: Conduct Key Clearance • Prior to assigning scores, we examine item functioning to: • Verify the accuracy of the computer-programmed scoring key • Evaluate the integrity of each item • What we examine • Item difficulty (p-values) • Point biserial correlations (pbi) • Hi/Lo Split (response distribution of upper and lower 45%) • Internal consistency reliability • Content of question stems and response options • Decisions we make based on item statistics and content review • Poorly performing items are eliminated • Alternative plausible responses receive full or partial credit IPAC Annual Conference 2011 42
Step 10: Conduct Key Clearance Questions we ask during key clearance: • Item difficulty (p-values) • What percent of candidates select the keyed response • Is the item overly difficult or too easy • Item point biserial correlations • Does highest pbi correspond to the keyed response • Variation within the distribution of response options • How often is each response option being endorsed • Are ―top‖ candidates endorsing the keyed response • Which response options are being endorsed by poorer performers • Internal consistency reliability • What would happen if a low performing item was dropped IPAC Annual Conference 2011 43
Page 1 and 2: In-Basket Design Developing and Imp
Page 3 and 4: What is the In-Basket Assessment
Page 5 and 6: The Steps • Step 1: Determine Cri
Page 7 and 8: Step 1: Determine Critical Tasks, D
Page 9 and 10: Step 1: Determine Critical Tasks, D
Page 11 and 12: Step 2: Collect Source Materials
Page 13 and 14: Step 2: Collect Source Materials Wo
Page 15 and 16: Step 3: Develop Overview and E-mail
Page 17 and 18: Step 4: Develop Performance Benchma
Page 23 and 24: Step 5: Write Multiple-Choice Quest
Page 33 and 34: Step 6: Confirm Fidelity and Job Re
Page 35 and 36: Step 6: Confirm Fidelity and Job Re
Page 37 and 38: Step 7: Assemble, Review, and Final
Page 39 and 40: Step 8: Prepare for Administration
Page 41: Step 9: Administer the In-Basket Ca
Page 45 and 46: References American Educational Res
Page 47 and 48: Questions IPAC Annual Conference 20
Page 49 and 50: Gast/Votraw -- In-Basket Design IPA
Page 51 and 52: Gast -- In-Basket Design PTC-SC Ann
Page 53: The ITEM as a whole should: It

In-Basket Design: Developing and Implementing an On-Line ... - IPAC

Create successful ePaper yourself

Delete template?

Save as template?