Abstracts - Earli

J 130 August 2007 17:00 - 18:20Room: -1.63Paper SessionAssessment of competenceChair:Patrik Scheinin, University of Helsinki, FinlandMeasuring measuring: An item response theory approachBrent Duckor, University of California, Berkeley, USAThe purpose of this study is to examine individual understanding of a particular evidence-basedframework for constructing measures (NRC, 2001). Given the broad range of misconceptions thatstill persist, experts have lamented the challenges for the field of measurement and assessment(Braun & Mislevy, 2005; Popham, 2000, 2004). Building on the model of expert-novice studies inhow individuals think and learn in other domains, the study explores the "building blocks"associated with proficiency in a new evidence-based approach to measurement, the ConstructingMeasures (Wilson, 2005) framework. The study employs mixed quantitative and qualitativemethods to generate and evaluate an instrument designed to measure CM proficiency. Theinstrument is based on a content analysis of the CM framework, and resulted in six dimensions ofproficiency. The open-ended and fixed response format test instrument was utilized to collectresponses from 72 participants. Guided by the 1999 Standards for Educational and PsychologicalTesting, analysis of validity evidence, consisting of evidence related to content, responseprocesses, internal structure, and relations to other variables, suggested that the instrument candetect meaningful differences in proficiencies on several dimensions. The partial credit Rasch itemresponse model employed fit the data acceptably well. Internal consistency indicators, such asperson separation (.87) and Cronbach’s alpha (.89), provided evidence for the instrument’sreliability. Results from further reliability analysis indicated high inter-rater agreement (.98) forscoring of the non-objective items. The relationship between individual CM proficiency and otherbackground variables was tested with multiple linear regression analysis. The results indicated thatboth research and consulting experience have a statistically significant (pHow to assess text, table and graph understanding? Strengths and shortcomings of teachers’ Ideason this point.Jesus Alonso-Tapia, University Autonoma of Madrid, SpainFermin Asensio, Institute of Secondary Education “Maria Zambrano”, SpainFermin Asensio, Institute of Secondary Education “Maria Zambrano”, SpainInmaculada Lopez, Institute of Secondary Education "Joaquin Araujo", SpainNuria Carriedo, Universidad Nacional de Educacion a Distancia, SpainFabio Rychecki, Universidade de Caxias do Sul, BrazilThe present work was carried out as part of a series aimed at identifying teachers’ ideas onassessment. The two studies presented examine teachers’ ideas about how text, table and graphunderstanding should be assessed to help students’ learning. First, the psychological models fromwhich teachers’ ideas should be evaluated were established after a careful review of literature onthe nature of each competence to be evaluated. Second, 68 secondary school teachers -38 fromSpain and 30 from Brazil- were asked to design assessment situations adequate for the mentionedpurpose. Results showed convergence of data coming from both countries, enlightened specific– 533 –