Abstracts - Earli

P 101 September 2007 11:00 - 12:20Room: 7.59Paper SessionAssessment of competenceChair:Gyöngyvér Molnár, University of Szeged, HungaryCompetency profiles from standard assessmentsClaus H. Carstensen, IPN Kiel, GermanyAndreas Frey, IPN Kiel, GermanyRecently for Germany it was decided to set up educational standards for several subject areas andseveral degrees of graduation with the goal to improve student performance. These standardsdefine educational goals in terms of competencies for each subject area and three different levelsof graduation. A newly developed standard test for Mathematics grade ten is analysed, in this testthe standards are organized in "big ideas" and "content related competencies". The presentcontribution addresses the question whether results can be reported simultaneously on five bigideas and six content related competencies of the test. The data to be analysed was collected on asecond day of testing following the PISA 2006 assessment in Germany. Using MultidimensionalIRT modelling for large scale assessment data, some alternatives of student competency profilesare modelled with different complexity. The paper addresses the question what differentialinformation we get from complex competency profiles, how reliable results can be reported andhow complex can profiles be reported with sufficient reliability.Educational standards for mathematics in Germany: Results for coder reliabilityAndreas Frey, Leibniz-Institute for Science Education (IPN), GermanyClaus H. Carstensen, Leibniz-Institute for Science Education (IPN), GermanyFollowing the mediocre German results of international large scale assessments like TIMSS, PISAor PIRLS the 16 German federal states recently decided to set up educational standards for severalsubject areas and several degrees of graduation with the goal to improve student performances.The standards define educational goals in terms of competencies. Whether these educationalstandards are met will be tested by means of standardized tests. The present contribution describesthe development of a standardized test to measure the attainment of the standard in mathematicsfor the "medium" graduation level (mittlerer Schulabschluss), usually obtained after grade ten,with a focus on coder reliability. Based on a faceted theoretical competency model an item set wasdeveloped and field tested as a national extension of PISA 2006 (N = 12000). Nearly half of the313 items were open ended, so that the students’ answers have to be coded before statisticalanalysis. Therefore 14 markers were trained on basis of a standardized coding guide. To ensurereliable measures the consistency of codes given by all coders for 180 responses to each of theopen ended items was examined within a generalisability theory framework. The observedvariance was decomposed into variance components for three main effects (Student, Coder, Item),three first level interactions (Student x Item, Student x Coder, Coder x Item) and a residual(including the second order interaction). The results show that the observed variance can mostly beexplained by effects that incorporate the variables Item and Student while the variance componentswhere coders are involved are negligibly small. Thus, there is no systematic coder effect and– 823 –