11READING ACHIEVEMENT IN 1991 AND 2000 - Pisa

More documents

Recommendations

Info

11 READING ACHIEVEMENT IN 1991 AND 2000ends of the difficulty scale to make a test equation between the two tests. It istherefore possible to measure the items from both studies on the same Raschscale; thereby making the two scales for the two different studies comparable.Figure 11.1 Distribution of difficulties (percent correct) for IEA and PISA items inBooklet 10IEA-RLPISAPercent Correct100 %90 %80 %70 %60 %50 %40 %30 %20 %10 %0 %0 20 40 60 80 100 120ItemsHaving done this, it remains to be seen if the combined sets of test 1 (IEA) andtest 2 (PISA) items in booklet 10 constitute one (so-called homogeneous) scalewith one latent σ-ability scale. If this can be accepted by adequate statisticaltests, the PISA items are thereby equated with the IEA items. This is, of course,a valid result for items in the supplementary booklet only. However, the resultwill also be valid for the PISA items in the other nine booklets as well, and forall items in the two IEA booklets. This is proved by the fact that both the PISAand IEA items have separately been ‘approved’ using the Rasch model.A first point of analysis is, however, to explore whether the IEA readingitems in the special 10 th booklet are on the same scale of difficulty as those inthe IEA 1991 study, in other words, whether the difficulty if the items, i.e. theθ-difficulty parameters in the Rasch model from IEA 1991, are the same as theθ-difficulty parameters found for the special 10 th booklet in PISA 2000. Itwould, of course, be a problem if the reading scale had changed since 1991, i.e.if the relative difficulties had changed. In order to control for this, it isnecessary to have access to old IEA 1991 data at the individual student anditem level. Approximately 80 items were re-used in the PISA 2000 study.The details of the statistical tests for consistency between the IEA readingscales will not be given here (Allerup 1995, 1997) but analysis shows, that the138
11 READING ACHIEVEMENT IN 1991 AND 2000IEA items have indeed kept their (relative) difficulties across the period. Theso-called ‘pop-B’ IEA reading scale has stayed the same across the years 1991to 2000. It is also worth mentioning that the national study ‘F2000’ (Allerup etal. 2001) made use of a subset of IEA items and found the same result. It issatisfying that the calibration (viz. the fit to the Rasch model) of a reading scalein 1991 seems to work years later.The 1991 calibration of an international reading scale involved Denmark asonly one part of the chain, but the international calibration required allcountries to accept successful ‘approval’ of the Rasch model. It is therefore tobe expected that the calibration can be extended to other countries as well.There are two parts to booklet 10, presented to the students with a break inbetween. More insight into the details of the tests are required in order to see ifthis situation adequately satisfies the criteria set by the Rasch model. However,numerical statistical tests based on the conditional distribution of test score 1,given that total score = test score 1 + test score 2, adequately and ‘objectively’(i.e. depending on the item difficulties θ only) confirm the impression gainedfrom the figures:The IEA items and the PISA items constitute one scale of difficulty, fromwhich a common student ability σ-scale can be estimated.This also appears to be valid when the two parts of booklet 10 are compared.Student abilities can therefore be estimated using any subset of the items,whether they originate from the IEA or the PISA study.11.3.4 Danish resultsFrom the successful fit of IEA 1991 and PISA 2000 reading items, animmediate question can be answered: Is the level of reading ability among1991 14 year olds the same as for 15 year olds in 2000? The answer can begiven either in terms of values on the ‘true’ latent σ-scale for student abilitiesor in terms of student scores. Without losing information these scores could betransformed to ‘percentages correct’ for each student. It is a matter of tastewhether one prefers one or the other set of values. Both studies conducted itemanalysis and reported their results on a Rasch scale. The choice is independentof the Rasch model analysis, but will set certain limits for the readerconcerning the interpretation of the results. In the case of the σ-scale scores, adifference between student achievements of, say ’10.5 Rasch score (σ-values)points’ will be the same, whatever the actual levels of the achievements of thestudents are. This will not be so if scores are reported as ‘percentage correct’,because a difference of, say, 5.5% between two students will not be the same if5.5% is the result of 53.5% - 48.0% or 97.5% - 92.0%. However, in nearly allinternational studies the σ-values have been used with the values centred andre-scaled to different points on the real axis, e.g.: international mean = 500,international standard deviation = 100.139
Page 1 and 2: 11 READING ACHIEVEMENT IN 1991AND 2
Page 3 and 4: 11 READING ACHIEVEMENT IN 1991 AND
Page 5: 11 READING ACHIEVEMENT IN 1991 AND

11READING ACHIEVEMENT IN 1991 AND 2000 - Pisa

Create successful ePaper yourself

Delete template?

Save as template?