Assessing Local Dependence in Educational Performance ...

Assessing Local Dependence in Educational Performance Assessmentswith Clustered Frce-Reslx~nse ItemsSteven FerraraMaryland State Department of EducationHuynh HuvnhUniversity of South CarolinaHeibatollah BaghiDelaware Department of Public InstructionThe purposes of this study are to (a} extend to polychotomous items a procedure foridentifying dependence in binary items that does not require assumptions of IRT-basedapproaches, (b) explore the magnitude of local dependence caused by clustering of freeresponseitems around a topical task or authentic reading passages in a large-scaleperformance assessment, and (c) probe contextual factors which may contribute to clusterdependence. Item clustering is common in performance-based or "authentic" assessments.Data for the study come from the 1991 administration of the Mm'yland School PerformanceAssessment Program. The results indicate that although there is very little correlationacross clusters, levels of sometimes substantial positive dependence exist for items withinclusters. Hypothesized explanations are made for the high levels of local dependencewhere it exists.Key words: Authentic assessment, item dependence, free response, multi-stepmath problem, open-ended items, performance-based assessment, reading passageI",,.O101'k'-Authors' notes: An earlier version of this paper was presented at the annual meetingof the National Council on Measurement in Education, San Francisco, April 1992. Wewish to thank Susan Ciotta, Trudy Collier, and June Danaher for their invaluable help withthis manuscript.

_ L . . . .As.~..uiall Lo~ IX-pcndcm,z - 1Assessing Lxx:al Dependence in Educational Pcrforrnance A~essmcntswith Clustered Free-Response ItemsA current trend in educational assessment is to move away from the exclusive useof forced-choice (e.g., multiple-choice) items and to rely more and more on free-responseor open-ended items. Such assessments are often referred to as "performanceassessmenls" or "authentic assessments" (e.g., Wiggins, 1989). Authentic assessmentsare intended to mirror exemplary strategic instruction and ate believed by proponents tomeasure higher-order thinking processes and knowledge integration (e.g., Frederiksen,1984; Shepard, 1991). In exemplary instruction, teachers often present backgroundinformation and promote connections among steps in the learning process to make thelearning more fluid and less arduous. Similarly, in authentic assessments the examineemay be presented with a context or purpose for completing the assessment activities. Inaddition, the set of items presented may have been selected and sequenced to lead theexaminee to a culmination such as an explanation, decision, or recommendation and finalexpression of understanding which reveals complex thinking. In such assessments, openendeditems are presented together and may be organized either around one or more readingpassages or in a multi-step task, in what we refer to here as "clusters."Performance assessments that contain such clusters of items may better assesshigher-order thinking and knowledge integration than do performance assessments whichare non-clustered collections of open-ended items. For example, in a reading cluster theexaminee may be asked to read one or more complete "authentic" (i.e., previouslypublished) texts and then respond to a series of assessment activities pertaining to bothtexts. In a mathematics cluster, a complex problem may be posed that requires multiple

,b,.." Assessing Local Dependence -.2steps to reach a solution. In all likelihood, the clustering of such free-response itcmswould create some level of dcpendence in the responses in each cluster.This likelihood of item dependence can be problematic. Local indcpendcnce mustbe achieved in order to meet the assumlaions of item respon~ theory (IRT) measurementmodels and to optimize test score reliability. Often, the prefened solution to itemdependence in test clusters is to sum the scores from dependent items into independent"testlets" (e.g., Waincr & Kiely, 1987) t rather than to delete or revise items untilindependence is achieved. Little is currently known about how much depende, nee mayexist in item clusters in performance assessments nor about the item characteristics that maycause dependence (cf. Yen, in press).Three purposes of this study arc to (a) extend to polychotomous items a procedurefor identifying dependence in binary items that does not require assumptions of IRT-basedapproaches, (b) explore the magnitude of local dependence caused by the clustering of freeresponseitems around a topical task or authcnfic reading passages in a large-scaleperformance assessment, and (c) probe contextual factors which may contribute to clusterdependence.Rclatcd SludiesSeveral researchers have noted passage dependence of standardized reading testitems requiring complex infcrences, including Hanna and Oaster (1980), Nicholas andB rookshire (I 987), and Scherich and Hanna (1977). ,Green and Langhorst (1986) notedthat some degree of both passage dependence and independence is to be expected in anytest of reading comprehension that covers a broad range of objectives.Ackerman (1987) and Ackerman and Spray(1986) used a model for itemdependence to simulate the effects of violating local independence on item calibration and

Asscssin$ Local Del~nClenc~ - 3ability estimation. They suggested that calibra~cl dependent itcrn parameters tended toov'er-estimatc the origirm[ itcrn parameters. Ability estimates ~'erc even less accurate as thedegree of dependence increased. Bell, Pattison, and Withers (1988) described a Ioglinearmodeling approach to examine local dependence of items within and between clusters.They found item dependence to be more marked within rather than between clusters, thatdependence was stronger in items ba.~:d on mathematical than on verbal material, and thatdependence increased with examinee ability.Wilson (1988) provides a proc~ure to assess local dependence of bioary itemswhen the responses fit a Rasch modcl and provides illustrations ba.~'d on 20 science items.First he calibrated binary items under the assumption of local independence. Next. hegrouped the binary itcms into clusters (i.e., what he called subtests or superitems) and thencalibrated via Masters' one-parameter Partial Credit model for i:x)lychotomous items(Masters, 1982_; Wright & Masters, 1982). Finally, he osserted local dependence whenthere were substantial discrepancies between the results of the two calibrations. Huynh(1993) provides a partial theoretical justification of the procedure proposed by Wilson.Finally, Yen (1984) proposed using the Q3 statistic to assess local dependenceunder assumptions of latent trait models. For an examinee with ability 0j on item j, let Xjbe the raw score, Ej be the expected score from a latent trait model, and let Dj = Xj - Ej bethe residual score. Then the Q3jj, index for itemsj andj' is the correlation between the tworesiduals Dj and D'j taken over all examinees. Yen noted that since item j is includedexplicitly in the raw score Xj and implicitly in the (estimated) expected score E, a negativebias is built into Q3. When local independence holds, Q3 is expected to be approximately-1/(n-l) where n is the number of binary items. Yen (in press) applied the Q3 statistic tostudy binary, and polychotomous items and found that the Q3 statistic performs adequatelyfor polychotomous items. However, the use of Q3 is limited to data which adequately fit alatent trait model.

- _Azz, cszing Local Delx.adeace - 4For a complex assessment program involving authentic activities such as theMaryland School Performance Assessment Program (MSPA~ described below), it isconceivable that complex, multi.dimensional latent traits may be needed to fully capture thenature of these items. Hence, it would be helpful to be able to study local dependence offree-response items without having to impose a particular latent trait model on the data.The methodology in this paper for assessing local dependence in free-response items isbased on previous work for multiple-choice items cited in Hamblcton (1988). Hamblctonsuggests investigating local independence b v checking correlation matrices for-examinees atdifferent intervals on the ability or test score scale. This study is based on real examineedata from the 1991 administration of the Marxland School Performance As~ssmentProgram.The Maq,'land School Performance Assessment Program (MSPAP)The 1991 administration of MSPAP included several parallel test forms withclustered free-response items." Each cluster of reading items is contained in a task (i.e., acluster of assessment activities) and pertains to one or more complete authentic texts oftenof several printed pages (e.g., from the children's magazines "Cricket," "Cobblestones,"and "Ranger Rick'). Each mathematics item cluster is a self-contained problem-solvingtask containing 3-5 assessment activities which arc organized around a theme (e.g.,"recycling aluminum cans to make money'), and leading to a final recommendation,decision, or other culmination. Efforts were made to write items so that examinees couldrespond to each item whether or not the,." had responded to other items in a cluster.However, there was no guarantee that the results of this effort would satisfy the conditionof local independence. Because of these complex contexts surrounding free-response

Reins, the level of dependence may be more visible in a free-response modc than in thcmore commonly used forced-choice mode.MSPAP is pan of a larger school reform effort initiated by the Maryland StateDepartment of Education. Performance on the MSPAP is used to evaluate schoolperformance and to provide information to guide school improvement efforts. Thc 1991MSPAP included assessments of widely distributed learning outcomes in reading, writing,language usage and mathematics in grades 3.5, and 8. All tasks in thc MSPAP rcquircbrief and extended open-ended responses to performance tasks: Examinees write, diagram,and sketch responses to tasks that focus on thcir ability to construct and extend meaningfrom what they read, construct and cxlcnd meaning through writing, and solve multi-stepmathematics problcms. Tasks arc designed to clicit examinees' thoughtful application ofknowledge, skills, and thinking processes. Examinee responses are ~ored by trainedteachers using scoring activity-specific keys for brief correct and incorrect responses;generic ~les for longer responses which are scored as correct/adequate, partiallycorrect/adequate, or incorrect/inadequate; and rubri~ for essays and other extendedresponses. Although each 1991 MSPAP test form was administered to a randomlyassigned group of at least 3,000 examinees, samples of approximately 1,400 examineeswere used in this study.Task developers were trained to minimize response dependence in the assessmentactivities they wrote for MSPAP assessment tasks. After the development phase, taskswere submitted to both judgmental and statistical analyses. One such judgmental reviewfocused on response dependence. The goals of this review were to (a) identify clusters ofinterdependent responses, and (b) treating interdependent clusters as testlets, determine ifan entire assessment form contained adequate numbers of independent responses to allowfor IRT scaling. (Details on scaling procedures are available in C'TB Macmillan/McGraw-Hill, 1992.) To identify interdependent responses within item clusters, reviewers read eachitem that was designed to elicit an examinee response and answered the questions, "Can

#.. Asse~ing Local ~ - 6examinees respond to this item whether or not they have (a) responded to previous or• subsequent items, and (b) answered corroctly or adequately to previous or subsequentitems?" An answer of "No" to either of these questions indicated a likely intcrdcpcndentcluster. For MSPAP scaling purposes, judgmentally dctcrmined intcrdcpendent clusterswere scaled as testlets only if Q3 statistics (Yen, 1984) indicated strong responseinterdependence and if creation of testlcts did not result in substantial item misfit. (Detailsare available in CTB Macmillan/McGraw-Hill, 1992.)i.McthodData Sets Used in this StudySeven data sets were systematically extracted from the MSPAP file ofapproximately 1.50,000 cases and served as the data base for this study. The first three damsets (RD36, RDSI, and RD81) consist of the responses to the reading examinations forgrades 3, .5, and 8. The next two data sets (MC31 and MC83) consist of the responses tothe mathematics content (MC) examinations for grades 3 and 8. Finally, the last two datasets (MP31 and MP8 l) contain the responses to the mathematics process (MP)examinations for grades 3 and 8.Each reading examination consists of clusters of items based on four readingpassages. Each mathematics examination consists of c]usicrs of items centered on ninetasks, cach task containing both mathematics content'and mathematics process items. TableI lists the number of items for each examination as wcll as the number of items in eachcluster.

0' Alseasinlt ~ ~ . "7In.sen Table 1 about hereThe upper panel ~Table 2 provides the summa~" statistics for each cxamination.The data indicate that the reading examinations are slightly easy for an examinee of averageability and that the distributions of the raw scores are ncgatively skcwed. The meanpercentage of maximum scores are 51% for RD36, 60% for RDSI, and 64% for RD81.On the other hand, the mathematics examinations are quite difficult for an examinee ofaverage ability and the distributions of the raw scores are positively skewed. The meanpercentage of maximum scores are 39% for MC31, 34% for MC83, 22% for MFr31, and29% for MPSI.Insert Table 2 about hemProceduresSorting subjects into homogeneous score groups. The first step in the data analysisfor an examination was to use the raw score on the examination to sort subjects into fivehomogeneous score groups of approximately the same size.Descriptive statistics for the score groups are found in the lower panel of Table 2.It may be seen that among the five score groups constructed from each examination, thereis more variability in the group at the skewed tail of the score range. In order to make theinter-item correlations in each examination more compatible across score groups, the

, Aclw.,~ng Local i::)epend¢m:e - 8extreme groups (Group I for the wading examinations and Group 5 for thc mathematicsexaminations) were deleted from all subsequent analyses.Overall procedure for dependence analysis. For cach score group, all inter-itemcorrelations were computed. The average within-cluster correlation was then computed forall inter-item correlations within a given common cluster. A frequency distribution wassubsequently constructed to give an overall picture of these correlati,ms. Then the averagebetween-cluster correlations were computed and tabulated for all inter-itcrn correlationsbetween any two given clusters. Finally, all clusters with mc, derate or substantial withinclusteraverage correlations were idcntificd and hypothesized explanations made on theprobable causes for this dependence.ResultsBetween-Clustcr Correlation AnalysesTable 3 reports frequency distributions of the average between-cluster correlationsfor the nine data sets. Although the negative correlations outnumber the positive ones,practically all the correlations are between -.07 and +.07. 3 Given the similarity between theaverage inter-item correlations used in this study and the Q3 statistic (Yen. in press), thenegative bias of Q3 is expected to hold true for these average correlations as well. Most ofthe average correlations for each of the nine test forms reported in the last row of Table 3are close to the quantity-I/(n-I) (the expected value of Q3 under local independence,described earlier), where n is the maximum raw score. Thus, these data provide strongevidence that the vectors of responses to the dusters can be assumed to be locallyindependent.

A~x~ng Local Dependence. 9Insert Table 3 about hereWithin-Cluster Correlation A nalv~sTable 4 reports frequency distributions for the average within-cluster eorrclations.Since almost all the correlations are above zero and several are atxwe +.07. the data clearlypoint to positive dependence among the items within several clusters.Insert Table 4 about hereTables 5 and 6 report the average within-cluster correlations for each cluster andscore group along with a qualitative description of the degree of Icx:al dependence. Thelocal dependence in Table 4 is described in Tables 5 and 6 as "moderate" if two or morescore groups have correlations in a cluster of at least .08. The description changes to"substantial" if there are at least two correlations of at least. 10. 4. t ~ . . . . . . . . . . . . . . . . . . . . . . . . . . . o . . . . . . .Insert Tables 5 and 6 about here

, .......... -- A=e==.s Lo=a ~ ' xoContent Descriptions for Clusters with Substantial ~Dcpcnd, c.n,,ceAccording to these criteria, dependence is identified a.~ moderate for five clustersand as substantial for seven clusters. For the illustrative purposes of this paper, di~ussionis ~o.vided only for the seven clusters with substantial dependence: RDSI, cluster 4(RDSI-4); RD81, cluster 3 (RD81-3); MC3I, cluster 3 (MC3 i-3); MC83, clusters 4 and 6(MC83-4, MC83~); and MP31, clusters 3 and 8 (MP'31-3, MP31-8). Table7 providesbrief content descriptions of the items in these clusters. In this section o/" the paper weprovide hypothesized explanations of the causes of dcpendcnce in these clusters.Insert Table 7 about hereReading cluster~.. RDSL-4 requires examinees to fill out an enrollment form ['or agymnastics class alter having read a complete sto~" about family relationships whose maincharacter is a young gid. These five items focus on examinee ability to read to perform atask. While it appears that examinees can answer each of these items independent of theother items, the information necessary, to answer the items can be found only in the stor?,'.Items on the 1991 MSPAP intended to assess reading to perform a task were intentionallydeveloped to be text dependent. Neither examincc prior knowledge nor conjecture wouldhelp examinees form responses to these items. This dependence on the sto~" to answerthese questions may be the source of the dependence among these items. The source ofdependency in these items appears to be similar to sources of dependency in earlieranalyses of reading comprehension items in multiple-choice tests (e.g., Green &

. )Assessing Local Dependence - IILanghorst, 1986; Hanna & Oaster, 19B0; Nicholas & Brookshire, 1987; Schcrich &Hanna, 19'77).RD81-3 requires examinees to answcr one comparison and onc contrast questionabout the main characters from an informational article and a short stor3", both with thetheme "meeting challenges." The interdcpcndence of the two items in this cluster may havetwo possible causes. First, both items require cxaminccs to think about the characters inboth reading selections. As with RD51-4, responding to these two items is highlydependent on the readings. Second, it is likely that cxaminccs would not be able to contrastthe two characters without first having drawn comparisons between them. Thus,examinees would not be ablc to answer the contrast question without first havingresponded somewhat successfully to the comparison question. In subsequent editions ofthe MSPAP, compare and contrast questions have been developed as single, rather thanseparate items.Mathematics content clustcrs. In MC31-3, examinees must read data regardingnumbers of aluminum cans collected over sevcral weekends, re-arr'angc the data in a table,perform two different calculations using data from the re-arranged table, determine the totalamount of money made from collecting the cans, and estimate how many more weekendsof collecting cans are necessa~" to earn $4.00. F.:caminccs would not be abic to completesubsequent calculations in this cluster without completing previous calculations nor makethe required estimate without completing all calculations. All questions in the cluster aredependent on the data given in the original table.All five responses in MC83-4 require examinees to understand the relationshipbetween the number of cuts made on a rectangular-shaped cake and the number of pieces ofcake that result. First, examinees describe the pattern in a given table (i.e., 1 cut in the cake= 2 pieces of cake, 2 cuts = 4 pieces, 3 cuts = 8 pieces). Then examinees extend the tablefor 4 and more cuts, identify from the extended table the fewest numbers of cuts necessaryto produce 32 pieces of cake, write an equation that represents the relationship between cuts

J,'Aue~in8 Local D=pendence - 1:o.and pieces of calce, predict the probability of finding an odd number of pieces of cake usin 8the equation, and explain [he predichon. 5 Responses to each item in this cluster are builtsequentially on previous responses and on understanding the pattern in the originaJ table.MC83-6 follows almost [he same pattern as MC:83-4, but requires cxaminccs tounderstand [he relationship between a serics of hexagons with a shared side and the numberof toothpicks required to create these hexagons. As above, cxarninccs are given thepattern, extend the pattern in a table, construct responds to questions from the new table,and write an equation. Also as above, responses to each item in this clustcr'arc builtsequentially on previous responses and on understanding the original pattcrn.Mathematics proccss.clustc~. Cluster MP3 l-3, a mathcmatics process cluster, islocated in [he same mathematics task as the mathematics con[cnt itcms in cluster MC.3 I-3.This task focuses on rccvcling aluminum cans. The two m.',thcmatics pr(x:ess questions inthis cluster are built on the previous thrcc content questions. The pr~x:css qucstions rcquireexaminees to solve a problem (i.e., estimate the numbcr of wcckcnds of collecting cansrequired to reach a dollar g.oal, based on thc prcvious calcul.',tions) and rc~son about theestimate (i.e., explain and justify the rc.',sonablcnc~ of the estimate). The dcpcndcnce ofthe tv,'o process items is probably a result of this estimation-explanation rclationship 6 andmay be magnified by their dependence on the three prcvious content items.The two process items in task 8 form the clustcr MP3 I-8. All content and processitems in this task. are based on given survey information on numbers of cxaminees whowould buy various types of items in a school store. Howcvcr, only the process items arestatistically dependent. As with MP3 I-3 above, examinees are required to solve a problem(i.e., decide, based on the £.ivcn sur~'cy data, which items to eliminate from the schoolstore) and reason (i.e., explain and justify [he decision). And also as above, theinterdependence of these two items may be magnified by their likely dependence on thecontent items in the task.

As,~,sinll Local l:~dmno=. 13DiscussionOur description of the contents of the seven item clusters with substantial lcx~!dependence points to some likely causes of item interdepcndcnce and suggest ways toavoid it. For example, test developers probably should expect compare-contrast items to belocally dependent because contrasting two or more objcct~ probably requires attention totheir similarities. In addition, items that are inu:ntionally interdependent, like themathematics content items here in which responses to items are built on each othersequentially, are likely to produce statistical local dependence. Also, reasoning itemswhich require examinees to explain and justify previous responses are also likely to belocally dependcnc Finally, any items that require attention to given information -- forexample, reading passages and the data tables in the mathematics tasks in this stud,," -- arelikely to be locally dependent to some degree.Yen (in press), in a comprehensive examination of local item dependence using datafrom the Comprehensive Tests of Basic Skills (CTBS/4) and the 1991 edition of theMSPAP, the same performance assessment used in this study, presents a compendium ofpossible causes of item dependence. She refers to these causes as external assistance orinterference; speededness; fatigue; practice; item or response format; passage dependence;item chaining; explanation of previous answer; scoring rubric or raters; and content.knowledge, and abilities. Several of the possible causes in her list appear consistent withthose hypothesized in this study (i.e., passage dependence, item chaining, and explanationof previous answer). Other possible causes in her list (i.e., external assistance orinterference, speededness, fatigue, practice, scoring rubric or raters) would not beidentified by examining items, as was done in this study. The other two causes in her list(item or response format; scoring rubric or raters; and content, knowledge, and abilities)were not hypothesized in this study.

~D dD~LAssessing Local Dependence - 17'Hanna, G.S., & Oaster, T.R. (1980). Studies of the ~riousnes.s of three threats topassage dependence. Educational and Psychological Measurement, CA, 583- 96.Huynh, H. (1993). On equivalence between a ixa'lial credit item and a set of independentRasch binary items. Manuscript submitted for publication.Masters, G.N. (1982). A Rasch model for partial credit scoring. Ps',r.'chomerrika, 49, 269-272.Nicholas, L.E., & Brookshire, R.H. (1987). Error analysis and passage dependence oftest items from a standardized test of multiple-sentence reading comprehension foraphasic and non-brain damaged adults. Journal of Speech and Hearing Disorders, 5.2,358-66.Scherich, H.H., & Hanna, G.S. (1977). Passage-dependence in the selection of readingcomprehension test items. Educational and Psvchological Measurement, 37, 991-7.Shepard, L. A. (1991). Psychometricians' beliefs about learning. EducationalResearcher, "0 2-16.Wainer, H., & Kiely, G.L. (1987). Item clusters and computer adaptive testing: A casefor testlets. Journal of Educational Measurement, 24 (3), 185-201.Wiggins, G. (1989). A true test: Toward more authentic and equitable assessment. Ph_.ADelta Kappan, 7_!, 703-713.

". ...... Assmins Local Dependence - ]8Wilson, Iv[. (1988). Detecting and interpreting local item dcpendcncc using a family ofRash models. Ap~ied Ps)'cholggical Measurement, 12, 353-64.Wright, B.D. & Masters, G.N. (1982). Rating .~,.olc anal)'sis. Chicago: Thc Univcrsity ofChic.ago Mesa Press.Yen, W.M. (I984). Effects of local itcm dcpendcnce on thc fit and cquating pefformanccof the three-parameter logistic model. Applicd P~'chological Mc~,~urcmcnt, 8, 125-145.Yen, W.M. (in press). Scaling performance asscssments: Slrategics for managing localitcm dcpcndcncc. Journal of Educational. Mc~urcmcn[.mac~teveXdepend.doc 7/25193

' - _ _ Aues,~in$ Local Dependence. 19Footnotes1 First introduced in the context of computer-adaptive tcsting, the term is used hcrcto refer to any group of items that is treated together a.s a single unit or super ilcm.9" In the MSPAP, stimuli that elicit examinee responses arc referred to asassessment "activities" to distinguish them from multiple-choice items. To faciiitatcdiscussion in this paper, such assessment activities will be rcfcrrcd to as "items."Similarly, "clusters" in this paper are referred to as "tasks" in the MSPAP.3 It should be noted that all items were used to group examinees.4 It may be noted that, for a sample size in the range of 1,600, the probability ofobtaining correlations of .08 and. l0 or highcr is less than .001 and therefore almostnegligible.5 This explanation is not dependent on the prediction because it is scaled as aprocess item while the prediction is scaled as a content item.6 The process items would probably also be statistically dependent on the contentitems in this and many other mathematics clusters. MSPAP mathematics content andmathematics process items are scaled separatcly to avoid this expected interdependence.

elOk- ~Table 1Distribution of-ltems in the Seven Dam ScL~Variable RD36 RD51 RD81 MC31 MC83 MI:'31 MP81Total N of items 30 31 27 38 47 28 29N of clusters 4 4 4 9 9 9 9N of items inCluster 1Cluster 2Cluster 3Cluster 4Cluster 5Cluster 6Cluster 7Cluster 8Cluster 98 8 910 10 i08 8 24 5 83 3 2 64 5 3 3B3 5 2 34 5 3 35 7 4 13 5 2 45 5 3 44 5 3 27 7 6 3

~u 4,Table 2Raw Score Descriptive Statistics for Score GroupsGroup Statistic RD36 RD51 RDSI MC31 MC83 M l::r31 MPSIAll subjectsN of students 1473Range 0-50Mean 25.57S.D. 11.55Pet. of max. 51.14Skewness -0.331347 1410 1461 1613 1461 16130-49 0-50 0-48 0.59 0.41 0-3229.56 31.87 18.65 ! 9.78 9.05 9.149.24 9.84 9.65 10.27 7.,56 5.7460.33 63.74 38.85 33.53 22.07 28.56-0.61 -0.83 0.43 0.67 , 1.12 0.74Score group1 Range 0-14 0-21 0-23 0-9 0.1 ! 0.1 0-3N of Ss 298 287 285 347 334 189 3222 ...... Range 15-23 22-27 24-30 10-14 12-19 2-5 4-6N of Ss 319 251 278 258 355 374 3893 Range 24-29 28-32 31-35 15-19 20-27 6-8 7-9N of Ss 275 289 327 286 335 217 2994 Range 30-35 33-37 36-39 20-27 28-37 9-14 10-13N of Ss 277 306 236 307 328 294 2695 Range 36-50 38-49 40-50 28-48 38-59 15-41 14-32N of Ss 286 213 278 261 315 278 296Note. Pet. of max. = mean raw score divided by maximum possible raw score (see top of range inrow 2 of upper panel above).

" OTable 3F.,'equency Distributions for BEtween-Cluster C00"elations Pooled Acrr~.~ Four Score GroupsCorrElation RD36 RD51 RD81 MC3 i MC83 MP31 MP81-0.14 1-0.12 1-0.11 1-0.10 1-0.08 2 1-0.0"/ ................................................................ I ............. 1 ............................. 5 ............. 4-0.06 1 3 I " 7 7-0.05 2 3 2 3 13 6-0.04 . . . . . . . 4 4 4 4 5 9 24-0.03 4 3 3 10 2 15 10-0,02 5 5 3 21 15 21 10-0.01 6 2 2 27 34 ~ 270.(30 ................................ 2 ............. 7 ............. 4 ........... 48 ........... 54 ........... 33 ........... 310.01 2 1 16 12 7 120.02 10 9 4 40.03 1 6 2 20.04 4 3 I 10.06 1 20.07 ................................................................................................................................ 1Nt:mber ofce,"rclationsAverage24 24 24 144 144 144 144-.017 -.020 -.029 -.006 -.003 -.019 -.020

ii I &Table 4Frequency Distributions for Within-Cluster Correlations Pooled Acrr~s Four Score GroupsCorrelation RD36 RD51 RD81 MC31 MC83 MF'31 MPSI-0.~ 1- ~ ° ~ H ~ ° ~ ° ° ~ ° ° ° ~ H * - ° ~ ° ~ Q ~ . . . . ~ . ~ o . ° ~ . ° . . . ~ . . ° ~ ° ° . ° . ° ~ . ~ . ° ~ ° ~ . ° . . ° . ~ . . .... ,, ......... , .......... . .............. ..--,.,. [-0.~ 1 1 !-0.02 ! ! 3 1-0.01 1 1 1 2 I0.00 ................................ 3 ............. 2 ............. 2 ............. 7 ............. 1 ........... "7 ............. 70.01 5 2 1 40.02 2 2 I0.03 2 2 2 60.04 3 2 10.05 1 1 I 30.06 40.07 ................................................................ 1 ............. 30.08 10.09 1 I0.I0 10.I1 10.12 1 !0.130.150.17 I0.180.200.22 I0.~0.270.30 10.32 30.33 10.34 10.454 3 42 " w14 2 42 1 42 34 l............. 4 ............. 1 ............. 13Il3 2111 111Number ofcorrelationsA v e rage16 16 16 36 36 34 310.025 0.050 0.076 0.036 0.070 0.051 0.025

a0.Table 6Average Within-Cluster Correlations in l=ach Score Group for the Mathematics Examinations,Sx maiioh- ...... Cluster Score Group Degree of2 3 4 5 dependenceMC31MC831 -0.O2 0.02 0.00 0.042 0.01 0.00 0.00 0.073 0.17 0.06 0.12 0.074 0.06 0.03 0.01 0.055 0.01 -0.01 0.08 0.066 0.07 0.09 0.03 0.057 0.03 -0.01 -0.03 0.038 -0.01 0.01 -0.04 -0.019 0.05 0.03 0.03 0.061 0.00 0.01 0.01 °0.042 0.04 0.03 0.06 0.013 0.04 0.08 0.08 0.074 0.17 0.22 0.23 O. 185 0.02 0.07 0.07 0.066 0.08 0.09 0.13 0.137 O. 10 0.05 0.06 0.058 0.15 0.07 0.06 0.039 0.02- 0.01 0.03 0.03SubstantialModerateSubstantialSubstantialMP311 * 0.07 0.08 0.08 Moderate2 0.01 0.00- 0.02 0.113 * -0.03 0.27 0.45 Substantial4 -0.02 0.00 0.09 O. 12 Moderate5 -0.01 -0.01 -0.01 O. I06 -0.03 0.03 -0.02 -0. 107 -0.03 0.03 -0.01 0.018 0.02 0.08 0.13 0.20 Substantial9 -0.04 0.01 -0.01 0.04MPSI1 0.01 0.01 0.01 0.032 °0.02 0.05 0.04 0.003 -0.01 0.04 0.01 0.074 0.08 0.04 0.02 0.066 * 0.03 0.138 O. 117 0.00 0.00 -0.01 0.058 -0.01 0.03 -0.08 -0.019 -0.03 0.04 0.03 0.05ModerateNote. Degree of dependence determined by number of correlations greater than .08 and. 10. See textfor further details. "*" indicates insufficient data to compute correlation.

Table 5Average Within-Cluster Correlations in Each Score Group for the Reading ExaminationsExamination Cluster Score Group Degree of2 3 4 5 dependenceRD36 1 0.05 0.130 0.01 0.042 0.01 0.00 -0.01 0.013 0.04 0.01 0.03 0.034 O. 12 0.04 0.09 0.01 ModcrateRD51 1 0.02 0.01 -0.02 0.032 0.00 -0.01 0.01 0.033 0.04 0.02 0.05 0.044 0.33 0.22 0.11 0.34 SubstantialRD81 1 0.05 0.00 -0.02 0.012 0.07 0.02 -0.03 0.003 0.32 0.32 0.32 0.304 O. 10 0.03 0.02 0.03SubstantialNote. Degree of depcndcncc determined by number of correlations greater than .08 and. 10. See textfor further details.

|," tTable 7Content Descriptions for Clusters, with Substantial Local DependenceF.~amination -- Cl~'ter Assessed Themeoutcome(s)-Responserequirement(s)1:li:)5I ................... 4Read toperform ataskFamilyrelationshipsFill in enrollmentform3Read forinformationand forliterawexperienceMcctingchallengesShort open-endedwritten response:compare and contrastMC31 3Estimation.arithmeticoperations,numberrelationsRec3.'clingaluminumcansArrange data in table;perform calculations;cstimateMC83 4Statistics,probability,patterns &relations,algebraCutting acakeDescribe pattern intable and writeequation; calculateprobabilityMC83 6Arithmeticoperations,geometD',measurement,patterns andrelations,algebraExploringpatternsShort open-endedwritten response, thentabulated; describerelationship and writean equation; drawhexagon to scaleMP'31 3Problemsolving,communication,reasoningRe~'clingaluminumCallsEstimate and explainestimate in wordsMP31 8Communication,reasoning,connectionsSchoolstoreDraw conclusion frombar graph; explainconclusion~m~d©p zables/pm 7/25/93.............................................. ............