Student Perceptions of Teaching Effectiveness - Harvard Graduate ...

More documents

Recommendations

Info

measurement error in calibrating reliability statistics. Recall that there are several survey items for each student in each composite index for each of the Seven C’s. The Seven C’s indices defined at the student level are used to create classroom-level measures that aggregate across students. Classroom-level measures, in turn, are embedded inside schools. The measurement model assumes that random factors introduce measurement error at each level of this system. Calculations that calibrate the measurement errors are the basis of reliability measures. For each Seven C’s index, the reliability represents the fraction of variation in that index that is true variation, not due to measurement error. Using this measurement model, indices for the Seven C’s constructed from MET survey data have proven highly reliable—in the range of 0.80 and above. They are also remarkably stable during the school year. To examine this stability, a special sample of 141 classrooms was surveyed in December 2009 and again in March 2010. Classroom-level indices were constructed for each month and compared. Corrected for measurement error, the correlations for given classroom quality indices, comparing the December and March values, ranged between 0.70 and 0.85. A more intuitive way to analyze the stability is the following. For each month, we used the survey responses for all 7 of the Seven C’s to form a single composite quality measure. Then, we used this measure to rank order the classrooms and form deciles. The bottom ten percent of classrooms in the ranking comprises the first decile, and so on. There is one set of deciles defined using the December data and another set defined using the March data. For the 141 classrooms in the analysis, 78 percent moved no more than one decile up or down between December and March. Ninety-three percent moved no more than two. Hence, there was great stability. At the same time, some classrooms did get better and some got worse. We know this because consistency between student responses within-classrooms at each point in time was sufficient to distinguish classrooms that changed from those that did not. In other words, for a minority of classrooms, we are able to reject the null-hypothesis that students perceived quality to be the same in March as in December. Dimensions of Teaching Quality We analyzed data from 1264 English Language Arts (ELA) classrooms and 1094 math classrooms to examine how strongly the Seven C’s distinguish dimensions of teaching quality. 1 The data were classroom averages of student survey responses. The goal was to learn more about what combinations of survey responses most strongly distinguish classrooms from one another. In addition to the Seven Cs, we included two other constructs in the analysis reported in Panel A of Exhibit 1. One was a three-item measure of how much time the class spent preparing for the state accountability exams (referred to as Test-Prep Emphasis on Exhibit 1). The other was a three item measure of Class Emotional Climate, measuring the degree to which students felt nervous, stressed or angry in the classrooms where they completed the MET survey. Panels A and B of Exhibit 1 summarize two distinct analyses. Each sheds light from a distinct vantage point (which we explain) on dimensions of classroom quality. The analysis summarized in Panel A included all of the survey items for each of the Seven C’s, as well as the Test Prep Emphasis and the 1 This is what researchers call “factor analysis.” Here, we explain the findings in nontechnical language. 4 | P a g e
Class Emotional Climate items. The analysis for Panel B included all of the items for six of the Seven C’s, excluding Control. Panel A: The Helicopter Vantage Point. Imagine sitting 300 feet above the ground in a helicopter looking down at a row of apartment buildings. You are trying to describe what you see. Some of the variation you observe—say, twenty percent of it—entails different styles of chimneys. Another ten percent entails different sized air conditioning units on the rooftops. Twenty percent is flat roofs and constituent materials. The remaining fifty percent might be the sides of the various buildings; but because of the angle of vision and the light, the features (e.g., windows, doors, siding materials, etc.) blur together and are difficult to distinguish. Exhibit 1: Dimensions of Teaching Distinguished by Student Survey Responses Secondary School Students in English Language Arts/Reading and Math PANEL A: Dimensional Analysis of Classroom Conditions English Language Arts (ELA) Math Dimensions: 1 2 3 4 5 1 2 3 4 5 The Five “Support” Cs Control (Press) Test-Prep Emphasis Challenge (Press) Class Emotional Climate Proportion of Variance 0.52 0.22 0.09 0.08 0.05 0.52 0.23 0.09 0.07 0.05 PANEL B: Dimensional Analysis of Six C’s other than Control. English Language Arts (ELA) Math Dimensions: 1 2 3 1 2 3 4 Care Clarify Challenge A: Effort Challenge B: Rigor Captivate Confer Consolidate Proportion of Variance 0.47 0.33 0.12 0.31 0.29 0.23 0.13 Classrooms with no fewer than 10 and no more than 35 students. Based upon principal factors analysis with orthogonal varimax rotation, using classroom-level means as the units of analysis. Separately for each subject (i.e., ELA and math), the initial factor analysis distinguished the Control index from the other six; Control loaded on one factor while the other six loaded on another. A second factor analysis using only items for the other six indices produced the distinctions shown in columns 1, 2 and 3. N=1264 Classrooms for ELA; N=1090 Classrooms for math. Panel A is like the view from the helicopter. First, just as the vantage point of the helicopter makes it difficult to distinguish windows, doors and textures on the sides of the buildings, the vantage point for Panel A makes it difficult to distinguishing the Five Support C’s from one another and also from 5 | P a g e
Page 1 and 2: Student Perceptions of Teaching Eff
Page 3: accepts nothing less than our full
Page 7 and 8: instead shows up in a supporting ro

Student Perceptions of Teaching Effectiveness - Harvard Graduate ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?