Philadelphia SBPA - Consortium for Policy Research in Education ...

CPRE 

CONSORTIUM FOR POLICY RESEARCH IN EDUCATION 

University of Pennsylvania •Harvard University •Stanford University 

University of Michigan •University of Wisconsin-Madison 

A Case Study of the Philadelphia Public School’s 

School-Based Performance Award Program 

Eileen Kellor, Allan Odden, and Eric Conti 

Consortium for Policy Research in Education 

University of Wisconsin-Madison 

1025 W. Johnson Street 

Madison, WI 53706 

(608) 263-4260 

http://www.wcer.wisc.edu/cpre 

September, 1999 

The research reported in this paper was supported by grants from the Pew Charitable 

Trusts (No. 97001184000), from the U.S. Department of Education, Office of 

Educational Research and Improvement, National Institute on Educational Governance, 

Finance, Policy-Making and Management, to the Consortium for Policy Research in 

Education (CPRE), the Wisconsin Center for Education Research, School of Education, 

University of Wisconsin-Madison (Grant No. OERI-R3086A60003) and from the 

Carnegie Corporation (No. B6520). The opinions expressed are those of the authors and 

do not necessarily reflect the view of the Pew Charitable Trusts, the National Institute on 

Educational Governance, Finance, Policy-Making and Management, Office of 

Educational Research and Improvement, U.S. Department of Education, the Carnegie 

Corporation, the institutional partners of CPRE, or the Wisconsin Center for Education 

Research. 

UW 

Wisconsin Center for Education Research, University of Wisconsin-Madison 

1025 West Johnson Street, Room 659, Madison, WI 53706-1796 •Phone 608.263.4260 •Fax 608.263.6448

Introduction 

Effective with the 1996-97 school year, the Philadelphia school district 

implemented a school-based performance award program as part of a broader strategy to 

improve student achievement throughout the district. This case will provide basic 

background information about the district and the circumstances leading up to the 

decision to establish a school-based performance award program. It will summarize the 

process used to design the program and provide an overview of the program’s design 

details and enablers that support the program’s goals. Finally, some evaluation findings 

will be reported and observations made regarding possible considerations for other 

jurisdictions considering a school-based performance award program. 

Background 

The Philadelphia Public Schools enrolls 214,000 students distributed across 259 

schools. The student population is 65% African American, 19% white, 11% Latino, and 

5% Asian. The district employs approximately 12,000 teachers at an average salary of 

$46,659, compared to the state of Pennsylvania average of $47,147. Per pupil spending 

in 1997 was approximately $6,860. Benchmark tests administered in 1996 established 

that less than half of the students displayed basic knowledge and skills in mathematics, 

reading, and science (Partnership for Reform, 1996). 

In 1983 the district entered into a desegregation agreement in which it agreed to 

address educational disparities as well as racial isolation (Schmidt, 1994a). Through the 

efforts of the district and other public and non-profit organizations, a variety of reform 

strategies were tried over the years. For example, beginning in 1988 the Philadelphia 

School Collaborative began to assist the district in its effort to restructure the city’s 

UW 



neighborhood high schools. The effort included implementing school-based 

management, shared decision-making at the school level, and generally identifying the 

administrative and procedural obstacles to restructuring the high schools and working 

with the district to remove those obstacles. One of the strategies included the creation of 

“charters”, or semiautonomous schools, within the high schools. Many of the problems 

encountered in the high school restructuring effort were mirrored at the lower grade 

levels, e.g., high student and teacher mobility, challenging labor-management relations, 

promotion policies that resulted in large numbers of students being over-age for their 

grade level, and chronic financial problems (Bradley, 1992). 

Early in 1994 a judge ruled that the district had failed to desegregate or to provide 

minority and poor students with equal educational opportunities. Although the district 

claimed that the disparities in achievement were separate from and irrelevant to the 

desegregation case, the judge stated that the educational differences were “the paramount 

and most fundamental issues presented.” The judge appointed a panel to help the district 

develop a plan that would address the achievement issues by some specific strategies 

identified by the judge, as well as others developed by the panel (Schmidt, 1994). Thus, 

the issue of improving student achievement became not only a socially desirable goal but 

also a court-mandated goal for the district. This backdrop contributed to the high level of 

importance and visibility of the district’s efforts to improve student achievement. 

At this same time the district was in the process of hiring a new superintendent. 

The previous superintendent had retired in the summer of 1993 after 11 years as 

superintendent. In June 1994, the Board announced that it had selected David Hornbeck, 

an educational consultant well known for promoting systemic, standards-based education 

UW 



eform, including the state of Kentucky’s comprehensive education reform program. 

Hornbeck had been one of two finalists for the position and was the only candidate 

remaining after the second candidate withdrew his application (Bradley, 1994). The 

community generally reacted positively to his selection and his strong commitment to 

improving student achievement. In fact, Hornbeck’s own contract included provisions 

that tied salary increases to student achievement goals and he made his appointment 

contingent upon being allowed to implement an accountability program (Bradley, 1994). 

One of Hornbeck’s first projects as superintendent was the development of a 

strategic plan for the district; the plan was titled Children Achieving. Around the same 

time the court-appointed panel issued its recommendations for restructuring 

Philadelphia’s schools, Hornbeck made public his proposal for a strategic plan for the 

district. The proposal was generally consistent with his statements about what he would 

do to improve Philadelphia’s schools that he made during the selection process. The 

strategic plan was developed through the efforts of seven task forces involving hundreds 

of educators and citizens. It included many of the same suggestions made by the courtappointed 

panel, including downsizing central administration and regional offices; 

implementing school-based management throughout the district, including local school 

councils of parents, teachers, and administrators; extending the paid work year for 

principals and teachers to allow for more professional development opportunities; 

reducing teacher-student ratios and creating smaller “learning communities” within 

schools; providing full-day kindergarten and prekindergarten; developing alternative 

schools and other programs for the most disruptive students; and also developing 

standards, assessment and accountability systems. The most racially isolated schools 

UW 



would be targeted initially and most of the reforms would be implemented within three 

years (Schmnidt, 1994b). 

In early 1995 the Annenberg Foundation announced that it was awarding a $50 

million challenge grant to the Philadelphia schools to support the comprehensive and 

systemic reform agenda laid out in Children Achieving. The grant required an additional 

$100 million to be raised, with $50 million to come from both the public and private 

sector. The Greater Philadelphia First organization, a coalition of leaders from major 

local corporations, would be the fiscal agent for the grant. The strong leading role of the 

superintendent in planning Philadelphia’s reforms, rather than forces external to the 

schools, was noted as a relatively unique aspect of Philadelphia’s Annenberg proposal. 

In fact, an individual involved in the drafting of Philadelphia’s proposal stated that the 

drafters made an effort to “find the elements of Mr. Hornbeck’s strategic plan that fit the 

best with Mr. Annenberg’s interests, in order to have a single comprehensive reform 

effort in Philadelphia in which all the pieces fit” (Sommerfeld, 1995). 

Ten major reforms were articulated in the strategic plan, Children Achieving. 

They were: 

1. Set high expectations for everyone. 

2. Design accurate performance indicators to hold everyone accountable for 

results. 

3. Shrink the centralized bureaucracy and let schools make more decisions. 

4. Provide intensive and sustained professional development. 

5. Make sure that all students are ready for school. 

UW 



6. Provide students with the community supports and services they need to 

succeed in school. 

7. Provide up to date technology and instructional materials. 

8. Engage the public in shaping, understanding, supporting and participating in 

school reform. 

9. Ensure adequate resources and use them effectively. 

10. Be prepared to address all of these priorities together and for the long term. 

In February 1995, Hornbeck announced plans to go ahead with aggressive reform 

measures (Schmidt, 1995). One of the specific strategies (Strategy 2) developed to 

achieve the strategic plan’s goals acknowledged the need for assessments and an 

incentive system. That strategy included three specific components: 

1. Implement a system of performance-based assessments tied to the new high 

standards for students; 

2. Design valid assessment for students of diverse language background. 

3. Design an incentive system for staff that links achievement by all students, 

including those whom the schools have historically failed, to real rewards and 

penalties. 

These three components created the basic framework for Philadelphia’s school-based 

performance award program. 

The focus on results and their publication, coupled with extrinsic incentives in the 

form of rewards and sanctions, are central to Philadelphia’s accountability system. The 

district believes that these factors are engines that drive school improvement for all 

students. Insisting that the 10 Children Achieving goals must work together as an 

UW 



integrated, comprehensive package, the thinking behind the accountability component is 

that it contributes to achievement in one of two ways: teachers will teach better because 

they desire the reward (either cash or public recognition) obtained for higher test scores, 

and teachers will teacher better because if student performance fails to improve, the 

teachers will be subject to various sanctions (Luhm, Foley and Corcoran, 1998). 

UW 



Program Design 

Although the basic principles of the program were set forth in the strategic plan, 

the development of the program’s details included hundreds of partners. The design 

process, including development of the actual curriculum content and student performance 

standards, made steady progress, with the work formally beginning in December 1995 

and the actual accountability measurement system completed by the end of November 

1996. The participants included 1,000 administrators, teachers, and parents from the 

district as well as over 2,500 citizens who attended 22 community meetings (one in each 

district cluster). During this time, a Philadelphia team was part of CPRE’s first schoolbased 

performance award design seminar (Odden, Heneman, Wakelyn and Protsik, 

1996). 

The actual design of the details began when the Standards Writing Teams began 

their work. In April and May of 1996, draft standards in English language arts, 

mathematics, science and the arts, were distributed for review, with review teams 

formally commenting on the first four sets of standards in the summer of 1996. In July 

1996, concurrent with the review of the draft standards, the district held a four-day 

professional development session on standards-based instruction for teams of teachers. 

The second draft of standards was distributed to all teachers in August 1996 and 

curriculum resources guides for grades K-4, 5-8 and 9-12 distributed to teachers in 

September. During October and November the details of the accountability system, the 

Performance Responsibility Program, were developed and approved by the Board of 

Education and public hearings on the proposed standards were held in each of the 

district’s 22 clusters. By the end of 1996 the Board adopted the first four sets of 

UW 



Recommended Content Standards, Benchmarks and Performance Examples. A resource 

guide for standards-based assessment and instruction was distributed to schools in 

February 1997. Development of the standards in the remaining areas (health and physical 

education, social studies, and world languages) continued, with final Board approval in 

July 1997 (CPRE, 1998). 

Elements of Performance 

The district’s accountability system is based on three performance indicators, 

which are combined into a single measure called the Performance Responsibility Index 

(PRI). The PRI includes three separate elements: student achievement scores in reading, 

match and science (60% of total score), graduation and promotion rates (20% of total), 

and student and staff attendance (20% of total score). For elementary and middle 

schools, a school’s promotion rate must average at least 95%; the graduation/persistence 

rate for high schools requires that an average of 95% of first-time 9 th graders must 

graduate from a Philadelphia school district high school four years later. With respect to 

student attendance, an average of 95% of the students in a school must be absent fewer 

than 10 days per school year to be considered “proficient” and an average of 95% of the 

school staff must be absent fewer than 10 days per school year (CPRE, 1998). 

The district selected the ninth edition of the Standard Achievement Test (SAT-9) 

to measure student achievement. The district chose the SAT-9 assessment because it is a 

criterion-reference test based on national standards and because it has a performancebased 

feature, not relying exclusively on multiple choice. Because the district’s own 

standards were based on national standards the district believed the SAT-9 fit well with 

its local standards. The first year of the program included the SAT-9 scores in reading, 

UW 



mathematics and science in grades 4, 8, and 11. The second year of the program added 

grades 3, 7, and 10 but these grades were not included in the accountability program 

during the first cycle (CPRE, 1998). 

The student and staff attendance measure was included to address concerns about 

serious teacher and student attendance problems. The persistence to graduation rate and 

promotion rate were included to ensure that students were not just dropping out and thus 

excluded from the accountability measures. In addition, those measures help to 

demonstrate that students are actually achieving, i.e., making progress and moving from 

grade to grade and eventually graduating. 

Calculating Change in Performance 

The following section documents how change in performance, i.e., improvements 

in student achievement) are calculated for each school. It also summarizes the methods 

the district uses to make the calculations fair, such as adjustments made or precautions 

taken to address special measurement issues and concerns. 

How change is measured 

A key concept of Philadelphia’s accountability program is that schools are not 

compared to other schools; each school is compared to its own baseline performance over 

time. Each of the three performance elements noted above has multiple performance 

levels. The SAT-9 scores are categorized into seven performance levels and the other 

two performance elements are categorized into six performance levels. Each 

performance level is given a weight, e.g., ranging from 0.0 points for Not Tested to 1.2 

points for Advanced level for the SAT-9 score categories. The value of the performance 

UW 



level is then multiplied by the percentage of students categorized at that level and the 

total score derived from the sum of the scores at each level. 

The twelve-year (one student generation) goal is that every school will have an 

average score of 95% across the three performance elements. This is measured by means 

of the PRI, which is a summary of a school’s results compared to its baseline 

performance in the 1995-96 school year. Using the 95% average as the ultimate goal, the 

district sets a target for each school every two years. The current accountability plan 

anticipates six two-year cycles. 

The SAT-9 academic scores are reported using seven performance levels. These 

performance levels are determined by the SAT-9 test reports and are similar to the 

National Assessment of Educational Progress (NAEP) performance levels. Harcourt 

Brace, the test developer, provides teacher evaluators with a rubric that outlines students’ 

performance levels. The standard report includes four levels of performance from 

advanced to below basic. However, the district was concerned that students making 

significant progress in the below basic performance level but not enough to attain the 

basic performance level would not be measured as having improved. To remedy this 

problem, the district and Harcourt Brace agreed to further delineate below basic 

performance into three separate performance levels. Thus, the seven levels are as 

follows: 

1. Advanced: superior performance on world-class standards. 

2. Proficient: solid performance on world-class standards. 

3. Basic: partial mastery of world-class standards. 

4. Below Basic III: inadequate mastery. 

UW 



5. Below Basic II: little mastery. 

6. Below Basic I: very little mastery. 

7. Not tested: students not having a valid test score. 

It should be noted that the ability to identify improvements in student achievement for 

students in the below basic category was an issue in Kentucky’s accountability program. 

That experience likely influenced Philadelphia’s decision to differentiate performance 

within the broader Below Basic category. In addition, Philadelphia’s program requires 

that the percentage of students categorized as performing below the “basic” level on the 

SAT-9 must decrease by at least 10 points, which means that at least some portion of 

students must move into at least the basic category for a school to receive an award. A 

school cannot meet its performance target unless performance improves significantly 

among the lowest performing students, thus a school may not “skim” the higherperforming 

students and rely on movement of these students from Basic to Proficient to 

Advanced levels of performance. 

The teacher and student attendance measures include the assignment of 

attendance levels to six categories with the same titles as those for student achievement. 

The calculation of the student attendance component of the PRI is as follows: 

1. Each student’s individual attendance rate and the student’s days (enrolled) are 

placed in one of the following categories: advanced (96-100%); proficient 

(95%); basic (85-94%); below basic III (80-84%); below basic II (75-79%); 

and below basic I (10-74); 

2. The total number of days enrolled by students in each category is summed and 

divided by the school’s total number of possible days to arrive at a PRI score 

UW 



for each category. For example, if the total number of possible days for a 

school was 30,000 and the number of days enrolled by students in the 

advanced category was 3,000, students in the advanced attendance level 

would account for 10% of the total possible days for the school 

3. Attendance rates in each category are then weighed by a factor. For example, 

the factor for the advanced category is 1.2, so the total subscore in the 

example above would be 12 (10 x 1.2). 

Staff attendance rates are based on the total number of staff and the percent of 

school days they attended. All teachers are included in the statistic, as well as 

administrative and instructional support staff. Food service workers and custodians are 

not included in the score. The same weighting system noted above is used by category, 

i.e., advanced is weighted 1.2; proficient is weighted by 1.0; basic is weighted by 0.8, 

etc., but the cut scores for each category are different. For example, any staff attendance 

under 93% is considered below basic, while student attendance under 85% is considered 

below basic. The differences between basic and advanced attendance are quite small, 

e.g., a teacher who attended 169 days out of a 180 day school year, one who attended 171 

days, and one who attended 173 days would each be included in a separate category. For 

example, a school in which 60% of staff attended 96% of school days would receive 72 

points toward its overall staff attendance goal, while the same proportion of teachers 

attending 94% of school days would receive only 48 points toward a school’s overall 

score (Luhm, Foley & Corcoran, 1998). 

An aspect of the staff attendance score that was controversial in some quarters 

involves who is included in the calculation. As noted above, some school support staff 

UW 



such as food service workers and custodians are excluded, but teaching staff on long-term 

leave for illness or maternity leave are not excluded. The district reports that this policy 

struck many principals as unfair. However, the intention of this policy was not to judge 

the validity of the absence but rather to stress the importance of having staff present to 

provide services to students. Moreover, by weighting attendance in the highest category 

by a factor greater than 1 (1.2), the effect of long-term absences on a school’s PRI can be 

balanced out by having more people in the “advanced” category of attendance. 

A final detail of the attendance scores is how they contribute to the overall 

calculation of the PRI. While the SAT-9 results for each subject area count individually 

as one component of the PRI score, staff and student attendance are averaged to create 

the Enabling Score. These two different indicators, staff and student attendance, were 

collapsed into one component to make a single composite score that counts for one-fifth 

of the total index score, with academic measures counting 60%. 

The remaining 20% of the PRI score is calculated using student promotion and 

persistence scores. Promotion measures the ability of staff to enable students in grades 1 

through 8 to move through at their appropriate grade level until high school. Persistence 

involves measuring the four-year graduation rate of the district’s 9 th grade students. The 

persistence rate measures the proportion of first-time 9th graders in school year 1993-94 

who graduate from a Philadelphia School District high school four years. The first group 

to which the persistence measure applied were 1992 9 th -graders. Multiple schools can get 

credit for student persistence if the student transfers between schools and remains on 

schedule to graduate. 

Making change fair 

UW 



A small number of schools did not participate in the first accountability cycle 

because they were new and did not have a baseline for 1995-96. A few other schools did 

not participate because they served unique student populations for which the PRI would 

not be an appropriate measure of school progress. The new schools are included in the 

second accountability cycle because baseline performance data is now available. For the 

remaining schools, appropriate performance indicators and measures are being 

developed so that they, too, will eventually be included in the accountability program. 

For the first year of cycle 1 of the program (1996-97), the SAT-9 score of every 

4 th , 8 th and 11 th grade student was counted toward the score of a school. Because of the 

issues of student mobility for the baseline year (1995-96) a student was assigned to the 

school where she/he spent the most time. Students in grades 4, 8 and 11 who do not 

complete have a composite score on the subject being measured are given a score of zero 

which affects how a school performs in terms of the accountability index. This policy is 

designed to reduce gaming by ensuring that school administrators do not inflate their 

scores by testing only those students whom they believe will perform well. Other 

scenarios for incomplete tests include: scores missing due to absence; students not taking 

both the multiple choice and the open-ended parts of the test; students not having met the 

publisher’s criteria for a serious attempt at the test; and students having been excluded 

from last spring’s testing because of special education or language minority status. 

Students who are classified as severely and profoundly impaired, as trainable 

mentally retarded, or are in the lowest level of English proficiency are exempted from 

testing and are not counted in the untested category. The policy for students classified at 

the next level of English proficiency is that they should be tested with appropriate 

UW 



accommodations, unless the teacher feels that the testing would be harmful to the child or 

the student enters the district after December 1. These students and those with mild 

disabilities are counted as untested if they do not take the test and students with mild 

disabilities are to be tested at their age-appropriate level with accommodations unless 

their individual education plan indicates differently. A school may decide to give a 

disabled student an out of grade level test, but this disqualifies the student and categorizes 

them as non-tested. 

The program’s method of calculating change in student achievement is intended 

to encourage the inclusion of all but a small proportion of students in the testing program. 

This goal may not have been well understood by everyone because there were concerns 

about possible manipulation of the results by excluding certain students or by deliberately 

keeping some students out of the testing program the first year. However, the program as 

a whole is intended to make the district accountable for the achievement of all students 

and not to “excuse” the district from improving the achievement of students with special 

needs, disabilities, or those with limited English proficiency. 

As noted earlier, the three SAT-9 tests in reading, math, and science comprise 

60% of the PRI score. The district reported that many educators made focused efforts to 

improve their students’ achievement on these tests. Despite these efforts to focus 

instruction to help students do better on the tests, there appeared to have been some 

concerns or lack of understanding about how these scores were used to calculate the 

overall index. 

One such controversy concerned the number of students considered not tested. In 

order to encourage the testing of all students regardless of ability the district made a 

UW 



policy decision to count all students categorized as “not tested” in each subject area as 

scoring zeros in calculating the overall score for a school. This discouraged schools from 

attempting to raise their performance on the SAT-9 by pushing low performers out. Yet, 

the “not tested” category is somewhat of a misnomer and implies that all of the students 

in that category were either absent or not included in the testing. 

The district reported 

that many of the students who were counted as not tested did participate in the 

examinations, but either did not complete all of the sub-tests for that subject area or for 

various reasons they did not receive a valid score on the test. The exact criteria applied 

varies by item type, but in general, to receive a valid composite score in each subject a 

child must: 

1. Attempt either three out of the first six items, or any ten multiple choice 

questions AND get one correct answer; and 

2. Attempt one open-ended question and be credited at least one point by the 

scorer. 

Some educators in the district believe that this policy unnecessarily penalizes 

schools with large special education populations whom they hypothesize had a higher 

tested, but invalid rate than other schools. The district indicated that many teachers 

expressed the frustrations their students experienced during the first year the tests were 

administered. However, the district also reported that other educators believed special 

education students are penalized if the system does nothing to encourage their inclusion 

in the testing process. Another way this issue could be looked at as it relates to the 

incentive program is that by including these students in the testing program from the 

outset, a school’s initial performance might indeed be lower than if the students were not 

UW 



included, however, in future years the school just needs to make improvement from that 

initial, lower base. (Refer to the “Evaluation Results” section for more discussion of this 

and related issues.) 

Reward System 

Philadelphia’s overall accountability program provides some form of rewards and 

sanctions for all levels of employees. Because the focus of this case is on the schoolbased 

performance award element of the accountability program, primary emphasis will 

be given to the reward system in that program. However, several other types of rewards 

and sanctions will be noted to demonstrate how the school-based performance award 

program is integrated into other elements of the overall accountability program. 

School-based performance award program 

Under the Performance Responsibility Program, individual schools may receive 

cash awards based on their performance and the extent of improvement demonstrated 

over the two year cycle. Schools that exceed their targets will receive public 

congratulations and an award of $1500 for each teacher and $500 for each other staff 

member. Schools that meet but do not exceed their targets are publicly recognized for 

their accomplishments but receive no award. Schools which improve beyond their 

baseline (1995-96) scores but fall short of their targets receive help from a team of 

educators who will review school information, assess school resources, and help find 

ways for schools to improve. 

Unlike some school-based performance award programs that grant awards to 

individual teachers, Philadelphia’s program grants the award to the entire school. 

Schools that receive an award are allocated a total amount that is available over a two- 

UW 



year period and may spend up to half of the amount the first year. The amount of money 

an eligible school receives is based on the number of staff members in the school’s staff 

allocation for the second year of the two-year cycle; for the first cycle, the staff allocation 

for 1997-98 was used to determine the amount of a school’s award. For teaching staff 

that are less than full-time the amount is pro-rated by the teacher’s full-time equivalency; 

for non-teaching staff the amount is pro-rated based on the number of hours the person 

worked at the school that year. Other than the prohibition on using the funds for staff 

bonuses the schools have wide latitude in deciding how the funds will be used. 

The decision on how to spend the money is made by the School Council with the 

general principle that the money must be spent on initiatives that will help the school 

continue to improve instruction and student achievement. In schools where there is no 

School Council, the decision on how to use the funds must be made by the principal, staff 

representatives and parents. Their involvement in the decision-making process is 

documented by their signatures on the form the principal submits to request release of the 

funds to the school’s budget. The forms must be received by the district before any funds 

are released to a school’s budget. The forms include detailed unit cost information for 

specific staff functions and other types of expenditures such as overtime, substitute costs, 

and professional development costs. The cost information helps the schools make 

informed decisions about how to use their award money and also provides a record of 

how the bonus funds were spent by the school. 

As regards sanctions for schools that do not achieve the specified improvement 

goals, to some extent the failure to meet the targets and to therefore miss the public 

recognition and bonus can be considered a sanction. The more meaningful sanction, 

UW 



however, is probably that schools that continue to decline can be reconstituted and at 

least 75% of the school’s staff replaced if a school fails to meet its short-term goals for 

two consecutive two-year accountability cycles. 

One final program element, the School Support Process, should be noted. The 

purpose of the program is to provide participating schools with intensive support and to 

monitor their ongoing progress in order to catch schools that appear to be on the road to 

decline and thus to prevent them from falling into sanction status. Each participating 

school is assigned a School Support Team, which is chaired by a district leader outside 

the school and the cluster and includes district staff, cluster staff, and parent and union 

representatives. The Team identifies the school’s areas of strength and need and makes 

specific recommendations to the school, cluster and central office as to what is needed to 

support student learning, including identification of progress milestones. The 13 schools 

participating in this program the first year were those that had a decline in Performance 

Index score and an increase in the proportion of students scoring below the Basic level on 

the SAT-9. The second year 12 schools were included, of which six were schools that 

had participated the previous school year and did not meet both parts of the Performance 

target. The other six were schools that declined on one of the performance targets; either 

the Performance Index fell or the percentage of students scoring below basic increased. 

At present, the district has the resources to support 12-15 schools as the program 

currently is designed (Office of Standards, Equity and Student Services, 1998). 

System-wide rewards and sanctions 

A related part of the overall accountability program is that the superintendent and 

his Cabinet are held accountable for improving student achievement through a set of 

UW 



yearly performance goals. System-wide SAT-9 scores account for 50% of their 

performance and their salary increases and bonuses are tied to these targets (Luhm, Foley 

and Corcoran, 1998). In addition, if system-wide achievement falls the superintendent 

and Cabinet can be penalized up to 5% of their annual pay. Cluster leaders are held 

accountable in a similar manner: their annual bonus and increases are directly tied to 

student achievement within their cluster, student achievement systemwide, and specific 

goals agreed on between the cluster leader and the Superintendent. Thus, for the top 

district management the progress (or lack thereof) towards the student achievement goals 

can have a personal, financial and relatively immediate impact. 

Program Funding 

For the first accountability cycle, $11,150,000 was allocated over two years. At 

$5,575,000/year in the operating budget, it represents .37% of the operating budget. This 

amount is significantly less than the 1% of budget amount often suggested by the CPRE 

Teacher Compensation Project as a possible guide. Awards totaling the full budgeted 

amount were granted based on the results of the first accountability cycle. 

It should be noted, however, that although the funding for the school-based 

performance award program in Philadelphia is not part of the $150 million created by the 

Annenberg Challenge, the Annenberg funds were undoubtedly influential in the 

individual school’s ability to meet the student achievement goals. In particular, the 

Annenberg funds were used to create the Children Achieving Challenge, which provides 

a wide range of technical support to the district and its partners. Thus to some extent the 

district’s direct costs for the school-based performance award program were largely the 

UW 



cost of the awards themselves, with many of the related costs, e.g., technical assistance to 

schools, supported at least in part by outside funds. 

Program Enablers 

As noted previously, the explicit shift to an accountability system for 

Philadelphia’s education system began in 1995 as part of the comprehensive reform plan 

outlined in the district’s strategic plan. However, as noted in the Background section, the 

district had begun various types of reform efforts even prior to the new superintendent’s 

arrival and the implementation of the strategic plan. Because the strategic plan outlined a 

comprehensive reform strategy of which the Performance Responsibility Program was 

but one element, it implicitly included many enablers that support the accountability 

program. The district-specific standards and curriculum resources mentioned previously 

were two types of supports for the program. Other supports are summarized briefly 

below. 

The Office for Best Practices was created with responsibility for identifying and 

disseminating information on current best practices in curriculum and instruction. 

Twenty-two (22) geographically based clusters were created; each cluster includes 8-15 

elementary and middle schools and a comprehensive high school. The clusters help 

support the decentralization of the district’s management structure and to reinforce the 

concept of smaller learning communities within the district and within schools 

themselves. Each cluster has staff to provide leadership and support for reform as well as 

an Equity Coordinator. The Equity Coordinator works in collaboration with the cluster 

team to ensure that all students have equal access to educational opportunities such as 

UW 



those mandated by law and program such as Title I, ESOL/Bilingual, and Desegregation 

(CPRE, 1998; Philadelphia Schools, 1999). 

Summer professional development institutes for teachers were provided and 

additional professional development opportunities provided throughout the year. 

Decisions for specific professional development opportunities became a school-level 

responsibility, with brokering and support at the cluster and district level. Family 

Resource Center Networks and associated staff were created and housed in the high 

schools to help forge connections between the families, the schools, and the social service 

system. More school nurses were authorized and placed in the schools on a full-time 

basis; child-care centers for school-age parents were created. These efforts and others 

were part of a broader strategy to have school financial and human resources work in 

unison with families, social services agencies and the community as a whole to help solve 

the social, medical, and financial problems that work against children’s learning (Cohen, 

1996; CPRE, 1998). More recent reform measures include full-day kindergarten for all 

five-year olds; a 300% increase in computers for a computer:student ratio of 1:10; 15,000 

new student volunteers, and the establishment of small learning communities in all 

schools. 

The 1994 contract with the teachers’ union included important new language that 

allowed the district to reconstitute schools that were continually low performing 

(Bradley, 1995). The district attempted to reconstitute two schools in 1997, although an 

arbitrator ruled that summer that the proper procedure had not been followed and the 

reconstitutions were voided. Nonetheless, that action resulted in the district and the 

union reaching some alternate agreements on seniority and transfer and apparently 

UW 



moving closer to a conceptual understanding that drastic changes might be needed to 

bring about the needed improvements to student achievement (Reinhard, 1997). 

Private and philanthropic funds have helped the district implement some of the 

reform plans that it otherwise might have been unable to. In addition, prominent business 

leaders have worked with the district to identify administrative and other types of 

changes that could save the district money. Although the district’s strategic plan is 

premised on the ability to provide and maintain these and other enablers, chronic 

financial issues have made it difficult for the district to carry out all of its plans and have 

threatened some of the initiatives already in place. The district has been engaged in a 

longstanding battle with the state of Pennsylvania over the state funding level; several 

lawsuits have been filed by the district against the state but thus far the district has not 

won any of them. Loans from two banks in June of 1998 ensured that the district’s 

budget would balance (White, 1998). Nonetheless, a report prepared for state lawmakers 

on the district’s financial situation suggested that the district could not sustain its 

operations in the long run and that projected expenditures would fall far short of the 

district’s revenues (Blair, 1998). 

In summary, the district has planned a comprehensive set of enablers that would 

support improved student achievement. Many of those enablers are already in place and 

others are pending. However, due to current and projected future financial concerns, it is 

uncertain whether all of the supports can be sustained in the long run. 

Evaluation Results 

The evaluation of the strategic plan has been ongoing since the program’s 

inception in 1995. Evaluations were conducted by the Consortium for Policy Research in 

UW 



Education (CPRE) from the University of Pennsylvania, as well as Research for Action 

(RFA), a non-profit organization engaged in education research and reform. In addition, 

an external advisory panel was appointed in 1998 and has provided additional review and 

analysis of how the accountability program is working. Highlights of these two formal 

evaluations are noted below, as well as certain concerns that were identified by the 

community. Finally, the results of the first accountability cycle are reported. 

CPRE-Penn Evaluation 

The following findings were taken from the evaluation of the program’s second 

year, 1996-97 (CPRE, 1997). The evaluators recommended that an expert panel should 

be appointed to review the PRI, monitor it over time, and advise the district. This would 

help district officials prove the quality of the system and build confidence in the results. 

It might also reassure teachers that they are not being treated unfairly. In addition, the 

district should put more effort into explaining the components of the PRI. For example, 

although only 51% of the teachers surveyed reported understanding the purpose of the 

PRI, 63% of those teachers believed that it had the potential to benefit their students. 

These data suggest that if district officials were to make a greater effort to educate 

teachers about the purpose of the PRI more teachers might see the potential benefit. 

Other recommendations included changing the calculation of the staff attendance 

variable so that long term illnesses would no longer be included. The qualitative data 

suggested that this might improve teacher and administrator attitudes toward the index. 

The SAT-9 categories (advanced, proficient, basic, and below basic) should not be used 

for the non-cognitive indicators in the accountability index because there is no empirical 

basis for the cut points used to assign schools to these categories. Their use eliminates 

UW 



much of the actual variation, yet exaggerates the importance of minor differences among 

schools. 

With respect to the academic areas, long-term targets might be based on state or 

national data and interim targets could be based on reasonable progress toward these 

goals. The actual changes in performance could be used in the index. To that end, a local 

panel of experts, including teachers and the union should work with the SAT-9 test 

publisher to review the alignment of the “revised” SAT-9 with Philadelphia’s standards. 

In addition, school district officials should move with “due haste” to pilot additional 

student performance indicators that can supplement the SAT-9, such as portfolios and 

course exams. 

The evaluators’ final in-depth analysis concluded that new programs often take 

more than two years to produce effects or increase achievement. Thus, it follows that a 

school that has adopted an appropriate course of action and is working hard to implement 

it may fail to reach its numerical target. For this reason, when district officials publicly 

identify schools as “low progress,” they should include information about action taken to 

improve performance. 

Community Concerns 

Between the 1995-96 and 1996-97 school years Philadelphia’s public school 

students significantly increased their achievement on the SAT-9 test, which led to 

increases in the overall school scores on the PRI. Ninety-two (92) schools reported more 

than a 5-point gain at the basic level or above, and over one-half of those had more than a 

10-point gain. The superintendent and others viewed these improvements as a major 

UW 



achievement and as affirmation that the Children Achieving reform agenda is a sound 

one (CPRE, 1997). 

However, some board members, press and teacher union representatives 

questioned the legitimacy of the increases. They variously argued that the scores 

increased simply because more students took the test, because the baseline was very low, 

or because of the methods used to calculate the index, e.g., testing different students each 

year and the lack of specific controls on testing accommodations for special education, 

disabled and limited English proficiency students. These questions were raised publicly 

when the superintendent met with the school board in January 1998 and have continued 

as chronic concerns, especially among teachers. These were reasonable questions to ask 

about the system, and the external advisory panel has since then considered most of those 

concerns (see following section). 

External Advisory Panel 

As recommended by the CPRE-Penn evaluators, an external advisory panel was 

appointed to review assessment and accountability practices in Philadelphia, with three 

specific objectives in mind: 1) reviewing the extent to which the practices promote the 

strategic plan objectives; 2) providing guidance to promote continuous improvement; and 

3) providing advice on whether or not and how the practices meet professional practices 

for assessment and accountability (Porter, 1998). The panel has five members and is 

chaired by Andrew Porter, a professor of education psychology at the University of 

Wisconsin-Madison and the director of the Wisconsin Center for Education Research. 

Other panel members are experts on accountability and assessment issues from several 

higher education institutions and research programs. Representatives from the school 

UW 



district participate in the meetings to provide information and respond to questions from 

the panel. 

The panel met three times in the 1998-99 school year and a summary was 

developed after each meeting to report the Panel’s deliberations to the Board of 

Education. The panel will continue to meet in the next school year to continue to review 

issues brought to them by the district as well as new issues that arise as the panel 

continues its work. 

The panel identified many elements of the accountability program that it views as 

strengths. The strengths include: 

• The general goal and approach, noting that if all 10 components of the strategy are 

accomplished there will definitely be a marked improvement in the quality of 

education provided to students in the district; 

• The strategy of putting different pieces in place over time rather than all at once is 

appropriate for the district; 

• The commitment to testing all core academic subjects rather than only one or two is 

appropriate; 

• Multiple parties (teachers, students, parents, community) will be held accountable; 

• The district is aligning its assessments with the district’s standards; 

• The program shows appropriate concern for providing accommodations to students 

with disabilities or limited English proficiency, within the broader goal of including 

and being held accountable for the performance of all students; 

• The strategy for getting all but a few students to participate in the accountability 

system seems promising and already seems to be having the intended effect; 

UW 



• More than one test will be used for the student accountability portion, rather than 

relying solely on one test to determine a high stakes outcome; 

• The establishment of incremental improvement levels within the “below basic” 

performance category is appropriate and allows the progress that schools make with 

the lowest-performing students to be recognized (A.C. Porter, personal 

communication, October 21, 1998). 

The panel has reviewed a number of aspects of the accountability program ranging from 

specific technical details to broad policy decisions. Several that are most germane to this 

case are summarized below. 

• Reliability of the school performance index: Concerns had been expressed about the 

stability with which schools are placed into the categories for reward, help, and 

reconstitution. The panel noted that the consistency will be improved in the second 

accountability cycle because school performance will be based on two adjacent grade 

levels, rather than the single grade level used in the first accountability cycle. The 

panel also recommended that the district and the panel continue to study this issue. 

(A. C. Porter, personal communication, October 21, 1998). 

• Treatment of nontested students: A primary concern in this area was the potential of 

artificially inflating the gains in the number of students tested by deliberately 

limiting the number of students tested in the first year of the accountability cycle and 

then increasing the number in the second year. The panel did not find evidence that 

such gaming had occurred. In addition, it noted that even if such gaming had 

occurred, it would only work in the first year because all students tested in the 

UW 



second year would be included in the baseline for the second accountability cycle (A. 

C. Porter, personal communication, January 22, 1999). 

• Accommodations of disabled, special education and limited English proficiency 

students: The panel believed that the district’s set of accommodations for these 

students seemed reasonable. It further stated that if the guidelines were internalized 

throughout the district they would lead to excellent procedures. As noted previously, 

this area had been controversial, with the undercurrent seemingly that there were no 

guidelines in place and that potentially too many opportunities for exclusion of these 

students existed and could be manipulated to improve a school’s performance (A. C. 

Porter, personal communication, January 22, 1999). 

• “Gaming” concerns in general: A major question posed by the Board of Education 

to the panel was whether the accountability system was set up and being 

administered in a way that offered opportunities for serious manipulation by school 

personnel and/or students. The panel concluded that if there had been any serious 

gaming problems they were a thing of the past. Regardless, data at the system level 

showed that even in the first accountability cycle there was positive movement both 

in the number of students tested and the student achievement levels. The panel also 

observed that the opportunities for gaming in the future are few, if any (A. C. Porter, 

personal communication, April 19, 1999). 

• Communication about the program: The panel noted that the accountability program 

is not particularly complicated and is based on reasonable pieces of school 

information. The panel would like to know more about how and what information 

UW 



about the program is being communicated to the public (A. C. Porter, personal 


Among the issues yet to be reviewed by the panel include the reasonableness of 

the student performance target of 95% proficient by the end of 12 years. It characterized 

that target as “enormously challenging” and further questioned whether a straight-line 

performance projection was appropriate. The panel is scheduled to begin meeting again 

the fall of 1999 and will continue to consider issues relating to the accountability and 

assessment program. The consideration of accountability program goals and details by 

an entity with no connection to or vested interest in the district should help build support 

for the program by providing an objective review forum (A. C. Porter, personal 


First accountability cycle results 

The first two-year accountability cycle was completed at the end of the 1997-98 

school year. The initial progress indicated at the end of the first year of the two-year 

cycle generally was sustained and improved upon significantly. Schools received awards 

if, by June 1998, they exceeded two-year performance targets pre-set based on the 1995- 

96 school year. Of the 249 schools included in the accountability program, 145 received 

awards totaling $11,150,000, with up to half of the award amount available for 

expenditure in 1998-99. The amount each school received was determined based on 

$1500 per teacher and other professional staff and $500 per para-professional and noninstructional 

staff, pro-rated on the basis of full-time equivalency. As noted previously, 

the awards could not be used for staff bonuses but other than that restriction the 

individual school had considerable latitude on how to spend the money. According to the 

UW 



district there was considerable overlap between the schools earning awards under the 

state’s two accountability programs (one for improvement in state test results and one for 

improvement in attendance) and those earning awards under the district’s program. Of 

the 53 schools that earned achievement awards from the state, all but one made 

substantial progress in the district’s accountability system and 41 of them also earned 

district awards. 

Evaluation Summary 

The entire reform agenda, including the accountability and incentive program, 

were controversial from the time Hornbeck was hired. Nevertheless, there has been 

continuous progress to implement all elements and student achievement appears to be 

improving. In fall 1998 the Philadelphia school board with strong support from the 

current mayor voted to extend Hornbeck’s contract until August 2001 despite some 

protests from the union, the city council, and Democratic mayoral contenders. In contrast 

to those who were unhappy with his performance, Hornbeck’s supporters praised his 

efforts to focus public schools on the mission of improving student achievement, of 

which the school-based performance award program is one mechanism (Keller & Olson, 

1999). Thus, it is likely that Philadelphia’s program will continue at least for the duration 

of Hornbeck’s tenure as superintendent which will allow at least one more accountability 

cycle to be completed and the continuing progress of Philadelphia’s students towards the 

long-term student achievement goals evaluated. 

UW 



Conclusion 

Philadelphia’s school-based performance award program is part of a 

comprehensive reform package, thus the program’s separate and direct effect on student 

achievement is difficult to isolate and it is arguable whether it would be appropriate to 

even attempt to do so. Many of the enablers that contribute to the success of the award 

program are appropriately intended to improve student achievement regardless of 

whether the results are tied to a formal performance award program. Finally, 

Philadelphia’s program is relatively new, with only one complete award cycle having 

been completed. There has been improvement in the achievement of Philadelphia’s 

students, including large increases in the number of students being tested, but the extent 

to which it will be sustained obviously cannot be predicted. 

Nonetheless, several observations can be made that other jurisdictions that are 

considering a school-based performance award program may want to consider. First, 

Philadelphia’s accountability program is part of a comprehensive education reform 

program that includes changes to the district’s administrative and decision-making 

structure, as well as specific curricular and instructional changes. As such it is not the 

sole or even the primary, means by which the district intends to improve student 

achievement, which we believe is the appropriate role for such a program. The 

performance award program was created to complement and support the district’s overall 

reform efforts with their emphasis on assessment and accountability, rather than to be the 

driving force of the reform effort. 

UW 



Second, Philadelphia’s context appears to pose some unique challenges that could 

influence the success of its performance award program. As noted previously, David 

Hornbeck made his acceptance of the superintendent position contingent on being able to 

implement an accountability program. Thus, although the community as a whole, as well 

as the teachers and administrators were involved in the design process, the initial impetus 

for the program did not necessarily come from within the district and the community 

itself. In addition, there was a relatively short time frame in which the program was 

designed and implemented, with all but a few schools participating from the outset. In 

retrospect, a pilot year might have allowed more support for such a program to solidify 

among all key stakeholders as well as for some of the more controversial and complex 

program details to have a public airing prior to the full implementation. However, given 

the low level of student performance in the district, it could just as easily be argued that 

speedy implementation was needed. 

Third, although the cost of the rewards provided as part of the Performance 

Responsibility Program was not that high, the costs of the enablers related to the 

education reform program were significant. These costs must be considered in the 

context of a district with chronic and longstanding financial concerns which regularly 

created a situation where the district had to cut existing programs, to say nothing about 

embarking on new programs. The presence of external funds to support the broader 

accountability program was undoubtedly an important factor in the schools’ ability to 

succeed and the ability of the district to sustain that level of funding may influence the 

long-term success of the program. 

UW 



Fourth, it appears that the labor-relations climate between the district and the 

teachers’ union is not one that could be characterized as “cooperative.” The union 

leadership does not appear to be supportive of the overall reform approach and does not 

believe it will succeed. In such a situation, it will be difficult for teachers to have a 

strong sense that they will be able to meet the improvement goals, or that there is even a 

reasonable expectation that they can meet the goals. The union leadership is not in 

complete agreement with the path the district is taking to education reform, with the 

strongest area of disagreement seemingly the accountability program. However, the 

union and the district do share some elements, such as lower class size, importance of 

maintaining discipline in the schools, etc. Nonetheless, without the full support of the 

involved parties the current reform effort, including the school-based performance award 

program, faces many challenges. 

Finally, Philadelphia’s accountability program is relatively new, with only one 

complete two-year award cycle under its belt. Thus, regardless of the challenges it has 

faced or the level of improvement to student achievement that remains, it must be 

acknowledged that it is still early in this particular education reform effort. It is probably 

not reasonable to expect that a large urban school district with a long history of poor 

student achievement and ineffective reform efforts can turn around a generation or more 

of problems in just a few years. Rather, the district’s progress and the role of the schoolbased 

performance award program in this progress, should be evaluated over a longer 

period of time. 

UW 



References 

Bradley, Ann (1992). New arrangements: reforming Philadelphia’s high schools from 

within. EdWeek [On-line]. 18 November. Available: http://www.edweek.com. 

Bradley, Ann (1997). Teachers: the reluctant recruits. EdWeek [On-line]. 19 February. 

Available: http://www.edweek.com. 

Cohen, Deborah L. (1996). Pencil me in. EdWeek [On-line]. 31 January. Available: 

http://www.edweek.com. 

Consortium for Policy Research in Education (1998). A second-year evaluation report of 

Children Achieving: Philadelphia’s education reform; executive summary 1996- 

97. February. Phildelphia: author. 

Keller, Bess & Olson, Lynn (1999). Philadelphia extends Hornbeck’s contract for 2 

years. EdWeek [On-line]. 10 February. Available: http://www.edweek.com. 

Luhm, Theresa, Foley, Ellen, and Corcoran, Tom (1998). The accountability system: 

defining responsibility for student achievement. April. CPRE: authors: 

Manzo, Kathleen Kennedy (1996). Philadelphia plan links student achievement, teacher 

pay. EdWeek [On-line]. 30 October. Available: http://www.edweek.com 

Miller, Julie A. (1993). Philadelphia embraces whole-school approach wholeheartedly. 

EdWeek [On-line]. 4 August. Available: http://www.edweek.com. 

Odden, Allan, Heneman, Herbert, Wakelyn, David J., and Protsik, Jean (1996). Schoolbased 

Performance Award Designs: A Case Study. October. University of 

Wisconsin-Madison. 

Office of Standards, Equity, and Student Services (1998). School Support Process: 

Interim Report. 12 February. Philadelphia: School District of Philadelphia 

UW 



Partnership for Reform (1996). Professional Responsibility Issue. Straighttalk About 

Philadelphia School Reform. Philadelphia: author 

Philadelphia Schools (1999). Supports for reaching the target. [On-line]. 16 June.. 

Available: http://www.philsch.k12.pa.us. 

Reinhard, Beth (1997). Arbitrator rejects overhaul plan for 2 Philadelphia schools. 

EdWeek [On-line]. 6 August. Available: http://www.edweek.com. 

Schmidt, Peter (1994a). Philadelphia schools shortchange minorities, court rules. 

EdWeek [On-line]. 16 February. Available: http://www.edweek.com. 

Schmidt, Peter (1994b). Philadelphia leaders join call for overhaul of schools. EdWeek 

[On-line]. 28 September. Available: http://www.edweek.com. 

Schmidt, Peter (1994c). Hornbeck setting sights on a new course for Philadelphia. 

Edweek [On-line]. 23 November. Available: http://www.edweek.com. 

Schmidt, Peter (1995). Hornbeck to push ahead with Philadelphia reforms. EdWeek 

[On-line]. 15 February. Available: http://www.edweek.com 

Sommerfeld, Meg. (1995). $50 million Annenberg Grant expected for Philadelphia. 

EdWeek [On-line]. 1 February. Available: http://www.edweek.com. 

UW

Philadelphia SBPA - Consortium for Policy Research in Education ...

Create successful ePaper yourself

Delete template?

Save as template?