March 2005 | EPI/TCP Book

Materials for news media: News release | Fact sheet

Jump to: Table of Contents | Introduction and Summary

PDF: Download table of contents and introduction

Audio: Listen to a March 30 national news conference call with the authors

---

Table of contents

Introduction and summary

Chapter 1. The reaction to the AFT’s report on charter school scores

Chapter 2. Can the ‘dust-up’ lead to a new consensus education research and policy?

Chapter 3. Problems with the critiques of the NAEP report by charter school supporters

• No schools, charter or regular public, should be evaluated using point-in-time score levels
• Shortcomings of ‘No Child Left Behind’ performance measures
• How charter school zealots helped create the NAEP charter sample

Chapter 4. Are charter school students more disadvantaged than regular public school students, and does this explain charter schools’ unexpectedly low NAEP scores?

• How selection bias complicates the evaluation of charter schools
• The NCES analysis of the charter school NAEP sample
• Comparative studies of charter and regular public school demographics in individual states
  • Arizona
  • California
  • Colorado
  • Connecticut
  • District of Columbia
  • Florida
  • Illinois
  • Massachusetts
  • Michigan
  • North Carolina
  • Pennsylvania
  • Texas
  • Wisconsin
• Summary of demographic data from state-level studies
• Are charter demographics over-stated because of a failure to offer the lunch program?
• Other evidence of the relative advantage or disadvantage of charter school students
• The KIPP case

Chapter 5. What we know about relative charter and regular public school student achievement

• The NCES’s own analysis of charter school NAEP scores
• Evidence from state-level studies regarding the achievement of charter school students
• On the policy views of researchers
• Description of state-level studies of charter school achievement
  • Arizona
  • California
  • Colorado
  • Connecticut
  • District of Columbia
  • Florida
  • Illinois
  • Michigan
  • North Carolina
  • Pennsylvania
  • Wisconsin
• Age-of-school influences on charter school student achievement
• The competition effect
• Segregation
• The Hoxby studies
• Summary of evidence on charter and regular public school achievement

Chapter 6. The philosophy of charter schools

• Are standardized test scores less important for charter schools, because charter schools will be shut down in any case if they don’t perform well?
• Are bureaucratic regulations and union rules the cause of low student achievement?

Chapter 7. Conclusion

Appendix A. Using different standards for evaluating charter and regular public schools

• Robert Lerner and the National Center for Education Statistics (NCES)
• Former U.S. Secretary of Education Roderick Paige and Deputy Secretary Nina Rees
• Jeanne Allen and the Center for Education Reform
• Rev. Floyd Flake
• Jay P. Greene; Kaleem Caire, and the Black Alliance for Educational Options
• Howard Fuller
• “Shoot the Messenger” reactions of charter school zealots

Appendix B. Alternative presentations of NAEP charter school demographic data

Endnotes

Bibliography

Acknowledgments

---

Introduction and summary

In the summer of 2004, a noisy controversy erupted over whether charter schools are more effective than regular public schools. The dust-up began when the American Federation of Teachers (AFT), known to support greater restrictions on charter schools, published test results from the federal government’s National Assessment of Educational Progress (NAEP). The data showed that average achievement is higher in regular public schools than in charter schools, both for students overall and for low-income students. The AFT’s report also noted that for black students, a group that many charter schools are specifically designed to serve, average achievement is no better in charter schools than in regular public schools.

The New York Times publicized this finding on its front-page. Immediately, the most zealous advocates of charter schools responded with a storm of criticism, including a full-page advertisement that they placed in the Times itself. These advocates did not deny that average test scores were higher in regular public schools than in charter schools. Rather, they claimed that the AFT report was methodologically flawed because it did not attempt to compare subsets of students who were truly similar in background and prior achievement. In particular, these advocates claimed that students attending charter schools are more disadvantaged than students attending regular public schools, and especially that black students in charter schools are more disadvantaged than black students in other public schools. If this were the case, then charter school students could have been expected to score lower than regular public school students even if charter schools were somewhat more effective. These charter school advocates claimed that charter schools are actually, on average, more effective, not less so, than regular public schools.

The controversy revealed an intense level of disagreement about the wisdom of policies to encourage charter schools. That the claims are so contradictory indicates how little consensus there is about:

• whether charter schools really are more effective than public schools;
• whether charter schools really do serve comparatively disadvantaged students;
• what kind of evidence is required to make judgments about the impact of charter schools on student learning; and
• what role charter schools can be expected to play in strategies to improve regular public schools.

Our aim in this book is to synthesize as comprehensively as possible all available evidence on the average effectiveness of charter schools relative to regular public schools. We conclude in Chapter 5 that, based on 19 studies, conducted in 11 states and the District of Columbia, there is no evidence that, on average, charter schools out-perform regular public schools. In fact, there is evidence that the average impact of charter schools is negative. This evidence of a negative effect comes particularly from those studies that use the strongest methodologies to discover causal effects, although the evidence of a negative effect is somewhat localized to specific states.

In pursuing this aim, it was essential that we first set standards for methodological quality. Children are not assigned at random to attend charter schools, so some attempt must be made to identify subsets of children attending charter and regular public schools who are as similar as possible in their prior characteristics, including academic achievement. Fairly clear standards for this kind of work have emerged in social science, and we describe these in Chapter 4. We also ask whether studies adhering more or less well to these standards produce similar or different results. With few exceptions, the general outlines of the story are similar: charter schools are no more effective than regular public schools on average and may, in fact, be less effective.

But do charter schools serve more disadvantaged students than those served by regular public schools? The answer to this question is somewhat complex. In many states, the fraction of charter school students who are black is somewhat higher than the fraction of regular public school students who are black. However, the black students attending charter schools in these states tend to be disproportionately better off socioeconomically than black students attending regular public schools.

The best studies of charter school effectiveness simultaneously remove the effects not only of race and socioeconomic factors but also of prior achievement and even a host of other, often unobservable differences (such as the educational levels of parents) between children attending the two types of schools. In these highest-quality studies in particular, the average effects of attending a charter school are null or negative. In Chapter 4 we compare, in detail, the kinds of students served by charter and regular public schools nationally and in studies done in 12 states and the District of Columbia.

Beyond synthesizing current evidence, our inquiry also explores a few of the policy implications of our findings about relative average charter school performance, and this requires us to re-evaluate some of the common rationales for supporting charter schools.

One argument is that charter schools liberate educators from bureaucratic regulations and union contracts that stifle creative educational improvements. We speculate that, while deregulation helps some educators devise good schools, it also enables others to devise bad and even corruptly managed schools. For example, while some charter schools can use freedom from normal certification requirements to hire unusually talented and dedicated teachers, other charter schools use this freedom to hire teachers who may be less qualified than teachers in regular public schools. We conclude that the evidence about average charter school performance is consistent with this wide range in the effects of deregulation. That charter schools are not substantially more effective, on average, than other public schools calls into question the view that bureaucracy and union contracts are major impediments to school improvement. It seems, based on the evidence, that deregulation and deunionization do not yield any bonanzas of learning, on average. If bonanzas are realized in some places, they are apparently offset by catastrophes in others.

A second argument is that charter schools are more accountable than regular public schools for their outcomes. This theory takes two forms. Some advocates of charter schools argue that, unlike regular public schools, charter schools will be closed by public authorities if their academic performance is inadequate. We show that evidence about actual charter school accountability processes does not support this assertion. Other advocates of charter schools argue that parental choice (the freedom of parents to choose better charter schools and to remove their children from low-performing ones) provides strong accountability. We suggest that to the extent charter schools rely on this mechanism of accountability, it should not be surprising that their average academic performance does not surpass that of regular public schools, for two reasons. First, parents may choose charter schools for other than academic reasons. Second, given how complex it is to assess academic performance (leading even experts to dispute the effectiveness of charter schools so vigorously), it is not surprising that parents would not always be able to discern a charter school that was more academically effective.

A third argument is that charter schools foster experimentation to see if novel educational approaches can produce good results. We do not deny that this is an important rationale for charter schools. But we note that, in any field, a spirit of experimentation is likely to produce many failures before (if ever) identifying successes. Researchers devise strategies for widespread experimentation to discover effective practices, not to produce average gains in outcomes — those may come later, when the policies identified as effective are implemented on a large scale. Charter schools might be successful in generating innovations that should be imitated, even if average charter school test scores are at or below those of regular public schools. This implies different criteria for evaluating the merits of charter schools than the claim — that average charter school test scores surely must be superior — advanced by those zealous charter school advocates who were most vociferous in attacking the AFT report.

Finally, a fourth argument is that competition from charter schools improves outcomes in regular public schools because educators in regular public schools are motivated to be more effective in order to avoid losing students to charter schools. This argument for charter schools, even if valid, would not require average charter school performance to be superior to that of regular public schools. Nonetheless, we find no evidence to support the claim of a positive competition effect of charter schools, although research in this area is not yet extensive.

A potentially encouraging result from the charter school dust-up of 2004 is that the policy community may now be better able to reach consensus on what standards are appropriate for judging evidence of educational effectiveness, not only of charter schools but of regular public schools in the nation, in states, and in districts. In particular, we note that many charter school advocates criticized the AFT report for failing to (or being unable to, given data limitations) properly adjust for student background characteristics and prior test scores when evaluating charter schools. We agree with this critique. B
ut we observe that some charter school advocates who were most vigorous in putting forward this critique have themselves been among the most outspoken opponents of making such adjustments when evaluating regular public schools and when comparing the educational effectiveness of states, schools, districts, and teachers. The dramatic change in the methodological standards of this group (detailed in Appendix A), revealed in responses to the AFT report, can increase the prospects for a more objective and fair review of public policy issues in education than we have experienced in the past. But this movement toward high methodological standards will succeed only if policy researchers apply them consistently, instead of adopting tough methodological standards only when convenient to support ideological positions. In particular, we urge that the standards set forth in the New York Times advertisement, placed by zealous charter school advocates in opposition to the AFT report (and reproduced in Chapter 1), be applied not only to charter school evaluation but to all school accountability policies at the federal and state levels, including those employed by the No Child Left Behind legislation.

In this book, we use two terms whose frequent repetition may be irritating to some readers. We apologize in advance for this irritation, but find it necessary nonetheless to use the terms. First, we often refer to the group of charter school advocates who have been most outspoken in their insistence that, regardless of good data, charter school performance must be superior to that of regular public schools. As one of the principal spokespersons for this group, Chester E. Finn Jr., described his and his colleagues’ reaction to the AFT report: “Charter supporters rushed to the barricades after last week’s AFT-coordinated blast in the New York Times.” For want of a better term, we call this group of barricade-rushers “charter school zealots.” We intend no disrespect to this group, and use “zealot” as Webster’s dictionary defines it: “someone who acts for a cause with excessive zeal (persistent, fervent devotion).” It is necessary to use a term for members of this group to distinguish them from many other supporters of charter schools whose devotion to charter schools is not excessive and who did not rush to the barricades following the release of the AFT’s report. Supporters of charter schools may have many reasons for their support, and these reasons do not require an a prioribelief that average charter school academic performance must be superior to that of regular public schools. These reasons might include beliefs that charter schools are a way to keep parents committed to public education by offering them more choice, a way to work around some or all of the administrative and union constraints that characterize many regular public schools, a way to keep some children in school who might otherwise be “lost,” or a way to involve parents more actively in decisions about their children’s education.

It is not the purpose of this book to evaluate in any depth the merits of these reasons for supporting charter schools or to propose policies regarding charter schools. We do, however, observe that any policy that permits parents to choose schools other than their neighborhood schools can involve costs as well as benefits, and that the difficult trade-offs involved in school choice have been too little discussed. For example, we note that if more academically able children exit their regular public schools in favor of charter schools (or, in the regular public sector, in favor of magnet or exam schools), this makes the task of neighborhood public schools more difficult because the students who remain will, on average, be less academically able and will lose the benefit of interaction with their more academically able peers. We also note that some evidence indicates that the existence of charter schools increases racial segregation in public schooling. These are not reasons to reject charter schooling, but policy deliberations must weigh these against the benefits claimed by charter school supporters.

There are also zealots who oppose charter schools. In this book, we aim to be fair and accurate, but we do not attempt to achieve an artificial “balance” by analyzing the zealotry of charter school opponents as well. Charter school zealots, for example, accuse the AFT of opposing charter schools at least partly because they threaten the union’s institutional interests. In examining the accuracy of the data analysis of NAEP charter school scores presented by the AFT, we do not find a need to examine the interests that may have motivated the AFT to perform this accurate analysis. Militant and unreflective charter school opposition, by the AFT or other influential policy makers, was not prominent in the dustup following the AFT’s report, and it is this controversy, and only this controversy, whose implications this book examines.

The other term we use repetitively is “on average” to describe data about charter and regular public schools. Without such a term, many readers may still appreciate that when data comparisons of charter and regular public schools are made, only averages are being described — there can be wide variation of achievement within a particular school (whether it is a charter or a regular public school), and there can be wide variation in the average achievement of schools that are charter schools and of schools that are regular public schools. But some readers may benefit from a reminder that a conclusion that charter school performance lags behind that of regular public school performance is not inconsistent with an observation that many charter schools may be far superior to typical regular public schools (and some may be greatly inferior). Or, typical charter schools may be superior to many regular public schools. Unfortunately, good data on school performance are so limited that we have almost no understanding of the variance of mean charter school academic achievement or of the variance of mean regular public school achievement. NAEP could not report such data, because NAEP reports test scores only of students, not of individual schools. And the state studies we examine, although they collect data on school mean performance levels, do not report standard deviations of these school means of performance, a statistic that would be needed to understand the extent to which average performance in charter schools is typical for charter schools generally. Because this is such a critical point, we keep it before the reader by frequently inserting the words “on average” in our discussion.

The co-authors of this book are not opponents, zealous or otherwise, of charter schools; among ourselves, we have a variety of ways in which we balance the costs and benefits of charter schools. The message of this book is not that charter schools have “failed,” but only that there is no reason to be surprised that their average performance apparently falls below that of regular public schools. We believe that a more reasoned discussion of education policy can proceed from this recognition.

---

About the authors

Martin Carnoy is a research associate of the Economic Policy Institute and professor of education and economics at Stanford University.

Rebecca Jacobsen is a research assistant of the Economic Policy Institute, a graduate student in politics and education at Teachers College, Columbia University, and formerly a teacher in New York City and Connecticut public schools.

Lawrence Mishel is president of the Economic Policy Institute and director of its education research and policy program.

Richard Rothstein is a research associate of the Economic Policy Institute and a visiting professor at Teachers College, Columbia University.

---

About EPI

The Economic Policy Institute was founded in 1986 to widen the debate about policies to achieve healthy economic growth, prosperity, and opportunity.

In the United States today, inequality in wealth, wages, and income remains historically high. Expanding global competition, changes in the nature of work, and rapid technological advances are altering economic reality. Yet many of our policies, attitudes, and institutions are based on assumptions that no longer reflect real world conditions.

With the support of leaders from labor, business, and the foundation world, the Institute has sponsored research and public discussion of a wide variety of topics: trade and fiscal policies; trends in wages, incomes, and prices; education; the causes of the productivity slowdown; labor market problems; rural and urban policies; inflation; state-level economic development strategies; comparative international economic performance; and studies of the overall health of the U.S. manufacturing sector and of specific key industries.

The Institute works with a growing network of innovative economists and other social science researchers in universities and research centers in the U.S. and abroad who are willing to go beyond the conventional wisdom in considering strategies for public policy.

About Teachers College Press

For over a century, Teachers College Press (the university press of Teachers College, Columbia University) has been committed to addressing the ideas that matter most to educators. Teachers College Press provides authoritative and practical resources for all participants in the education process, including teachers, teacher educators, researchers, academics, administrators, school board members, policymakers, parents, and students. Teachers College Press publishes many cutting-edge, critically acclaimed books, videos, and CD-ROMs in the subject areas of educational leadership and policy, language and literacy, early childhood education, math and science, social studies, teacher education, school reform, multicultural education, urban education, sociology, and much more.

Copyright © 2005. Published simultaneously by the Economic Policy Institute and Teachers College Press.