October 2007 | EPI book
Vouchers and Public School Performance
A Case Study of the Milwaukee Parental Choice Program
Martin Carnoy
Stanford University, School of Education
Frank Adamson
Stanford University, School of Education
Amita Chudgar
Michigan State University, School of Education
Thomas F. Luschei
Florida State University, College of Education
John F. Witte
University of Wisconsin, LaFollete School of Public Affairs
jump to: Table of contents | Executive summary | Introduction
Table of Contents
Acknowledgements
Executive Summary
Introduction
Chapter 1 The Milwaukee Parental Choice Program (MPCP)
Chapter 2 What do we know about research on competition effects?
Chapter 3 Revisiting the competition effect in Milwaukee using inter-cohort test scores
Chapter 4 Estimating competition effects within Milwaukee using intra-cohort test score gains
Chapter 5 Discussion
Conclusion
Appendix
Endnotes
Bibliography
Acknowledgements
The authors would like to thank Deb Lindsey, the Director of the Division of Assessment and Accountability of the Milwaukee Public School District and Jim Bloom, Senior School Data Support Specialist in that same division, who helped guide us through the data. Special thanks to John Gardner, a member of MPS?s school board when this research was initiated, and Phil Olsen at the Wisconsin Department of Public Instruction, Office of Educational Accountability. We are indebted to Anneliese Dickman of Milwaukee?s Public Policy Forum, and Emily Van Dunk, formerly of the Forum and now at the Milwaukee County Budget Office, who know more about Milwaukee?s voucher program than almost anyone and were incredibly generous with their advice and data. Joydeep Roy of the Economic Policy Institute devoted a lot of time going through this study and making many insightful suggestions, as did a number of anonymous reviewers. Of course, none of these helpful people should in any way be held responsible for our analysis.
Executive Summary
Since the 1980s, as part of a broader market-oriented agenda, school choice and vouchers have become an increasingly important part of America?s educational reform policy debate. At stake are the possibility of greatly improving American education and fundamental issues of who has responsibility for the quality of educational provision and of the governance of the educational system.
The debate is rooted in ideological differences between market proponents, who attach greater importance to individual choice, and supporters of a publicly run educational system, who place greater importance on equity, commonality, and public accountability. Yet, the debate is also conducted in the empirical arena, where data are used to test the claims of proponents on both sides.
This study is an attempt to contribute to that empirical debate by revisiting the issue of whether competition improves student achievement in traditional public schools, a key argument for educational markets. Our study uses data from an extensive choice reform in Milwaukee?s Public School District (MPS), a district with the typical educational problems of an American urban center, but unusual in that it has had a voucher plan, the Milwaukee Parental Choice Program (MPCP), targeted at low-income students since 1990.
After the experimental phase of the voucher plan ended in 1995, Wisconsin?s legislature approved the expansion of the plan, capping it at 15% of public school enrollment, or about 15,000 students. In 1998, the Wisconsin Supreme Court upheld the legislative action allowing vouchers to be used in religious schools. This immediately led to rapid growth of Milwaukee voucher takers and increased the number of schools accepting vouchers. By the 2002-03 school year, more than 11,000 students were using vouchers to attend private schools, and more than 100 schools were accepting voucher students. Twenty-six were new schools that had entered the market specifically to take advantage of voucher students. In 2006, after a compromise between the legislature and the governor, the cap was raised to 22,500 voucher students. At the same time, enrollment in Milwaukee public schools has declined by more than 7%.
A full eight years after the school district expanded the voucher program, it is still not possible to measure whether voucher students in Milwaukee perform better or worse than their counterparts who remain in public schools. The Wisconsin legislature made no provision to evaluate the expanded voucher plan. However, one of the main arguments for increasing educational choice is that choice induces competition among schools vying for students, and this improves student performance in all schools, private and public. The competitive effects of choice should be especially notable in low-income urban areas that are characterized by low achievement and inefficient, bureaucratic public school systems. So even without information on student performance in private schools, we can test whether student performance is improving in Milwaukee?s neighborhood public schools and whether, other things being equal, student performance increases more in those schools most affected by student flight to private schools and other educational alternatives, such as charter and magnet schools. Our focus was the elementary level, where most voucher use occurred.
We used two methods to test for improved student performance resulting from increased competition. The first method, already employed by Caroline Hoxby and, in a modified form, by Rajashri Chakrabarti, compares the fourth-grade test scores of students in three categories of Milwaukee public schools in the 1996-2004 period with each other and with a control group of schools outside of Milwaukee. This analysis tests whether the introduction of large-scale vouchers in 1998 had a significant impact on the performance of Milwaukee?s public school students. It confirms the earlier results showing a large improvement in Milwaukee in the two years following the 1998 expansion of the voucher plan to religious schools. However, we also confirm that little positive improvement took place in later years even as enrollment declined in Milwaukee?s neighborhood schools and the number of voucher applications continued to increase. This raises questions about whether traditional notions of competition among schools explain these increased scores in the two years immediately after the voucher plan was expanded.
The second method estimates the effects of various measures of competition (such as the number of voucher schools and available voucher places near a public school) on gain scores of public school students
in fourth and fifth grades. Using this method, we find essentially no evidence that students in those traditional public schools in Milwaukee facing more competition achieve higher test score gains. Student test score gains are generally not significantly related to various indicators of direct competition associated with families? increased potential choice of schools or their actual exercise of choice for their school-aged children. Neither the number of private schools within a mile of a public school, nor the relative number of voucher places nearby, nor the relative number of voucher applications from the public school has a positive effect on the mathematics or language arts gains that students make in either the fourth or fifth grade.
Thus, our analysis finds little or no indication that pupils in those Milwaukee public schools that have more school choice possibilities nearby made significantly greater year-to-year gains in primary school tests than pupils in other Milwaukee public schools. The effect of other variables, such as staying in the same public school from year to year, seem to have a more consistent (albeit small) positive impact on student performance, particularly on language arts test score gains.
Nor did we find evidence that students realize higher test score gains in schools suffering greater recent-past losses in enrollment. Enrollment losses are the purported mechanism pushing public school personnel to work harder to keep students from leaving. This result is particularly interesting because the Milwaukee Public School District has long threatened and now has carried out school closings because of declining enrollment and continued poor test score results in a number of traditional public schools.
Our failure to find observable competition effects could not be attributed to the overlay of cream-skimming on competition. The cream-skimming offset would occur if ?better? students were leaving public neighborhood schools for private, charter, and magnet competitors, negatively influencing test scores in neighborhood schools and counteracting any positive effect of competition. The average proportion of students eligible for free and reduced-price lunch and the proportion of minority students in Milwaukee?s lower income schools changed little in the years after 1998. There appears to be some cream-skimming of students into magnet schools, since the average proportion of disadvantaged minority pupils in magnet schools is much higher than in the low-middle-income Milwaukee schools, but the proportion eligible for free or reduced-price lunch is lower.
Our results are therefore mixed. Overall, Milwaukee public schools made a one-time gain versus other Wisconsin schools with somewhat comparable ethnic/social class composition. Yet, students in Milwaukee schools facing more competition from private voucher schools made no greater gains. We conclude from these somewhat contradictory results that the observed improvement in public school test scores associated with the implementation of a greatly expanded voucher plan in 1998 was probably a response to the threat of increased competition. Yet, no one has been able to explain what schools did to raise student performance. We speculate that school personnel in Milwaukee, confronting the possibility of greater competition, began paying much greater attention to the state test and focusing on improving students? performance on the test. Once having done that, however, it was much more difficult to make further improvements despite increased competition from voucher and charter schools.
We also conclude that for the choice argument to be convincing, advocates need to show more consistent and sustained improvements in student learning and should be able to explain at the operational level how choice induces schools to improve student performance. A more concrete explanation of this one-time increase in test scores is especially needed because the Milwaukee case is one of the few where a significant positive competition effect on student achievement has been found. This study shows that the traditional competition indicators do not seem to provide this explanation. If choice can, at best, produce a one-time improvement, particularly an improvement due to schools taking standardized tests more seriously, this is an effect that can probably be produced by other (possibly lower cost) policies and incentives. Thus, we need a much better explanation of what occurs in both public and private/charter schools when such choice plans are introduced.
Introduction
The quality of American public schooling has long been a major policy issue. Since the 1980s, as part of a broader market-oriented agenda, school choice and vouchers have become an increasingly important part of that educational reform policy debate. At stake are the possibility of greatly improving American education and fundamental issues of responsibility for the quality of educational provision and of the governance of the educational system.
Choice advocates contend that parents? right to choose their child?s school increases their sense of empowerment and well-being, and that increased competition for students among schools, public and private, results in all schools improving their educational delivery and raising student achievement. These are noble goals. Nevertheless, vouchers for private education conflict with widely held views of education as a public service, charged with creating an integrated society and in giving all children, regardless of social class or ethnic group, common educational experiences. In the case of vouchers that can be used in private religious schools, such plans also conflict with deeply held views about the separation of Church and State. Further, opponents of educational markets argue that the main educational outcome of vouchers for private schools is to exacerbate sorting of students by social class, race, and level of parents? motivation.
The debate is rooted in ideological differences between market proponents, who attach greater importance to individual choice, and supporters of a publicly run educational system, who place greater importance on equity, commonality, and public accountability (Levin 2002). Yet, the debate is also conducted in the empirical arena, where data are used to test the claims of proponents on both sides. This report is an attempt to contribute to that empirical debate by revisiting the issue of whether competition improves student achievement in traditional public schools, a key argument for educational markets. Our study uses data from an extensive choice reform in Milwaukee?s Public School District (MPS), a district with the typical educational problems of an American urban center, but unusual in that it has had a voucher plan since 1990?the Milwaukee Parental Choice Program (MPCP)?that expanded to include private religious schools in 1998.
There have been a number of empirical studies of competition effects of voucher and charter school programs on student performance in U.S. public schools (see Belfield and Levin 2002 for a review; also Chakrabarti 2005), and in several other countries with national voucher or choice plans, such as Chile, New Zealand, and Sweden.1 Although most such studies suggest, at best, a positive but small effect of private and charter school competition on student achievement, some research in the United States has claimed a much larger impact (Hoxby 1994; Hoxby 2003).
In this report, we make a somewhat different argument concerning vouchers and competition than has been made in the past. We suggest that a voucher program on a large scale, such as introduced in Milwaukee in the late 1990s, may have had a positive effect on student achievement in Milwaukee?s traditional public elementary schools (as argued by Hoxby (2
003) and Chakrabarti (2005)), but that this effect seems to have been a one-time response of all public schools to a change in contextual conditions rather than a continuous and differentiated improvement based on the degree of competition in the Milwaukee school market (see Van Dunk and Dickman 2003).
Our study replicates school-based research by Hoxby (2003) and Chakrabarti (2005) that shows Milwaukee schools raising fourth-grade achievement relative to a comparison sample of schools outside Milwaukee after the expansion of the voucher program in 1998-99. But we go beyond their broad conclusions to test the proposition that direct competition from voucher schools was the main driver of increases in test scores. We study individual student achievement gains in neighborhood public schools facing varying degrees of competition within the MPS.2 Intra-cohort gain analysis allows us to: 1.) estimate competition effects on school ?value added? for several cohorts of Milwaukee?s primary school students; 2.) test alternative measures of competition; and 3.) test the hypothesis that students in those public schools facing greater competition from voucher schools had higher achievement gains.
We present the study in five parts: in the first section we describe the evolution of the Milwaukee Parental Choice Plan; in the second section we review the literature on vouchers and other school choice programs, including the empirical research on the Milwaukee voucher plan; in the third section we present empirical estimates of the possible impact of competition from voucher schools on student inter-cohort achievement changes in the MPS compared to similar Wisconsin schools outside Milwaukee; in the fourth section we estimate competition effects on intra-cohort achievement gains in Milwaukee?s public schools; and in the final section we draw policy conclusions from our empirical results.
For a printer-friendly version of this report, click here.
