NBER Reporter 2014 Number 1: Research Summary
School Assignment and School Effectiveness
Parag Pathak *
A growing number of U.S. households have the opportunity to send their children to public schools outside of traditional neighborhood boundaries. Over the last decade there has been a proliferation of research on the design of centralized choice systems intended to make it easier for children to exercise choice. Millions of students have been assigned to schools using mechanisms either directly or indirectly inspired by academic work.
In recent research with several co-authors, I explore the equity, efficiency, and incentive properties of these choice systems. Aside from these properties, centralized assignment generates valuable data and quasi-experimental variation that can be used for evaluation of various educational practices and policies. I have worked with several researchers to exploit this variation to study productivity differences between schools and school models.
Immediate Acceptance
One of the most common school assignment systems is based on the concept of immediate acceptance: when applicants apply to a school, they are - offered a seat immediately if they qualify. A mechanism based on this principle was in place in Boston until 2005, and hence it is commonly known as the Boston mechanism. 1 A large number of Local Education Authorities in England also employed this mechanism, - called First Preference First.
One issue with this mechanism is that applicants do not have the incentive to rank their desired schools truthfully. That is, ranking a competitive school first may harm a student's chances at lower-ranked schools, creating strategic pressure on the applicant. Should an applicant take a risk at the school she really wants, or instead rank a safe choice first? In work with Tayfun Sönmez, I show that if families do not understand these incentives and rank their choices truthfully, then sophisticated families who understand the rules of the game benefit at the expense of the unsophisticated. 2
The poor incentive properties of immediate acceptance systems led authorities in Chicago to abandon their allocation scheme for the city's elite selective high schools in 2009. Officials in the Chicago Public Schools (CPS) observed that students with higher test scores were denied admission to their second-choice school, even though they had higher scores than students who ranked the school first. After eliciting preferences from more than 14,000 participants, CPS announced a new mechanism and asked participants to re-rank their choices. The new mechanism is a serial dictatorship where the highest-scoring student is assigned to her top choice, the next highest scoring student is assigned to her top choice among remaining schools, and so on. What is particularly surprising about this switch is that the new mechanism also did not have straightforward incentives because it limited the number of choices students could rank. Students could only rank four out of nine possible choices, necessitating strategic calculations on which choices to list and which ones to drop. In the subsequent year, they switched to a system with the same underlying algorithm, but allowed students to rank six schools.
A few years earlier, by an Act of Parliament, authorities in England outlawed First Preference First arrangements citing concerns - that the procedure is unfair to unsophisticated participants. Following this legal ruling, many districts adopted variants of the deferred acceptance algorithm, known in England as Equal Preferences. 3 Using this procedure, first formally studied by David Gale and Lloyd Shapley in 1962, applicants start by applying to their - first choice. Schools tentatively accept their preferred applicants up to capacity and reject the rest. Any rejected student applies to his next most preferred choice, and schools update their set of provisional acceptances by comparing these new proposals to students tentatively held over from the previous round. The algorithm terminates when there are no new proposals from rejected students.
The key idea is that assignments are deferred until there are no new proposals, and only then are they finalized. Unlike the First Preference First system, a student ranking a school second can displace one who ranks it first, if the school prefers that student. The reason it is called Equal Preferences is that when schools receive proposals, they do not discriminate among applicants based on where they were ranked on the applicant's preference form. As in the Chicago case, the Local Education Authorities that adopted Equal Preferences often limited the number of choices students could rank. Table 1 describes some of these transitions. 4
Table 1: School Admission Reforms
| School District | Reform Year | Old Rule | New Rule | More or Less Manipulable? |
| Boston Public Schools | 2009 | Boston | GS | Less |
| Chicago Selective Public HS | 2009 | Boston (list 4 choices) | SD (list 4 choices) | Less |
| Chicago Selective Public HS | 2010 | SD (list 4 choices) | SD (list 6 choices) | Less |
| Denver Public Schools | 2012 | Boston (list 2 choices) | GS (list 5 choices) | Less |
| Seattle Public Schools | 1999 | Boston | GS | Less |
| Seattle Public Schools | 2009 | GS | Boston | More |
| England - Newcastle | 2005 | Boston (list 3 choices) | GS (list 3 choices) | Less |
| England - Manchester | 2007 | FPF (list 3 choices) | GS (list 3 choices) | Less |
| England - Surrey | 2010 | GS (list 3 choices) | GS (list 6 choices) | Less |
Note: Boston refers to the Boston mechanism, FPF refers to First Preference First mechanisms, GS refers to the student-proposing deferred acceptance algorithm of Gale and Shapley, and SD refers to a serial dictatorship.
Sönmez and I develop a way to rank systems based on their propensity toward manipulation.5 Our approach makes it possible to evaluate whether the new systems are less manipulable than their predecessors. While our criterion is non-consequentialist, it allows for relative comparisons of two systems without ideal incentive properties. As shown in Table 1 it also has important positive content where, with the exception of Seattle in 2009, every example involves the adoption of a less manipulable system according to our measure.
Design of School Lotteries
An important issue in student assignment systems involves resolving situations where two applicants have identical claims for school seats, but there is only one seat left. This can happen, for instance, when two students obtain the same priority at a school because they reside in the school's walk zone, and there are more walk-zone applicants than seats. One might suspect that using separate lotteries at each school would be more fair than a single lottery because under a single lottery, if an applicant has a better lottery number than another applicant, that remains true at each school. However, together with Atila Abdulkadiroğlu and Alvin Roth, I show that a single lottery draw across all schools has better properties than school-specific lottery draws when using deferred acceptance.6 In the case of New York City where there are 90,000 applicants each year, more than 2,000 additional applicants obtain their first choice with a single lottery draw compared to school-specific draws. 7
Another popular mechanism is based on Gale's top trading cycles (TTC) algorithm. Roughly speaking, this procedure endows students with schools and allows them to trade with one another in an ordered market where trades among top choices occur before trades among lower choices. Suppose Ann wants school 1 as her top choice but has the highest priority at school 2, while Bob wants school 2 as his top choice but has the highest priority at school 1. In the TTC algorithm, Ann and Bob would trade their assignments. In 2012, the OneApp assignment system used in the Recovery School District in New Orleans employed a mechanism based on TTC. 8 In general, there is no preferred way to conduct lotteries for TTC. Together with Jay Sethuraman, I show that in the special case where schools do not have priorities, the allocations produced with a single lottery draw and with school-specific draws are identical.9
Boston's Choice Plan
Much of the initial work on student assignment was motivated by Boston's iconic school choice system, and it continues to inspire new scientific developments. In Boston and elsewhere, students wish to attend schools close to their home, especially at elementary school entry points. Districts recognize this by prioritizing applicants in the school's walk zone, a geographic area surrounding the school. On the other hand, such policies can increase segregation across schools as students who live near highly desired schools fill up the seats and prevent those from outside the neighborhood from having an opportunity to attend.
To ensure that out-of-neighborhood applicants - have an opportunity to attend a particular school, many choice systems follow Boston's in having a slot-specific priority structure. In Boston, for half of the school seats, applicants with walk-zone priority are ordered ahead of those who do not have walk-zone priority. For the other half, students from the walk zone are treated in the same way as students from outside the zone. This 50-50 split represents a compromise between those in favor of neighborhood schools and those favoring more choice.
When a student is eligible to attend a school both because of walk-zone priority and because of the district-wide assignment rule, the assignment mechanism must deal with another type of indifference. Since students care only about their school assignment, they are indifferent about whether they consume a walk-zone or a non-walk-zone slot. The mechanism's precedence order specifies the order in which slots are depleted. Together with Umut Dur, Scott Kominers, and Sönmez, I - show that student precedence has dramatic consequences for achieving distributional objectives. 10 In Boston, for instance, the precedence rule entirely undermined the intended effect of the 50-50 policy and the outcome was nearly identical to that without walk-zone priority at all. The reason is that applicants first depleted walk-zone slots before non-walk-zone slots. A walk-zone applicant who did not obtain a walk-zone slot competes with the general pool of applicants for non-walk-zone slots, but only after this applicant has been rejected from the walk-zone pool. This rejection induces a form of adverse selection - the applicant is rejected so he must have an unusually bad lottery number - that renders rejected walk-zone applicants not competitive for non-walk-zone slots. As a result, almost no students from the walk zone are assigned to the non-walk-zone slots, undermining the 50-50 compromise.
We develop a framework to study these features of slot-specific priorities and identify counterfactual policies that more faithfully implement policy goals. As a result of our work, Boston - substantially changed its walk-zone policy in 2014.
Boston has also completely redesigned how it determines the set of options students are allowed to rank on their choice menu. Until 2014, residents were restricted to applying to schools in one of three zones of the city and a handful of citywide schools. In 2014, the city adopted a zone-free plan where choice menus are customized based on an applicant's address. The choice menus are designed to ensure that each student is able to apply to - enough of the closest highly rated schools. Peng Shi and I use historical choices expressed in Boston to estimate models of school demand. We use these models to extrapolate the choices applicants would make under these new choice menus. Our results were discussed by school officials and played a significant role in the adoption of the new plan. We plan to update these predictions in a two-part project that will evaluate the performance of structural models of demand forecasting. Because our predictions were made in advance of the policy change, there is no scope for post-analysis bias. 11 We intend to revisit our predictions after applicants have expressed new choices in the spring of 2014, and to use the new data to assess the strengths and weaknesses of counterfactual prediction using discrete choice models of school demand.
Measurement of School Effects
Much of the excitement about school assignment mechanisms comes from the potential to engineer practical solutions that might improve welfare. In my view, an equally important role of common enrollment systems is in producing valuable data that can be used to evaluate the impact of various educational initiatives.
A longstanding question in education has been about the effects of attending charter schools, which are publicly funded schools with enhanced autonomy. When a charter school is over-subscribed, in many jurisdictions students are admitted via lottery. Records on schools' admissions in decentralized and uncoordinated systems tend to be poorly kept and infrequently audited. Together with several co-authors, I collect admissions records from Boston-area charter schools and study the effects of attending an over-subscribed charter school on short-run measures of student achievement. We find large and significant test score gains for charter lottery winners in middle and high school. 12 In subsequent work, I find that charter lottery winners at Boston high schools increase SAT and AP scores, along with evidence of a substantial shift from two- to four-year colleges. 13 In contrast, in work with Joshua Angrist and Christopher Walters, I find more mixed evidence on the performance of charter schools outside of urban areas of Massachusetts. 14
Charters are not assigned centrally in Boston, though they are now beginning to be assigned together with traditional district schools in unified enrollment systems in cities like Denver, Newark, and New Orleans. Alternative schools known as exam schools, which group together the highest-achieving students in the district, are centrally assigned in many cities based on admissions test scores. Together with Abdulkadiroğlu and Angrist, I exploit admissions discontinuities to measure the value of attending schools with high-achieving peers. On a wide range of academic outcomes, we find that marginal applicants who are accepted at exam schools do not score higher on subsequent performance metrics, such as standardized tests, than their near-peers who did not matriculate at exam schools.15
Another school model I have investigated using lottery-based variation in a centralized match is the small high school. Together with Abdulkadiroğlu and Weiwei Hu, we exploit variation in New York City's high school match to study the effects of attending an over-subscribed small high school, which typically has fewer than 500 students across grades 9 to 12. Unlike charter schools, these schools are run with teachers who are part of the city's collective bargaining agreement. Students are much more disadvantaged than typical New York City high school students. Our results offer some of the first evidence that traditional district schools can produce achievement gains comparable to high-achieving charter schools. 16 Based on surveys, many small high schools have similar characteristics to high-achieving charter schools including high expectations and data-driven instruction. These results highlight the potential for within-district reform strategies to substantially improve student achievement.
* Pathak is co-director of the NBER's Working Group on Market Design and a Research Associate in the NBER's Programs on the Economics of Education, Industrial Organization, and Public Economics. He is an Associate Professor of Economics at MIT.
1. A. Abdulkadiroğlu and T. Sönmez,-"School Choice: A Mechanism Design Approach," American Economic Review, 93 (2003), pp. 729-47.
2. P. A. Pathak and T. Sönmez,-"Leveling the Playing Field: Sincere and Sophisticated Players in the Boston Mechanism," American Economic Review, 98 (2008), pp. 1636-52.
3. D. Gale and L. Shapley,-"College Admissions and the Stability of Marriage," American Mathematical Monthly, 69 (1962), pp. 9‒15.
4. Table 1 is reproduced from P. A. Pathak and T. Sönmez, "School Admissions Reform in Chicago and England: Comparing Mechanisms by their Vulnerability to Manipulation," NBER Working Paper No. 16783, February 2011, and American Economic Review, 103 (2013), pp. 80-106. This paper provides further description of the mechanisms referenced in the table.
5. Pathak and Sönmez, 2013, op. cit.
6. A. Abdulkadiroğlu, P. A. Pathak, and A. E. Roth, "Strategy-proofness versus Efficiency in Matching with Indifferences: Redesigning the New York City High School Match," NBER Working Paper No. 14864, April 2009, and American Economic Review, 99 (2009), pp. 1954-78.
7. See Table 1 in Abdulkadiroğlu, Pathak, and Roth, 2009, op. cit.
8. A. Vanacore, "Centralized Enrollment in Recovery School District Gets Tryout," New Orleans Times-Picayune, April 16, 2012.
9. P. A. Pathak and J. Sethuraman, "Lotteries in Student Assignment: An Equivalence Result," NBER Working Paper No. 16140, June 2010, and Theoretical Economics, 6 (2011), pp. 1-17.
10. U. M. Dur, S. D. Kominers, P. A. Pathak, and T. Sönmez, "The Demise of Walk Zones in Boston: Priorities vs. Precedence in School Choice," NBER Working Paper No. 18981, April 2013.
11. P. A. Pathak and P. Shi, "Demand Modeling, Forecasting, and Counterfactuals, Part I," NBER Working Paper No. 19859, January 2014.
12. A. Abdulkadiroğlu, J.D. Angrist, S.M. Dynarski, T. J. Kane, and P. Pathak, "Accountability and Flexibility in Public Schools: Evidence from Boston's Charters and Pilots," NBER Working Paper No. 15549, November 2009, and Quarterly Journal of Economics, 126 (2009), pp. 699-748.
13. J. D. Angrist, S. R. Cohodes, S. M. Dynarski, P. A. Pathak, and C. R. Walters, "Stand and Deliver: Effects of Boston's Charter High Schools on College Preparation, Entry, and Choice," NBER Working Paper No. 19275, July 2013.
14. J. D. Angrist, P. A. Pathak, and C. R. Walters, "Explaining Charter School Effectiveness," NBER Working Paper No. 17332, August 2011, and American Economic Journal: Applied Economics, 5 (2013), pp. 1-27.
15. A. Abdulkadiroğlu, J. D. Angrist, and P. A. Pathak, "The Elite Illusion: Achievement Effects at Boston and New York Exam Schools," NBER Working Paper No. 17264, July 2011, and Econometrica, 82 (2014), pp. 137-96.
16. A. Abdulkadiroğlu, W. Hu, and P. A. Pathak,-"Small High Schools and Student Achievement: Lottery-Based Evidence from New York City," NBER Working Paper No. 19576, October 2013.