Sixty-three years ago on Wednesday, the Supreme Court prohibited school segregation. In the South, Brown v. Board of Education was enforced slowly and fitfully for two decades; then progress ground to a halt. Nationwide, black students are now less likely to attend schools with whites than they were half a century ago. Was Brown a failure?

Not if we consider the boost it gave to a percolating civil rights movement. The progeny of Brown include desegregation of public accommodations and the mostly unhindered right of African Americans to compete for jobs, to vote, and to purchase or rent homes. Brown’s greatest accomplishment was its enduring imprint on the national ethos: the idea of second-class citizenship for African Americans, indeed for any minority group, is now universally condemned as a violation of the Constitution and of American values. None of these transformations came easily, and none are complete, but none would have happened were it not for Brown.

Yet the decision could not accomplish its stated purpose. Today, nearly half of all black students attend majority black schools, with over 70 percent in high-poverty school districts. New York is the most segregated state: two-thirds of its black students attend schools that are less than 10 percent white. A growing number are “integrated” with low-income Hispanics and other recent immigrants, but still isolated from the mainstream.

Because schools remain segregated, we have little chance to substantially boost the achievement of black children, especially those from low-income families. Of course, some children will always surmount their disadvantages and excel. But when separate schools concentrate students who are in poorer health and more frequently absent, who may be homeless or in unstable housing, and whose parents are less-educated, achievement lags when teachers are overwhelmed by non-academic challenges.

If more than a few pupils in a single classroom act-out from the stresses of economic insecurity, neighborhood violence, or parental incarceration, behavioral issues steal time from instruction. Added social, psychological, and academic services for these children can reduce (but not eliminate) their challenges, but funding for such services is inadequate everywhere.

Seemingly permanent segregation is not what we expected. In 1954, a few hours after Brown was announced, Thurgood Marshall, leader of the NAACP’s Legal Defense Fund, told reporters that it would take, at most, five years for schools to desegregate nationwide.

He didn’t anticipate the massive resistance of Southern states to the decision, yet that’s no longer the most important factor impeding integration. Rather, schools remain segregated mostly because their neighborhoods are segregated. Had civil rights lawyers been able to attack neighborhood rather than school segregation, they would have accomplished more for educational equality than by focusing on schools directly.

When its school desegregation drive began during the Depression, the NAACP was struggling, with members’ contributions counted more often in dimes than dollars. But in 1930, a wealthy benefactor gave the organization an extraordinary grant of $100,000 (about $1.4 million in today’s currency) to fight school segregation.

The NAACP’s top lawyer was Howard University law school dean Charles Hamilton Houston. His protégé was Thurgood Marshall. Concluding that a frontal challenge to K-12 segregation was too radical, they began by suing to admit African Americans to public law schools — figuring that if Supreme Court justices understood nothing else about equality, they might comprehend that adequate legal education was necessary to practice law.

By 1935, Houston and Marshall had persuaded a court to order the whites-only University of Maryland law school to admit African Americans. The civil rights lawyers then proceeded to force desegregation of other graduate schools. Only after a 15-year campaign were they ready to attack K-12 segregation, in lawsuits leading to Brown. The gripping tale of their perseverance, courage, and realism was masterfully recounted in Richard Kluger’s 1975 book, Simple Justice.

Philanthropic funding alone did not lead the NAACP to focus on school, not housing segregation. When their campaign began, Franklin Roosevelt’s New Deal had not yet begun to enforce metropolitan segregation. When it did so, it employed administrative guidelines, less conspicuous than laws segregating schools. The federal government’s residential race policy was explicit, unhidden, and just as unconstitutional as school segregation laws, but it only unfolded gradually as the nation recovered from the Depression and then suburbanized.

As I describe in my recent book, The Color of Law, the policy began in 1933 with the New Deal’s Public Works Administration that built segregated housing in cities where segregation had not previously been known. Initial projects were for middle-class white families who had lost homes in the Depression. Civil rights leaders had to press to get any housing for black families, so were less focused on its segregation. Projects constructed for African Americans were always segregated—in the North, Midwest, and West, not only in the South.

At the time, white ethnic immigrants, rural migrant and black workers frequently lived in walking distance from downtown factory jobs. Certainly there were racially and ethnically homogeneous pockets, but rigid separation was rare. Even in the South, urban neighborhoods were more integrated than today. In Atlanta, the Public Works Administration razed a downtown neighborhood, “the Flats,” whose population was almost evenly split between whites and blacks. But the government replaced it with public housing for white families only.

In his autobiography, Langston Hughes described his integrated Cleveland neighborhood high school where, during World War I, he dated a Jewish girl and his best friend was Polish. But during the New Deal, federal and local government built separate projects for whites and blacks in that same neighborhood — and in cities across the country — creating and reinforcing a pattern of racial separation that might not otherwise have evolved.

Then, during World War II, workers of both races flocked to jobs in defense plants, but found no lodging available. To keep assembly lines running, the federal government had to create housing. The West Coast’s largest shipbuilding center was Richmond, California. Few African Americans lived there before the war (some domestic servants worked for white families), so there was no established segregation. By war’s end, Richmond’s population had multiplied tenfold, with inflexible segregation established by racially separate war housing. Just south of Richmond in Berkeley, black-designated buildings were placed along the railroad tracks while white-designated buildings were placed inland near shopping and white residential areas. The pattern, too, was repeated almost everywhere.

Suburbanization for whites-only, by explicit federal command, followed. It is well-known that the Federal Housing Administration had a “redlining” policy, generally refusing to insure mortgages in black neighborhoods. Less familiar is that the Federal Housing Administration and the Veterans Administration guaranteed development loans for entire suburban communities, provided that no homes be sold to African Americans.

One of the first and largest was Levittown, 17,000 low-cost homes in Nassau County, New York. William Levitt could never have independently amassed the capital for its design and construction. Instead, he obtained government-guaranteed bank loans by agreeing to exclude black buyers and to place language in every deed prohibiting re-sales to African Americans. The Federal Housing Authority also insisted upon zoning ordinances that banned future mixed-income development. Such practices everywhere created suburban white nooses around increasingly black cities.

When middle-class whites left for federally subsidized and racially exclusive suburbs, their monthly mortgage payments were often less than their prior public housing rents. By the mid-1950s, almost every whites-only public housing project had vacancies, while black-designated projects had long waiting lists. Soon, with whites no longer needing it, all public housing was opened to African Americans. As industry left cities for suburbs where nearby whites could find employment, poverty became more concentrated in urban black neighborhoods and public housing was transformed from a middle-class program addressing housing shortages to a welfare program for the (mostly minority) poor.

In 1948, the Supreme Court prohibited courts from evicting black families who moved to white neighborhoods and in 1968, the Fair Housing Act banned future housing discrimination. The Act’s enforcement has been imperfect, but some African Americans have successfully integrated suburban communities. But even with perfect enforcement, most would have been, and were, unable to do so. Racially restricted single-family homes that sold in the 1940s and ‘50s for twice national median income now sell for three times that much. Working-class black families who could have bought suburban homes when they were first built can no longer afford them.

Most Americans have forgotten this history, aided by Supreme Court opinions that declare residential segregation to be “de facto,” resulting mostly from private discrimination. But de facto segregation is mythical. In reality, neighborhood segregation has resulted from intentional government policy, as unconstitutional as the “de jure” school segregation imposed by Southern legislatures prior to 1954. The governmentally-sponsored residential segregation imposed by federal, state, and local authorities in the mid-twentieth century established a segregated landscape in every metropolitan area that has never been remedied.

Once the Supreme Court decided Brown, children of either race could attend their neighborhood schools if the decision were obeyed. But if courts were now to recognize that residential segregation is rooted in unconstitutional policy, undoing it would be daunting. Children can easily walk to nearby schools but families can’t easily pick up and move to integrated and now-unaffordable neighborhoods, when no longer barred from doing so.

On May 17, 1954, when Thurgood Marshall and his colleagues predicted rapid school desegregation, they said that they would now turn attention to housing. But their resources were soon consumed by defending Brown, as state after state sabotaged it. And it was already too late for straightforward attacks on neighborhood segregation — the government’s scheme to segregate metropolitan areas was by then mostly complete, with residential patterns solidified.

Dismantling de jure residential segregation is incomparably more difficult now than it would have been 70 or 80 years ago. But that’s no excuse for avoiding it. Unless we desegregate neighborhoods, Brown’s promise of integrated education will remain unattainable.

The Color of Law asserts that “letting bygones be bygones” is not a policy worthy of a constitutional democracy. The achievement gap with which educators struggle can never be closed until we recognize that some of the most important education policy dilemmas cannot be addressed in isolation. Fundamentally, education policy is housing policy.

This blog post originally appeared in the Washington Post.