Brookings H-1B Report’s Flawed Analysis & Flawed Process

The Brookings Institution issued a new report on Friday about the H-1B program—a temporary foreign worker program for “skilled” occupations, meaning those that require a college degree—and then issued corrections to it almost immediately afterwards.

The report claims to include a wage analysis on “new data” that, “suggests that the H-1B program helps to fill a shortage of workers in STEM [science, technology, engineering and math] occupations.”  

There are two critical problems with the report’s analysis of these data:

First, the data are proprietary, meaning the data are exclusively held by the authors, thus no one can critique or review the study’s presentation of the data or its findings. (The authors obtained the data from two other researchers, who first obtained it through a Freedom of Information Act request.) This kind of approach, where researchers use data that are not available publicly, means that the data and subsequent analysis can never be checked, leaving out a critical step in the scientific process.

This step is critical because mistakes happen. In fact, just hours before it was to be released the authors shared the study with me. They did not ask for a critical review, and it would have been impossible for me to conduct one given the short time frame (typically a reviewer is given two weeks to provide input), and more importantly, I did not (and still do not) have access to the proprietary data.

As I skimmed the report I found a few peculiar items in the study. Since I have spent more than a decade working on the H-1B program, and these data are presented at an aggregate level by U.S. Citizenship and Immigration Services (USCIS) on an annual basis, I’m quite familiar with them.

The Brookings researchers had only 105,000 H-1B workers in their data but USCIS reported (in its 2010 Characteristics of Specialty Occupations report) that almost 193,000 H-1B workers were approved that year. It turns out that the Brookings study was missing nearly 88,000 workers, or approximately 45% of the data, in their analysis. This is the exact same data, so there is no logical reason for the discrepancy.

I pointed out this serious discrepancy to the authors, and they dismissed my concerns. After I repeatedly pointed out why their data didn’t make any sense—after all, USCIS was reporting the same data with many more workers—the Brookings authors realized I was correct.

They had erroneously thrown out 45% of good data before even beginning their analysis. Subsequently, their study was changed on the fly to include the data. This was an honest albeit elementary mistake, since anyone who works on the H-1B program is familiar with the size of the program.

Mistakes get made in the course of research, which is why it is critical to have multiple pairs of eyes scrutinizing the data and the analysis. But when researchers use proprietary data, as the Brookings report does, it precludes these critical steps in the scientific process.

Without public access to the data there is no way for the Brookings report to be properly reviewed.

Second, notwithstanding the major data problems in the report, the study makes many questionable analytic choices. I will only point to one of them here.

The report attempts to evaluate whether H-1B workers are underpaid. The key issue is to design your analysis so that you are comparing apples to apples. But the Brookings study fails this basic standard in a number of ways.

Its key table on wage analysis (Table 1) tries to make a comparison of H-1B wage levels against comparable American workers.

However, it does so by using a very expansive definition of comparable American workers. In calculating “American Wages” it includes all computer occupations, including technician level workers like Computer Support Specialists. This category is a significant share of all computer occupations and has relatively very low wages. By including the low wage categories in its analysis, Brookings is systematically lowering the reported American wages, thereby making H-1B wages appear higher than they are.

The upshot is that the authors are making an apples to oranges comparison.

The Brookings report adds no value to the question of H-1B wages. USCIS actually provides more detailed and more useful tables of the same data in its annual report (for example, see Tables 10, 11, and 12 here).

The U.S. Government Accountability Office attempted a similar exercise with the same data set that Brookings is using (but from an earlier year, 2008). After describing the various weaknesses in the data (see pages 82-83) the report concluded: “In light of these limitations, caution should be used in interpreting differences found in comparing estimated 2008 median U.S. citizen worker salaries and the median salaries for H-1B worker petitions submitted in 2008.” Brookings does nothing to allay or account for, or even acknowledge these limitations.

And it is well known that the H-1B program is used for cheaper labor.

Industry veteran Neeraj Gupta testified before the Senate Judiciary Committee on April 22 that most H-1Bs are being used for cheaper labor. Just a few weeks ago, a Wall Street Journal article about proposed changes to the H-1B program reported that “Indian IT professionals working in the U.S. are typically paid about 25% less than their American counterparts.” And some H-1B employers themselves have told government auditors that they hire H-1Bs because they can be paid less (for example, see page 4 of this report).

I will post a more detailed critique of the Brookings data and report later this week. For another take on the weaknesses in the report’s methodology, see U.C. Davis Professor Norm Matloff’s analysis from his email newsletter.


  • twinsfan1100

    The Brookings Institute is corrupt. The Brookings Institute receives millions of dollars of funding from the Bill and Melinda Gates Foundation.

    The Bill and Melinda Gates Foundation, under the false claim that they are engaged in philanthropy, pays the Brookings Institute to concoct studies that reach predetermined conclusions that are beneficial to Bill Gates’ financial interests at Microsoft.

    For years Microsoft, which Bill Gates is the founder and Chairman of the Board, has been one of the leaders in the use of H-1B visas. Gates has been laying off thousands of US workers annuallly and backfilling the jobs, previously held by the displaced workers, with workers from low cost labor centers, primarily India and Communist China for the purpose of 1) reducing labor costs and 2) moving US technology, industry, and jobs to the same low cost labor centers, primarily India and Communist China.

  • US Citizen

    Regarding “I will post a more detailed critique of the Brookings data and report later this week.”, is there some place I can read this on the web?

  • twinsfan1100

    It should be noted that Bill Gates, who until recently was the Chairman of the Board at Microsoft, is among the most generous sponsors of the Brookings Institute. Microsoft has long been one of the largest users of the H-1B visa while simultaneously laying off US workers.

  • econdataus

    Good article. I was stunned to see such a poorly researched paper with Brookings’ name on it. Until I see evidence to the contrary, I now put as much trust in their work as I do the Heritage Foundation. I have to wonder if they aren’t affected by their funding from Microsoft. In any case, the paper by Norm Matloff that you mentioned brought up another point which I had wondered about after reading about the Brookings paper. I had also looked at data from the American Community Survey, one of the sources for the Brookings numbers, and posted some results at http://econdataus.com/svworkers.html . As can be seen, about half of the software developers in Silicon Valley are non-citizens (a proxy for H-1B visa workers). The percentage is much lower in California as a whole, however, just about 38 percent. Did the Brookings authors make any correction for the fact that a larger percentage of H-1B visa workers were in the very expensive Silicon Valley? According to Matloff, “Brookings DIDN’T consider region–their analysis had no region variable. And that factor matters a LOT.”

    Also, Matloff mentions that Brookings made no mention of a number of better, contrary studies, including some where employers were asked about wages directly. He states, “…based on interviews with some H-1B employers, Salzman reported that H-1B workers in jobs requiring lower levels of IT skill received lower wages, less senior job titles, smaller signing bonuses, and smaller pay and compensation increases than would be typical for the work they actually did.” This bothers me about much of the pro-H-1B visa commentary such as at http://www.fwd.us/ or http://www.competeamerica.org/ . They give sources for none of their claims and make no mention of valid, conflicting opinions. These two sites seem to just offer to put readers names on prewritten letters (don’t worry your pretty little head about what they say) and send them to their representatives.

    One last very disturbing thing that you mentioned was that much of the data used is proprietary. That is the same problem that led to the Reinhart Rogoff debacle described at http://usbudget.blogspot.com/2013/05/is-there-debtgdp-threshold-at-90.html . As mentioned there, I think that all studies should be ignored or treated as interesting ideas until they can be verified. Anyhow, did you ever post a more detailed critique of the Brookings data and report as you had hoped and, if so, where is it? In any case, you and Matloff did do a good service in analyzing the Brookings report.

  • econdataus

    Good article. I was surprised to see such a poorly researched paper with Brookings name on it. I have to wonder if they aren’t affected by their funding from Microsoft. In any case, the paper by Norm Matloff that you mentioned brought up another point which I had wondered about after reading about the Brookings paper. I had also looked at data from the American Community Survey, one of the sources for the Brookings numbers, and posted some results at http://econdataus.com/svworkers.html . As can be seen, about half of the software developers in Silicon Valley are non-citizens (a proxy for H-1B visa workers). The percentage is much lower in California as a whole, however, just about 38 percent. Did the Brookings authors make any correction for the fact that a larger percentage of H-1B visa workers were in the very expensive Silicon Valley? According to Matloff, “Brookings DIDN’T consider region–their analysis had no region variable. And that factor matters a LOT.”

    Also, Matloff mentions that Brookings made no mention of a number of better, contrary studies, including some where employers were asked about wages directly. He states, “…based on interviews with some H-1B employers, Salzman reported that H-1B workers in jobs requiring lower levels of IT skill received lower wages, less senior job titles, smaller signing bonuses, and smaller pay and compensation increases than would be typical for the work they actually did.” This bothers me about much of the pro-H-1B visa commentary such as at http://www.fwd.us/ or http://www.competeamerica.org/ . They give sources for few of their claims and make no mention of valid, conflicting opinions. These two sites seem to just offer to put readers names on prewritten letters (don’t worry your pretty little head about what they say) and send them to their representatives.

    One last very disturbing thing that you mentioned was that much of the data used is proprietary. That is the same problem that led to the Reinhart Rogoff debacle described at http://usbudget.blogspot.com/2013/05/is-there-debtgdp-threshold-at-90.html . Of course, people should be free to release results obtained from proprietary data. However, such studies should ignored or treated as interesting ideas until that they can be independently verified.