The Urban Institute released a new analysis yesterday of the impact of a bill that Congress passed last year to repeal large parts of the Affordable Care Act (ACA). Urban’s analysis is based on many uncertain assumptions, including the implausible one that the incoming Trump administration and Congress, despite numerous campaign promises, will not provide any flexibility for people to purchase non-ACA-compliant products after repeal. Urban’s projections should be treated with significant skepticism because of the large uncertainty about its assumptions as well as substantial mistakes Urban has made in the past about the impact of the ACA.

Urban Has Been Largely Wrong About the ACA’s Coverage Impact

In December 2010, the Urban Institute simulated the provisions of the ACA as if fully implemented in 2010,projecting that there would be more than 23 million people enrolled in the ACA exchanges. In fact, exchange enrollment this year will average only about 10 million people. So, this Urban projection overestimated actual enrollment by 130%.

In January 2015, the Urban Institute estimated 2016 exchange enrollment in states with federal exchanges. They made this projection to warn about how many people would be effected if the Supreme Court ruled in a case that year that subsidies were not allowed in the 34 states with federal exchanges. Urban estimated there would be 13.6 million such enrollees in 2016. As of June 30, 2016, there were only about 7.7 million people enrolled in exchanges in those 34 states. This Urban projection overestimated actual enrollment by 77%. While the Supreme Court did not disallow subsidies in federal exchanges, such a ruling would have had much less effect than Urban projected and warned about.

Urban also significantly erred in its projection of the make-up of exchange enrollment. As the table below shows, in its January 2015 projection, Urban estimated that about one-third of enrollees would have income below 200% of the federal poverty level (FPL)—about $24,000 for a single person—and that about one-quarter of enrollees would have income above 400% of the FPL—about $48,000 for a single person. It turned out that in both 2015 and 2016, roughly two-thirds of exchange enrollees had income below 200% of the FPL and only 2% of enrollees had income above 400% of the FPL.

Conclusion

The Urban Institute was not the only organization that made large errors in projecting the ACA’s impact. TheCongressional Budget Office, the Centers for Medicare and Medicaid Services, and the Rand Corporation also made substantial and similar errors. The ACA is a big and complicated law and modeling its effect is a difficult exercise loaded with uncertainty.

In short, the main collective flaw was the assumption that the individual mandate would be a lot more effective than it has been. In reality, the coverage offered in the exchanges is a really bad deal for people who don’t receive large subsidies to purchase it. Millions of people have made the economically rational decision to pay the mandate penalty or seek an exemption rather than purchase the unattractive coverage.

Urban’s most recent projections should be discounted given that its past modeling produced such large errors of the ACA’s impact and that its new projections rest upon numerous uncertain and unrealistic assumptions. More broadly, policy should not be driven by the outputs of economic models. Policymakers’ key focus should instead be on realigning incentives and removing barriers to care. Then, let markets work, as they do throughout the rest of the economy, to deliver consumers better health care at lower cost.