Assessing the Quality of Regulatory Analysis

Congress and the executive branch have attempted to improve the quality of regulatory decisions by adopting laws and executive orders that require agencies to analyze benefits and costs of their decision options. This paper assesses the quality and use of regulatory analysis accompanying every economically significant regulation proposed by executive-branch regulatory agencies in 2008 and 2009. It considers all analysis relevant to the topics covered by Executive Order 12866 that appears in the Regulatory Impact Analysis document or elsewhere in the Federal Register notice that proposes the rule.

Our research team used a six-point qualitative scale to evaluate each regulation on 12 criteria grouped into three categories: (1) Openness: How easily can a reasonably intelligent, interested citizen find the analysis, understand it, and verify the underlying assumptions and data? (2) Analysis: How well does the analysis define and measure the outcomes the regulation seeks to accomplish, define the systemic problem the regulation seeks to solve, identify and assess alternatives, and evaluate costs and benefits?; and (3) Use: How much did the analysis affect decisions in the proposed rule, and what provisions did the agency make for tracking the rule’s effectiveness in the future?

We find that the quality of regulatory analysis is generally low, varies widely, and did not change much with the change of administrations between 2008 and 2009. The principal improvements across all regulations occurred on the Openness criteria. Budget or "transfer" regulations, which define how the federal government will spend money or collect revenues, have much lower quality analysis than other regulations. Use of analysis is correlated with its quality, and use of analysis fell in 2009 after controlling for the quality of the analysis. Regulations implementing Recovery Act spending programs have better provisions for retrospective analysis than other transfer regulations.