Evaluating the Quality and Use of Regulatory Impact Analysis

Executive Order 12866, which governs regulatory analysis and review in the executive branch, requires federal agencies to conduct a regulatory impact analysis (RIA) for all economically significant regulations. (An “economically significant” regulation has economic effects of at least $100 million annually or satisfies other criteria listed in the executive order.) The purpose of this analysis is to understand the problem the regulation aims to solve, identify alternative solutions, and assess the benefits and costs of the alternatives, in order to create regulations that solve real problems at a reasonable cost.

In 2009, the Mercatus Center at George Mason University initiated the Regulatory Report Card project to assess how well executive branch agencies conduct regulatory analysis. In a new study for the Mercatus Center, senior research fellow Jerry Ellig uses the data from this project to examine the quality of RIAs for all economically significant, prescriptive regulations proposed between 2008 and 2013. Report Card evaluations reveal that regulatory agencies often adopt regulations that affect several hundred million Americans and impose billions of dollars of costs without knowing whether the regulation will solve a significant problem, whether a more effective solution exists, or whether a more targeted solution could achieve the same result for a lower cost. Extensive statistical analysis of the scores suggests that institutional reforms are the most promising means of improving the quality and use of regulatory analysis.

REGULATORY REPORT CARD

Based on the analysis principles laid out in Executive Order 12866, the Mercatus Center’s Regulatory Report Card project evaluates the extent to which each proposed economically significant regulation is accompanied by analysis that accomplishes the following objectives:

1. assessing the nature and significance of the problem the regulation seeks to solve,
2. identifying a wide variety of alternative solutions,
3. defining the benefits the agency seeks to achieve and assessing each alternative’s ability to achieve those outcomes, and
4. identifying each alternative’s opportunity cost to society.

A good RIA does not just calculate monetary costs and benefits, but identifies, verifies, and explains the source of the problem and demonstrates with evidence why the proposed solution is the most effective means of solving the problem. In the Regulatory Report Card evaluation, a regulation’s analysis can earn a total of 5 points on each of the four criteria listed above for a total of 20 possible points.

REPORT CARD RESULTS

The Report Card scores were low on average and highly variable. Out of the maximum possible 20 points, the average score for the 130 economically significant regulations proposed between 2008 and 2013 was 10.7. The highest score achieved was 18 points. There are several areas of particular concern:

• Scores were consistently low for the analysis of the systemic problem. Only 13 percent of the regulations had significant evidence demonstrating the existence and cause of the problem.
• Only 19 percent were accompanied by analysis that considered a wide range of different solutions or levels of stringency.
• Just 24 percent of RIAs contained reasonably thorough assessments of price effects, and just 12 percent contained reasonably thorough analysis of costs due to behavioral changes.
• Of the 130 regulations analyzed, only one included a reasonably complete framework for retrospective analysis.

Like prior evaluations of the quality of regulatory analysis, the Report Card evaluations reveal significant deficiencies in the use of analysis. Where there are differences in the findings between prior evaluations and the Report Card, the Report Card typically offers less optimistic evaluations. This is owing to the fact that the Report Card relies on a more in-depth qualitative assessment rather than a checklist.

EXPLAINING HIGH VARIATION IN ANALYSIS QUALITY

Just a few agencies, producing 23 percent of the regulations in the sample, have significantly lower-quality analysis than the rest of the executive agencies. Regression analysis shows that the party controlling the administration and agency-specific effects are not sufficient to explain the wide range in RIA quality. There are several factors, however, that help to explain the variance in quality and use of analysis:

• Administrations tolerate less transparent analysis and less thorough explanations of how agencies used the analysis from agencies that are more likely to share their policy preferences.
• Midnight regulations (those finalized in the lame-duck period of a presidency) and potential midnight regulations have lower-quality analysis, regardless of which party controls the executive branch.
• When the Office of Information and Regulatory Affairs has greater influence in the administration, regulations have higher scores for quality of analysis, transparency of analysis, and agency explanations of how the analysis affected decisions.
• Regulations that are more politically visible may have better analysis, more extensive provision for retrospective analysis, and more extensive explanations of how the agency used the analysis in decisions.
• Statutory constraints that specify regulatory decision-making criteria are correlated with the quality of analysis. This suggests that agencies can and do comply with statutory standards that specify the topics an RIA must address.

CONCLUSION

The Regulatory Report Card reveals that the quality and use of regulatory analysis by federal agencies do not live up to the standards articulated in executive orders and guidance the Office of Management and Budget has created for agencies. The average Report Card score for recent regulations barely exceeds 50 percent. When regulations affect hundreds of millions of Americans and impose billions of dollars in costs, RIAs should demonstrate that agencies know whether a significant problem exists, understand the root cause of the problem, and understand the benefits and costs to society of alternative solutions.

The Report Card findings are directly relevant to the ongoing debate over reform of the federal regulatory process. The current system produces serious deficiencies in the quality of analysis, which lead to uninformed regulatory decisions. Agencies rarely make provision for retrospective analysis when issuing new regulations. New regulation is not an impediment to better analysis, and so it should be feasible to hold all agencies to the same high standards for analysis.