Explaining the Regulatory Report Card Criterion

In the following 12 videos, one for each of the 12 criteria in the Report Card, senior scholars Jerry Ellig and John Morrall explain, criterion by criterion, how an evaluation team determines whether a Regulatory Impact Analysis (RIA) lives up to the standards put forth in the Executive Order 12866 and the Office and Management Budget Circular A-4.

Criteria 1 through 4 cover questions about the openness of the RIA, including clarity and quality of documentation. Criteria 5 through 8 deal with the quality of the RIA's analysis. Criteria 9 through 12 assess whether the agency used the RIA in crafting the proposed rule or made provisions for retrospective analysis after already adopted rules.

For more, please see the Regulatory Report Card home page.

Criterion 1: How easily were the RIA, the proposed rule, and any supplementary materials found online?

  • How easily can the proposed rule and RIA be found on the agency's web site?
  • How easily can the proposed rule and RIA be found on Regulations.gov?
  • Can the proposed rule and RIA be found without contacting the agency for assistance?

Criterion 2: How verifiable are the data used in the analysis?

  • Is there evidence that the analysis used data?
  • Does the analysis provide sufficient information for the reader to verify the data?
  • How much of the data are sourced?
  • Does the analysis provide direct access to the data via links, URLs, or provision of data in appendices?
  • If data are confidential, how well does the analysis assure the reader that the data are valid?

Criterion 3: How verifiable are the models and assumptions used in the analysis?

  • Are models and assumptions stated clearly?
  • How well does the analysis justify any models or assumptions used?
  • How easily can the reader verify the accuracy of models and assumptions?
  • Does the analysis provide citations to sources that justify the models or assumptions?
  • Does the analysis demonstrate that its models and assumptions are widely accepted by relevant experts?
  • How reliable are the sources? Are the sources peer-reviewed?

Criterion 4: Was the agency's analysis comprehensible to an informed layperson?

  • How well can a non-specialist reader understand the results or conclusions?
  • How well can a non-specialist reader understand how the analysis reached the results?
  • How well can a specialist reader understand how the analysis reached the results?
  • Are the RIA and relevant portions of the Federal Register notice written in ―plain English‖? (Light on technical jargon and acronyms, well-organized, grammatically correct, direct language used.)

Criterion 5: How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them?

  • How well does the analysis clearly identify ultimate outcomes that affect citizens' quality of life?
  • How well does the analysis identify how these outcomes are to be measured?
  • Does the analysis provide a coherent and testable theory showing how the regulation will produce the desired outcomes?
  • Does the analysis present credible empirical support for the theory?
  • Does the analysis adequately assess uncertainty about the outcomes?

Criterion 6: How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve?

  • Does the analysis identify a market failure or other systemic problem?
  • Does the analysis outline a coherent and testable theory that explains why the problem (associated with the outcome above) is systemic rather than anecdotal?
  • Does the analysis present credible empirical support for the theory?
  • Does the analysis adequately assess uncertainty about the existence and size of the problem?

Criterion 7: How well does the analysis assess the effectiveness of alternative approaches?

  • Does the analysis enumerate other alternatives to address the problem?
  • Is the range of alternatives considered narrow or broad?
  • Does the analysis evaluate how alternative approaches would affect the amount of the outcome achieved?
  • Does the analysis adequately address the baseline—what the state of the world is likely to be in the absence of further federal action?

Criterion 8: How well does the analysis assess costs and benefits?

  • Does the analysis identify and quantify incremental costs of all alternatives considered?
  • Does the analysis identify all expenditures likely to arise as a result of the regulation?
  • Does the analysis identify how the regulation would likely affect the prices of goods and services?
  • Does the analysis examine costs that stem from changes in human behavior as consumers and producers respond to the regulation?
  • Does the analysis adequately address uncertainty about costs?
  • Does the analysis identify the approach that maximizes net benefits?
  • Does the analysis identify the cost-effectiveness of each alternative considered?
  • Does the analysis identify all parties who would bear costs and assess the incidence of costs?
  • Does the analysis identify all parties who would receive benefits and assess the incidence of benefits?

Criterion 9: Does the proposed rule or the RIA present evidence that the agency used the Regulatory Impact Analysis?

  • Does the proposed rule or the RIA assert that analysis of outcomes, benefits, the systemic problem, alternatives, or costs affected any decisions?
  • How many aspects of the proposed rule did the analysis affect?
  • How significant are the decisions the analysis affected?

Criterion 10: Did the agency maximize net benefits or explain why it chose another option?

  • Did the analysis calculate net benefits of one or more options so that they could be compared?
  • Did the analysis calculate net benefits of all options considered?
  • Did the agency either choose the option that maximized net benefits or explain why it chose another option?
  • How broad a range of alternatives did the agency consider?

Criterion 11: Does the proposed rule establish measures and goals that can be used to track the regulation's results in the future?

  • Does the RIA or the Federal Register notice contain analysis or results that could be used to establish goals and measures to assess the results of the regulation in the future?
  • In the RIA or the Federal Register notice, does the agency commit to performing some type of retrospective analysis of the regulation's effects?
  • Does the agency explicitly articulate goals for at major outcomes the rule is supposed to affect?
  • Does the agency establish measures for major outcomes the rule is supposed to affect?
  • Does the agency set targets for measures of major outcomes the rule is supposed to affect?

Criterion 12: Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so?

  • Does the RIA or Federal Register notice demonstrate that the agency has access to data that could be used to assess some aspects of the regulation's performance in the future?
  • Would comparing actual outcomes to outcomes predicted in the analysis generate a reasonably complete understanding of the regulation's effects?
  • Does the agency suggest it will evaluate future effects of the regulation using data it has access to or commits to gathering?
  • Does the agency explicitly enumerate data it will use to evaluate major outcomes the regulation is supposed to accomplish in the future?
  • Does the analysis demonstrate that the agency understands how to control for other factors that may affect outcomes in the future?