Section 1: What Was Evaluated?
To evaluate the quality and use of regulatory analysis, the research team reads the preamble to each proposed rule and the accompanying RIA. In some cases, agencies produce additional analysis in technical support documents, which we also consider. For some rules, agencies are also required to prepare a Regulatory Flexibility Analysis that assesses the effects on small entities and examines alternatives that might reduce the burden on them. We include the Regulatory Flexibility Analysis to the extent that it has content relevant to our evaluation criteria.
This approach is broader than just reading the document or section of the Federal Register notice explicitly labeled "Regulatory Impact Analysis." It is necessary for two reasons.
First, different agencies organize the content differently in different rules. Sometimes the RIA is a separate document only referenced or summarized in the Federal Register preamble. It may also be in a separate section of the preamble. Some parts may be in a separate section of the preamble, and other parts may be in other parts of the preamble that discuss justifications for the regulation. Analysis may also occur in the preamble without even being referenced in the RIA. Reading all of this material makes the evaluation task much more difficult, but it allows us to give the agency credit for doing good analysis of outcomes, costs, systemic problems, and alternatives regardless of where the analysis appears.
Second, we assess the agency's commitment to using regulatory analysis to make regulatory decisions and assess their effects in the future. If this commitment is documented anywhere, it is usually found in the preamble to the rule, where the agency discusses the rationale and justification for issuing the regulation. This is especially true for measures involving retrospective analysis of the regulation's actual effects; these almost always appear in the preamble to the rule, rather than the RIA (when they are discussed at all).
Section 2: Scoring System
2008-2012 Scoring System
For regulations evaluated between 2008 and 2012, we evaluated the regulatory analyses based on 12 criteria, grouped into three categories:
- Openness: How easily can the informed layperson find the analysis, understand it, and verify the underlying assumptions and data?
- Analysis: How well does the analysis define and measure the outcomes or benefits the regulation seeks to accomplish, define the systemic problem the regulation seeks to solve, identify and assess alternatives, and evaluate costs and benefits?
- Use: How much did the analysis affect decisions in the proposed rule, and what provisions did the agency make for tracking the rule's effectiveness in the future?
Below is a full list of the 12 criteria:
Openness
- Accessibility: How easily were the Regulatory Impact Analysis (RIA), the proposed rule, and any supplementary materials found online?
- Data Documentation: How verifiable are the data used in the analysis?
- Model Documentation: How verifiable are the models and assumptions used in the analysis?
- Clarity: Was the agency’s analysis comprehensible to an informed layperson?
Analysis
- Outcomes: How well does the analysis identify the desired benefits of other outcomes and demonstrate that the regulation will achieve them?
- Systemic Problem: How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve?
- Alternatives: How well does the analysis assess the effectiveness of alternative approaches?
- Benefit-Cost Analysis: How well does the analysis assess costs and benefits?
Use
- Use of Analysis: Does the proposed rule of the RIA present evidence that the agency used the RIA?
- Net Benefits: Did the agency maximize net benefits or explain why it chose another option?
- Measures and Goals: Does the proposed rule establish measures and goals that can be sued to track the regulation's results in the future?
- Retrospective Data: Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so?
Jerry Ellig and Patrick McLaughlin's paper "The Quality and Use of Regulatory Analysis in 2008," provides additional detail on the kinds of questions considered under each criterion.
For each criterion, the evaluators assign a score ranging from 0 (no useful content) to 5 (comprehensive analysis with potential best practices). Thus, each analysis has the opportunity to earn between 0 and 60 points. In general, the research team uses the guidelines below for scoring. Because the Analysis criteria involve so many discrete aspects of regulatory analysis, we developed a series of sub-questions for each of the four Analysis criteria and award a 0–5 score for each sub-question. These scores are then averaged to calculate the score for the individual criterion.
Post-2012 Scoring System
After 2012, in response to feedback received from readers and presentation audiences, we streamlined the scoring system and rearranged some of the evaluation questions to create a more intuitive scoring system. The post-2012 scoring system eliminates Criteria 1, 2, 3, 4, 11, and 12, substitutes 4 new criteria (1. Systemic Problem, 2. Alternatives, 3. Benefits, 4. Costs), and renumbers Criteria 9 and 10 as Criteria 5 and 6.
The post-2012 scoring criteria are:
- Systemic Problem: How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve?
- Alternatives: How well does the analysis assess the effectiveness of alternative approaches?
- Benefits (or other Outcomes): How well does the analysis identify the benefits or other desired outcomes and demonstrate that the regulation will achieve them?
- Costs: How well does the analysis assess costs?
- Use of Analysis: Does the proposed rule or the RIA present evidence that the agency used the Regulatory Impact Analysis in any decisions?
- Cognizance of Net Benefits: Did the agency maximize net benefits or explain why it chose another alternative?
Each criterion receives between 0 and 5 points, for a maximum possible score of 30 points.
What Do the Scores Mean?
5 - Complete analysis of all or almost all aspects, with one or more "best practices"
4 - Reasonably thorough analysis of most aspects and/or shows at least one "best practice"
3 - Reasonably thorough analysis of some aspects
2 - Some relevant discussion with some documentation of analysis
1 - Perfunctory statement with little explanation or documentation
0 - Little or no relevant content
The qualitative nature of the evaluation may be the most controversial aspect—especially since it seems to contrast with the objective and quantitative orientation of much benefit-cost research. A benefit of our qualitative approach is that it provides a richer and potentially more accurate evaluation of the actual quality of the analysis. As OIRA (2008, 19) notes, "Objective metrics can measure whether an agency performed a particular type of analysis, but may not indicate how well the agency performed this analysis." For example, rather than just asking whether the analysis considered alternatives or counting the number of alternatives considered, we can give an analysis a higher score if it considered a wider range of alternatives. Instead of just asking whether the agency named a market failure, we also assess whether the agency provides plausible evidence that the market failure exists, awarding a higher score based on how convincing the evidence is. Another benefit of the qualitative approach is that it encourages agencies to find the best way to do analysis that can inform decisions, instead of treating regulatory analysis as a "check the box" compliance exercise.
The main drawbacks of qualitative evaluation are that the results can be more subjective, less transparent, and harder to replicate. Several aspects of our research design seek to keep these costs within tolerable limits. We designed the evaluation process to achieve a consistent understanding of which practices deserve which kind of score, and evaluators took notes justifying each score. The entire research team underwent extensive training, evaluated several of the same proposed regulations and accompanying RIAs, compared scores, and discussed major differences until we achieved a consensus on scoring standards. For questions that were particularly difficult to evaluate, we developed written descriptions of the practices that would qualify for various scores in most cases, realizing that these descriptions could never be comprehensive. Each analysis are scored by two team members, with discussion to achieve consensus when scores differed significantly. Each author also reviews the other's scores and notes and then discusses and resolves differences to ensure that all documents are evaluated consistently on all questions.