Score: 19 / 60
RULE SUMMARY
The Forest Service, U.S. Department of Agriculture (USDA), is proposing to establish a state-specific rule to provide management direction for conserving and managing inventoried roadless areas on National Forest System (NFS) lands in Colorado. A proposed rule was published in the July 25, 2008, Federal Register. In response to public comment on the 2008 proposed rule and a revised petition submitted by the state of Colorado on April 6, 2010, the Forest Service is publishing a new proposed rule.
METHODOLOGY
There are twelve criteria within our evaluation within three broad categories: Openness, Analysis and Use. For each criterion, the evaluators assign a score ranging from 0 (no useful content) to 5 (comprehensive analysis with potential best practices). Thus, each analysis has the opportunity to earn between 0 and 60 points.
| Criterion | Score |
|---|---|
Openness | |
| 1. How easily were the RIA , the proposed rule, and any supplementary materials found online? The NPRM and the environmental impact assessment, but not the RIA, turn up with a keyword search in regulations.gov. There is no link to the RIA in the NPRM. Much of the economic analysis is actually in the environmental impact analysis, which is linked. The USDA home page has a landing page for the roadless rule that has links to all the relevant documents, but this is difficult to find without a phone call to the agency. | 1/5 |
| 2. How verifiable are the data used in the analysis? Most data are not directly sourced in the RIA; the RIA refers readers to the environmental impact analysis. The environmental impact analysis says that full supporting documents are available at the Forest Service office in Lakewood, CO; some are available online. Data in the environmental impact statement are usually sourced to public documents or internal databases that are not very transparent. | 2/5 |
| 3. How verifiable are the models and assumptions used in the analysis? The RIA mostly presents conclusions from the environmental impact statement without page references. It usually makes assertions about cause and effect (e.g., some tree cutting reduces fire damage or increases safety) with no citation to studies or sources. The RIA includes a short list of references, most of which do not have links. Some parts of the environmental impact statement simply list assumptions or conclusions with no documentation; others are well-documented. The environmental impact statement has an extensive list of references, about one-quarter of which are linked. | 2/5 |
| 4. Was the analysis comprehensible to an informed layperson? The text of the 16-page executive summary does not clearly explain what the alternatives are or what kinds of activities count as costs versus benefits. Many unfamiliar acronyms and jargon make understanding difficult. The environmental impact analysis is much better written and much more informative than either the NPRM or the RIA. Results are hard to understand because the results of the three alternatives to the baseline are not presented as differences from the baseline. | 2/5 |
Analysis | |
| 5. How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them? | 3/5 |
| Does the analysis clearly identify ultimate outcomes that affect citizens’ quality of life? Potential benefits named in the RIA include a list of private benefits from permitted activities, non-market environmental benefits, and changes in Forest Service costs. A table lists nine "values" offered by public commenters that reflect both private and environmental benefits. It is not always clear whether these count as benefits or costs of the regulation. | 4/5 |
| Does the analysis identify how these outcomes are to be measured? Except for the value of output from mineral leases, the outcomes are not usually measured. The RIA mostly describes how the alternatives would affect the amount of activity (e.g., tree cutting, road construction) that might affect these outcomes. Most of the RIA's assessments (e.g., more or less risk of fire, threats to species, water quality) presume an automatic link between activity and outcome that is often documented in the environmental impact statement but not the RIA. The RIA claims effects are described qualitatively due to the absence of site-specific proposed projects. | 3/5 |
| Does the analysis provide a coherent and testable theory showing how the regulation will produce the desired outcomes? There are a lot of assertions in the RIA about what will lead to what without much of a theory explaining why. The environmental impact assessment has better explanations of how road building and tree cutting affect various outcomes. | 4/5 |
| Does the analysis present credible empirical support for the theory? Only the environmental impact assessment cites some sources (but not much data) in support of claims that various activities will have various effects. Rarely is there enough information to determine if the particular differences between alternatives can be expected to generate large or small effects. | 2/5 |
| Does the analysis adequately assess uncertainty about the outcomes? The RIA notes that economic projections are approximations but does not offer an uncertainty analysis. It also notes that production of commodities or amenity services will vary with changing market conditions. | 2/5 |
| 6. How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve? | 0/5 |
| Does the analysis identify a market failure or other systemic problem? This should have been easy to do but was not done. Rulemaking is a response to a petition from the state of Colorado seeking state-specific regulations to manage roadless areas. The "Purpose and Need" section lists many reasons someone should manage the roadless areas but does not describe a systemic problem with the state of current management. It just says the state and the Forest Service agree there is a need. The 2008 RIA came closer to naming a systemic problem. | 1/5 |
| Does the analysis outline a coherent and testable theory that explains why the problem (associated with the outcome above) is systemic rather than anecdotal? The RIA might imply that management of roadless areas under existing "forest plans" might not adequately reflect state-specific and local needs (as the 2008 RIA implied in regard to its baseline 2001 rule). This might have been developed into some theory explaining inadequacy of the baseline approach, but this was not done. | 0/5 |
| Does the analysis present credible empirical support for the theory? The only evidence is an assertion that the Forest Service and the state agree a regulation is needed. | 0/5 |
| Does the analysis adequately assess uncertainty about the existence or size of the problem? No relevant content. | 0/5 |
| 7. How well does the analysis assess the effectiveness of alternative approaches? | 4/5 |
| Does the analysis enumerate other alternatives to address the problem? The environmental impact statement identifies seven alternatives, four of which are chosen for further analysis. RIA examines four alternatives, one of which is the baseline. | 5/5 |
| Is the range of alternatives considered narrow (e.g., some exemptions to a regulation) or broad (e.g., performance-based regulation vs. command and control, market mechanisms, nonbinding guidance, information disclosure, addressing any government failures that caused the original problem)? The alternatives are very narrow, with little scope for flexibility. Analysis considers two different inventories of roadless areas, different amounts of acreage restricted to "upper tier," and several different sets of authorized or prohibited activities, such as tree cutting and roads for oil, gas, and mineral leases. The three regulatory alternatives are all variations on the same basic approach, rather than different forms of regulation. Differences in the amount of road construction and tree cutting appear to be very small over several million acres. Thus, the alternatives vary primarily in scope of the rule from status quo to most expansive application, as well as restrictiveness of permitted activities. | 3/5 |
| Does the analysis evaluate how alternative approaches would affect the amount of the outcome achieved? The analysis identifies outcomes produced by each alternative for each issue considered. This was usually ordinal, since not much was measured. But the assessment is quite extensive, given the large number of values considered. | 4/5 |
| Does the analysis adequately address the baseline? That is, what the state of the world is likely to be in the absence of federal intervention not just now but in the future? The baseline is no state-specific regulation, with all roadless areas managed according to "forest plans." The RIA does not explain what this means (unlike the 2008 RIA); the environmental impact study has a brief explanation. The analysis offers many comparisons of the alternatives but does a poor job of showing the net effects of the alternatives relative to the baseline. One page of text does this comparison for oil, gas, and coal production. | 3/5 |
| 8. How well does the analysis assess costs and benefits? | 2/5 |
| Does the analysis identify and quantify incremental costs of all alternatives considered? The RIA mostly reports changes in activity, not costs. Effects on mineral production and on other values under different alternatives could have been presented as opportunity costs if the RIA had properly compared the three alternatives to the baseline. Unlike the 2008 RIA, it does not quantify value of property at risk from fire, a potential cost of restricting access. It is surprising the RIA did not employ any techniques for quantifying or monetizing the value of non-market goods. | 2/5 |
| Does the analysis identify all expenditures likely to arise as a result of the regulation? The only expenditures discussed are Forest Service expenditures. Changes in expenditures in roadless areas are assumed to be matched by reductions in other areas due to budget constraints. No assessment of expenditures related to opportunity costs, such as cost of forest fires. | 3/5 |
| Does the analysis identify how the regulation would likely affect the prices of goods and services? Price effects for mining/energy are considered, and the Forest Service concludes these are negligible. The preamble states that there will be no noticeable change in prices since changes in production will not occur until well into the future. Nothing else considered, even though wild fire risk could have a big effect on housing and insurance prices. | 1/5 |
| Does the analysis examine costs that stem from changes in human behavior as consumers and producers respond to the regulation? Changes in human behavior are discussed as far as how the range of alternatives will affect industry and human enjoyment of wildlife. Due to the unclear baseline, it is not always clear which outcomes count as benefits and which ones count as costs. | 2/5 |
| If costs are uncertain, does the analysis present a range of estimates and/or perform a sensitivity analysis? The analysis states that projections are uncertain, and to the extent that this statement applies to outcomes that might be opportunity costs, it is an acknowledgement of cost uncertainty. | 1/5 |
| Does the analysis identify the alternative that maximizes net benefits? There is no single net benefit figure. An extensive table lists how the different alternatives accomplish more or fewer of various activities or outcomes. So it is a step toward a net benefit comparison. | 2/5 |
| Does the analysis identify the cost-effectiveness of each alternative considered? Since outcomes were not usually quantified, no analysis of cost-effectiveness was done. | 0/5 |
| Does the analysis identify all parties who would bear costs and assess the incidence of costs? The RIA calculates total value of oil, gas, and coal development; jobs; labor income; and state and local mineral lease revenues. These address only one effect of the rule, on mineral/energy development. We had to infer that these are the main cost categories because the analysis does not explain which values count as costs and which ones count as benefits. Maps and tables suggest how some costs and benefits may be distributed geographically. | 3/5 |
| Does the analysis identify all parties who would receive benefits and assess the incidence of benefits? Values associated with roadless characteristics are listed and commenters who advocated these values are discussed. We had to infer that these are the main benefit categories because the analysis does not explain which values count as costs and which ones count as benefits. The analysis discusses benefits to wildlife and how scenery improves the quality of people's lives. Also some discussion of how industry benefits under each alternative. Maps and tables suggest how some costs and benefits may be distributed geographically. | 2/5 |
Use | |
| 9. Does the proposed rule or the RIA present evidence that the agency used the analysis? The RIA is mentioned in the NPRM but never cited as a reason for any decision. The proposed rule is a revision of a 2008 proposed rule. The 2008 rule never became final because the state of Colorado decided to revise its petition after soliciting public comments on the proposed federal rule. The NPRM states that the proposed rule reflects public comments and the state's revised petition. The RIA does not appear to have influenced the decisions about the regulation, and the Forest Service does not claim to have used it. | 1/5 |
| 10. Did the agency maximize net benefits or explain why it chose another alternative? The RIA provides some basis for assessing net benefits qualitatively. However, neither the RIA nor the NPRM explains how the Forest Service made tradeoffs among the various values. No basis is explained for making the changes from 2008, other than that Colorado revised its petition in response to public comments. Clearly, analysis of net benefits played no role here. | 1/5 |
| 11. Does the proposed rule establish measures and goals that can be used to track the regulation's results in the future? No measures or goals were established. With some additional work and decisions about priorities, the RIA could be used to establish measures and goals, but measures and goals are not immediately obvious. | 0/5 |
| 12. Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so? The RIA contains no discussion of data to be used to measure future performance. The RIA suggests that the Forest Service may have access to data on some but by no means all outcomes/values affected by the regulation. These could be used to determine whether the rule is having the desired effects. | 1/5 |
| Total | 19 / 60 |
Additional details
- Agency
- Department of Agriculture
- Regulatory Identification Number
- 0596-AC74
- Agency Name
- USDA
- Rule Publication Date
- 04/15/2011
- Comment Closing Date
- 07/14/2011