Score: 25 / 60
RULE SUMMARY
The Food Safety and Inspection Service (FSIS) is proposing regulations requiring continuous inspection of catfish and catfish products. FSIS is proposing these regulations to implement provisions of the Food, Conservation, and Energy Act (Farm Bill) of 2008. The proposed regulations are intended to ensure that catfish products distributed in commerce are wholesome; not adulterated; and properly marked, labeled, and packaged.
MONETIZED COSTS & BENEFITS (AS REPORTED BY AGENCY)
| Dollar Year | 2004 | |
| Time Horizon (Years) | 10 | |
| Discount Rates | 3% | 7% |
| Expected Costs (Annualized) | $10.6 million | |
| Expected Benefits (Annualized) | ||
| Expected Costs (Total) | ||
| Expected Benefits (Total) | ||
| Net Benefits (Annualized) | ||
| Net Benefits (Total) |
METHODOLOGY
There are twelve criteria within our evaluation within three broad categories: Openness, Analysis and Use. For each criterion, the evaluators assign a score ranging from 0 (no useful content) to 5 (comprehensive analysis with potential best practices). Thus, each analysis has the opportunity to earn between 0 and 60 points.
| Criterion | Score |
|---|---|
Openness | |
| 1. How easily were the RIA , the proposed rule, and any supplementary materials found online? On regulations.gov and usdoj.gov, the proposed regulation can be found easily via an RIN or keyword search. The NPRM has a link to the RIA and to the risk assessment, which is necessary to fully understand the benefits of the rule. A link to the NPRM can also be found on the USDA web page by searching for "catfish inspection" and then clicking on the link to the Office of Catfish Inspection. | 5/5 |
| 2. How verifiable are the data used in the analysis? FSIS use of data is largely limited to calculating the cost of the proposed rule. However, this data consisted of wholesale price information that did not clearly cite the source except to say it came from the Catfish Market Statistics. Wage rates used in the cost calculation are sourced to the analysis for another rule with no explicit document cited. The few government statistical sources used are linked. Many assumptions about the time needed to perform tasks are based on "expert opinion" or just stated as assumptions with no source given. The risk assessment often uses USDA poultry data because salmonella data for catfish are not available. Risk assessment data are sourced and usually linked. | 2/5 |
| 3. How verifiable are the models and assumptions used in the analysis? When academic or government studies are used, citations are given, occasionally with links. Models in the risk assessment are sourced to academic studies, which are fully cited but usually not linked. The analysis used an @RISK stochastic simulation model to conduct an uncertainty analysis on salmonella illnesses. The sensitivity analysis rests on the assumption that implications of avoiding salmonella illnesses can be drawn upon for other possible, although citation for this assumption is not provided. | 4/5 |
| 4. Was the analysis comprehensible to an informed layperson? Frequent use of some unfamiliar jargon and acronyms made the analysis difficult to follow in places. A great deal of data on the catfish industry are presented in narrative form without a clear indication of the significance of all of the detail provided; a summary table would be better. The risk assessment is quite technical, but an expert could follow it. | 3/5 |
Analysis | |
| 5. How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them? | 3/5 |
| Does the analysis clearly identify ultimate outcomes that affect citizens’ quality of life? An outcome would be a reduction in illnesses, due to a reduction in salmonella, E. coli, and other possible contaminants. | 5/5 |
| Does the analysis identify how these outcomes are to be measured? The analysis discusses a reduction in illnesses and associated monetized illness-related costs. The risk assessment calculates a range of possible reductions in salmonella illnesses. The RIA performs a break-even analysis but does not explicitly project how many illnesses the regulation might prevent or explain why the numbers from the risk assessment were not used. At one point, the analysis notes it is difficult to attribute disease outbreaks to catfish or to other sources. Piecing these two documents together, it is unclear how outcomes could be measured and monetized. | 3/5 |
| Does the analysis provide a coherent and testable theory showing how the regulation will produce the desired outcomes? The general chain of causation is that verification of an establishment's food-safety system leads to a reduction in pathogens, which reduces illnesses. No theory is articulated that explains how the specific provisions would assist in achieving the identified outcomes. Analysis asserts that compliance rates for many steps are already high, but effectiveness depends on how well all of the elements are performed. This might be a step toward explaining how the regulation produces results, but it is not really explained. | 1/5 |
| Does the analysis present credible empirical support for the theory? The RIA does not. The risk assessment presents statistics showing that salmonella contamination in poultry fell by slightly more than half after HACCP regulations were implemented, but it notes that other factors may explain part of this change. It is also not clear whether poultry processors had voluntary programs pre-HACCP similar to those employed by catfish processors. In sum, there may be some evidence suggesting the regulation will be effective, but the size of the effect is not clear. The sensitivity analysis provides information on how benefits may accrue depending on the effectiveness of the proposed rule, but no probability of such effectiveness is provided. | 2/5 |
| Does the analysis adequately assess uncertainty about the outcomes? The risk assessment acknowledges substantial uncertainties about the effectiveness of inspection. It estimates a range of results assuming inspection is 10 percent, 50 percent, and 90 percent effective and assuming that it takes varied numbers of years for inspection to achieve these levels of effectiveness. No argument is provided for how likely any given level of effectiveness would be. Sensitivity analysis was conducted for many of the data inputs. | 3/5 |
| 6. How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve? | 0/5 |
| Does the analysis identify a market failure or other systemic problem? "Need for regulatory action" section simply states that the regulation is necessary to implement a mandate in the 2008 Farm Bill. No specific market failure or big health problem is articulated by FSIS. The health problem provided is anecdotally based. FSIS reports about 66 illnesses and 8 hospitalizations since 1990. According to the risk assessment, there are either 1,764 or 2,308 salmonellosis cases (depending on the definition of catfish used), but the average probability of illness is 1.5 x 10^-6 per serving. | 0/5 |
| Does the analysis outline a coherent and testable theory that explains why the problem (associated with the outcome above) is systemic rather than anecdotal? No theory of a systemic problem is presented. Indeed, the risk assessment notes that salmonella illness from catfish is rare because catfish is cooked before being eaten. | 0/5 |
| Does the analysis present credible empirical support for the theory? Much evidence presented suggests no systemic problem. About 18 out of 23 catfish slaughtering facilities already pay the National Marine Fisheries Service for inspections. All 23 have HACCP plans. Buyers and independent third parties also conduct reviews and audits of production processes. There have been just 7 catfish-related disease outbreaks with 66 illnesses between 1973 and 2007. The risk analysis states that salmonella illness from catfish is an "uncommon event," there has been one suspected outbreak in the past 20 years, which made 10 people ill. NPRM cites some other instances in which contaminated catfish may have caused illness, but the evidence is not clear. | 0/5 |
| Does the analysis adequately assess uncertainty about the existence or size of the problem? Since a systemic problem was not really articulated, there is no discussion of uncertainty about it either. | 0/5 |
| 7. How well does the analysis assess the effectiveness of alternative approaches? | 2/5 |
| Does the analysis enumerate other alternatives to address the problem? The RIA lists as "alternatives" two alternative inspection schedules: an inspector continuously present in the plant, or an inspector visits each day. The NPRM states that the USDA considered and rejected a command-and-control approach. Such a system would involve using specific programs, techniques, equipment, and so forth. Instead, FSIS claims that the proposed rule has used the proposal that focuses more on the verification of an establishment's food-safety system. However, this distinction is not clear as various sectors of the catfish industry are required to make adjustments in sanitation, label approval, documentation, and recordkeeping. There are also two different "scenarios" that cover larger or smaller numbers of fish based on alternative definitions of "catfish." | 3/5 |
| Is the range of alternatives considered narrow (e.g., some exemptions to a regulation) or broad (e.g., performance-based regulation vs. command and control, market mechanisms, nonbinding guidance, information disclosure, addressing any government failures that caused the original problem)? These are narrow tweaks on the same regulatory approach. The difference between the command-and-control alternative and the proposed rule is unclear. | 2/5 |
| Does the analysis evaluate how alternative approaches would affect the amount of the outcome achieved? The RIA does not estimate the amount of outcome for the regulation or any of the alternatives. The breakeven analysis is conducted for both species scenarios, but not for the inspection alternatives that are actually listed in the "alternatives" section. This is also the case for the estimates of illnesses in the risk assessment. | 1/5 |
| Does the analysis adequately address the baseline? That is, what the state of the world is likely to be in the absence of federal intervention not just now but in the future? The analysis notes (on p. 51), "The emphasis on the use of client-driven food safety requirements and third party audits would ensure that the share of establishments with verified, effective food safety management practices would further increase over time." But legislation requires a rulemaking. Since outcomes are not calculated in the RIA, there is no real outcome baseline. The RIA acknowledges one important factor that would affect the baseline but does not really follow through with it in an assessment of outcomes. The risk assessment calculates a salmonella illness baseline based on poultry data prior to adoption of HACCP regulations for poultry. There is no discussion of whether poultry processors used voluntary inspections at the time, so it is not clear if this is a comparable baseline. | 1/5 |
| 8. How well does the analysis assess costs and benefits? | 2/5 |
| Does the analysis identify and quantify incremental costs of all alternatives considered? Cost calculations attempt to identify compliance costs that are incremental to what many producers are already doing. Costs are calculated for the two species scenarios but not for the two alternatives listed in the "alternatives" section. | 3/5 |
| Does the analysis identify all expenditures likely to arise as a result of the regulation? Calculations cover labor costs, equipment costs, and office space in plants provided for government inspectors. Cost of office space includes remodeling, utilities, and so forth, but not opportunity cost. Similarly, import reinspection calculations seem to ignore opportunity costs. Effects on foreign producers not calculated. | 4/5 |
| Does the analysis identify how the regulation would likely affect the prices of goods and services? The analysis estimates that the rule would increase costs by about 1/10 of 1 cent per pound, compared to wholesale prices of 40 cents–$1.14 per pound. (This may be misleading since many firms are already close to being in compliance, and so the cost per pound will vary by firm.) There is no discussion of whether or how this might be passed through to consumers. | 3/5 |
| Does the analysis examine costs that stem from changes in human behavior as consumers and producers respond to the regulation? There is no discussion of a decrease in catfish produced due to higher costs. The RIA does not examine the effect of the regulation on foreign catfish supply or prices. Additionally, no examination of the consumer response to higher prices is considered. | 0/5 |
| If costs are uncertain, does the analysis present a range of estimates and/or perform a sensitivity analysis? The cost analysis asserts a range of figures for 10th and 90th percentiles but does not explain how they were calculated. | 1/5 |
| Does the analysis identify the alternative that maximizes net benefits? Benefits are not projected, so net benefits were not calculated. A break-even analysis shows how many cases of salmonella would have to be prevented to justify the costs, but only for the alternative chosen. | 1/5 |
| Does the analysis identify the cost-effectiveness of each alternative considered? Cost-effectiveness is not calculated because avoided illnesses were not projected. Instead, a break-even analysis was conducted to find the necessary amount of illnesses averted to have the benefits equal the costs. | 2/5 |
| Does the analysis identify all parties who would bear costs and assess the incidence of costs? The analysis divides the incidence of cost between Ictaluridae and Siluriformes catfish. The analysis calculates costs to small businesses but concludes the effect is not substantial because it is a small percentage of revenues. No division of costs is considered among domestic and foreign catfish providers. Although a higher price per pound of catfish is provided, the analysis does not offer a clear examination of whether consumers will pay higher catfish prices. The RIA notes in passing that the regulation would reduce costs for firms currently paying the FSIS for inspections, and this cost is a large share of the estimated costs. This suggests the regulation will create transfers between firms that are not documented in the analysis. | 2/5 |
| Does the analysis identify all parties who would receive benefits and assess the incidence of benefits? The RIA mentions that per-capita catfish consumption is higher in the South but does not follow up on this to examine differential incidence of benefits. There is no discussion of how the ultimate benefits would be broken down among different segments of the population. Total benefits were not really estimated anyway. | 1/5 |
Use | |
| 9. Does the proposed rule or the RIA present evidence that the agency used the analysis? Use of statistics in the proposed rule leads the reader to believe that cost-benefit analysis played a minor rule in publishing the proposed rule. Depending on whether catfish are defined as Siluriformes or Ictaluridae, the proposed rule would have to avert roughly 790 or 590 illnesses, respectively, over a ten-year time frame. These benefits seem unlikely considering there have been only 66 illnesses in the last 21 years. The RIA suggests that baseline improvements had to be ignored when drafting the proposed rule because "the alternative of no rulemaking is unavailable since the 2008 Farm Bill requires FSIS to issue regulations for mandatory inspection of catfish and catfish products." The Department has discretion in defining catfish. The NPRM claims that the USDA considered its risk assessment and benefit-cost analysis, but it does not explain how these affected any specific decisions. | 2/5 |
| 10. Did the agency maximize net benefits or explain why it chose another alternative? Net benefits were not calculated. The RIA did perform a break-even analysis but not really to evaluate alternatives. It says the break-even analysis "illustrates" potential benefits, but it does not offer an assessment of how plausible the result is or whether that justifies the regulation. | 1/5 |
| 11. Does the proposed rule establish measures and goals that can be used to track the regulation's results in the future? No commitment to goals or measures. Some goals or measures could be developed based on the methods in the "illustrative" risk assessment, but it would take some additional work to do this. | 0/5 |
| 12. Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so? The risk assessment mentions that relevant data on catfish are scarce but suggests that the inspection regime would facilitate collection of data that could be used to evaluate the regulation's effects and assess options in the future. The data for illnesses and hospitalizations were collected by the Center for Disease Control and presumably could be tracked. | 1/5 |
| Total | 25 / 60 |
Additional details
- Agency
- Department of Agriculture
- Regulatory Identification Number
- 0583-AD36
- Agency Name
- Department of Agriculture
- Rule Publication Date
- 02/24/2011
- Comment Closing Date
- 06/24/2011
- Dollar Year
- 2004
- Time Horizon (Years)
- 10