- | Regulation Regulation
- | Policy Briefs Policy Briefs
- |
Flightcrew Member Duty and Rest Requirements
Proposed Rule
Score: 26 / 60
RULE SUMMARY
The FAA is proposing to amend its existing flight, duty and rest regulations applicable to certificate holders and their flightcrew members. The proposal recognizes the growing similarities between the types of operations and the universality of factors that lead to fatigue in most individuals. Fatigue threatens aviation safety because it increases the risk of pilot error that could lead to an accident. The new requirements, if adopted, would eliminate the current distinctions between domestic, flag and supplemental operations. The proposal provides different requirements based on the time of day, whether an individual is acclimated to a new time zone, and the likelihood of being able to sleep under different circumstances.
METHODOLOGY
There are twelve criteria within our evaluation within three broad categories: Openness, Analysis and Use. For each criterion, the evaluators assign a score ranging from 0 (no useful content) to 5 (comprehensive analysis with potential best practices). Thus, each analysis has the opportunity to earn between 0 and 60 points.
| Criterion | Score |
|---|---|
Openness | |
| 1. How easily were the RIA , the proposed rule, and any supplementary materials found online? Keyword and RIN searches on regulations.gov turn up the Federal Register notice. It is misfiled under "rules" rather than "proposed rules." The Federal Register notice says the RIA is in the docket, and it is. A link to the notice is also 4 clicks from the DOT home page (select "Federal Aviation Administration," then "rulemaking," then "recently published rulemaking documents." | 4/5 |
| 2. How verifiable are the data used in the analysis? Six airlines furnished confidential data on flight crew scheduling to assist in modeling costs. Accident data used to estimate benefits apparently come from federal sources. Explicit source documents rarely indicated for data. Many recitations of National Transportation Safety Board findings that ultimately reflect the board's judgment, so they would be hard to test. | 2/5 |
| 3. How verifiable are the models and assumptions used in the analysis? Several literature reviews on fatigue are cited in the RIA. The Federal Register notice cites more studies and says a complete bibliography is in the docket. The analysis presents simulation results, but it is not clear how the simulations were done or whether their assumptions are reasonable. Provisions of the proposed regulation were "scored" for their ability to prevent accidents, but the scoring method is a black box with no citation to explanations. | 2/5 |
| 4. Was the analysis comprehensible to an informed layperson? The exposition, while generally good, seemed jargony at times and was overlong in dealing with anecdotes. Analysis offers a straightforward explanation of how costs were estimated. Accident calculations are explained, but a little hard to follow. More technical statistical analyses (but not explanations of simulations) are relegated to appendices so the lay reader doesn't have to struggle through them. | 4/5 |
Analysis | |
| 5. How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them? | 3/5 |
| Does the analysis clearly identify ultimate outcomes that affect citizens’ quality of life? Reduction in fatalities, injuries, and property damage due to a reduction in fatigue-related aviation accidents. | 5/5 |
| Does the analysis identify how these outcomes are to be measured? The number of accidents and dollar value of avoided accident costs are quantified. How the accident costs are divided between fatalities, injuries, and property damage is not clear. | 3/5 |
| Does the analysis provide a coherent and testable theory showing how the regulation will produce the desired outcomes? Fatigue causes accidents, so adoption of regulations intended to prevent fatigue will reduce accidents. The regulation attempts to impose monitoring and training requirements on carriers and crew to control crew behavior (eg, staying up late the night before a flight). The scoring system that assesses the regulation's effectiveness in preventing accidents is a black box, so it is unclear whether that system is based on coherent theories that take human behavior into account. An important causality issue is unaddressed: Fatigued pilots were found to be more likely to be involved in accidents but is it not at least plausible that pilots who are predisposed to fly when fatigued are also less diligent in general? | 4/5 |
| Does the analysis present credible empirical support for the theory? Instead of empirical analysis, the regulation is "scored," according to scoring systems DOT claims are widely-recognized, to estimate the % of accidents it would have prevented. International standards are mentioned in the Federal Register notice, but no analysis of their effects is presented. | 1/5 |
| Does the analysis adequately assess uncertainty about the outcomes? Simulations estimate a range of possible future accidents and accident-related costs. They are run using data on fatigue-related accidents and all accidents. High, low, and "best" estimates are presented. The analysis also presents alternative benefit figures using alternative values of a statistical life. However, this uncertainty analysis is not documented very well, so it is hard to assess its quality. | 4/5 |
| 6. How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve? | 1/5 |
| Does the analysis identify a market failure or other systemic problem? No explanation of why airlines or crews take fatigue-related risks, given airlines' liability and crews' direct exposure to danger. Thus, it is not clear whether this is a systemic problem or the result of a few "bad actors." | 0/5 |
| Does the analysis outline a coherent and testable theory that explains why the problem (associated with the outcome above) is systemic rather than anecdotal? No theory explains airlines' or crews' choices. Both are (apparently) making bad decisions for unexplained reasons. Does scheduling of takeoff/landing slots play any role here, and if so do other regulations affect that? | 0/5 |
| Does the analysis present credible empirical support for the theory? Evidence is cited that fatigue impairs performance, and anecdotal evidence shows that fatigue has caused some accidents. Analysis recites accounts of accidents in which officials determined fatigue was a cause. Analysis compares accident rates in first ten hours of crew duty with higher rates beyond tenth hour. It also shows there is a higher accident rate between midnight and 4:00 am. But since there was no coherent theory explaining airlines' and crews' decisions, there was no behavioral theory to test. | 2/5 |
| Does the analysis adequately assess uncertainty about the existence or size of the problem? Some of the benefit simulations can be interpreted as acknowledging uncertainty about the size of the problem. | 2/5 |
| 7. How well does the analysis assess the effectiveness of alternative approaches? | 3/5 |
| Does the analysis enumerate other alternatives to address the problem? The analysis considers several more restrictive scenarios. The Federal Register notice mentions several other suggestions from interest groups, but they are not analyzed. | 4/5 |
| Is the range of alternatives considered narrow (e.g., some exemptions to a regulation) or broad (e.g., performance-based regulation vs. command and control, market mechanisms, nonbinding guidance, information disclosure, addressing any government failures that caused the original problem)? These alternatives involve small deviations from the basic regulation. | 2/5 |
| Does the analysis evaluate how alternative approaches would affect the amount of the outcome achieved? No. It assumes all alternatives would be equally effective and only evaluates costs. | 2/5 |
| Does the analysis adequately address the baseline? That is, what the state of the world is likely to be in the absence of federal intervention not just now but in the future? National Transportation Safety Board collects data on crashes, but the agency goes forward on the basis that crashes will occur at the same rate as before which, if wrong, would make it very hard to measure the number of crashes that didn't happen because of the rule. Based on historical data, the analysis projects the number of fatigue-related accidents that could be expected during the next 20 years. This implicitly assumes the next 20 years will repeat historical experience in the absence of the regulation. Simulations are run to estimate a range of possible accidents and accident-related costs. | 2/5 |
| 8. How well does the analysis assess costs and benefits? | 2/5 |
| Does the analysis identify and quantify incremental costs of all alternatives considered? Costs appear to be incremental. They are calculated for the chosen alternative and several other tweaks. For some reason, the calculations are in the regulatory analysis section of the Federal Register notice but not in the RIA. | 3/5 |
| Does the analysis identify all expenditures likely to arise as a result of the regulation? The RIA incorporates evidence provided by airlines, which would indicate a reasonably comprehensive list of costs. Analysis identifies four types of costs: flight operations, scheduling reliability, fatigue training, and cost of rest facilities. The last includes forgone passenger revenue from seats used as rest facilities. | 5/5 |
| Does the analysis identify how the regulation would likely affect the prices of goods and services? No relevant discussion. Analysis stops at how airlines and flight crews are affected. In addition to cost pass-through to consumers, there may be effects on competition. Larger airlines may have different costs than smaller airlines, and if larger airlines (which have more scheduling flexibility) benefit, there could be unanticipated costs from reducing competition. | 0/5 |
| Does the analysis examine costs that stem from changes in human behavior as consumers and producers respond to the regulation? The cost analysis reflects scheduling changes airlines would make in response to the regulation. No discussion of costs passed through to consumers. This may be especially important if the costs are passed through to consumers, reducing air travel and potentially prompting some passengers to substitute auto for air travel. | 2/5 |
| If costs are uncertain, does the analysis present a range of estimates and/or perform a sensitivity analysis? A few acknowledgements of some cost uncertainties. No formal analysis or alternative estimates that reflect cost uncertainty. | 1/5 |
| Does the analysis identify the alternative that maximizes net benefits? Net benefits could be calculated from the benefit and cost information. However, benefits were assumed to be identical for all alternatives. | 2/5 |
| Does the analysis identify the cost-effectiveness of each alternative considered? Cost-effectiveness is not explicitly addressed. Effectiveness of alternatives is assumed to be identical, so higher-cost alternatives are automatically less cost-effective. | 1/5 |
| Does the analysis identify all parties who would bear costs and assess the incidence of costs? Airlines are assumed to bear all costs. Some costs are calculated separately for different industry segments (commercial passenger, large cargo, etc.). The analysis offers some qualitative reasons the regulation may disproportionately affect small entities, but asserts that no alternatives would accomplish the goals of the regulation. No discussion of how costs might be passed through to consumers or shifted to workers. | 2/5 |
| Does the analysis identify all parties who would receive benefits and assess the incidence of benefits? No significant analysis of benefit incidence. The analysis mentions that the regulations might result in a transfer from airlines to flight crews, because crews will get more rest. Not clear if this will happen or if other contract terms will adjust to cancel out this benefit to crews. | 1/5 |
Use | |
| 9. Does the proposed rule or the RIA present evidence that the agency used the analysis? Regulation appears motivated primarily by the findings of a review committee established to make recommendations for this regulation. These were likely influenced by the research on fatigue and performance mentioned in the RIA. The regulatory review section of the Federal Register notice considers some more stringent alternatives that are more expensive but simply assumed to be just as effective. Not clear if these were seriously considered or added on after the decisions had been made. | 2/5 |
| 10. Did the agency maximize net benefits or explain why it chose another alternative? Net benefits (actually negative) could be calculated for the chosen alternative, though the figure is not explicitly given. Costs were calculated for some marginal alternatives that were assumed to be equally effective; it is not clear how reliable any resulting net benefit figures would be. Little or no evidence that net benefits were a serious consideration. | 1/5 |
| 11. Does the proposed rule establish measures and goals that can be used to track the regulation's results in the future? No measures or goals established. National Transportation Safety Board will continue to track fatigue-related accidents, but this is distinct from tracking the outcome of the rule. The FAA makes no provision for that. | 1/5 |
| 12. Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so? Airlines must report some data to ensure they are complying with the regulation, but the Federal Register notice makes no explicit provision to gather data to evaluate the regulation's effects. Future accident data could be used to measure effects. | 1/5 |
| Total | 26 / 60 |
Additional details
- Agency
- Department of Transportation
- Regulatory Identification Number
- 2120-AJ58
- Agency Name
- Department of Transportation
- Rule Publication Date
- 09/14/2010
- Comment Closing Date
- 11/15/2010