Score: 23 / 60
RULE SUMMARY
The Secretary of Education proposes priorities, requirements, definitions, and selection criteria for the Race to the Top Fund. These priorities, requirements, definitions, and selection criteria may be used in any year in which this program is in effect.
The Secretary of Education proposes priorities, requirements, definitions, and selection criteria for the Race to the Top Fund. These priorities, requirements, definitions, and selection criteria may be used in any year in which this program is in effect.
METHODOLOGY
There are twelve criteria within our evaluation within three broad categories: Openness, Analysis and Use. For each criterion, the evaluators assign a score ranging from 0 (no useful content) to 5 (comprehensive analysis with potential best practices). Thus, each analysis has the opportunity to earn between 0 and 60 points.
| Criterion | Score |
|---|---|
Openness | |
| 1. How easily were the RIA , the proposed rule, and any supplementary materials found online? 1810-AB07 can be found on regulations.gov using the RIN and a keyword search, as well as on the Department of Education website. On the Department of Education website, the regulation can be found by entering either a keyword or the RIN into the search box on the top right section of their home page. In the search results, the Department of Education provides a direct link to the RIA and the rule associated with it. | 5/5 |
| 2. How verifiable are the data used in the analysis? Very little data is used to assess the main results of the proposal. | 0/5 |
| 3. How verifiable are the models and assumptions used in the analysis? The analysis cites several peer-reviewed studies in support of the claim that teacher quality affects student achievement, but teacher credentials are not a good predictor of quality. This claim supports the requirement that states cannot have legal barriers to using student performance to evaluate teacher quality. Most other claims are unsupported by theory or empirical research. | 2/5 |
| 4. Was the analysis comprehensible to an informed layperson? The proposal is fairly readable. There are only a few unfamiliar acronyms. Some of the layers of specific requirements get quite detailed and are hard to keep straight. Since there was very little actual analysis, it is hard to say how easily the analysis is to comprehend. The reader cannot tell what reasoning/evidence led the department to the conslusions that are reflected in the proposal. | 2/5 |
Analysis | |
| 5. How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them? | 2/5 |
| Does the analysis clearly identify ultimate outcomes that affect citizens’ quality of life? Improved student academic achievement, including specifically targeted groups of students (economically disadvantaged students, students from major racial and ethnic groups, students with disabilities, and students with limited English proficiency). | 4/5 |
| Does the analysis identify how these outcomes are to be measured? Some performance measures refer to the National Association of Educational Progress; others refer to yet-to-be-developed measures of student achievement or "growth." The analysis does not actually project how much the regulation or the spending will improve outcomes, by whatever measure. | 3/5 |
| Does the analysis provide a coherent and testable theory showing how the regulation will produce the desired outcomes? The proposal claims that requiring states to undertake certain reform strategies will maximize the chances of improved student achievement. In most cases; this is simply asserted with no elaboration of underlying theory. | 1/5 |
| Does the analysis present credible empirical support for the theory? Substantial empirical research is cited only to support the claim that high-quality teachers make a big difference, but teacher credentials are not correlated with high quality. The analysis would be much better if this kind of research were cited in support of all fo the proposal's claims. | 2/5 |
| Does the analysis adequately assess uncertainty about the outcomes? The department's main goal in establishing performance measures appears to be monitoring the states' performance, rather than evaluating the effectiveness of this program's expenditures. This is an implicit admission that results are uncertain. There is no formal analysis of uncertainty. | 1/5 |
| 6. How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve? | 1/5 |
| Does the analysis identify a market failure or other systemic problem? Unlike other regulations issued in 2009 governing education grants, this one does not make much of a case that the problem is large. The nature of the school reforms the department is trying to promote could be taken as a claim that education problems are systemic (eg, related to institutional structure, method, widespread practices, etc. rather than just anecdotal). There is no explanation of why the nature of the problem requires a federal response. | 1/5 |
| Does the analysis outline a coherent and testable theory that explains why the problem (associated with the outcome above) is systemic rather than anecdotal? The need for and effectiveness of most of the reforms are simply asserted. Many of the reforms do address systemic/institutional issues. | 1/5 |
| Does the analysis present credible empirical support for the theory? No relevant discussion. | 0/5 |
| Does the analysis adequately assess uncertainty about the existence or size of the problem? No relevant discussion. | 0/5 |
| 7. How well does the analysis assess the effectiveness of alternative approaches? | 1/5 |
| Does the analysis enumerate other alternatives to address the problem? The only alternative specifically mentioned is giving out the grants without such specific requirements. The proposal claims other requirements were considered, but these are not elaborated or analyzed. | 2/5 |
| Is the range of alternatives considered narrow (e.g., some exemptions to a regulation) or broad (e.g., performance-based regulation vs. command and control, market mechanisms, nonbinding guidance, information disclosure, addressing any government failures that caused the original problem)? The extent of the reforms suggests the requirements may make some difference, but without a good baseline assessment of what states would do in the absence of the requirements, it's hard to tell how much difference they would make. | 1/5 |
| Does the analysis evaluate how alternative approaches would affect the amount of the outcome achieved? Assertions that the requirements are necessary could be interpreted as a claim that there would be less reform without the requirements. | 1/5 |
| Does the analysis adequately address the baseline? That is, what the state of the world is likely to be in the absence of federal intervention not just now but in the future? No clear baseline is discussed. | 0/5 |
| 8. How well does the analysis assess costs and benefits? | 1/5 |
| Does the analysis identify and quantify incremental costs of all alternatives considered? The proposal outlines the federal costs. It mentions states will bear some costs but only quantifies the paperwork burden of applying. | 3/5 |
| Does the analysis identify all expenditures likely to arise as a result of the regulation? It identifies federal expenditures, but not state expenditures. The latter is an important omission given that one of the criteria is whether the state has a larger percent of its budget available for education this year than it had last year. | 2/5 |
| Does the analysis identify how the regulation would likely affect the prices of goods and services? No relevant discussion. | 0/5 |
| Does the analysis examine costs that stem from changes in human behavior as consumers and producers respond to the regulation? No relevant discussion. | 0/5 |
| If costs are uncertain, does the analysis present a range of estimates and/or perform a sensitivity analysis? No relevant discussion. | 0/5 |
| Does the analysis identify the alternative that maximizes net benefits? Benefits are not calculated. | 0/5 |
| Does the analysis identify the cost-effectiveness of each alternative considered? Benefits are not calculated. | 0/5 |
| Does the analysis identify all parties who would bear costs and assess the incidence of costs? Federal taxpayers, states, and local education authorities bear the costs. State costs are not quantified except for the paperwork burden of applying. | 1/5 |
| Does the analysis identify all parties who would receive benefits and assess the incidence of benefits? Students ultimately benefit, and the proposal mentions targeted groups of students several times. But none of this is predicted, measured, or divided up among groups of students. "The benefits conveyed on a State through its receipt of a grant will greatly exceed those costs. In addition, even States that apply but are unsuccessful in the competitions may derive benefits, as the process of working with LEAs and other stakeholders on the State application may help accelerate the pace of education reforms in the State." | 1/5 |
Use | |
| 9. Does the proposed rule or the RIA present evidence that the agency used the analysis? There is very little analysis, so it is hard to conclude that the agency used much of the analysis. The reform proposals may be based on some kind of analysis of the effectiveness of various reforms that is not discussed at all in this document. One requirement—that states may not have legal barriers to use of information on student achievement to evaluate teachers—is directly linked to relevant research. | 2/5 |
| 10. Did the agency maximize net benefits or explain why it chose another alternative? Benefts are not calculated, so net benefits are not calculated either. The proposal identifies how to measure success—student acheivement—but did not estimate how these requirements or the spending would affect student achievement. The focus on reforms the department believes will be most effective could be interpreted as some kind of sensitivity to net benefits. | 1/5 |
| 11. Does the proposed rule establish measures and goals that can be used to track the regulation's results in the future? For criteria that involve future reforms, the proposal enumerates performance measures in some cases and requires states to develop performance measures in other cases. The enumerated performance measures are a mix of outcomes (eg, student achievement) and inputs/processes (eg, state funding for education). States must submit annual reports. States must agree to participate in a national evaluation of the program, if the department decides to conduct one, and the proposal seeks comment on whether states should be required to conduct their own evaluations. This is not quite a commitment to evaluate major outcomes in the future, but it's close. | 3/5 |
| 12. Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so? The indicated performance measures show what types of data states are supposed to supply. The proposal also requires states to make their data available for evaluation of the program. | 3/5 |
| Total | 23 / 60 |
Additional details
- Agency
- Department of Education
- Regulatory Identification Number
- 1810-AB07
- Agency Name
- Department of Education
- Rule Publication Date
- 07/29/2009
- Comment Closing Date
- 08/28/2010