Score: 19 / 60
RULE SUMMARY
The Secretary of Education (Secretary) proposes priorities, requirements, definitions, and selection criteria under the Investing in Innovation Fund. The Secretary may use these priorities, requirements, definitions, and selection criteria for competitions of the Investing in Innovation Fund for fiscal year (FY) 2010 and later years. We intend for the priorities, requirements, definitions, and selection criteria to support the efforts of local educational agencies (LEAs) and nonprofit organizations (as defined in this notice) that have strong track records of improving student achievement (as defined in this notice) to expand their work; identify, document, and share best practices; and take successful practices ‘‘to scale.’’
METHODOLOGY
There are twelve criteria within our evaluation within three broad categories: Openness, Analysis and Use. For each criterion, the evaluators assign a score ranging from 0 (no useful content) to 5 (comprehensive analysis with potential best practices). Thus, each analysis has the opportunity to earn between 0 and 60 points.
| Criterion | Score |
|---|---|
Openness | |
| 1. How easily were the RIA , the proposed rule, and any supplementary materials found online? The RIA is part of the preamble. It can be found on regulations.gov using the RIN. A keyword search requires checking multiple documents and folders. On the Department of Education's website, a keyword search of "Investing in Innovation" leads to a page of information about this program, but not to the actual RIA. Also, a RIN search from the Department of Education's homepage brings up a link to the actual preamble that contains the RIA. | 4/5 |
| 2. How verifiable are the data used in the analysis? There are no "data" in the analysis that informs the decisions. It appears some of the decisions are based on research which is cited, but not linked. The analysis of the effects on small entities does cite some figures showing the number of organizations involved and mentions the sources. | 1/5 |
| 3. How verifiable are the models and assumptions used in the analysis? The preamble cites research supporting four of the eight funding priorities. There are no models/assumptions in the RIA section. | 3/5 |
| 4. Was the analysis comprehensible to an informed layperson? The preamble and RIA are reasonably understandable. However, the RIA contains no analysis of alternatives; it simply asserts that the chosen alternatives are the best. One must read the entire preamble and then infer why the chosen requirements are the best. | 3/5 |
Analysis | |
| 5. How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them? | 2/5 |
| Does the analysis clearly identify ultimate outcomes that affect citizens’ quality of life? "The purpose of the program is to provide competitive grants to applicants with a record of improving student achievement, in order to expand the implementation of, and investment in, innovative practices that are demonstrated to have an impact on improving student achievement or student growth (as defined in this notice) for high-need students (as defined in this notice), as well as to promote school readiness, close achievement gaps, decrease dropout rates, increase high school graduation rates, and improve teacher and school leader effectiveness." This statement implicates many important outcomes. | 5/5 |
| Does the analysis identify how these outcomes are to be measured? It is clear that many of these otucomes are measurable and are measured. The analysis, however, does not explicitly measure them for the purpose of projecting how the regulation will affect the outcomes. | 3/5 |
| Does the analysis provide a coherent and testable theory showing how the regulation will produce the desired outcomes? The regulation does not explicitly state but assumes that the government funds can incentivize groups to innovate new ways of educating students, and those innovations will improve student performance. The preamble occasionally provides some brief logic, but nothing extensive. | 2/5 |
| Does the analysis present credible empirical support for the theory? The regulation provides citations of various articles that discuss what variables affect student performance but does not provide evidence of how federal funding affects those variables. So, evidence is provided that verfies some links in the chain of causation but not all. | 2/5 |
| Does the analysis adequately assess uncertainty about the outcomes? No relevant discussion. | 0/5 |
| 6. How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve? | 1/5 |
| Does the analysis identify a market failure or other systemic problem? These more specific guidelines are necessary to "focus on the educational reform and innovation activities most likely to raise student achievement and eliminate persistent disparities in achievement across different populations of students." Not much context provided to help the reader understand why this is a big problem or why it is a systemic problem. | 1/5 |
| Does the analysis outline a coherent and testable theory that explains why the problem (associated with the outcome above) is systemic rather than anecdotal? No explicit theory is presented. The nature of some of the reforms suggests there might be some systemic problems. | 1/5 |
| Does the analysis present credible empirical support for the theory? Some of the cited research might suggest that there is a systemic problem somewhere. | 0/5 |
| Does the analysis adequately assess uncertainty about the existence or size of the problem? No relevant discussion. | 0/5 |
| 7. How well does the analysis assess the effectiveness of alternative approaches? | 0/5 |
| Does the analysis enumerate other alternatives to address the problem? The regulation defines the priorities for the Innovation fund and states that other definitions were considered, but the regulation does not enumerate the other options. | 1/5 |
| Is the range of alternatives considered narrow (e.g., some exemptions to a regulation) or broad (e.g., performance-based regulation vs. command and control, market mechanisms, nonbinding guidance, information disclosure, addressing any government failures that caused the original problem)? It is impossible to tell because no alternatives are enumerated, except for awarding grants under the department's customer criteria, which are not specified in the preamble or the RIA. | 0/5 |
| Does the analysis evaluate how alternative approaches would affect the amount of the outcome achieved? No relevant discussion. | 0/5 |
| Does the analysis adequately address the baseline? That is, what the state of the world is likely to be in the absence of federal intervention not just now but in the future? No relevant discussion. | 0/5 |
| 8. How well does the analysis assess costs and benefits? | 1/5 |
| Does the analysis identify and quantify incremental costs of all alternatives considered? It identifies total federal funding only and does not examine costs of alternatives. | 1/5 |
| Does the analysis identify all expenditures likely to arise as a result of the regulation? The regulation describes the costs that organizations will expend applying for funds and states the amount of funds available to grant. | 4/5 |
| Does the analysis identify how the regulation would likely affect the prices of goods and services? No relevant discussion. | 0/5 |
| Does the analysis examine costs that stem from changes in human behavior as consumers and producers respond to the regulation? No relevant discussion. | 0/5 |
| If costs are uncertain, does the analysis present a range of estimates and/or perform a sensitivity analysis? No relevant discussion. | 0/5 |
| Does the analysis identify the alternative that maximizes net benefits? The analysis asserts that the proposed requirements are better than the status quo, using logic suggesting the net benefits would be higher, but this is not really developed or substantiated. | 1/5 |
| Does the analysis identify the cost-effectiveness of each alternative considered? There is no relevant discussion in the RIA. The preamble mentions that the department would like to fund proposals that are more cost-effective. | 1/5 |
| Does the analysis identify all parties who would bear costs and assess the incidence of costs? The regulation estimates how many local education agencies and nonprofits will apply. It is those parties who will bear the cost of applying. The sole discussion of incidence is this assertion: "The costs of carrying out activities would be paid for with program funds and with matching funds provided by private-sector partners. Thus, the costs of implementation would not be a burden for any eligible applicants, including small entities." | 3/5 |
| Does the analysis identify all parties who would receive benefits and assess the incidence of benefits? Grants are supposed to focus on high-need students. There is no other discussion of incidence of benefits and no estimate of how many recipients will receive grants. | 1/5 |
Use | |
| 9. Does the proposed rule or the RIA present evidence that the agency used the analysis? The proposal has very little analysis. The department appears to have used some research to identify some of its preferred requirements. The proposal appears to reflect the department's strong desire to use evidence-based criteria to award funds, but the notice and RIA do a very poor job of presenting the research that led the department to its decisions, so it's hard to tell. | 1/5 |
| 10. Did the agency maximize net benefits or explain why it chose another alternative? The RIA does not calculate benefits or net benefits. The department does express a preference for funding proposals that are more cost-effective. | 1/5 |
| 11. Does the proposed rule establish measures and goals that can be used to track the regulation's results in the future? Grant recipients must cooperate with any evaluations the department decides to conduct. This suggests an interest in conducting some type of retrospective analysis. | 2/5 |
| 12. Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so? There is no explicit discussion, and little in the RIA that an evaluation could be based on. | 0/5 |
| Total | 19 / 60 |
Additional details
- Agency
- Department of Education
- Regulatory Identification Number
- 1855-AA06
- Agency Name
- Department of Education
- Rule Publication Date
- 10/09/2009
- Comment Closing Date
- 11/09/2009