Score: 31 / 60
RULE SUMMARY
The U.S. Secretary of Education proposes requirements for School Improvement Grants authorized under section 1003(g) of Title I of the Elementary and Secondary Education Act of 1965, as amended (ESEA), and funded through both the Department of Education Appropriations Act, 2009 and the American Recovery and Reinvestment Act of 2009 (ARRA). The proposed requirements would define the criteria that a State educational agency (SEA) must use to award school improvement funds to local educational agencies (LEAs) with the lowest-achieving Title I schools that demonstrate the greatest need for the funds and the strongest commitment to use those funds to provide adequate resources to their lowest-achieving Title I schools in order to raise substantially the achievement of the students attending those schools. The proposed requirements also would require an SEA to give priority, through a waiver under section 9401 of the ESEA, to LEAs that also wish to serve the lowest-achieving secondary schools that are eligible for, but do not receive, Title I funds. Finally, the proposed requirements would require an SEA to award school improvement funds to eligible LEAs in amounts sufficient to enable the targeted schools to implement one of four specific proposed interventions.
METHODOLOGY
There are twelve criteria within our evaluation within three broad categories: Openness, Analysis and Use. For each criterion, the evaluators assign a score ranging from 0 (no useful content) to 5 (comprehensive analysis with potential best practices). Thus, each analysis has the opportunity to earn between 0 and 60 points.
| Criterion | Score |
|---|---|
Openness | |
| 1. How easily were the RIA , the proposed rule, and any supplementary materials found online? 1810- AB06 can be found on regulations.gov using the RIN and a keyword search, as well as on the Department of Education website. On the Department of Education website, the regulation can be found by entering either a keyword or the RIN into the search box on the top right section of their home page. In the search results, the Department of Education provides a direct link to the proposed rule, which contains the RIA. | 5/5 |
| 2. How verifiable are the data used in the analysis? There is very little supporting data in the first place; but there are some citations where data is actually used. The analysis cites one report from the Mass Education Insight and Research Institute on the cost of achieving a school turnaround. | 1/5 |
| 3. How verifiable are the models and assumptions used in the analysis? the analysis cites a report from the Mass Education Insight and Research Institute in support of claims that the four proposed interventions are effective. | 2/5 |
| 4. Was the analysis comprehensible to an informed layperson? The entire preamble is fairly easy to understand, except for some very detailed llists of requirements. The amount of actual analysis is very, very small. It is easy to read, but because there is little explanation of analysis, it is not clear whether some of the conclusions are actually justified. | 3/5 |
Analysis | |
| 5. How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them? | 3/5 |
| Does the analysis clearly identify ultimate outcomes that affect citizens’ quality of life? Better education for children of low-income families: "The benefits, then, would be more effective schools serving children from low income families and a better education for those children." | 4/5 |
| Does the analysis identify how these outcomes are to be measured? The regulation lists achievement measures like test scores and graduation rates, plus intermediate ("leading") indicators like discipline incidents and students enrolled in advances coursework. | 5/5 |
| Does the analysis provide a coherent and testable theory showing how the regulation will produce the desired outcomes? Regulations target the funds to low-achieving schools, and the funds must be used for one of four interventions: (1) turnaround (replace principal, at least 50 percent of staff, change governance structure), (2) restart (close school and reopen under management of an organizaiton selected through a rigorous review process, (3) closure (close school and send students to high-achieving schools), (4) transformation (address four areas critical to transforming as school). Causal links between these actions and student achievment should have been explained. | 4/5 |
| Does the analysis present credible empirical support for the theory? It cites a report from the Mass Education Insight and Research Institute and a report (authorship unclear) on implementation of the No Child Left Behind Act. There may be research supporting the claim that these four interventions are effective, but it is not discussed in any depth. the probably of success is not clear: "Although there are noted examples of successful school reforms, the vast majority of the lowest performers have not changed course, either because they have received insufficient support or because interventions have been ineffective." | 2/5 |
| Does the analysis adequately assess uncertainty about the outcomes? There is some implicit acknowledgement of uncertainty in the department's call for school authorities to monitor results and see which interventions are working. But there is no analysis of uncertainty in regard to the outcomes expected from this regulation. Data will be used to track results in the future, but in the meantime, there's almost no prediction of how much of what type of results the rule will actually produce. | 1/5 |
| 6. How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve? | 1/5 |
| Does the analysis identify a market failure or other systemic problem? "The proposed requirements are needed to implement the School Improvement Grants program in FY 2009 . . . to achieve its objective of supporting comprehensive and effective efforts by LEAs to overcome the challenges faced by low-achieving schools that educate concentrations of children living in poverty." The preamble claims that achievement in low-achieving schools really is very low, that these schools have failed to turn around for years, and that low-achieving secondary schools are a big cause of the dropout problem. So the department has at least asserted a big problem, but the systemic nature of the problem is a mystery. | 2/5 |
| Does the analysis outline a coherent and testable theory that explains why the problem (associated with the outcome above) is systemic rather than anecdotal? There is no explicit theory given. The list of interventions seems to suggest that the schools' main problems are personnel, structure, governance, and other institutional factors. It does not explain why these interventions have not been adopted previously in these schools, or why absence of federal funding is the only constraint. | 1/5 |
| Does the analysis present credible empirical support for the theory? The preamble presents a few figures suggesting that the problem is big. One statement implies that lack of federal funds may be a problem: "There are close to 2,000 high schools in the country in which graduation is at best a 50/50 proposition. However, Department data indicate that fewer than half of these schools currently receive Title I, Part A funds." | 2/5 |
| Does the analysis adequately assess uncertainty about the existence or size of the problem? No relevant discussion. | 0/5 |
| 7. How well does the analysis assess the effectiveness of alternative approaches? | 1/5 |
| Does the analysis enumerate other alternatives to address the problem? One alternative is considered: give out the money with no requirements that target it for specific interventions in low-income schools. "A likely alternative to promulgation of the requirements proposed in this notice would be for the Secretary to allocate the FY 2009 school improvement funds without setting any regulatory requirements governing their use . . . LEAs would likely not use all the funds for activities most likely to result in a real turnaround of those schools and significant improvement in the educational outcomes for the students they educate." | 2/5 |
| Is the range of alternatives considered narrow (e.g., some exemptions to a regulation) or broad (e.g., performance-based regulation vs. command and control, market mechanisms, nonbinding guidance, information disclosure, addressing any government failures that caused the original problem)? Not much of an alternative is considered. It offers no justification for why the method chosen is the most effective approach; the method that is chosen is fairly prescriptive in its overall structure but does allow for implementation details to be carried out at the local level (although local implementation is kind of a given). | 1/5 |
| Does the analysis evaluate how alternative approaches would affect the amount of the outcome achieved? There is no analysis, just assertions that the money will not be used as effectively if the department does not impose requirements. | 1/5 |
| Does the analysis adequately address the baseline? That is, what the state of the world is likely to be in the absence of federal intervention not just now but in the future? There is no explicit analysis, but the department appears to assert that not much will change in the absence of this rule: "Although there are noted examples of successful school reforms, the vast majority of the lowest performers have not changed course, either because they have received insufficient support or because interventions have been ineffective." | 1/5 |
| 8. How well does the analysis assess costs and benefits? | 2/5 |
| Does the analysis identify and quantify incremental costs of all alternatives considered? The analysis only identified federal appropriations, not costs associated with the strings. Since there is no real analysis of alternatives, costs of alternatives are not calculated. | 2/5 |
| Does the analysis identify all expenditures likely to arise as a result of the regulation? Federal payments to states; state and local costs—applying for grants, distributing the grants to LEAs, ensuring compliance with the proposed requirements, and reporting to the Department, and LEA costs of applying for subgrants and implementing the interventions. The analysis notes that any costs will be financed through the grant funds, so the requirements should not be a net burden on state or local authorities. | 4/5 |
| Does the analysis identify how the regulation would likely affect the prices of goods and services? No relevant discussion. | 0/5 |
| Does the analysis examine costs that stem from changes in human behavior as consumers and producers respond to the regulation? No relevant discussion. | 0/5 |
| If costs are uncertain, does the analysis present a range of estimates and/or perform a sensitivity analysis? The preamble seeks comment on how to reduce any burdens. | 2/5 |
| Does the analysis identify the alternative that maximizes net benefits? Net benefits are not calculated. The preamble asserts that the chosen alternative will be most effective. | 1/5 |
| Does the analysis identify the cost-effectiveness of each alternative considered? See above. | 1/5 |
| Does the analysis identify all parties who would bear costs and assess the incidence of costs? The analysis notes that targeting the funds might reduce funding for less needy schools, but this effect is likely small because the total pot of money is so much larger than in the past: "The proposed requirements will have a distributional impact on the allocation of school improvement funds nationally . . . larger proportion . . . flowing to . . . the lowest-achieving schools . . . However, because the FY 2009 appropriation for the program is much larger than the appropriation for FY 2008, the negative impact on the latter category of LEAs may be minimal." Analysis is conducted to see how many small entities the regualtion will affect. | 3/5 |
| Does the analysis identify all parties who would receive benefits and assess the incidence of benefits? Ultimate beneficiaries would be students at the worst schools, and the performance data focus on their performance. | 2/5 |
Use | |
| 9. Does the proposed rule or the RIA present evidence that the agency used the analysis? The decision to focus the funding on four types of interventions in low-income schools appears to be based on some research about the effectiveness of various interventions. The research is not explained in depth, but it doubtful the department would have come up with these four options if it had not consulted some research. The department at least sought comment on ways to reduce the burden or otherwise comply with regulatory analysis requirements: "We invite you also to assist us in complying with the specific requirements of Executive Order 12866 and its overall requirement of reducing regulatory burden that might result from these proposed requirements. Please let us know of any further ways we could reduce potential costs or increase potential benefits while preserving the effective and efficient administration of this program." | 4/5 |
| 10. Did the agency maximize net benefits or explain why it chose another alternative? Net benefits are not calculated. The underlying logic of the decision to focus on specific interventions in the worst schools seems to be driven by something akin to net benefits logic—the idea that this is where/how the money will do the most good. | 2/5 |
| 11. Does the proposed rule establish measures and goals that can be used to track the regulation's results in the future? The department does not set student achievement goals, but it articulates measures adn requires state or local authorities to set goals. Grants are renewable for two years if the school is on track meeting its student achievement goals. The preamble notes that this information should be useful to local authorities to track implementation, to state authorities to evaluate progress, and "Analyses of these data at the national level would inform the Nation’s collective knowledge of what works in turning around our lowest-achieving schools." This is not quite a commitment to evaluate the regulation, but comes pretty close. | 3/5 |
| 12. Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so? "The Secretary proposes to collect data in three general categories: (1) Interventions (those an LEA is implementing); (2) Leading Indicators (instructional minutes per school year and teacher attendance); and (3) Student Achievement Outcomes (average scale scores on State assessments, in the aggregate and disaggregated by subgroup as identified in 34 CFR 200.13(b)(7), and number of students enrolled in advanced coursework)." A chart lists the data. | 4/5 |
| Total | 31 / 60 |
Additional Details
- Agency
- Department of Education
- Regulatory Identification Number
- 1810-AB06
- Agency Name
- Department of Education
- Rule Publication Date
- 08/26/2009
- Comment Closing Date
- 09/25/2009