From The Washington Post...

Link to Original Story

The Answer Sheet
By Valerie Strauss

Question raised about new KIPP study

No sooner had a new study been released saying that middle school students in the charter school network called Knowledge Is Power Program significantly outperform their public school peers on reading and math tests than critics raised an important question about the report.

Mathematica Policy Research issued the study yesterday, my colleague Bill Turque reported, about the KIPP network, which now has 82 schools that serve children from low-income households. Seven are in the District, including three middle schools -- KEY, WILL and AIM academies -- which are among the highest performing on the DC-CAS standardized tests.

The study looked at 22 KIPP middle schools including AIM and KEY, and discovered that by seventh grade, half of them showed growth in math scores equal to an additional 1.2 years of school. Reading gains for KIPP were not as dramatic but still significant, the researchers reported, reflecting an additional three-quarters of a year of growth.

However, an initial analysis of the study conducted by Professor Gary Miron of Western Michigan University found that attrition data appears to have been misrepresented.

Miron conducted his initial analysis at the request of the Think Tank Review Project, a collaboration of the Education and Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University. Here’s a post from the project on Miron’s conclusions:

A key finding of the study is that attrition at KIPP schools is not much different from attrition at comparable conventional public schools. This finding is important because past research about KIPP suggests that selective attrition -- struggling students disproportionately leaving, with more successful students staying and then scoring well on tests -- may give KIPP a substantial boost.

However, an initial analysis of the report by Professor Gary Miron of Western Michigan University concludes that this initial study report misrepresents the attrition data.

According to Miron, "While it may be true that attrition rates for KIPP schools and surrounding districts are similar, there is a big difference: KIPP does not generally fill empty places with the weaker students who are moving from school to school. Traditional public schools must receive all students who wish to attend, so the lower-performing students leaving KIPP schools receive a place in those schools."

In contrast, Miron explains, "The lower performing, transient students coming from traditional public schools are not given a place in KIPP, since those schools generally only take students in during the initial intake grade, whether this be 5th or 6th grade."

The KIPP study’s description of attrition only considers half the equation, when comparing KIPP schools to matched traditional public schools. The researchers looked at the attrition rates, which they found to be similar - in the sense of the number of students departing from schools. But they never considered the receiving or intake rate. Even though the researchers agree that the students who are mobile are lower performing, they do not take into account the reality that KIPP schools do not generally receive these students.

Professor Miron conducted his own quick analysis, using the Common Core database, and concluded that there is a 19% drop in enrollment in KIPP schools between grades 6 and 7 and a 24% drop in enrollment between grades 7 and 8. (This analysis only included KIPP schools that had enrollments in all three grades). In comparison, traditional public schools in these grades maintain the same enrollment from year to year.

While Miron’s review questions about the validity of this report’s particular findings, this is solely because of this single problem. In other ways, he found the study to be rigorous and high quality, promising to be even better in subsequent years of the evaluation. Those future reports can, and Miron hopes, will address the questions raised here and also about students retained in grade.

Importantly, Miron is also not saying that the KIPP schools do poorly. Those schools provide about 50% more instructional time and place rigorous demands on students and their families.

"We have every reason to believe that KIPP likely does a great job with the low-income students of color who wish to attend and who have relatively supportive parents who can do things like drive them to Saturday school," Miron says.

But he does question whether this is a viable model for larger numbers of students, and he also wonders whether the different departure and receiving policies may make matters worse for students who are left behind or who later leave KIPP schools. How would the KIPP model work if students who cannot handle the rigorous KIPP demands could not move to conventional public schools?

In a separate e-mail, Professor Kevin Welner, director of the Education and Public Interest Center (EPIC) at the University of Colorado at Boulder, told me:

I hope Gary’s raising of this important issue is received as it was intended. This was a difficult study to undertake, and in many ways the Mathematica researchers did an excellent job. Gary’s comments below make that clear. There seems to be a tendency among advocates for school models such as this to react very defensively instead of welcoming criticism and finding ways to address genuine issues. In this case, Gary focused in on the validity of a key claim -- an attrition claim that has been trumpeted by KIPP’s advocates -- which is substantially undermined by the researchers’ failure to account for the differences in "receiving" policies. This was almost surely a source of bias, resulting in a more positive result for KIPP.