How well do Minnesota’s schools really compare?

This op ed appeared in the Spring Valley Tribune on September 11th, 2019

In our new report The State of Minnesota’s Economy: 2018, we note that our state’s highly educated population is a plus in terms of economic growth. In the much quoted U.S. News ranking, Minnesota comes in 7th nationally.

But a new report from the Cato Institute, ‘Fixing the Bias in Current State K–12 Education Rankings‘, argues that there is much wrong with such rankings. According to the authors, Stan Liebowitz and Matthew Kelly,

The most popular and influential state education rankings fail to provide an “apples to apples” comparison between states. By treating states as though they had identical students, they ignore the substantial variation present in student populations across states. Conventional rankings also include data that are inappropriate or irrelevant to the educational performance of schools. Finally, these analyses disregard government budgetary constraints. Not surprisingly, using disaggregated measures of student learning, removing inappropriate or irrelevant variables, and examining the efficiency of educational spending reorders state rankings in fundamental ways.

Liebowitz and Kelly note that “direct measures of the entire student population’s understanding of academic subject matter, such as those from the NAEP, are the most appropriate measures of success for an educational system”. But measures such as U.S. News’ include data on graduation rates and SAT and ACT college entrance exam test scores. But, as the authors note, “graduation is not necessarily an indication of actual learning, and only those students wishing to pursue a college degree tend to take standardized tests like the SAT and ACT”. “Focusing on NAEP data”, they write, “thus avoids selection bias while more closely measuring a school system’s ability to improve actual student performance.”

Furthermore, students from different socioeconomic and ethnic backgrounds tend to perform differently on NAEP tests regardless of the state they are in but the popular state rankings usually ignore this. “[S] such aggregation”, Liebowitz and Kelly write, “often renders conventional state rankings as little more than a proxy for a jurisdiction’s demography.” The facts are that kids from some socioeconomic groups arrive at school with backgrounds, skills, and life experiences, which handicap them relative to others. It follows that a state with a smaller share of kids in these categories will have a built in advantage when it comes to measuring educational outcomes. A state, like Minnesota, that does well on such aggregated scores might be benefiting from its socioeconomic make up rather than any great achievement by its education system.

Liebowitz and Kelly provide the following example

According to U.S. News, Iowa ranks 8th and Texas ranks 33rd in terms of pre-K–12 quality. U.S. News includes only NAEP eighth-grade math and reading scores as components in its ranking, and Iowa leads Texas in both. By further including fourth grade scores and the NAEP science tests, the comparison between Iowa and Texas remains largely unchanged. Iowa students still do better than Texas students, but now in all six tests reported for those states (math, reading, and science in fourth and eighth grades). To use a baseball metaphor, this looks like a shut-out in Iowa’s favor.

But this is not an apples-to-apples comparison. The characteristics of Texas students are very different from those of Iowa students; Iowa’s student population is predominantly white, while Texas’s is much more ethnically diverse. NAEP data include average test scores for various ethnic groups. Using the four most populous ethnic groups (white, black, Hispanic, and Asian), at two grade levels (fourth and eighth), and three subject-area tests (math, reading, science), there are 24 disaggregated scores that could, in principle, be compared between the two states in 2017. This is much more than just the two comparisons—eighth grade reading and math—that U.S. News considers.

Given that Iowa students outscore their Texas counterparts on each of the three tests in both fourth and eighth grades, one might reasonably expect that most of the disaggregated groups of Iowa students would also outscore their Texas counterparts in most of the twenty exams given in both states. But the exact opposite is the case. In fact, Texas students outscore their Iowa counterparts in all but one of the disaggregated comparisons. The only instance where Iowa students beat their Texas counterparts is the reading test for eighth grade Hispanic students. This is indeed a near shut-out, but one in Texas’s favor, not Iowa’s.

When the NAEP scores are disaggregated, Iowa’s ranking falls from 17th on the aggregated ranking to 32nd. That of Texas, by contrast, rises from 35th to 6th.

What about Minnesota? We ranked 7th with U.S. News and 4th on the aggregated NAEP scores. But, when we disaggregate these scores to account for the make up of the student body, our state slumps to 33rd.

Minnesota’s education rankings are generally a source of pride for the state. This new research suggests that we ought to think again. I’ve written previously that, before latching on to a state ranking, you should take a look at how that ranking was constructed. The same goes for state education rankings.

John Phelan is an economist at the Center of the American Experiment.