Flawed survey used to support adding a second Amtrak train between St. Paul and Chicago

The Midwest Interstate Passenger Rail Commission (MIPRC) recently surveyed students, faculty, and staff at three Winona-area colleges and universities on their usage and views of Amtrak passenger rail.  The survey supplements one administered to 30 other colleges and universities across the Midwest.

The results led to this Star Tribune headline: “Millenials are all aboard for passenger rail.”   If a Google search is an accurate measure of the MIPRC media hits, then the Star Tribune is the only major news outlet to make headline news out of this survey. Maybe that’s because every other news organization decided a voluntary online SurveyMonkey survey didn’t carry enough weight to be real news.

Voluntary online surveys are a popular and low-cost way for organizations to get some general feedback.  The feedback can be meaningful, but everyone knows the results are not at all scientific.

Being voluntary, these online surveys don’t elicit random, representative results.   The MIPRC survey results are undoubtedly biased in favor of people who have an interest in trains.  Why waste time taking a survey on trains if you’ve never taken the train?  As a University of Texas online guide to statistics explains, “voluntary response samples oversample people who have strong opinions and undersample people who don’t care much about the topic of the survey.”

Ignoring the inherent bias in the survey, the MIPRC concluded the responses “suggest a latent passenger pool exists in Winona (and by extension, much of southeast Minnesota and parts of neighboring western Wisconsin) for a second daily Twin Cities‐Chicago train.”

And here we have another survey faux pas.  Extrapolating survey results to other populations not surveyed is usually frowned upon.  The folks on the campuses surveyed don’t at all represent the rest of the southeast Minnesota and western Wisconsin population in terms of important things like age and socio-economic status.

The Star Tribune article focused of its ink on this claim that the survey supports adding a second daily Amtrak train between St. Paul and Chicago.  Twice the article notes that over half of the respondents said “they would take the train more if additional service were available.”   But that’s not exactly what the students said.  They really said they would be “more likely” and most couched that by saying only “somewhat more likely.”

When you think about, it’s pretty dumb to ask whether more frequent train service would “make you more likely, less likely or equally likely to travel by Amtrak as you do now?”  How could adding a second train possibly make it less likely or even equally likely to take the train?  It seems like general rule of probability that more of something makes it more likely to happen.  If that’s not a general rule, it should be.

So, add biased survey wording to the biased sampling and extrapolation problems with the MIPRC survey.

Even the Minnesota Department of Transportation (MnDOT) appears willing to ignore the shortcomings of the survey.  The Star Tribune quotes Dan Krom, director of MnDOT’s Passenger Rail Office, expressing hope that the “survey will bolster the case for more rail service.”  MnDOT posted a press release expressing the same sentiment.

These MnDot comments suggest the agency might not be taking the most objective approach to studying whether adding a second train makes sense.  Hopefully, MnDOT’s partner—the Wisconsin Department of Transportation—can keep the study grounded.