The Topline: You Can't Make This Stuff Up

Last week's revelation that a widely cited political science paper was probably based on faked data has reverberated around the opinion research community. The paper, published in Science, purported to demonstrate that a conversation with a gay political canvasser could change views on same sex marriage.

The size and durability of the change in opinion had raised both hopes and eyebrows among LGBT activists, political organizers more generally, and political and social science researchers. The study was picked up by the popular media, including the NPR program This American Life.

It all came crashing down when another team of researchers attempted to replicate the study. They quickly concluded that the follow-up online surveys, which tracked changes in the opinions of voters who had been canvassed by gay or straight volunteers earlier in the experiment, had probably never happened. Faced with mounting evidence of fabrication, one of the co-authors has requested a retraction from Science. The New York Times wrote of the incident, "The case has shaken not only the community of political scientists but also public trust in the way the scientific establishment vets new findings."

Data fabrication in surveys is not a new problem and is not confined to the data from one high-profile paper. Most efforts to detect fabrication focus on interviewer "curb-stoning", which gets its name from interviewers filling in paper and pencil questionnaires, sometimes literally sitting on the curb, rather than going door-to-door to survey actual respondents. Even here, more work is needed. But the focus on interviewers as the source of fabrication has meant the survey research community has missed other threats and needs to catch up.

This recent episode demonstrates is that fabrication can happen anywhere along the chain of data collection, processing, and analysis. In this most recent case, a clever fabricator apparently copied an already existing dataset, altering it just enough to avoid detection, at least for a while. Another kind of apparent fabrication is familiar to those who follow political polling, where survey firms Research 2000 and Strategic Vision seemed to have falsified results of surveys during the 2008 presidential cycle.

These cases are different still from the issues that exist in difficult research environments, mostly overseas, where face-to-face surveys are conducted with paper and pencil. The dangers of conducting surveys in these places are real, and give rise to the temptation to take shortcuts by inventing data rather than collecting it. Fabrication in these environments is the subject of several forthcoming papers, including one of my own with several co-authors.

The threat domestically is, despite last week's revelations, probably lower than elsewhere. But if this episode has shown anything, it's that more attention is needed across the board and throughout the entire survey research process. I have been involved in a series of events on data fabrication that has been quietly unfolding over the last year, organized by a combination of statistical and survey research organizations. The revelations during the sessions have made one thing clear. We have a problem.

An article from 1946 frames the current issue well: "The [fabrication] problem must be solved if the opinion research technique is to preserve its status as a reliable tool of inquiry." Sadly, this is as true today as it was then.

-Steve Koczela

THE CROSSTABS

Census

Last week the U.S. Census released its 2014 population estimates for cities and states across the country.

The Boston Globe compiled the data for Massachusetts into a handy map at the bottom of this article about Watertown, the fastest-growing community in the Commonwealth.

Opiates

The Globe partnered with the Harvard school of public health for a survey on opiate addiction in Massachusetts.

We asked some questions along the same lines for WBUR back in May 2014.

Around that time, MPG's Steve Koczela wrote about the lack of good data on the scope of the problem. The available data has since improved substantially.

MBTA Data

The MBTA has published new information about absenteeism at the agency, providing a much clearer and comprehensive picture of the scope of the problem.

We had written previously for CommonWealth about problems with the absenteeism figures in Governor Baker's special MBTA panel's report.

Ari Ofsevit takes issue with Jim Stergios's math on Commuter Rail ridership and expansion.

We wrote earlier that ridership stats for the Commuter Rail point in many different directions.

2016 Presidential Primaries 

 A new national IVR poll of Republican primary voters by Vox Populi for the Daily Caller finds Marco Rubio and Scott Walker out in front of the rest of the field.

 Over at FiveThirtyEight, Harry Enten uses Rick Santorum as a yardstick to measure the rest of the GOP field, which he asserts is stronger than in 2012.

 Bernie Sanders' website's 404 page is peak Bernie Sanders.

 Also, here is a picture of Bernie Sanders holding a can of Heady Topper, just because.

Miscellany

 Market surveys show that Tom Brady's personal brand has taken a big hit post-Deflategate.

 A new survey finds that Latinos with uncertain immigration status are less likely to engage in civic life, including visiting a doctor or taking public transit.

 Americans value their privacy but have little confidence that private business and the government can keep their data secure, according to a new survey from Pew.

 Over at The UpShot, Brendan Nyhan notes that wealthy Americans are fine with free trade, and seem to be getting their way with the Trans-Pacific Partnership.

 Gallup is the latest in a string of polls to find support for same-sex marriage at a record high. HuffPollster has a round-up.

 As we congratulate multimedia reporter/trivia lord Mike Deehan on his move to WGBH, Pew looks at the decline of full-time statehouse reporters across the country.

- - - - - - - - - - - - - - - - - - - - - - - Nerd Alert Tearline- - - - - - - - - - - - - - - - - - - - - - -

 For Wayback Wednesday, FiveThirtyEight is out with an online documentary about Jesse Ventura's successful bid to be governor of Minnesota.

 Fresh from the annual AAPOR conference, market researcher Annie Petit vents on how political and social science researchers are well behind her field in terms of adopting new ways of conducting surveys.