Questioning statistical significance

When discussing statistical significance in class I always preface my discussion by highlighting the arbitrary nature of accepted p-values. “If you had cancer,” I ask my students, “and a doctor said that there was a 94% chance that a particular treatment would cure you, would you take it?” They would, they assert. But a 94% chance isn’t good enough for social science research. Neither, suggests Valen Johnson, a statistician at Texas A&M, is a 95% chance, or a 98% chance. As discussed by John Timmer at Ars Technica, Johnson mathematically links Bayesian statistics to probabilities:

The math then allows a direct comparison between the probability values. In his comparison, scientific standards seem pretty weak. The 95 percent certainty corresponds to a Bayesian evidence threshold of between three and five, which Johnson notes is typically considered “positive evidence”—but it falls well below the values considered to be “strong evidence.” It takes 99 percent certainty to get there.

…

Johnson concludes that if we assume that only one-half of the hypotheses should give us a positive result, then “these results suggest that between 17 percent and 25 percent of marginally significant scientific findings are false.” If we assume the proportion of correct hypotheses is larger—which we might, given that scientists are usually pretty clever about the hypotheses they choose to test—then the problem gets even more pronounced. Overall, Johnson’s suggestion is simple: raise the statistical rigor all around. Demand that experiments produce a p value of 0.005 or smaller. And be even pickier about results that we consider highly significant. There is a cost to this, in that you need bigger samples to achieve the higher statistical rigor. In his example, you’d have to double the sample size. That’s no problem if you’re breeding bacteria and fruit flies, but it will add a lot of time and expense if your project involves mice.

Or, of course, humans. One implication that Timmer notes for increased significance thresholds is that those with small sample sizes would have to consider discussing non-significant results, potentially undermining our blind faith in statistical significance. While that would be nice, in the world we actually live in the more likely outcome is that individual or small-scale research would be even more difficult to conduct successfully. Good luck getting that NSF grant!

Read Full Post »

War games

A recent post by D.B. Grady at The Atlantic criticizes the ad campaign for Modern Warfare 3, the latest in Activision’s hugely successful Call of Duty video game series. In part, it reads:

The advertisement trivializes combat and sanitizes war. If this were September 10, 2001, maybe it wouldn’t be quite so bad. Those who are too young to remember Vietnam might indulge in combat fantasies of resting heart rates while rocket-propelled grenades whiz by, and of flinty glares while emptying a magazine into the enemy. But after ten years of constant war, of thousands of amputees and flag-draped coffins, of hundreds of grief-stricken communities, did nobody involved in this commercial raise a hand and say, “You know, this is probably a little crass. Maybe we could just show footage from the game.”

Responding to this, Ben Kuchera at Ars Technica notes that the problem isn’t with the advertising but with the game itself:

For gamers, there is nothing new or striking about how the ad shows war, because that’s the way the game shows war: we wear the skin of a soldier and take part in armed conflict as if it were a thrill ride. We design our in-game avatars, and we virtually kill people in locations based on the real world, with dramatic music and a presentation that seems to tell us the game is a very serious thing. All the while, we’re cheering on our kill-streaks and laughing as bodies fly hither and yon. From the outside looking in, or if you’re not familiar with war games, it is a very disturbing way to spend your free time.

…

The imagery used in the ad may be shocking to non-gamers, but the ad itself isn’t the problem. The popularity of this sanitized, no-consequences form of virtual war is what should have critics talking. Activision didn’t have to create a commercial to sell millions of copies of Modern Warfare 3, the game would have been a monstrous hit without this commercial; gamers have been drowning themselves in pixelated bloodshed and gleeful violent for decades.

When we discuss ways that the media leads to desensitization it is important to remember which medium is leading the charge. Increasingly, this medium is video games. This doesn’t mean that we can’t enjoy violent games any more than we shouldn’t enjoy The Bourne Identity, but I think it does mean that the dialog surrounding these games needs to mature so that we can think critically about their effects.

 

Read Full Post »