• Home
  • About
  • Contact
  • Good places to start

Memoirs of a SLACer

sociological views on life and the liberal arts

Feeds:
Posts
Comments
« Lily Allen attacks tropes, others attack Lily Allen
The paradox of GoldieBlox »

Questioning statistical significance

November 19, 2013 by John

When discussing statistical significance in class I always preface my discussion by highlighting the arbitrary nature of accepted p-values. “If you had cancer,” I ask my students, “and a doctor said that there was a 94% chance that a particular treatment would cure you, would you take it?” They would, they assert. But a 94% chance isn’t good enough for social science research. Neither, suggests Valen Johnson, a statistician at Texas A&M, is a 95% chance, or a 98% chance. As discussed by John Timmer at Ars Technica, Johnson mathematically links Bayesian statistics to probabilities:

The math then allows a direct comparison between the probability values. In his comparison, scientific standards seem pretty weak. The 95 percent certainty corresponds to a Bayesian evidence threshold of between three and five, which Johnson notes is typically considered “positive evidence”—but it falls well below the values considered to be “strong evidence.” It takes 99 percent certainty to get there.

…

Johnson concludes that if we assume that only one-half of the hypotheses should give us a positive result, then “these results suggest that between 17 percent and 25 percent of marginally significant scientific findings are false.” If we assume the proportion of correct hypotheses is larger—which we might, given that scientists are usually pretty clever about the hypotheses they choose to test—then the problem gets even more pronounced. Overall, Johnson’s suggestion is simple: raise the statistical rigor all around. Demand that experiments produce a p value of 0.005 or smaller. And be even pickier about results that we consider highly significant. There is a cost to this, in that you need bigger samples to achieve the higher statistical rigor. In his example, you’d have to double the sample size. That’s no problem if you’re breeding bacteria and fruit flies, but it will add a lot of time and expense if your project involves mice.

Or, of course, humans. One implication that Timmer notes for increased significance thresholds is that those with small sample sizes would have to consider discussing non-significant results, potentially undermining our blind faith in statistical significance. While that would be nice, in the world we actually live in the more likely outcome is that individual or small-scale research would be even more difficult to conduct successfully. Good luck getting that NSF grant!

Share this:

  • Facebook
  • Twitter
  • Tumblr
  • More
  • Email
  • Pinterest
  • Reddit

Like this:

Like Loading...

Related

Posted in The Ivory Tower, The Publication Gauntlet, The State of Sociology | Tagged Ars Techinca, Bayesian Statistics, John Timmer, Memoirs of a SLACer, p-Values, Statistical Significance, Valen Johnson |

  • Pages

    • About
    • Contact
    • Good places to start
  • Previous Posts

    November 2013
    M T W T F S S
     123
    45678910
    11121314151617
    18192021222324
    252627282930  
    « Oct   Dec »
  • Categories

  • Blogs I Read

    • Conditionally Accepted
    • Crooked Timber
    • Female Science Professor
    • Historiann
    • orgtheory.net
    • scatterplot
    • Small Pond Science
    • Soc'ing Out Loud
    • Sociology in Focus
    • Sociology Source
    • Sociology Toolbox
    • Tenured Radical
    • The Society Pages
    • whatisthewhat
  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

  • Meta

    • Register
    • Log in
    • Entries feed
    • Comments feed
    • WordPress.com
  • Site Meter

    Site Meter

Blog at WordPress.com.

WPThemes.


Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
  • Follow Following
    • Memoirs of a SLACer
    • Join 178 other followers
    • Already have a WordPress.com account? Log in now.
    • Memoirs of a SLACer
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
loading Cancel
Post was not sent - check your email addresses!
Email check failed, please try again
Sorry, your blog cannot share posts by email.
%d bloggers like this: