The broken review system

A while back, Philip Cohen posted on some of the problems with the peer review system in sociology, sharing the story of a paper that underwent 13 peer reviews over several years in the publication gauntlet. Although the paper’s findings were essentially unchanged by this process, each reviewer apparently thought that the paper could be framed in a different, and better, way. As Cohen says:

Most (or all) of the reviewers were sociologists, and most of what they suggested, complained about, or objected was about the way the paper was “framed,” that is, how we establish the importance of the question and interpret the results. Of course framing is important – it’s why you’re asking your question, and why readers should care (see Mark Granovetter’s note on the rejected version of “the Strength of Weak Ties”). But it takes on elevated importance when we’re scrapping over limited slots in academic journals, so that to get published you have to successfully “frame” your paper as more important than some other poor slob’s.

Cohen also cites problems with the journal system and its speed and arbitrary nature, but I think that the issue of framing is particularly important because journal editors appear to be letting reviewers asking a version of the “why didn’t you write the paper I would have written?” conference question arbitrarily prevent the publication of otherwise-worthy papers. This is particularly problematic for graduate students and those of us who work at teaching-oriented institutions and don’t typically have numerous papers under review at once.

Cohen proposes an alternate peer review system, but barring major changes in the system, I think that editors can take immediate steps to address this issue. When somebody asks the “why didn’t you write the paper I would have written?” question at a conference, everybody else in the audience rolls their eyes and recognizes the problem. Journal editors need to be more forceful about recognizing these problems themselves, providing clearer review guidelines and ensuring that framing is not the single most important factor in their decisions.

Via: Scatterplot

—

“Like” Memoirs of a SLACer on Facebook to receive updates and links via your news feed and use the comments to complain about their framing.