The broken review system

A while back, Philip Cohen posted on some of the problems with the peer review system in sociology, sharing the story of a paper that underwent 13 peer reviews over several years in the publication gauntlet. Although the paper’s findings were essentially unchanged by this process, each reviewer apparently thought that the paper could be framed in a different, and better, way. As Cohen says:

Most (or all) of the reviewers were sociologists, and most of what they suggested, complained about, or objected was about the way the paper was “framed,” that is, how we establish the importance of the question and interpret the results. Of course framing is important – it’s why you’re asking your question, and why readers should care (see Mark Granovetter’s note on the rejected version of “the Strength of Weak Ties”). But it takes on elevated importance when we’re scrapping over limited slots in academic journals, so that to get published you have to successfully “frame” your paper as more important than some other poor slob’s.

Cohen also cites problems with the journal system and its speed and arbitrary nature, but I think that the issue of framing is particularly important because journal editors appear to be letting reviewers asking a version of the “why didn’t you write the paper I would have written?” conference question arbitrarily prevent the publication of otherwise-worthy papers. This is particularly problematic for graduate students and those of us who work at teaching-oriented institutions and don’t typically have numerous papers under review at once.

Cohen proposes an alternate peer review system, but barring major changes in the system, I think that editors can take immediate steps to address this issue. When somebody asks the “why didn’t you write the paper I would have written?” question at a conference, everybody else in the audience rolls their eyes and recognizes the problem. Journal editors need to be more forceful about recognizing these problems themselves, providing clearer review guidelines and ensuring that framing is not the single most important factor in their decisions.

Via: Scatterplot

—

“Like” Memoirs of a SLACer on Facebook to receive updates and links via your news feed and use the comments to complain about their framing.

Read Full Post »

How to review a manuscript

Since I was recently asked to review a paper, it seems like a good time to take a look at “The Art of Manuscript Reviewing.”  Although it is aimed at historians, the advice it contains is equally applicable to sociologists.  You should check out the entire thing (it is short!), but the list includes:

• Recognize what the author has accomplished
• Be honest
• Be concrete and constructive
• Don’t take over
• Be attentive to questions of audience
• Be gracious

All of these are points that I would appreciate as an author.  Incidentally, since I have only published sporadically in the past few years and I’m certainly not the go-to “expert” on any topic, I rarely get asked to review papers.  One piece of reviewing advice I’ve received from a senior scholar that I’m unlikely to be able to follow is that I should spend only two hours reviewing an article: one hour to read it and one hour to provide comments.

Via: Edge of the American West

Read Full Post »

“Blind” peer review

The combination of a recent paper submission and a post by Tina at Scatterplot have caused me to wonder about the age-old question of “blind” peer review.  The question, of course, is whether peer reviews can truly be blind in the days of online conference information (sometimes including papers) and internet search engines.  This question came up at Orgtheory a while back, with the definitive follow-up poll suggesting that most people do look up the authors of papers they are reviewing, either before or after the review.  Obviously, older individuals may be less likely to respond to Orgtheory polls and similarly less likely to look up authors in this way, but it is still likely that a blind review will not be blind to all reviewers.

Given that blind peer review is blind for a reason, it seems that we have a problem.  Sure, a few high-profile scholars might be recognizable by their writing style, theoretical perspectives, or citations, but the vast majority of sociologists do not have that problem.  I wonder, for example, how being an unrecognized name from an unrecognized school will affect me when my reviewers attempt to take the blinders off of the peer review process.

Given these issues, it seems that we have a few options.  One is to give up on the illusion of blind reviews and sign both submissions and reviews.  Another option is to take the blinding process further by removing titles as well, since titles are likely the easiest way to search for a paper that has been presented at a conference.  Other options include preventing our conference presentations from being archived online and throwing out the whole presentation and publication model and moving to communes organized by John Galt.  I’m not sure which of these would be most effective.

Read Full Post »

Dear Journal Editor

Dear Journal Editor,

A while back, I received an R&R at your journal.  Maybe you’ve experienced this yourself – an invitation to revise your paper based on the feedback of anonymous reviewers and then resubmit the paper to the same journal that issued the invitation.  If you have experienced this, then you likely know that R&Rs can be a lot of work.  Obviously, an R&R is no guarantee that a paper will eventually be accepted, but it does imply that a paper will be given a fair appraisal upon resubmission.

It turns out that your tenure as editor began while I was hard at work on the aforementioned revisions such that the R&R was invited by the previous editor and the resubmission was submitted to you.  I recognize that a journal editor has considerable influence over the direction of a given journal but I believe that a journal editor also has a responsibility to finish the work that the previous editor has started.  You can imagine, then, how interested I was in receiving word from you three and a half months after my resubmission, which is a relatively quick turn around for reviewers these days.  You can also imagine how surprised I was to find that after three and a half months you had decided to inform me that you would, in fact, not be sending my resubmission to reviewers because it was not in line with the type of research that you were interested in publishing, despite the work of previous reviewers, the previous editor, and myself.

Based on this turn of events, it is reasonable for you to expect me to be bitter.  Thanks to the work of the reviewers at your journal, my paper quickly received a conditional acceptance at another journal where it was recently published.  Publication was a nice validation of the hard work I had put into this paper but my validation did not stop there.  In fact, you may have seen the discussion of my paper in newspapers across the country and you may have heard interviews discussing the paper on the radio.  The paper, it turns out, has gotten quite a bit of media attention, much of which mentions the name of the journal where the paper was published.  That journal is not your own.  I write to you, then, not because I am bitter but because I wish to inform you that, in your effort to further limit the type of work that is published in your journal, you lost.

More generally, situations like this reinforce my decision to work at an institution where tenure depends more on a combination of factors than on the whims of journal editors.  While you are certainly not The Paris Review, I hope that you noticed my work somewhere in the press and felt a pang of regret that your journal was not getting the attention that your previous editor and reviewers made possible.

Sincerely,

John Smith

Read Full Post »

Finding the good in bad reviews

Following up on my post about reviewing bad papers that Jeremy gave the Scatterplot bump, I completed my review while being as gentle as I could.  Although I did not search for the paper’s title before completing my review, I did so afterward and came up with nothing, supporting my belief that this work had not been previously presented in any form and leading me to suspect that it was written by a graduate student.  While most students in my own graduate program would not dream of submitting a paper for review that has not been read by numerous graduate students and professors, it is becoming increasingly clear that not all graduate student experiences are the same.

At any rate, shortly after completing the review I received a message from the journal’s editor containing the decision (a much-needed rejection) and the text of all of the reviews.  I thought that this was excellent, as it gave me an opportunity to see how others had framed their attempts to let the author down gently and also to compare the points that were raised.  It actually took me a moment to identify which of the reviews was my own because a number of them began by pointing out how interesting the idea was and how promising the research could be if done in a completely different way.

Incidentally, a friend of mine reviewed a paper for the same journal around the same time and was horrified that she received the same type of message because she worried that others might negatively judge her review.  Of course, the reviews were blind, so I had a hard time seeing how a chance to compare her own review with those of others could have possibly reflected negatively on her.  I tried pointing out how excited I had been at the opportunity to see how others approached the review but this failed to convince her that the practice was a good one.  (Maybe she has her own pseudonymous blog with a bizarro-world version of this post.)  Given my relative lack of reviewing experience, I’m not sure how common this practice is but my sense is that the answer is “not very.”  While opinions about the practice are split based on my sample size of two, as a young scholar I greatly appreciated the chance to gain a bit more insight into the reviewing process.

Read Full Post »

Bad reviews

Today I read a bad paper that I will be giving a bad review.  I haven’t reviewed many papers, so I am hoping that I have just had bad luck so far, but it seems that every time I agree to review a paper it has serious flaws.  While the authors were kind enough to double space their papers and include page numbers, I could have done without these things if those double-spaced and numbered pages were filled with coherent arguments, paragraphs, or even sentences.

Despite my strong beliefs that these papers were bad, I sympathize with the authors who receive my comments.  As the recipient of a number of bad reviews of my own papers, I try to couch my criticisms between statements that “I think you have some really good ideas” and “your methodological approach is interesting.”  Unfortunately, these statements probably don’t do much to lessen the blow when they are followed by “…but your poor writing prevents me from knowing for sure” and “…but seriously flawed.”

The worst thing about this is that I always have high hopes for the papers I agree to review.  The paper I read today had been sitting on my desk for a while while I worked on dissertation-related matters, so I was looking forward to the chance to read about an interesting topic that was loosely related to my own area of research.  While my hopes weren’t met, I will still feel bad when I submit my suggestion to reject the paper.  Still, I would have felt worse if the author had bothered to proofread it before sending it to total strangers.

Read Full Post »

The publication gauntlet

There is an interesting post on Inside Higher Ed about Michèle Lamont’s How Professors Think: Inside the Curious World of Academic Judgment, which examines peer review panels.  As a graduate student, I’ve been curious about peer review processes in general, as I’ve experienced successes and failures in peer-reviewed publications and grant competitions.

I suspect that little will change in regard to these processes in my future life as a junior faculty member, but I wonder how my choice of institution will affect my desire to run the publication gauntlet.  Coming from a “publish or perish” department, I have a strong desire to get my work published in order to contribute to the body of sociological knowledge.  I also want to publish in well-regarded journals in order to increase the chance that others will actually be aware of my contributions.  At the liberal arts school where I will be employed, however, expectations for publication are much lower than for junior faculty in my current program and the fact that a paper went through peer review is more important than the name on the front of the journal.

Higher expectations for research often come with fewer teaching obligations and graduate student collaborators, allowing faculty to maintain multiple projects and submit a lot of papers for review.  At most liberal arts schools, teaching loads are higher and there are no graduate students with whom to collaborate, which I expect results in fewer concurrent projects and fewer submissions for review.  As a student I have watched papers go through numerous review cycles at multiple journals before receiving an R&R or conditional acceptance.  In one case, this process took years.  A large part of the reason for this was that the authors submitted their paper first to a top generalist journal, then to highly-ranked specialty journal, then to another specialty journal, before being accepted at yet another specialty journal that, while still good, does not have as much cachet as the earlier destinations.

Although I want to publish in highly-ranked journals, I am unlikely to have years to devote to the publication of a single paper as a junior faculty member at a liberal arts school.  In time, I wonder if this desire will fade in favor of running a more forgiving gauntlet that is equally supported by my tenure review committee while fewer outside of my institution are aware of my contributions to sociological knowledge.

Read Full Post »