The broken review system

A while back, Philip Cohen posted on some of the problems with the peer review system in sociology, sharing the story of a paper that underwent 13 peer reviews over several years in the publication gauntlet. Although the paper’s findings were essentially unchanged by this process, each reviewer apparently thought that the paper could be framed in a different, and better, way. As Cohen says:

Most (or all) of the reviewers were sociologists, and most of what they suggested, complained about, or objected was about the way the paper was “framed,” that is, how we establish the importance of the question and interpret the results. Of course framing is important – it’s why you’re asking your question, and why readers should care (see Mark Granovetter’s note on the rejected version of “the Strength of Weak Ties”). But it takes on elevated importance when we’re scrapping over limited slots in academic journals, so that to get published you have to successfully “frame” your paper as more important than some other poor slob’s.

Cohen also cites problems with the journal system and its speed and arbitrary nature, but I think that the issue of framing is particularly important because journal editors appear to be letting reviewers asking a version of the “why didn’t you write the paper I would have written?” conference question arbitrarily prevent the publication of otherwise-worthy papers. This is particularly problematic for graduate students and those of us who work at teaching-oriented institutions and don’t typically have numerous papers under review at once.

Cohen proposes an alternate peer review system, but barring major changes in the system, I think that editors can take immediate steps to address this issue. When somebody asks the “why didn’t you write the paper I would have written?” question at a conference, everybody else in the audience rolls their eyes and recognizes the problem. Journal editors need to be more forceful about recognizing these problems themselves, providing clearer review guidelines and ensuring that framing is not the single most important factor in their decisions.

Via: Scatterplot

—

“Like” Memoirs of a SLACer on Facebook to receive updates and links via your news feed and use the comments to complain about their framing.

Read Full Post »

Senior scholars know best, except when they don’t

Jessica at Scatterplot recently posted some good advice about imaginary “perfect” jobs. She writes:

There’s a tenure-track academic job I hear students talk about – one with work-life balance and a forty-hour work week and at least two weeks (but hopefully an entire summer) of carefree, completely unplugged vacation; one where you have all the autonomy and prestige of a professor, along with job security and a professional level paycheck, but there aren’t external pressures on your time except for those that you select because they’re consistent with your values and life goals…that job – that does not exist. And, even if it did, you would not increase your chances of landing such a job by eschewing the professional advice of faculty or colleagues because they are seen as somehow biased toward a different kind of job, one that just doesn’t fit you or your life goals.

As I said above, I think this is good advice but the “do what senior scholars tell you to do in order to be successful” line of reasoning falls apart when so many senior scholars don’t understand other types of jobs or have outdated ideas of what various types of jobs entail (or even what is required to get jobs like theirs). If you want to work at a SLAC, for example, especially a high-ranking SLAC, publications are essential, so advice that a student will fade to obscurity in one of those jobs is ridiculous. Too many advisors still want to see their students replicate their careers, acting as if other types of careers are beneath them.

One could argue that Eric Grollman’s success in getting an excellent liberal arts job after initially aiming for an R1 is a strong example in favor of the idea that there is only one track, but the pressures that he reports facing from his committee members about even interviewing at liberal arts jobs show that this system still has flaws. I was fortunate not to receive these sorts of messages from my committee members, but a current colleague reports that her dissertation advisor neglected to provide her with any advice on negotiating her job offer from our institution because the advisor hoped that a “real” job offer would come along. That some students know early on what type of job they would like to pursue but still receive these sorts of messages undermines the value of advice in other areas.

In some ways, I could be seen as an example of the type of grad student that Jessica mentions in the comments, where she says, “This is about the students who don’t aspire to a life like the faculty in their grad programs – people who they (erroneously) believe work 80 hours a week all year long and have no life outside of work.” I started grad school around the time that a large number of junior faculty members were hired and watched them go through a grueling tenure process that included the very real threat of being denied tenure unless they could publish in ASR or AJS. I knew that I did not want that kind of experience, but this doesn’t mean that I didn’t think I would have to work to get a job or afterward or that I didn’t seek a strong grounding in theory and methods, as I took more than the required number of courses in each.

Just as Jessica provides advice for students, I would like to provide some advice for faculty who deal with graduate students: listen to them. Consider their career goals and give them advice that will maximize the chances that they will realize those goals while necessarily keeping an eye on their general marketability given that few of them will end up at the types of institutions they seek. If you start your mentoring by assuming that they want to emulate your career, though, and criticizing any desire to do otherwise, be aware that you are discrediting any future advice you will give.

Oh, and one more thing: When your graduate students are on the job market, get your damn letters of recommendation done early and often. There is no excuse for mentoring somebody throughout the entire academic process only to hang them out to dry by not fulfilling your duty as an advisor.

—

“Like” Memoirs of a SLACer on Facebook to receive angry rants about what professors should do via your news feed.

Read Full Post »

The strength of weak ties made stronger by rejection

In keeping with the theme of the early struggles of now-famous sociologists, Shamus has a post at Scatterplot featuring Mark Granovetter’s rejection letter and reviews from ASR for his paper on the strength of weak ties, then called “Alienation Reconsidered: The Strength of Weak Ties.” As Granovetter notes in Shamus’s post, the framing changed significantly between the version that was rejected by ASR and the version that was accepted by AJS.

Read Full Post »

Yes, it is there, too: Academic false consciousness and academic journals

As if academic false consciousness wasn’t bad enough at our institutions and among graduate students, it appears that it is also present at academic journals. Last week, Olderwoman at Scatterplot posted about receiving a packet of five reviews of the same article, stating:

Although five was over the top and freaked me out, it has become pretty common now for me as a reviewer to get a packet with four reviews. No wonder we regular reviewers are feeling under the gun. The old calculation of two or even three reviews per article has gone by the wayside. The pressure for fast turnaround and the high turn-down or non-response rate among potential reviewers has led editors to send out articles to extra reviewers in the hopes of ending up with at least the minimum two or three.

But this is a death spiral. As a frequently-sought reviewer I get at least four requests a month, sometimes as many as eight, and I used to get more before I got so crabby.  When I was young and eager, I was reviewing an article a week [and thus, by the way, having a huge influence on my specialty area], and I know some people who are keeping that pace. But at some point you burn out and say “no more.” I, like all other frequently-sought reviewers I know, turn down outright the requests from journals I don’t know for articles that sound boring, and then save up the other requests and once a month pick which articles I want to review. So the interesting-sounding articles from good journals get too many reviewers, while the boring-sounding articles from no-name journals get none. If journal editors respond to the non-response by reviewers to boring-sounding articles by sending out even more reviewer requests per article, our mailboxes will be flooded even more and the non-response rate and delayed-response rate by reviewers will go up even more. Senior scholars are asked to review six to eight (or more?) articles per month. You have to say no to most of the requests.

And then we have the totally out of hand R&R problem. I think it is completely immoral to send an R&R to ANY new reviewers. I know a young scholar with a perfectly good paper who is now on the 4th (!!!!) iteration of an R&R from ASR. Not because she has not satisfied the original reviewers, but because the editors keep sending each revision to a new set of reviewers in addition to the original reviewers and, of course, the new reviewers have a different perspective and a new set of suggestions for the paper, some of which cover ground that was gone over in one or more of the previous revisions. Not to mention the problem that R&R memos are now longer than the original articles!!  We are no longer a discipline of article publishing, we are turning into a discipline of R&R memo-writing.

She proposes several ground rules that she thinks would help the problem and that reminded me a bit of Gary Fine’s discussion of similar problems as editor of Social Psychology Quarterly.

Fabio followed her post with one of his own, talking specifically about ASR and the number of R&Rs that are given:

This issue has arisen with respect to the American Sociological Review, the flagship journal of the American Sociological Association. The ASR has been giving R&R’s to many submitted articles, much more than average, and they are soliciting many reviews per article. It has also been sending articles through multiple rounds of revisions, leading to articles being held at the journal for years. Since they seem to accept to same number of articles per year (about 40), that implies that the multiple rounds of revision do not lead to publication for many authors. Here is my response to that post:

I am asking the American Sociological Review to curtail this practice. In writing this, I have no personal stake in this matter. I do not have any papers under review, nor has the ASR accepted my previous submissions. I only write as a member of the profession, senior faculty at a top 20 program, a former managing editor of an ASA journal (Sociological Methodology), former associate editor of the American Journal of Sociology, occasional board member for various journals, author, and reviewer.

The inflated R&R policy is damaging sociology in a few ways. First, by continually R&R’ing papers that have little chance of publication, the ASR is “trapping” papers that may be perfectly suitable for specialty journals or other outlets. Thus, inflated R&Rs keep good research out of the public eye for years. You are suppressing science.

Second, inflated R&Rs damage the reputation of the ASR itself. The goal of a flagship journal is to be very picky. When people hear that a paper has been invited for revision, they believe that the editors think that the paper is of great merit and wide relevance. Inflated R&Rs undermine that perception.

Third, you are damaging people’s careers. By trapping papers, you preventing papers from being resubmitted to other journals that can help their careers. Also, R&R invitations are often seen as signs of intellectual progress, especially for doctoral students and junior faculty. By lumping together strong and weak papers, you are debasing the “currency” of the R&R. When people see “R&R at American Sociological Review,” they no longer know what to think and that pollutes the junior level job market.

Fourth, you are wasting precious time. Reviewers are usually full time faculty who teach, mentor graduate and undergraduate students, do administrative work, conduct research, and have full family lives. Thus, when you ask for a fourth reviewer, or a invite a paper for a third round of R&R, you are taking up many, many scarce resources.

Olderwoman, Fine, and Fabio all make valid points that need to be addressed by editors as well as their reviewers. More than any of the other instances of academic false consciousness, this seems like something that can be addressed quickly and relatively easily. Let’s do it.

Read Full Post »

On Major Procrastination Disorder in 1751

Although it had not yet been recognized by psychiatrists (or graduate students), this article from Scatterplot indicates that Major Procrastination Disorder was present in 1751. I think it is time to add a “Nothing is New” category. Other entries: snark and technology, wasting time with media, and wireless reading devices.

Read Full Post »

A reminder about quantitative data for first-year grad students in love with the GSS

They can be just as “fuzzy” as qualitative data.

Read Full Post »

Excuse me, do you have this analogy in Call of Duty?

Over at Scatterplot, Drek points to a blog post by John Scalzi that likens being born as a straight white male to playing a video game like World of Warcraft on the lowest difficulty setting. Here’s a taste:

This means that the default behaviors for almost all the non-player characters in the game are easier on you than they would be otherwise. The default barriers for completions of quests are lower. Your leveling-up thresholds come more quickly. You automatically gain entry to some parts of the map that others have to work for. The game is easier to play, automatically, and when you need help, by default it’s easier to get.

…

[I]t’s certainly possible someone playing at a higher difficulty setting is progressing more quickly than you are, because they had more points initially given to them by the computer and/or their highest stats are wealth, intelligence and constitution and/or simply because they play the game better than you do. It doesn’t change the fact you are still playing on the lowest difficulty setting.

You can lose playing on the lowest difficulty setting. The lowest difficulty setting is still the easiest setting to win on. The player who plays on the “Gay Minority Female” setting? Hardcore.

And maybe at this point you say, hey, I like a challenge, I want to change my difficulty setting! Well, here’s the thing: In The Real World, you don’t unlock any rewards or receive any benefit for playing on higher difficulty settings. The game is just harder, and potentially a lot less fun. And you say, okay, but what if I want to replay the game later on a higher difficulty setting, just to see what it’s like? Well, here’s the other thing about The Real World: You only get to play it once. So why make it more difficult than it has to be? Your goal is to win the game, not make it difficult.

I like this analogy, but I wish it was presented through something like Call of Duty or Madden that a wider variety of (likely male) students spend their time playing.

Read Full Post »

Giving the Grad Skool Rulz

I recently purchased copies of Fabio’s Grad Skool Rulz for myself and some students who are planning to go to grad school next year (and yes, I paid for the “copies” that I gave to students). The question I always have about packaged collections of things that originated on the internet is whether it is worthwhile to purchase something that is essentially available for free. In this case, I think that the ability to send the entire package as a PDF that a student can save somewhere is probably a better way of delivering information than saying “there’s a series of blog posts about grad school – look them up!” Since none of the undergrads that I know regularly read sites like Scatterplot, OrgTheory, or Crooked Timber, this is also a way to introduce them to the world of academic blogging that they will surely become familiar with when they are procrastinating during grad school.

Overall, I think that Fabio does a nice job of discussing things that grad students should know. I went to grad school in a supportive environment where these issues could be openly discussed with advisors, but not everybody is so lucky. For prospective students, this makes choosing a graduate program incredibly important. For those who are already enrolled, the Rulz should help navigate potentially uncertain waters.

Read Full Post »

“Blind” peer review

The combination of a recent paper submission and a post by Tina at Scatterplot have caused me to wonder about the age-old question of “blind” peer review.  The question, of course, is whether peer reviews can truly be blind in the days of online conference information (sometimes including papers) and internet search engines.  This question came up at Orgtheory a while back, with the definitive follow-up poll suggesting that most people do look up the authors of papers they are reviewing, either before or after the review.  Obviously, older individuals may be less likely to respond to Orgtheory polls and similarly less likely to look up authors in this way, but it is still likely that a blind review will not be blind to all reviewers.

Given that blind peer review is blind for a reason, it seems that we have a problem.  Sure, a few high-profile scholars might be recognizable by their writing style, theoretical perspectives, or citations, but the vast majority of sociologists do not have that problem.  I wonder, for example, how being an unrecognized name from an unrecognized school will affect me when my reviewers attempt to take the blinders off of the peer review process.

Given these issues, it seems that we have a few options.  One is to give up on the illusion of blind reviews and sign both submissions and reviews.  Another option is to take the blinding process further by removing titles as well, since titles are likely the easiest way to search for a paper that has been presented at a conference.  Other options include preventing our conference presentations from being archived online and throwing out the whole presentation and publication model and moving to communes organized by John Galt.  I’m not sure which of these would be most effective.

Read Full Post »

Forgetting what they’ve learned

A recent double post by Nathan Palmer at Scatterplot and The Sociology Source (which shows up in my RSS reader as “Blog”) tackles the issue of making social facts understandable for students.  As Palmer states:

I tell my class to imagine that I have just handed back their graded tests for them to review. I tell them that the class average was a 72%. This, I tell them, is an empirical social fact. The trend or in this case the average for the entire class was 72%.

Then I ask them, “would it make sense if one of you told me ‘the average can’t be a 72% because I got a 96% on my test’?” They laugh at the ridiculousness of this question. “Well when I present to you empirical social facts and you say to me ‘well I know this one guy who doesn’t do what your research says’ or ‘well that’s not true in my experience, so your social fact must be wrong’ you are basically arguing that because you got a 96% the class average can’t be a 72%” Many heads nodding in unison. They get it.

This seems like an excellent way to make this point, given the number of students who have told me that research findings aren’t “true” because they had different experiences.  It reminded me, however, of something else I encountered recently – former sociology students who have forgotten what they learned about social life as a result of the dreaded “real world.”

It seems that the social facts we teach students can be overcome by a few years of job experience.  Former sociology students who gained an in-depth understanding of the long-standing discrimination against blacks, for example, may claim that they are the victims of “reverse discrimination” when they can’t find a job in a recession.  Similarly, knowledge of the burden of the second shift may be overcome by a man who finds that his wife will do the laundry herself if he waits long enough.

While students in our classrooms seem to grasp the concepts we teach, these concepts are often counter to the stereotypical norms of our society.  Once they get it, the larger question becomes how we can get them to keep it.

Read Full Post »