The American Economic Association's decision to switch from double-blind to single-blind reviewing has attracted much attention, for example:
http://chronicle.com/blogs/ticker/leading-economics-journals-drop-double-blind-peer-review/33462
These articles do a good job of setting out various sides of the argument. Those who advocate double-blind reviewing maintain referees' decisions should not be biased by preconceptions about the authors. (They haven't, but could, note that double-blind refereeing is being dropped as American pre-eminence in academic publishing is sliding).
Those who argue against say that double-blind reviewing prevents referees from identifying potential conflicts of interests. Also, in practice refereeing is never actually double-blind because it is almost always possible to discover who the author of a paper is through an internet search.
I've set up a little survey to collect the views of WCI readers on this issue:
https://www.surveymonkey.com/s/DKMB5L8
Once I have a few responses, I'll provide you with some more analysis.
Worthwhile Canadian Initiative 
I don’t think any double-blind reviewing happens in economics. Not posting one’s working papers online is not reasonable; it’s how we communicate with our peers given the long and variable lags in the publication process. The AEA decision is just acknowledging the state of affairs. I think most people would support a double-blind standard if possible, but I just don’t see how it could possibly be implemented.
Andrew: “I don’t think any double-blind reviewing happens in economics.”
Most weekends I do a crossword puzzle. I could just flip the page to see the answers to the puzzle, but I don’t. That’s not part of the rules of the game. (I could also google answers, but I don’t do that either.)
Sure, people generally can find out the identity of an author. But do they? I don’t know. (I’ll add that to the quiz).
I filled out the survey but it reveals one of my own biases. I think double-blind benefits women because I presume relatively unknown female economists are scrutinized more thoroughly and harshly. If it turns out female authors are instead subject to a lower standard in an effort to give female economists a louder voice, double-blind could hinder them from a count-the-publications perspective…
More seriously, I am still close enough to my student days to remember that in rare cases the identity of the student altered the grade. It should not happen but it does, and people who claim not to be influenced by knowledge of the author of lying to themselves. A-students are A-students and B-students are B-students.
As for conflicts of interest, if they are not declared by the authort surely the addition of all AEA readers will do a much better job than the addition of just the reviewers?
That said, this rule change depends on how it is implemented. Is the AEA decision a reflection of the difficulty in guaranteeing double-blind? There is a big difference between being told “Please review Frances’ paper” and “Pleae review this paper; it’s already posted online and you may have seen drafts before but please review it anyway even if you know who the author is.”
Frances: I think most people google the paper to know the author.
The rule is if the author is unknown, the referee will check more the details to make sure everything is fine.
if the author is well-known and known to do carefull paper. The referee don’t have to triple check details…
In general, when I’m refereeing, I prefer not to know who the author is.
Here’s a Bayesian analysis.
Suppose you are a referee (or an editor) trying to evaluate the quality of a paper as best you can. You read the paper (read the referees’ reports) and make your best estimate of the quality of the paper on that basis. But you are aware that your estimate cannot be perfectly accurate. Perhaps the quality of the paper is better, or worse, than you think it is, based on your reading of the paper (or reading those reports).
Suppose you knew the identity of the author, but hadn’t (yet) read the paper (or the referees’ reports). You would have some prior estimate, based on your beliefs about the quality of the author, on whether the paper was likely to be of high quality.
A good Bayesian would say that you should base your posterior estimate of the quality of the paper on both your reading of the paper (reading the referees’ reports) and on your prior estimate, based on the author’s quality.
In principle, you should use the information given by who the author is in exactly the same way that you would use your prior knowledge of the probability of having condition X when you get tested by the doctor for condition X. See Mike’s post on amniocentesis: http://worthwhile.typepad.com/worthwhile_canadian_initi/2011/05/probability-theory-question-of-the-day.html
(Has this point about Bayesian reasoning already been made in one of those links Frances gave?)
To put it more controversially, what most people call “prejudice”, Bayesians call a “prior”.
You might respond that it is the referee’s job to report the likelihood function, and the editor’s job to incorporate the prior. But then someone might respond to that response that the referees might have a better estimate of the author’s quality within a particular area than the editor.
(I can’t decide whether I would accept this Bayesian reasoning, BTW. The exact same point would seem to apply in principle to a judge or jury knowing the past criminal record of a defendant.)
So economics is admitting that isn’t a science? Or am I missing something?
reason, there’s tons of subjectivity in the publication process of all fields. If this means economics is not a science then neither is physics or math or anything else.
the fact is, a large proportion of referee comments are to get their own work referenced.
At the same time, I’ll tell you a story from math (true story from my dad): he and co-authors submit a paper to a journal, the referee recomends rejection based on the result being too trivial. The editor lets them re-submit, they delete roughly half the detail in their proofs and the referee recomends acceptance (different refs I suppose but still…)
John: “if the author is well-known and known to do carefull paper. The referee don’t have to triple check details…”
Your comment is the most persuasive argument I’ve seen yet for double-blind refereeing.
Nick: “A good Bayesian would say that you should base your posterior estimate of the quality of the paper on both your reading of the paper (reading the referees’ reports) and on your prior estimate, based on the author’s quality.”
In hiring, if we applied that kind of reasoning using gender or racially based criteria, that would be discrimination. (“People of ethnicity X, on average, publish more so, all observable characteristics being equal, on average one is better off hiring someone of ethnicity X”). But, when hiring, we do use information about where a person obtained his/her PhD in precisely this way.
Yet, as reason said, isn’t the idea of scientific study that the quality of work can be objectively measured?
Adam P: “a large proportion of referee comments are to get their own work referenced.” As an editor, that’s not been my experience. I am amazed by the quality and thoughtfulness of many of the comments referees send me. The problem is when I have to ask 10 people before I can find someone to referee a paper.
Well, I do remember having to do some acrobatics in order to incorporate some tangential digression that was clearly a referee’s bugbear. Oh well, he got a citation and I got the publication.
My own experience with the AER’s double-blind was that my co-author Pascal St-Amour and I had to waive it after a referee asked for clarification on some points that were covered in another paper we were working on. We had to put out a working paper version and let him know where to find it.
Actually, that whole AER refereeing process was a nightmare. One referee was mildly positive, and the other wanted a whole different paper. We worked like dogs on the revision – and both referees changed their minds. The one who liked the original didn’t like the revision, and vice-versa.
The version that got published ended up being almost identical to the original submission, with a footnote added pointing to the other working paper. (Which eventually came out in the JBES.)
When refereeing I don’t just evaluate the paper, but I also look for any evidence of plagiarism — hence the Internet searches. On more than a few occasions I’ve found authors overstating the originality of their results, or even copying and pasting entire passages from their previously published, but uncited, papers. Nine times out of ten this reveals the identity of the author, so in this sense I agree with the AEA: Double-blind refereeing is great in principle, but in practice it is difficult to implement with current technology.
I’ve refereed a number of papers in the past and you can usually tell who the author(s) are because they cite themselves and their advisors heavily. If anyone gets a paper with 3 or more references to me in it, they can be 100% sure I wrote it.
Of course, this method doesn’t work too well if the author is someone like, say, Greg Mankiw, who is going to be heavily cited anyway. But if a virtual unknown keeps getting cited then you can be pretty sure that virtual unknown is the one who wrote the paper.
Frances:
I think there is a fundamental difference between a crossword and avoiding looking up the answers because it is a self challenge. Yes, it shows that people can have restraint, but the degree depends on the motivation for the restraint. People may be less self-motivated for restraint when refereeing a paper.
Generally:
Don’t academic journals now act as a stamp of approval for the ideas and papers that are published? They are not necessary for distribution or to enter in to an academic conversation. They are explicitly a gate keeping mechanism. Outsiders can now use other tools to promote their ideas and enter into conversations. If their ideas are good enough, presumable they will be picked up in the profession, and then the gate-keepers at the journals will have to admit them. Of course, this is still a highly biased and reputation based process.
Whitfit: “Don’t academic journals now act as a stamp of approval for the ideas and papers that are published? They are not necessary for distribution or to enter in to an academic conversation. They are explicitly a gate keeping mechanism.”
I’m not sure whether you’re attempting to defend single-blind review or making a damning indictment of the enter academic publication game? Perhaps both?
Mike: “But if a virtual unknown keeps getting cited then you can be pretty sure that virtual unknown is the one who wrote the paper.”
Or should you just reject a paper by someone not smart enough to hide their (low-rank) identity on principle?
“Or should you just reject a paper by someone not smart enough to hide their (low-rank) identity on principle?”
Ha! As a game theory, I’m embarrassed that I hadn’t thought of that.
Game theorist.
In epidemiology, double blind is the exception rather than the rule. Some very high status places will leave in a revealing detail (like which research ethics board approved the study) that break part of the blinding. I slightly prefer double blind (despite a natural curiousty to know what did the research) and I think I review better that way. I have also seen the unblinded review model in medical science (BMC journals) and there the reviews tended to be much more positive than in single or double blind journals.
All that being said, in my exact area of concentration I can almost always pick out the exact research group based on the data they used, the details they focus on, and the writing style used. Once or two it was a new group and I was just stumped (and looked forward to finding out when the paper was published). We don’t usually have working papers, either, which also changes things.
One thing that I have encountered that I don’t think is ideal is that when the reviewer is not blinded they will occasionally contact the authors directly. Rarely has this been a good thing.
I remember my first article as a referee; by the second line of the abstract you knew the heavyweight he was. By the fourth he was almost shouting: “Dare us to turn it down and face me at your next grant application!”
Main problem was that it was not really in my field and I didn’t felt comfortable reviewing it.I called the editor who told me : ” That’s why we choose you. He can’t cut your grant…”
Nick:
“(I can’t decide whether I would accept this Bayesian reasoning, BTW. The exact same point would seem to apply in principle to a judge or jury knowing the past criminal record of a defendant.)”
Judges and juries do it. Previous record is always presented. “Goes to credibility , your Honor.”
There is an old saying in French police and judicial circles: three acquittals equal one guilty verdict.
Double blind peer review was the best we could do in snail-mail times. Today, you should publish your working paper in blog form and let the thread rip. When it calms down, there you are and the editor can say yes or no.
In effect a Delphi.
http://en.wikipedia.org/wiki/Delphi_method
Frances:
“I’m not sure whether you’re attempting to defend single-blind review or making a damning indictment of the enter academic publication game? Perhaps both?”
I’m not sure – more of a positive than normative statement really. I’m just not sure how much it matters in a world where ideas and findings are blogged about, posted in working paper form, discussed and otherwise disseminated ideas before getting to the publishing stage. Of course, this may only describe a small portion of economic papers of the type published in the AER. But, it seems to me that there is a benefit to having a discussion about a paper with different people, in as broad and open a process as is reasonable.
I moved from economics to law, and it seems to me that many people post and discuss papers before they get to the publishing stage, and that anyone that would be able to adequately evaluate a paper might have already read it at the review stage.
Although I find it more and common that I’ve already seen the papers that I’m sent to referee, I’ve also done double-blind referee reports where I was unable to find the author’s identity.
On at least one occasion I had to know his/her/their identity, because I had seen a paper on the same topic with a different title; if they were two distinct papers, I felt the anonymous one was not original enough to warrant publication. In the end, the editor told me that the paper was simply a revised and retitled version of the paper that I had seen.