You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

How to offer peer review comments

6 Post author: lukeprog 04 April 2011 04:24PM

I was recently asked by an editor to offer (single-blind) peer review comments for some upcoming academic work on the Singularity. Having never done this before, I sought out some literature on how to do it.

In case others find themselves in this position, here are some helpful papers I found:

But I know that many Less Wrongers have been engaged in the peer review process for many years. If so, please offer your own comments or link to other helpful instructions!

Comments (3)

Comment author: Vladimir_M 05 April 2011 08:11:00AM 6 points [-]

Considering your knowledge about the topic, the usual level of care and effort you put into your writing, and your apparent enthusiasm for the task, I am pretty sure that you'll do a better job than at least 99% of reviewers in practice.

In my experience from inside the academic sausage factory, most peer reviews are quite awful. (Bryan Caplan is one academic who openly agrees.) This is not very surprising considering that the work is difficult, tedious, unpaid, uncredited, without any penalties for doing a bad job, and usually passed on to otherwise overworked, underpaid, and apathetic grad students. If you just manage to understand the paper and offer any sensible criticism and suggestions, that will be magnificent by the usual standards.

With this in mind, these papers you cite strike me as rather detached from how things really work in practice. Such high and elaborate intellectual standards might be realistic for reviewers writing publicly and with some incentive to do a good job, but definitely not for the way anonymous peer review works presently.

Comment author: James_Miller 04 April 2011 04:49:50PM 5 points [-]

There is a huge variance in the type of comments reviewers give. Keep in mind that because of the large number of submissions the Singularity book has received the editors will, I'm guessing, be rejecting most abstracts and so if you think the abstract should be accepted you need to provide compelling reasons to justify your decision.

One of the best ways to help an author would be to give him a full citation or link to writings that strongly relate to his topic but which he appears unaware of.

Reviewing an abstract is very different from reviewing a full paper so if you're doing the former don't necessarily rely on the advice of people explaining how you should do the latter.

Comment author: nerzhin 05 April 2011 01:58:39PM 2 points [-]

There was a very interesting discussion of exactly this question (at least as it relates to the mathematics community) on mathoverflow recently.