You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Lumifer comments on Open Thread August 31 - September 6 - Less Wrong Discussion

5 Post author: Elo 30 August 2015 09:26PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (326)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 03 September 2015 04:38:37PM *  12 points [-]

If MIRI doesn't publish reasonably frequently (via peer review), how do you know they aren't wasting donor money?

How did science get done for the centuries before peer review? Why do you place such weight on such a recently invented construct like peer review (you may remember Einstein being so enraged by the first and only time he tried out this new thing called 'peer review' that he vowed to never again submit anything to a 'peer reviewed' journal), a construct which routinely fails anytime it's evaluated and has been shown to be extremely unreliable where the same paper can be accepted and rejected based on chance? If peer-review is so good, why do so many terrible papers get published and great Nobel-prize-winning work rejected repeatedly? If peer review is such an effective method of divining quality, why do many communities seem to get along fine with desultory use of peer review where it's barely used or left as the final step long after the results have been disseminated and evaluated and people don't even bother to read the final peer-reviewed version (particularly in economics, I get the impression that everyone reads the preprints & working papers and the final publication comes as a non-event; which has caused me serious trouble in the past in trying to figure out what to cite and whether one cite is the same as another; and of course, I'm not always clear on where various statistics or machine learning papers get published, or if they are published in any sense beyond posting to ArXiv)? And why does all the real criticism and debate and refutations seem to take place on blogs & Twitter if peer-review is such an acid test of whether papers are gold or dross, leading to the growing need for altmetrics and other ways of dealing with the 'post-publication peer review' problem as journals increasingly fail to reflect where scientific debates actually are?

I've said it before and I'll said it again: 'peer review' is not a core element of science. It's barely even peripheral and unclear it adds anything on net. For the most part, calls for 'peer review' are cargo culting. What makes science work is replication and putting your work out there for community evaluation. Those are the real review by peers.

If you are a donor who wants to evaluate MIRI, whether some arbitrary reviewers pass or fail its papers is not very important. There are better measures of impact: is anyone building on their work? have MIRI-specific claims begun filtering out? are non-affiliated academics starting to move into the AI risk field? Heck, even citation counts would probably be better here.

Comment author: Lumifer 03 September 2015 06:39:37PM 3 points [-]

How did science get done for the centuries before peer review?

Mostly by well-off people satisfying their personal curiosity. Other than that, by finding a rich and/or powerful patron and keeping him amused :-D

I agree that the cult of peer review is overblown. But does MIRI produce any relevant and falsifiable output at all?

Comment author: Jiro 03 September 2015 08:48:26PM 0 points [-]

How did science get done for the centuries before peer review?

I would answer differently than you: "Very inefficiently and with lots of errors".

Comment author: Good_Burning_Plastic 08 September 2015 12:05:50PM 3 points [-]

I would answer differently than you: "Very inefficiently and with lots of errors".

As opposed to quick, reliable present-day peer-reviewed science? ;-)

Comment author: Lumifer 03 September 2015 08:55:06PM 3 points [-]

"Very inefficiently and with lots of errors"

Well, not that this has changed...

Comment author: Douglas_Knight 06 September 2015 06:48:51PM 1 point [-]

What leads you to that conclusion? When do you think peer review began and how do you judge efficiency before and after?