= 27d567bc2d48d9f40e9ccc20ea370b15)
gwern comments on Self-Congratulatory Rationalism - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (395)
This is one of the loonier[1] ideas to be found on Overcoming Bias (and that's quite saying something). Exercise for the reader: test this idea that sharing opinions screens off the usefulness of sharing evidence with the following real-world scenario. I have participated in this scenario several times and know what the correct answer is.
You are on the programme committee of a forthcoming conference, which is meeting to decide which of the submitted papers to accept. Each paper has been refereed by several people, each of whom has given a summary opinion (definite accept, weak accept, weak reject, or definite reject) and supporting evidence for the opinion.
To transact business most efficiently, some papers are judged solely on the summary opinions. Every paper rated a definite accept by every referee for that paper is accepted without further discussion, because if three independent experts all think it's excellent, it probably is, and further discussion is unlikely to change that decision. Similarly, every paper firmly rejected by every referee is rejected. For papers that get a uniformly mediocre rating, the committee have to make some judgement about where to draw the line between filling out the programme and maintaining a high standard.
That leaves a fourth class: papers where the referees disagree sharply. Here is a paper where three referees say definitely accept, one says definitely reject. On another paper, it's the reverse. Another, two each way.
How should the committee decide on these papers? By combining the opinions only, or by reading the supporting evidence?
ETA: [1] By which I mean not "so crazy it must be wrong" but "so wrong it's crazy".
Verbal abuse is not a productive response to the results of an abstract model. Extended imaginary scenarios are not a productive response either. Neither explains why the proofs are wrong or inapplicable, or if inapplicable, why they do not serve useful intellectual purposes such as proving some other claim by contradiction or serving as an ideal to aspire to. Please try to do better.
As I said, the scenario is not imaginary.
I might have done so, had you not inserted that condescending parting shot.
Your real world scenario tells you that sometimes sharing evidence will move judgements in the right direction.
Thinking that Robert Hanson or someone else on Overcoming Bias hasn't thought of that argument is naive. Robert Hanson might sometimes make arguments that are wrong but he's not stupid. If you are treating him as if he would be, then you are likely arguing against a strawman.
Apart from that your example also has strange properties like only four different kind of judgements that reviewers are allowed to make. Why would anyone choose four?
It is a lot more than "sometimes". In my experience (mainly in computing) no journal editor or conference chair will accept a referee's report that provides nothing but an overall rating of the paper. The rubric for the referees often explicitly states that. Where ratings of the same paper differ substantially among referees, the reasons for those differing judgements are examined.
The routine varies but that one is typical. A four-point scale (sometimes with a fifth not on the same dimension: "not relevant to this conference", which trumps the scalar rating). Sometimes they ask for different aspects to be rated separately (originality, significance, presentation, etc.). Plus, of course, the rationale for the verdict, without which the verdict will not be considered and someone else will be found to referee the paper properly.
Anyone is of course welcome to argue that they're all doing it wrong, or to found a journal where publication is decided by simple voting rounds without discussion. However, Aumann's theorem is not that argument, it's not the optimal version of Delphi (according to the paper that gwern quoted), and I'm not aware of any such journal. Maybe Plos ONE? I'm not familiar with their process, but their criteria for inclusion are non-standard.
That just tells us that the journals believe that the rating isn't the only thing that matters. But most journals just do things that make sense to them. The don't draft their policies based on findings of decision science.
Those findings being? Aumann's theorem doesn't go the distance. Anyway, I have no knowledge of how they draft their policies, merely some of what those policies are. Do you have some information to share here?
For example that likert scales are nice if you want someone to give you their opinion.
Of course it might sense to actually do run experiments. Big publishers do rule over 1000's of journals so it should be easy for them to do the necessary research if the wanted to do so.
That was excessive, and I now regret having said it.