= 27d567bc2d48d9f40e9ccc20ea370b15)
JGWeissman comments on 3 Levels of Rationality Verification - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (182)
A school that reports to each student their class ranking easily games this test. The test could even favor schools that don't teach students enough to question an arbitrary class rank.
Also, this doesn't consider the possibility that students can be good rationalists, but don't interact with enough of the other students to make a good assessment of their relative strengths.
Good rationalists, taken as a group, shouldn't be systematically optimistic.
They should be if they want to win in practice, as opposed to just getting theoretically-correct answers. See, e.g., the studies referenced in Seligman's "Learned Optimism", that show optimists consistently out-perform pessimists (i.e., realists) in a wide variety of fields and endeavors.
(Of course, Seligman's definition of optimism may be different from yours.)
Perhaps we can still test for this systematic optimism, while filtering for the noise I objected to, by instead of asking a "yes" or "no" question, asking for the probability that the student is in the top 50%. Treat the sum of these probabilities as the count of "yes" answers in the original version. Then a rational student should be able to account for his ignorance of other students in his answer.
This is even easier to game: assuming the school has any merit, any individual you ask should have good incentive to simply say "50%" guaranteeing a perfect score. The very first time you used the test it might be okay, but only if nobody knew that the school's reputation was at stake.