Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Diadem comments on Initiation Ceremony - Less Wrong

49 Post author: Eliezer_Yudkowsky 28 March 2008 08:40PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (91)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Diadem 04 September 2014 08:35:13PM 5 points [-]

I know this is a very old story, but I have some thoughts on it I wanted to share.

Let me first share an experience that I think everybody who has ever seriously studied math (or any complicated subject) has had. You're working on a difficult math problem, say a complicated differential equation. You are certain your method is correct, but still your answer is wrong. You've checked your work, you've double checked it, you've checked it again. Your calculation seems flawless.. Finally, in desperation, you ask a friend for help. Your friend takes one glance at your work, smiles, and says: "Four times five does not equal twelve"... Oh. Yeah. Right. Good point.

We all make mistakes. Even very skilled people sometimes make elementary mistakes. Brennan in the story is doing a calculation that is very trivial for him, but it is still possible. Even if he can't see a flaw, can't even imagine a flaw, that doesn't mean the odds are zero.

Yes, they are certainly very small. Brennan is saying "The odds of me making a mistake are very small, so I am confident I am correct". But this is the Bayesian Conspiracy, not the Frequentist Conspiracy. Brennan should be asking: "Given that someone has clearly made a mistake, what are the odds of me having made it, instead of every other person in the Conspiracy. The answer is obvious.

Thus, Brennan fails as a Bayesian, and should not be accepted into the Conspiracy.

And I am not merely making a pedantic point here. This is a very important point for the real world as well. Yes, standing up to peer pressure is important, but only when it is rational. Global warning deniers also think they are standing up to peer pressure. Creationists also think they are standing up to peer pressure. And often for the exact same reason that Brennan is doing so, in this story. They thought about the issue themselves, they may even know a thing or two about it, and they really can not see any flaw in their logic, so they stick with it, convinced the odds of them having made a mistake are very small, forgetting about the huge prior.

This is actually my first post on this site. I have read quite a bit, but not everything, so I hope I am not inadvertently saying something that has been discussed before. I couldn't find anything, and I think it's an important point.

Comment author: RichardKennaway 04 September 2014 09:00:45PM 1 point [-]

Brennan should be asking: "Given that someone has clearly made a mistake, what are the odds of me having made it, instead of every other person in the Conspiracy.

But in fact, in the story neither of those hypotheses hold. No-one is making a mistake.

Comment author: Vaniver 04 September 2014 09:48:52PM 2 points [-]

This is actually my first post on this site.

Welcome!

Given that someone has clearly made a mistake, what are the odds of me having made it, instead of every other person in the Conspiracy. The answer is obvious.

But this assumes that the Conspirator is telling him the truth, instead of testing him. I think Brennan is right in considering alternative hypotheses about the Conspirator's motives.

And I am not merely making a pedantic point here. This is a very important point for the real world as well.

There are questions of 'epistemic hygiene' here. If I hold a belief because someone else I trust holds that belief, I need to be careful that I don't give other people the impression that I'm providing independent verification of that belief instead of just importing their belief. If Brennan calculated a different answer, him telling the group that will allow them to converge more quickly to the correct belief (even if he acts on the group consensus belief, because that's the one that he trusts more than his private belief!).

Comment author: Diadem 04 September 2014 11:19:33PM 1 point [-]

Thanks!

You are right that there is also the scenario that the test givers are lying (which in this case turns out to be the truth). But this is not something Brennan in the story considers, so it can not have affected his decision. So he arrived at the correct answer, but did so by faulty logic. His two errors (not considering one possible scenario, and assigning wrong odds to the two scenarios he does consider) just happen to cancel out. It would certainly be a way to fix this story: Let Brennan first realize that he should trust everybody else over himself, and then realize that the examiner may be lying.

Though there remains a problem. If the conspirators are lying, it is not clear what answer they want. It may be a test to see if he can withstand peer pressure, but it might also be a test to see if he is willing to entertain the notion of being wrong!

Finally, yes, you are absolutely right that holding a believe because others hold it does not constitute proof. So perhaps the most rational answer would be: "My own independent calculation tells me that the answer is 2 in 9, and for the purpose of establishing a consensus opinion on this question, that is my answer. However I do not think that my evidence is enough to shift the consensus opinion away from the answer of 1 in 6, and thus this is what I shall consider the correct answer, despite my own intuition".

Comment author: VAuroch 05 September 2014 01:06:40AM *  1 point [-]

But this is not something Brennan in the story considers,

But it is! He recalculates - aloud, which makes him less likely to repeat a mistake, and more likely to catch it if he does - and then, reaching the same conclusion as before, gives it again. He thinks he might have made a mistake, which is a reasonable thought, so he works it out again.

In other words,

It may be a test to see if he can withstand peer pressure, but it might also be a test to see if he is willing to entertain the notion of being wrong!

but he does the correct thing for both of those scenarios. He entertains the notion of being wrong, and calculates his odds publicly, where anyone could point out to him if he has found that 4 times 5 is 12, but, finding the same conclusion as before, he stands up to peer pressure.