MichaelBishop comments on Survey Results - Less Wrong

48 Post author: Yvain 12 May 2009 10:09PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (210)

Comments (210)

Sort By: Old

Show:

You are viewing a single comment's thread. Show more comments above.

Comment author: MichaelBishop 13 May 2009 03:10:10PM * 5 points [-]

I believe in the wisdom of crowds, but I also think that your actions were screwing up the results.

If you weren't going to take a question seriously, I wish you wouldn't have answered it at all.

ADDED: I decided not to downvote you because I don't want to discourage being honest/forthcoming.

Comment author: kpreid 13 May 2009 04:18:03PM 4 points [-]

0% confidence should mean zero weight when computing the results, no?

Comment author: orthonormal 13 May 2009 06:28:06PM 1 point [-]

That's an interesting idea, but I think Yvain just averaged the answers without regard to confidence.

Comment author: MichaelBishop 13 May 2009 07:22:20PM 2 points [-]

Yes, but what was the point of that survey question? Among other things, it could assess a) the distribution of the survey takers accuracy, b) the distribution of the survey takers calibration, c) the relationship of accuracy and calibration to other personal characteristics.

I don't mean to make an overly-big-deal about this, and I appreciate thomblake's other contributions to the LW community, but because he didn't really give us his best guess about when the lightbulb was invented, he reduced our ability to learn all these things.

Comment author: thomblake comment score below threshold [+] (18 children)

Comment author: MichaelBishop 13 May 2009 07:49:58PM 1 point [-]

I'm intrigued. Please point me to a discussion of these issues or make a top level post.

Comment author: dfranke 13 May 2009 08:07:09PM 2 points [-]

Sounds to me like simply a rejection of the Bayesian interpretation of probability based on the usual frequentist objections.

Comment author: Nominull 13 May 2009 09:38:22PM 5 points [-]

I thought you were a believer in the wisdom of crowds, though? The magic of collaborative estimation doesn't occur if everybody who isn't absolutely sure shuts up.

Comment author: Craig_Morgan 14 May 2009 02:34:50AM 1 point [-]

At the time of this comment, thomblake's above comment is at -3 points and there are no comments arguing against his opinion, or why he is wrong. We should not downvote a comment simply because we disagree with it. Thomblake expressed an opinion that differs (I presume) from the community majority. A better response to such an expressed opinion is to present arguments that correct his belief. Voting based on agreement/disagreement will lead people not to express viewpoints they believe differ from the community's.

Comment author: jimrandomh 14 May 2009 02:52:22AM * 3 points [-]

We should not downvote a comment simply because we disagree with it.

This sounds great in theory, but other communities have applied that policy with terrible results. Whether I agree with something or not is the only information I have as to whether it's true/wise, and that should be the main factor determining score. Excluding disagreement as grounds for downvoting leaves only presentation, resulting in posts that are eloquent, highly rated, and wrong. Those are mental poison.

Comment author: JGWeissman 14 May 2009 04:07:25AM 5 points [-]

When someone honestly presents their position, and is open to discussing it further, there is no need to down vote their comment for being wrong. In fact, it is counterproductive. By discouraging them from expressing their incorrect position, you do not cause them to relinquish it. By instead explaining why you think it is wrong, you help them to adopt a better position. And if it happens that they were right and you were wrong, then you have the opportunity to learn something.

I tend to down vote comments that are off topic, incoherent, arrogant, or present a conclusion without support.

I tend to up vote comments when they are eloquent, insightful, and correct, or sometimes, when they say pretty much what I was planning to say.

Comment author: Z_M_Davis 14 May 2009 04:49:16AM 3 points [-]

While I agree that voting shouldn't be based strictly on agreement/disagreement, voting is supposed to be an indicator of comment quality, with downvotes going to poorly-argued comments that one would like to see less of. It is worth bearing in mind that the more mistaken a conclusion is, the less likely one is to encounter strong comments in support of that conclusion.

If someone were to present specific, clearly-articulated arguments purporting to show that popular notions of accuracy and calibration are mistaken, that might well deserve an upvote in my book. But above, thomblake seems to be rejecting out of hand the very notion of decisionmaking under uncertainty, which seems to me to be absolutely fundamental to the study of rationality. (The very name Less Wrong denotes wanting beliefs that are closer to the truth, even if one knows that not everything one believes is perfectly true.) I've downvoted thomblake's comment for this reason, and I've downvoted your comment because I don't think it advances the discourse to discourage downvotes of poor comments.

Comment author: thomblake 14 May 2009 02:16:09PM 0 points [-]

rejecting out of hand the very notion of decisionmaking under uncertainty

Nope. I'm something of a Popperian. On things I care about, I find the best position I can and act as though I'm 100% certain of it. When another position is shown to be superior, I reject the original view entirely.

There are some circumstances where we need to make a decision without anything we can feel that strongly about, but I think that in most circumstances, bringing 'probability' into the process isn't helpful. Humans just aren't built to think like that, and I'd rather use just plain 'judgment'.

Comment author: Z_M_Davis 14 May 2009 06:48:25PM 1 point [-]

On things I care about, I find the best position I can and act as though I'm 100% certain of it. When another position is shown to be superior, I reject the original view entirely.

Thank you for the clarification. Although frankly, I don't see how that could possibly work. I mean, suppose someone flips a coin that you know to be slightly biased towards heads. Would you be willing to bet a thousand dollars that the coin comes up heads?

Okay, coinflip examples are always somewhat contrived (although I really could offer you that bet), but we can come up with much more realistic scenarios, too. Say you've just been on a job interview that went really well, and it seems like you're going to get the position. Do you therefore not bother to apply anywhere else, acting with 100% certainty that you will get the job? Or I guess you could alternatively say that you simply "don't know" whether you'll get the position--but can "I don't know" really be a complete summary of your epistemological state? Wouldn't you at least have some qualitative feeling of "The interview went well; I'll probably get hired" versus "The interview did not go well at all; I probably won't get hired"?

Humans just aren't built to think like that

I certainly agree that humans aren't built to naturally think in terms of probabilities, but I see no reason to believe that human-default modes of reasoning are normative: we could just be systematically stupid on an absolute scale, and indeed I am rather convinced this is the case.

bringing 'probability' into the process isn't helpful

I agree that it would be incredibly silly to try to explicitly calculate all your beliefs using probability theory. But a qualitative or implicit notion of probability does seem natural. You don't have to think in terms of likelihood ratios to say things like "It's probably going to rain today" or "I think I locked the door, but I'm not entirely sure." Is this the sort of thing that you mean by the word judgment? In any case, even if bringing probability into the process isn't helpful, bringing in this dichotomy between absolutely-certain-until-proven-otherwise and complete ignorance seems downright harmful. I mean, what do you do when you think you've locked the door, but you're not entirely sure? Or does that just not happen to you?

Comment author: JGWeissman 14 May 2009 07:29:57PM 3 points [-]

I mean, suppose someone flips a coin that you know to be slightly biased towards heads. Would you be willing to bet a thousand dollars that the coin comes up heads?

Well, that is what Bayesian decision theory would suggest you do, provided your utility function is linear with respect to money.

But, to illustrate the problem with acting as though you were 100% certain in your best theory, suppose I offer you the following bet. I will roll an ordinary 6 sided dice, and if the result is between 1 and 4 (inclusively), I will pay you $10. But if the result is 5 or 6, you will pay me $100. So you see that getting a result between 1 and 4 is more likely than getting a 5 or a 6, so you treat it as certain, so you accept my bet which you assign an expected value of $10. But really, the expected value is (2/3)$10 - (1/3)$100 = -$80/3. On average, you lose about $27 with this bet.

The problem here is that, by acting as though you are 100% sure, you give no weight to the potential costs of being wrong (including the opportunity of cost of the potential benefits of a different decision).

Comment author: Z_M_Davis 14 May 2009 10:16:29PM 1 point [-]

Well, that is what Bayesian decision theory would suggest you do, provided your utility function is linear with respect to money.

Right; I wasn't thinking. Your example is better.

Comment author: thomblake 15 May 2009 05:43:26PM 0 points [-]

I was talking about ordinary circumstances. I've never bet money on the roll of a die, nor shall I. If it were to come up, I might well do the sort of analysis you suggest, as probability seems like it's correctly applied to die rolling. Can you think of a better example, that might actually occur in my life?

continue this thread »

Comment author: jimrandomh 14 May 2009 02:41:24AM 1 point [-]

When providing what you think is the correct answer, there is still some probability that you're mistaken. That probability could be 10^-10, but it can't be zero. And when answering "I don't know", you can still guess, and produce a probability that your guess was correct. Lumping all low probabilities of being correct into a single qualitative judgment of "I don't know" sometimes makes sense, but sometimes a concrete probability is useful so you should know how to generate one.

Comment author: randallsquared 13 May 2009 04:31:14PM 1 point [-]

I believe in the wisdom of crowds, but I also think that your actions were screwing up the results.

This seems contradictory. Care to explain?

Comment author: orthonormal 13 May 2009 05:11:32PM 6 points [-]

The "wisdom of crowds" would only apply if everyone is trying to actually get the answer right, and if the errors of incompetence are somewhat random. A large number of intentional pranksters (or one prankster who says "a googolplex") can predictably screw up the average by introducing large variance or acting in a non-random fashion.

MichaelBishop comments on Survey Results - Less Wrong

Article Navigation

Comments (210)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (210)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login