Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Mass_Driver comments on The Optimizer's Curse and How to Beat It - Less Wrong

44 Post author: lukeprog 16 September 2011 02:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (81)

You are viewing a single comment's thread.

Comment author: Mass_Driver 16 September 2011 07:07:48AM 5 points [-]

consider a decision problem in which there are k choices, each of which has true estimated [expected value] of 0.

Lukeprog, if I've understood you correctly, then this is no good; this is a corner case. The question to be answered here is whether we should expect a "common sense" executive who favors plans with a high prior estimate to do better than a "technical" analyst who favors plans that perform well according to the formal estimation criteria. By assuming that all prior estimates are identical except for bias, this assumption ensures that the technical analyst will win. This, however, begs the question. One could just as easily assume that there is large variation in the true expected values, and that the formal criteria will always produce an estimate of 0, in which case the common sense executive will always win.

Am I missing something? I like the topic; I would enjoy reading about which approach we should expect to perform better in a typical situation.

Comment author: Nisan 16 September 2011 09:16:15PM 5 points [-]

I think the case where all the choices has a "true expected value" of 0 is picked out merely to illustrate the problem.

Comment author: lukeprog 16 September 2011 11:08:25PM 0 points [-]

Yes.

Comment author: Mass_Driver 17 September 2011 12:06:25AM 4 points [-]

That's fine; you're more than welcome to illustrate the problem, and your analysis does in fact do that. It does it very well; your writing, as always, is very lucid.

However, you finish the article by claiming that Bayesian analysis can correct for the problem, and this is something that (I don't think) you even begin to show. Bayesian analysis solves the corner case, but does it bring any traction at all on a typical case?

Comment author: [deleted] 16 September 2011 04:29:55PM 2 points [-]

Put another way, the higher the variance in the true payoffs, the less relevant the curse. This is the flipside of: the more accurate the estimates, the less relevant the curse.

Comment author: RobinZ 16 September 2011 03:49:54PM 0 points [-]

I think it's worse than that: Karnofsky's problem is that he has to compare moderate-mean low-variance estimates to large-mean large-variance estimates, but lukeprog's solution is for comparing the estimate to the result in cases where the variance is equal across the board.