http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=6577844

"Speculations concerning the first ultraintelligent machine" started with the dubious sentence "The survival of man [sic] depends on the early construction of an ultraintelligent machine." That might have been appropriate during the Cold War. Today I suspect that the word survival should be replaced by extinction.

I.J. Good (1998)

This is called the optimizer's curse. There are standard ways to solve the problem, e.g. multilevel modeling.

Yeah. I hope to eventually write a next-round reply to some of what you've said here, what Beckstead said there, and what Karnofsky said here and here. But I'm not sure when I'll get to that; it's a ways down my to-do list.

http://link.springer.com/chapter/10.1007/978-94-009-6340-5_10

PDF.

Also see Michael Nielsen's ideas in Reinventing Discovery.

Good post! I think that are still some disagreements to be worked out with regard to how this works out in practice, but I appreciate your contribution to the dialogue.

Only say 'rational' when you can't eliminate the word.

Thanks!