Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Vladimir_Golovin comments on Free to Optimize - Less Wrong

25 Post author: Eliezer_Yudkowsky 02 January 2009 01:41AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (80)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Vladimir_Golovin 02 January 2009 08:30:35PM 1 point [-]

EY: The desire not to be optimized too hard by an outside agent is one of the structurally nontrivial aspects of human morality.

The vast majority of optimization-capable agents encountered by humans during their evolutionary history were selfish entities, squeezing their futures into their preferred regions. Given enough evolutionary time, any mutant humans who didn't resist outside manipulation would end up 'optimized' to serve as slave labor in favor of the 'optimizers'.

EY: would be the gift of a world that works on improved rules

Yes, just plug the most important holes (accidental death, unwanted suffering, illness, justice, asteroids, etc.), and leave people have fun.