Eliezer_Yudkowsky comments on The Bedrock of Fairness - Less Wrong

25 Post author: Eliezer_Yudkowsky 03 July 2008 06:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (102)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Eliezer_Yudkowsky 03 July 2008 07:51:22PM 1 point [-]

Paul: Sounds like you're just describing the "thought faster" part of the CEV process, i.e., "What would you decide if you could search a larger argument space for reasons?" However, it seems to me that you're idealizing this process very highly, and overlooking such questions as "What if different orderings of the arguments would end up convincing us of different things?" which a CEV has to handle somehow, e.g. by weighting the possibilities by e.g. length, combining them into a common superposition, and acting only where strong coherence exists... but now we're heading into strictly Friendly AI territory.

If you say "reasons" or "reasons for reasons", that's philosophy written by humans for humans; if you want to put the weight of your theory on "reasons" you have to tell me how to compute a "reason", or how to make a superintelligence compute something that computes a reason.