billswift comments on What I Think, If Not Why - Less Wrong

25 Post author: Eliezer_Yudkowsky 11 December 2008 05:41PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (100)

Sort By: Old

You are viewing a single comment's thread.

Comment author: billswift 12 December 2008 03:37:00PM 0 points [-]

I see a possible problem with FAI in general. The real world is deterministic on the most fundamental level, but not even a super-powerful computer could handle realistic problems at that level, so it uses stochastic, Bayesian, probabilistic, whatever you want to call them methods to model the apparent randomness at more tractable levels. Once it start uses these methods for other problems, what is to stop it from applying them to its goal-system (meta-ethical or whatever you want to call it)? Not in an attempt to become unFriendly, but to improve its goal-system when it realizes that there may be room for improvement in that system also. And it discovers that Friendliness isn't a necessary part of itself but was programmed in and can be modified.