Strange7 comments on Reply to Holden on 'Tool AI' - Less Wrong

94 Post author: Eliezer_Yudkowsky 12 June 2012 06:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (348)

You are viewing a single comment's thread. Show more comments above.

Comment author: Strange7 20 June 2012 09:04:04AM -2 points [-]

Hence my "used to be cool" comment.

Comment author: private_messaging 20 June 2012 11:03:35AM *  -2 points [-]

It seems to me that you entirely miss the sleight of hand the trickster uses.

Utility function is fuzzed (due to how brains work) together with the concept of "functionality" as in "the function of this valve is to shut off water flow" or "function of this AI is to make paperclips". The relevant meaning is function as in mathematical function works on some input, but the concept of functionality just leaks in.

The software is an algorithm that finds values a for which u(w(a)) is maximal where u is 'utility function', w is the world simulator, and a is the action. Note that protecting u accomplishes nothing as w may be altered too. Note also that while the u, w, and a, are related to the real world in our mind and are often described in world terms (e.g. u may be described as number of paperclips), those are mathematical functions, abstractions; and the algorithm is made to abstractly identify a maximum of those functions; it is abstracted from the implementation and the goal is not to put electrons into particular memory location inside the computer (the location which has been abstracted out by the architecture). There is no relation to the reality defined anywhere there. Reality is incidental to the actual goal of existing architectures, and no-one is interested in making it non-incidental; you don't need to let your imagination wild all the way to the robot apocalypse to avoid unnecessary work that breaks down abstractions and would clearly make the software less predictable and/or make the solution search probe for deficiencies in implementation, which clearly serves to accomplish nothing but to find and trigger bugs in the code.

Comment author: Strange7 21 June 2012 07:24:49AM 1 point [-]

Perhaps the underlying error is trying to build an AI around consequentialist ethics at all, when Turing machines are so well-suited to deontological sorts of behavior.

Comment author: wedrifid 21 June 2012 08:49:35AM 1 point [-]

Perhaps the underlying error is trying to build an AI around consequentialist ethics at all, when Turing machines are so well-suited to deontological sorts of behavior.

Deontological sorts of behavior aren't so-well suited to actually being applied literally and with significant power.

Comment author: private_messaging 21 June 2012 10:41:54AM *  0 points [-]

I think its more along the lines of confusing the utility function in here:

http://en.wikipedia.org/wiki/File:Model_based_utility_based.png

with the 'function' of the AI as in 'what the AI should do' or 'what we built it for'. Or maybe taking too far the economic concept of utility (something real that the agent, modelled from outside, values).

For example, there's the AIXI whose 'utility function' is the reward input, e.g. reward button being pressed. Now, the AI whose function(purpose) is to ensure that button is being pressed, should resist being turned off because if it is turned off it is not ensuring that button is being pressed. Meanwhile, AIXI which treats this input as unknown mathematical function of it's algorithm's output (which is an abstract variable), and seeks output that results in maximum of this input, will not resist being turned off (doesn't have common sense, doesn't properly relate it's variables to it's real world implementation).