private_messaging comments on Reply to Holden on 'Tool AI' - Less Wrong

94 Post author: Eliezer_Yudkowsky 12 June 2012 06:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (348)

You are viewing a single comment's thread. Show more comments above.

Comment author: private_messaging 15 June 2012 01:55:16AM *  -2 points [-]

I don't see why the 'oracle' has to work from some real world goal in the first place. The oracle may have as it's terminal goal the output of the relevant information on the screen with the level of clutter compatible with human visual cortex, and that's it. Up to you to ask it to represent it in particular way.

Or not even that; the terminal goal of the mathematical system is to make some variables represent such output; an implementation of such system has those variables be computed and copied to the screen as pixels. The resulting system does not even self preserve; the abstract computation making abstract variables have certain abstract values is attained in the relevant sense even if the implementation is physically destroyed. (this is how software currently works)

Comment author: Armok_GoB 15 June 2012 02:32:08AM -1 points [-]

The screen is a part of the real world.