You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Incorrect comments on Yet another safe oracle AI proposal - Less Wrong Discussion

2 Post author: jacobt 26 February 2012 11:45PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (33)

You are viewing a single comment's thread. Show more comments above.

Comment author: Incorrect 28 February 2012 04:17:31AM *  0 points [-]

Because it is safe to ask a non-agent Oracle questions which do not have dangerous answers (or create dangerous information while calculating the answer). On the other hand, an Oracle that behaves as an agent is unsafe to ask any question because it might convert the planet to computronium to calculate the answer.

Comment author: orthonormal 28 February 2012 04:28:55AM 0 points [-]

OK, so you're using "behaves as an agent" to mean what I mean by "cares about our universe". It doesn't sound like we disagree on substance (i.e. what such a program would do if it worked as proposed).