wedrifid comments on The mathematics of reduced impact: help needed - Less Wrong

10 Post author: Stuart_Armstrong 16 February 2012 02:23PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (94)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 20 February 2012 07:12:20AM 0 points [-]

As the current ultimate authority on AI safety I am curious if you would consider the safety profile of this oracle as interpreted here to be along the lines I describe there. That is, if it could actually be constructed as defined it would be more or less safe with respect to its own operation except for those pesky N bits and what external entities can do with them.

Unless I have missed something the problem with attempting to implement such an AI as a practical strategy are:

  • It is an infinity plus one sword - you can't just leave those lying around.
  • The research required to create the oracle is almost all of what it takes to create an FAI. It requires all of the research that goes into FAI for CEV research - and if the oracle is able to answer questions that are simple math proofs then even a significant part of what constitutes a CEV implementation would be required.