Wei_Dai comments on Safe questions to ask an Oracle? - Less Wrong

2 Post author: Stuart_Armstrong 27 January 2012 06:33PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (41)

You are viewing a single comment's thread.

Comment author: Wei_Dai 23 February 2012 01:09:29AM 3 points [-]

This seems like a really bad question. If we consider an Oracle AI and its human users as a system, then the factors that contribute to risk/safety include at least:

  1. design of the OAI
  2. its utility function
  3. background knowledge it's given access to
  4. containment methods
  5. the questions
  6. what we do with the answers after we get them

All of these interact in complex ways so a question that is safe in one context could be unsafe in another. You say "interpret the question as narrowly or broadly as you want" but how else can we interpret it except "design an Oracle AI system (elements 1-6) that is safe"?

Besides this, I agree with FAWS that we should (if we ought to be thinking about OAI at all) be thinking about how we can use it to reduce existential risk or achieve a positive Singularity, which seems a very different problem from "safe questions".