wedrifid comments on What if AI doesn't quite go FOOM? - Less Wrong

11 Post author: Mass_Driver 20 June 2010 12:03AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (186)

Comments (186)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 23 June 2010 08:30:22PM *  5 points [-]

You're familiar with CEV so I'll try to reply with the concepts from Eliezer's CEV document.

Defining Friendliness is not the life-or-death problem on which the survival of humanity depends. It is a life-or-death problem, but not the life-or-death problem. Friendly AI requires:

  1. Solving the technical problems required to maintain a well-specified abstract invariant in a self-modifying goal system. (Interestingly, this problem is relatively straightforward from a theoretical standpoint.)
  2. Choosing something nice to do with the AI. This is about midway in theoretical hairiness between problems 1 and 3.
  3. Designing a framework for an abstract invariant that doesn't automatically wipe out the human species. This is the hard part.

PhilGoetz does not have a framework for a well specified abstract invariant self-modifying goal system. If Phil was "seeming to be growing too powerful too quickly" then quite likely the same old human problems are occurring and a whole lot more besides.

The problem isn't with your values, CEV<Phil>, the problem is that you aren't a safe system for producing a recursively self improving singularity. Humans don't even keep the same values when you give them power let alone when they are hacking their brains into unknown territory.

Comment author: dlthomas 16 November 2011 05:09:47PM 0 points [-]

When talking about one individual, there is no C in CEV.

Comment author: wedrifid 16 November 2011 05:24:36PM *  0 points [-]

I use 'extrapolated volition' when talking about the outcome of the process upon an individual. "Coherent Extrapolated Volition" would be correct but redundant. When speaking of instantiations of CEV with various parameters (of individuals, species or groups) it is practical, technically correct and preferred to write CEV<X> regardless of the count of individuals in the parameter. Partly because it should be clear that CEV<wedridid> and CEV<all nerds> are talking about things very similar in kind. Partly because if people see "CEV" and google it they'll find out what it means. Mostly because the 'EV' acronym is overloaded within the nearby namespace.

AVERAGE(3.1415) works in google docs. It returns 2.1415. If you are comparing a whole heap of aggregations of a feature, some of which only have one value, it is simpler to just use the same formula.

Comment author: dlthomas 17 November 2011 01:55:19AM 1 point [-]

Seems reasonable.

Powered by Reddit Powered by Reddit