You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Giles comments on CEV-inspired models - Less Wrong Discussion

7 Post author: Stuart_Armstrong 07 December 2011 06:35PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (41)

You are viewing a single comment's thread.

Comment author: Giles 08 December 2011 06:26:40PM 0 points [-]

What about recursive CEV?

Start off with CEV-0. I won't go into how that is generated, but it will have a lot of arbitrary decisions and stuff that seems vaguely sensible.

Then ask CEV-0 the following questions:

  • How should CEV-1 go about aggregating people's preferences?
  • How should CEV-1 deal with non-transitive or non-independent preferences?
  • How should CEV-1 determine preferences between outcomes that the subject could never have imagined?
  • What should CEV-1 do if people lack the expertise to judge the long-term consequences of their preferences?
  • Should CEV-1 consider people's stated or revealed preferences, or both?
  • Should CEV-1 consider preferences of non-human animals, people in comas, etc.?
  • How should CEV-1 deal with people who seem to be trying to modify their own preferences in order to game the system? (utility monsters/tactical voting)

... and so on. The answers to these questions then make up CEV-1. And then CEV-1 is asked the same questions to produce CEV-2.

Various different things could happen here. It could converge to a single stable fixed point. It could oscillate. It could explode, diverging wildly from anything we'd consider reasonable. Or its behavior could depend on the initial choice of CEV-0 (e.g. multiple attractive fixed points).

Explosion could (possibly) be avoided by requiring CEV-n to pass some basic sanity checks, though that has problems too (the sanity checks may not be valid, i.e. they just reflect our own biases. Or they may not be enough - they act as constraints on the evolution of the system but it could still end up insane in respects we haven't anticipated).

Some other problems could be resolved by asking CEV-n how to resolve them.

I'm not sure how to deal with the multiple stable fixed points case. That would seem to correspond to different cultures or special interest groups all trying to push whichever meta-level worldview benefits them the most.

Comment author: Stuart_Armstrong 10 December 2011 03:43:01PM 0 points [-]

Once CEV-n becomes a utility function, it will generally (but not always) get stuck there for ever.