You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Alon comments on Anti-Pascaline agent - Less Wrong Discussion

4 Post author: Stuart_Armstrong 12 March 2015 02:17PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (24)
Tags: probability utility wager pascal agent ai

Comments (24)

You are viewing a single comment's thread.

Comment author: Alon 08 May 2016 12:32:55PM *  0 points [-]

It seems this type of AI might cheat and set ε=0 using the following method:

  1. create an "insurance company" subagent (see below for what it is and why creating it has a utility that is >ε).
  2. give the insurance company a few utilons (paper clips) to hold as hostages.
  3. set a rule that all future subagents will be able to communicate the insurance company.
  4. the insurance company will "buy" all the reward that has a probability < ε from subagents at roughly its fair price (its utility)
  5. all subagents won't ignore events that happen with probability < ε because their very existence is has a probability of 1 to increase their utility (via the insurance company).

alternatively for any ε you set, it will be profitable for the AI to create a new version of itself with the same utility function and ε'=ε/2 than give the new AI all the resources it has and commit suicide.

Comment author: Stuart_Armstrong 09 May 2016 11:30:05AM 0 points [-]

alternatively for any ε you set, it will be profitable for the AI to create a new version of itself with the same utility function and ε'=ε/2 than give the new AI all the resources it has and commit suicide.

This doesn't seem true. A ε/2 AI will take risks, looking for higher utility, that the ε AI wouldn't.

Powered by Reddit Powered by Reddit