Perplexed comments on Friendly to who? - Less Wrong

2 Post author: TimFreeman 16 April 2011 11:43AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (38)
Tags: morality friendly ai consensus fai

Comments (38)

You are viewing a single comment's thread.

Comment author: Perplexed 16 April 2011 06:51:49PM 3 points [-]

B: ... "I'll make an FAI that cares about every human equally, no matter what they do." ... Would you help me build that?

A: Well, it fits with my intuitive notion of morality, but it's not clear what incentive I have to help.

At this stage, I think the dialog goes astray by missing the real practical and political reason for CEV. The correct question is "Would you actively oppose me?" The correct answer is, "Well, I don't see how I could reasonably expect anything much better than that, so ..., no, I suppose I won't actively oppose you." And the difficult problem is convincing a rather large fraction of mankind to give the correct answer.

Comment author: timtyler 17 April 2011 04:23:07PM 2 points [-]

The correct question is "Would you actively oppose me?" The correct answer is, "Well, I don't see how I could reasonably expect anything much better than that, so ..., no, I suppose I won't actively oppose you."

The rich and powerful won't care for CEV. It pays no attention to their weath. They might as well have wasted their time accruing it.

Since the rich and powerful are high on the list for funding the R&D behind intelligent machines, they are likely to find a way to fund something that pays more attention to their preferences.

The "I don't see how I could reasonably expect anything much better" seems likely to be a failure of the imagination.

Comment author: Perplexed 17 April 2011 05:28:49PM 1 point [-]

The rich and powerful won't care for CEV. It pays no attention to their weath.

Not necessarily so. Quoting Eliezer: "A minor, muddled preference of 60% of humanity might be countered by a strong, unmuddled preference of 10% of humanity." So any good Marxist will be able to imagine the rich and powerful getting their way in the computation of CEV just as they get their way today: by inducing muddle in the masses.

The "I don't see how I could reasonably expect anything much better" seems likely to be a failure of the imagination.

And here I was considering it a victory of reason. :)

Comment author: timtyler 17 April 2011 09:30:48PM *  1 point [-]

Quoting Eliezer: "A minor, muddled preference of 60% of humanity might be countered by a strong, unmuddled preference of 10% of humanity." So any good Marxist will be able to imagine the rich and powerful getting their way in the computation of CEV just as they get their way today: by inducing muddle in the masses.

There's little reason for them to bother with such nonsense - if they are building and paying for the thing in the first place.

CEV may be a utilitarian's wet dream - but it will most-likely look like a crapshoot to the millionaires who are actually likely to be building machine intelligence.

Comment author: timtyler 17 April 2011 09:41:54PM *  0 points [-]

The "I don't see how I could reasonably expect anything much better" seems likely to be a failure of the imagination.

And here I was considering it a victory of reason. :)

It seemed as though you were failing to forsee opposition to CEV-like schemes. There are implementation problems too - but even without those, such scenarios do not seem very likely to happen.

Comment author: TimFreeman 17 April 2011 12:34:32AM 1 point [-]

Thanks, I agree. It's good to see that this multiplayer game notion of morality leads to a new insight that I didn't build into it.

Powered by Reddit Powered by Reddit