Eliezer_Yudkowsky comments on Decision Theory FAQ - Less Wrong

52 Post author: lukeprog 28 February 2013 02:15PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (467)
Tags: decision_theory

Comments (467)

You are viewing a single comment's thread. Show more comments above.

Comment author: davidpearce 01 March 2013 10:07:33PM 1 point [-]

Eliezer, I'd beg to differ. Jane does not accurately model the world. Accurately modelling the world would entail grasping and impartially weighing all its first-person perspectives, not privileging a narrow subset. Perhaps we may imagine a superintelligent generalisation of http://www.guardian.co.uk/science/2013/feb/28/brains-rats-connected-share-information http://www.guardian.co.uk/science/brain-flapping/2013/mar/01/rats-are-like-the-borg With perfect knowledge of all the first-person facts, Jane could not disregard the strong preference of the cow not to be harmed. Of course, Jane is not capable of such God-like omniscience. No doubt in common usage, egocentric Jane displays merely a lack of altruism, not a cognitive deficit of reason. But this is precisely what's in question. Why build our canons of rational behaviour around a genetically adaptive delusion?

Comment author: Eliezer_Yudkowsky 01 March 2013 11:47:01PM 4 points [-]

See also:

"The Sorting Hat did seem to think I was going to end up as a Dark Lord unless [censored]," Harry said. "But I don't want to be one."

"Mr. Potter..." said Professor Quirrell. "Don't take this the wrong way. I promise you will not be graded on the answer. I only want to know your own, honest reply. Why not?"

Harry had that helpless feeling again. Thou shalt not become a Dark Lord was such an obvious theorem in his moral system that it was hard to describe the actual proof steps. "Um, people would get hurt?"

"Surely you've wanted to hurt people," said Professor Quirrell. "You wanted to hurt those bullies today. Being a Dark Lord means that people you want to hurt get hurt."

Harry floundered for words and then decided to simply go with the obvious. "First of all, just because I want to hurt someone doesn't mean it's right -"

"What makes something right, if not your wanting it?"

"Ah," Harry said, "preference utilitarianism."

"Pardon me?" said Professor Quirrell.

"It's the ethical theory that the good is what satisfies the preferences of the most people -"

"No," Professor Quirrell said. His fingers rubbed the bridge of his nose. "I don't think that's quite what I was trying to say. Mr. Potter, in the end people all do what they want to do. Sometimes people give names like 'right' to things they want to do, but how could we possibly act on anything but our own desires?"

"Well, obviously," Harry said. "I couldn't act on moral considerations if they lacked the power to move me. But that doesn't mean my wanting to hurt those Slytherins has the power to move me more than moral considerations!"

Powered by Reddit Powered by Reddit