Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Richard_Hollerith comments on That Alien Message - Less Wrong

111 Post author: Eliezer_Yudkowsky 22 May 2008 05:55AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (164)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Richard_Hollerith 23 May 2008 08:00:00PM 4 points [-]

RI asks,

how moral or otherwise desirable would the story have been if half a billion years' of sentient minds had been made to think, act and otherwise be in perfect accordance to what three days of awkward-tentacled, primitive rock fans would wish if they knew more, thought faster, were more the people they wished they were...

Eliezer answers,

A Friendly AI should not be a person. I would like to know at least enough about this "consciousness" business to ensure a Friendly AI doesn't have (think it has) it. An even worse critical failure is if the AI's models of people are people.

Suppose consciousness and personhood are mistaken concepts. Well, since personhood is an important concept in our legal systems, there is something in reality (namely, in the legal environment) that corresponds to the term "person", but suppose there is not any "objective" way to determine whether an intelligent agent is a person where "objective" means without someone creating a legal definition or taking a vote or something like that. And suppose consciousness is a mistaken concept like phlogiston, the aether and the immortal soul are mistaken concepts. Then would not CEV be morally unjustifiable because there is no way to justify the enslavement -- or "entrainment" if you want a less loaded term -- of the FAI to the (extrapolated) desires of the humans?

Comment author: SforSingularity 15 August 2009 01:22:48PM 1 point [-]

Suppose consciousness and personhood are mistaken concepts.

This is almost certainly the case IMO.

Comment author: pnrjulius 09 April 2012 05:12:44AM 0 points [-]

Well, as it stands, words like "person" and "consciousness" do pick out certain things (Bill Gates is a conscious person) and leave out other things (this ballpoint pen is not a conscious person). So there must be SOME meaning there, even if it's a confused or morally irrelevant meaning.

Hence, they're not as mistaken as the idea of the immortal soul, since that is as far as I can tell incoherent. (It's what moves my body... but it is non-physical? What does that MEAN?)

They could be as mistaken as the aether or phlogiston, but that's actually not so bad. These are scientific concepts that turned out to be wrong and were replaced by better concepts (relativity and oxidation reactions respectively) that explained the same phenomena and also explained many other phenomena. So if we replace "consciousness" in this way, we'll replace it with something that works just as well in the obvious cases and works even better in the tricky ones. (I am hopeful that we will do exactly this.)