People don't want to believe that you can control an AI, for the same reason they don't want to believe that their life stories could be designed by someone else. Reactance. The moment you suggest that a person's life can only go one way, they want it to go another way. They want to have that power. Otherwise, they feel caged.
People think that humans have that power. And so they believe that any truly human-level AI must have that power.
More generally, people think of truly, genuinely, human level minds as black boxes. They don't know how the black b...
Where is this noirish Eliezer when he's writing about the existence of free will and non-relativist moral truths?
Great post.
You seem to think of personal identity (PI) as a brittle thing, easily broken.
I want to note that the issue you raise, about whether PI is thick or thin (e.g. thick to the point of brittleness), seems to divide compatibilists and incompatibilists: compatibilists think PI is thick, incompatibilists thin. Consistent with my interpretation, you both (i) defend a thick notion of PI and (ii) strongly sympathize with compatibilism. Note that Daniel Dennett, another compatibilist (whom you seem fond of), raises many of the same objections about peopl... (read more)