Tim_Tyler comments on The Magnitude of His Own Folly - Less Wrong

26 Post author: Eliezer_Yudkowsky 30 September 2008 11:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (127)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Tim_Tyler 02 October 2008 07:18:00AM 0 points [-]

No such thing [as an AI society] for many (most?) possible AIs; just a monolithic maximizer.

We might attain universal cooperation - but it probably wouldn't be terribly "monolithic" in the long term. It would be spread out over different planets and star systems. There would be some adaptation to local circumstances.

Could I become superintelligent under a Sysop?

The CEV document is littered with the term "human", "humanity" and the "human species" - but without defining what they mean. It seems terribly unlikely that our distant descendants will classify themselves or each other as "humans" - except perhaps as a term of abuse. So: once all the "humans" are gone, what happens then?

Also, if a human can change into a superintelligence - and remain a valued person - why can't a valued superintelligence be created from scratch? Is it because you were once DNA/protein you get special treatment? IMO, the future dominant organisms would see such views as appalling substrate chauvanism - what you are made of is an implementation detail, not who you really are. Is it because of who your ancestors were? That's biblical morality - the seventh son of the seventh son, and all that. People will be judged for who they are, not for who they once were, long, long ago.