Tim_Tyler comments on The Magnitude of His Own Folly - Less Wrong

26 Post author: Eliezer_Yudkowsky 30 September 2008 11:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (127)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Tim_Tyler 01 October 2008 07:20:00AM -1 points [-]

it seems likely that most of the difference between people's visions of the future stems from differing cultural/memetic backgrounds, character flaws, lack of information and time, etc.

Indeed, but our cultural background is the only thing that distinguishes us from cavemen. You can't strip that off without eliminating much that we find of value. Also, take the luddite/technophile divide. That probably arises, in part, because of different innate abilities to perform technical tasks. You can't easily strip that difference off without favouring some types of nuclear genetics over others.

Obviously, this isn't a terminal problem - society today does its best to please some majority of the population - a superintelligence could act similarly - but ineviably there will be some who don't like it. Some people don't want a superintelligence in the first place.

It all seems rather hypothetical, anyway: this is the benevolent-agents-create-superintelligence-for-the-good-of-all-humanity scenario. I don't list that as among the plausible outcomes on http://alife.co.uk/essays/the_awakening_marketplace/ Even the inventor of this scenario seems to assign it a low probability of playing out. The history of technology is full of instances of inventions being used to benefit the minorities best placed to take advantage of them. Is there any case to be made for such a utility function ever actually being used?