Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Zubon comments on Amputation of Destiny - Less Wrong

26 Post author: Eliezer_Yudkowsky 29 December 2008 06:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (66)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Zubon 29 December 2008 06:21:32PM 11 points [-]

Point of clarification: if human ascension to Mind status is possible, and speeding that ascension is within the power of the NFSI, how are you avoiding having at least one human mind ascend to main character status well ahead of the rest of the species?

At least one of the current six billion squishy things is going to want to enter the godmode code and ascend immediately, and if not them then one of the other trillions of Earth organisms that could be uplifted. Even if the NFSI limits the rate of ascension to the eudaimonic rate, that will vary between people; given six billion rolls of the dice (and more rolls every day), someone will have the value "really really fast" for his/her personal eudaimonic rate. Anything worth waiting for is worth having right now.

The effect seems like passing the recursive buck a very short distance. Humans create a computer that can but will not make all human efforts superfluous; the computer can and does uplift a human to equal capacities; at least one human can and may make all human efforts superfluous. Perhaps CEV includes something like, "No one should be able to get (much) smarter (much) faster than the rest of us," but restricting your intelligence because I am not ready for anyone that smart is an odd moral stance.

Comment author: xxdanbrowne 25 January 2011 09:23:09PM 0 points [-]

What's interesting to me is how much I believe the original poster has missed about the culture. Sure there are features lacking from culture society such as it having not too many significant challenges for either the humans or the Minds.

What you have gotten incorrect however, is that the human part of the Culture DOES need the minds. The main factor behind the collapse of all human civilizations up till now has been the "will to power" of the humans who seek glory through military means who have ultimately caused the destruction of many cultures (small c) in the past. Human nature being what it is, there will always be those who seek to topple the status quo which benefits everyone in order to set things up to benefit only themselves or satisfy their own personal whims whatever the case may be. It is my position that human culture NEEDS something like the Minds in order to protect itself from such usurpers and barbarians.

Additionally, it is quite clearly stated throughout the culture novels that though many of the Minds see humans as being animals, there are, nevertheless a handful of humans living in the Culture at any one time who are capable of making the right decision every time. The Culture has a great deal of respect for these humans.

Back in the real world, if you ask the question: "why would any sufficiently advanced AI tolerate the existence of humans if not just because it likes them?" The answer to that is Adam Smith. Even in the case of two countries where country A outcompetes country B in productivity at all possible classes of product, it is STILL more efficient to have country B produce a limited number of products and trade them to country A. That is likely to continue to be the case in the future.

The above is only one possible requirement for Minds living with humans. There are other arguments for keeping them around.

The only place where I can agree 100% is asking the question "why don't humans simply become Minds?"

That would seem to me to be a reasonable choice to make on your deathbed.

Comment author: DanielLC 16 January 2013 06:50:15AM 4 points [-]

Even in the case of two countries where country A outcompetes country B in productivity at all possible classes of product, it is STILL more efficient to have country B produce a limited number of products and trade them to country A.

This isn't a matter of productivity. It's a matter of efficiency. If a Mind can construct 1000 widgets or 2000 gadgets in a year, and a human can only construct one widget or one gadget, it would be best for the Minds to produce gadgets and the humans to produce widgets, but if a Mind can construct 1000 widgets or 2000 gadgets with one unit of resources, and a human can only construct one widget or one gadget, you're better off just having the Mind make everything.

Even if they're producing thoughts, the humans are made of matter that can be used to make Minds. If nothing else, you can choose between having more humans or Minds. Two Minds can produce 1000 widgets and 2000 gadgets, but a human and a Mind can only produce 501 widgets and 1000 gadgets.